I wrote a blog article regarding advanced map with Python earlier and I found multi threading process using map with ThreadPoolExecutor. If you want to run multiple long running processes at the same time in multiple threads, this example may be for you. Here is the sample code.
from concurrent import futures
from concurrent.futures import ThreadPoolExecutor
from itertools import repeat
def long_running_function(self, a, n):
return a ** n
start_time = time.time()
n = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
with ThreadPoolExecutor(max_workers=3) as executor:
fs = map(executor.submit, repeat(self.long_running_function), repeat(2), n)
results = futures.wait(list(fs))
for result in results.done:
if result.exception() is not None:
elapsed_time = time.time() - start_time
long_running_function takes 2 parameters. a is a fixed parameter and n is for each element of the n list.
So if you executed the
long_running_function sequentially for each element, it would take 30 seconds because the function sleeps for 3 seconds but using 3 threads at a time, it took only 12 seconds. 10 elements should equal to 4 batches, hence 12 seconds. If there are hundreds or thousands of items, you can imagine this method can be a huge gain.
test_advanced_map function, it instantiates
max_workers value 3. It means 3 threads run at the same time. Depending on your program, you can increase the number but be careful not to increase the number too much because it can hog your memory.
And using map function, you execute the
executor.submit function. You pass
long_running_function as a fixed parameter as a function to be executed. And then, you pass repeat(2) and n for the parameter a for the
long_running_function. You can execute a function for each element of list using map function in Python. I wrote about it before here.
Lastly, this method can handle returned results and unhandled exceptions. I believe this is the best way to do multi threading in Python. I could be wrong, so please comment if you have better ideas! 🙂