Python Multi-processing with example

Using multiprocessing module

             | Multi-args  Concurrence   Blocking    Ordered-results
--------------------------------------------------------------------
map | no yes yes yes
map_async | no yes no yes
apply | yes no yes no
apply_async | yes yes no no
starmap | yes yes yes yes
starmap_async| yes yes no no

Multi-processing with apply_async(variant of apply)

  • It assigns a job to the worker pool but doesn't wait around for the result. Instead, it returns a placeholder. We can use the placeholder to get the actual result by calling result_placeholder.get().
  • Takes a function as a first argument and a tuple of arguments for that function as a second argument.
IT WASN'T JUST A PUPPY
IT WASN'T JUST A PUPPY
IT WASN'T JUST A PUPPY
YOU WANTED ME BACK...
YOU WANTED ME BACK...
YOU WANTED ME BACK...
['Hello: john_wick_1', 'Hello: john_wick_1', 'Hello: john_wick_3']
Completed in 1.175609827041626 seconds
IT WASN'T JUST A PUPPY
IT WASN'T JUST A PUPPY
IT WASN'T JUST A PUPPY
YOU WANTED ME BACK...
YOU WANTED ME BACK...
YOU WANTED ME BACK...
['Hello: john_wick_1', 'Hello: john_wick_1', 'Hello: john_wick_3']
Completed in 1.1662490367889404 seconds
>>> import os
>>> print(f"Number of CPU available: {os.cpu_count()}")
Number of CPU available: 16

Multi-processing vs multi-threading

Difference between multithreading and multiprocessing?

Conclusion

References

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store