Python Multiprocessing Tutorial: Run Code in Parallel Using the Multiprocessing Module
Why Stack Overflow recommends it
Stack Overflow users link it when answering or asking about parallelizing CPU-bound Python work, and it is the tutorial many askers follow before posting their own multiprocessing errors.
Recommended by 10 people across 10 different months, 2019 to 2022.
What it is
Corey Schafer's 44-minute tutorial on running code in parallel in Python with the multiprocessing module. It also shows how to batch-process high-resolution images with concurrent.futures.ProcessPoolExecutor, with code available for download.
What people said
Recommends Artur Müller Romanov · 2020-04 · Stack Overflow · on “multiprocessing [concurrent.futures.ProcessPoolExecutor] not calling function correctly”
Hello fellow programmers, I'm new to multiprocessing. I've watched this guide on multiprocessing: [this video] and am trying to make the code work. Basically what this does is read a bunch of images and save them to the processed folder. link to zip (130MB): [LINK] The whole code: import time import concurrent.futures from pathlib import Path from PIL import Image, ImageFilter parent_path = Path(__file__).parent output_path = parent_path.joinpath('processed') def process_image(img_name): img = Image.open(img_name) img = img.filter(ImageFilter.GaussianBlur(15)) img.thumbnail(size) img.save(f'{output_path.joinpath(img_name)}') print(f'{img_name} was processed...') def main(): img_names = [ 'photo-1
Recommends Omid Ki · 2020-04 · Stack Overflow · on “How can I speed up this "for" loop?”
As martineau said, you can use threading to speed up your program. Just import the "threading" library and add threding.Thread in your class's parentheses and name your function "def run(self):" then put whatever you like your thread to do inside that function and wherever you want to start your thread use your object name and then ".start()" this way it will start your thread. Most of the times it is not a good idea to do this but if you have to then you should do it. By the way if your program is more I/O (input and output) bound then it is better to use threads, if it is CPU bound it should be better for you to use multi-processing. You can also watch these tutorials for better understand
Recommends Jens · 2020-04 · Stack Overflow · on “Multiprocessing a big amount of data in python”
Fo an easy handling of multiprocessing you can use the concurrent.futures module. Python Documentation: Concurrent Futures [LINK] Before I´m explaining every aspect, there is a great and easy video tutorial with sample code (easy to adapt): [this video] For handling many tasks with multiple processors or threads i recommend the Queue module. Python Documentation: Queue [LINK] from queue import Queue #Create Queue object q = Queue() #Put item to queue q.put("my value") #Get and process each item in queue and remove it while not q.empty(): myValue = q.get()
Recommends Boyke Ferdinandes · 2020-05 · Stack Overflow · on “Code does not run multiprocessing functions in python 3.8”
literally copied and pasted your code, it run as what you expected. to print "sleeping 1 sec" followed by "done sleeping". Finished in 0.0 second(s) Sleeping 1 second... Sleeping 1 second... Done Sleeping... Done Sleeping... I know that youre following this tutorial: [this video] it is a good one indeed, but you also need to listen to his explanation. he is really good at it. he also explain how to turn to this output: Sleeping 1 second... Sleeping 1 second... Done Sleeping... Done Sleeping... Finished in 1.01 second(s) *note that the 'finished' is on the last row
Recommends Jose Moreno · 2020-09 · Stack Overflow · on “Can not use queue.put with keras model”
check this [this video] in your case is better used a multiprocessing pool for a more clean code, do any errors occur in the execution?,
Recommends Matiiss · 2020-10 · Stack Overflow · on “Running non Tkinter code along side Tkinter code”
i wont be able to tell but here is a tutorial for threading [LINK] with real life example and tutorial for multiprocessing [this video] with real life example.
Recommends Arthur Harduim · 2020-11 · Stack Overflow · on “How to do simultaneous downloads with Selenium Python”
I think a process pool would be perfect for your case. Here is the digest from concurrent.futures import ProcessPoolExecutor def download_from_tab(tab_num): pass with ThreadPoolExecutor() as executor: for tab_num in range(number_of_tabs): _ = executor.submit(download_from_tab, tab_num) The code above will open a new process for each tab and start downloading things in parallel. You can find the oficinal documentation here: [LINK] There is also an awesome video that shows step-by-step how to use processes for exactly this: [this video]
Recommends Alexander Hunter · 2021-04 · Stack Overflow · on “I need to perform an "expensive" loop on a table with millions of rows. What technology (programming language or something else) should I use?”
If you're willing to stay with python I would probably recommend using the multiprocessing module. Corey Schafer has a good tutorial on how it works here [this video] . Multiprocessing is a bit like threading however uses multiple interpreters to complete the main task unlike the threading module which switches through each thread to execute a line of code independently. Divide up the work with however many cores you have on your CPU with this: import os cores = os.cpu_count() This should speed up the work load by dividing up the work across your entire computer.
Recommends Colin · 2021-06 · Stack Overflow · on “What determines order of printed results when counting with two threads?”
as far as I am aware the variation is determined by the thread allocation and other aspects such as current CPU workload. Below is some code that I have been using for multiprocessing, maybe it will be of some use for you. import concurrent.futures def SomeFunction(): ProcessList = [] for i in range(1,11): ProcessList.append(i) return ProcessList if __name__ == "__main__": with concurrent.futures.ProcessPoolExecutor() as exec: # loop determines the number of processes to run results = [exec.submit(SomeFunction) for _ in range(2)] for f in concurrent.futures.as_completed(results): r = f.result() print(r) Output is: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] If you need to
Recommends NedNoodleHead · 2022-08 · Stack Overflow · on “TKINTER - How to keep the main loop from waiting a validate command?”
I had a similar problem, I fixed it with threading: import threading def my_function(): something_somewhat_intensive() def my_function_thr(): thr = threading.Thread(target=my_functioon) thr.start() Then you call my_function_thr() to begin that process on a thread. I don't know how this interacts with the import 'subprocess' at all, but this fixed a lot of problems that I had. It should also be mentioned that if your process is resource intensive, and a bit more than like saving a file or something, use Multiprocessing [this video] . For a lot of processes, especially with just wanting the Tkinter window to not freeze, I do recommend threading.