Parallel Programming

The threading module uses threads, the multiprocessing module uses processes. The difference is that threads run in the same memory space, while processes have separate memory. This makes it a bit harder to share objects between processes with multiprocessing. Since threads use the same memory, precautions have to be taken or two threads will write to the same memory at the same time. This is what the global interpreter lock is for.

Spawning processes is a bit slower than spawning threads. Once they are running, there is not much difference.

threading

thread pool

pool = multiprocessing.pool.ThreadPool()

apply

pool.apply(func[, args[, kwds]])
  • return what is defined in the function
  • call function with arguments and keywords
  • synchronous execution, block until gets result

apply_async

pool.apply_async(func[, args[, kwds[, callback[, error_callback]]]])
  • return a result object
  • call function with arguments, keywords, callback function, error_callback function
    • if the function completes, the callback is applied
    • if the function fails, the error_callback is applied
  • asynchronous execution, do not block

map

pool.map(func, iterable[, chunksize])
  • return what is defined in the function
  • parallel version of built-in function map()
  • only support one iterable argument
  • block until gets result
  • chop the iterable object into chunks, submit them to the thread pool as separate tasks

map_async

pool.map_async(func, iterable[, chunksize[, callback[, error_callback]]])
  • return a result object.
  • call function with arguments, keywords, callback function, error_callback function
    • if the function completes, the callback is applied
    • if the function fails, the error_callback is applied
  • asynchronous execution, do not block

close

pool.close()
  • prevent any more tasks from being submitted to the pool
  • once all the tasks have been completed the worker threads will exit.

terminate

pool.terminate()
  • stop the worker threads immediately without completing work
  • terminate() will be called if the pool object is collected by garbage collector

join

pool.join()
  • wait for the worker threads to exit
  • must call close() or terminate() before calling join()

multiprocessing

process pool

pool = multiprocessing.Pool()
  • the same as threading

MPI

results matching ""

    No results matching ""