Package wsatools :: Module Parallel

Module Parallel

An alternative implementation of multiprocessing.Pool.map that is faster, more robust and which also works for class methods and non-pickleable functions.

Usage

Interface is the same as the map function. The first argument is the name of a function that takes only a single argument. The second argument is an iterable sequence for which every item will become an argument to the function given in the first argument:

   from wsatools.Parallel import parallel_map
   results = parallel_map(self.runOnData, [1,2,3])

Author: R.S. Collins

Organization: WFAU, IfA, University of Edinburgh

Note: Gratuitously taken from this site: http://www.astropython.org/snippet/2010/3/Parallel-map-using-multiprocessing

Functions

[hide private]

worker(f, ii, chunk, out_q, err_q, _lock)
A worker function that maps an input function over a slice of the input iterable.

source code

run_tasks(procs, err_q, out_q, num)
A function that executes populated processes and processes the resultant array.

source code

parallel_map(function, sequence, numcores=None)
A parallelised version of the native Python map function that utilises the Python multiprocessing module to divide and conquer sequence.

source code

Variables

[hide private]

__package__ = 'wsatools'

Function Details

[hide private]

worker(f, ii, chunk, out_q, err_q, _lock)

source code


A worker function that maps an input function over a
slice of the input iterable.

:param f    : callable function that accepts argument from iterable
:param ii   : process ID
:param chunk: slice of input iterable
:param out_q: thread-safe output queue
:param err_q: thread-safe queue to populate on exception
:param _lock: thread-safe lock to protect a resource
             ( useful in extending parallel_map() )

run_tasks(procs, err_q, out_q, num)

source code

A function that executes populated processes and processes the resultant array. Checks error queue for any exceptions.

:param procs: list of Process objects :param out_q: thread-safe output queue :param err_q: thread-safe queue to populate on exception :param num : length of resultant array

parallel_map(function, sequence, numcores=None)

source code

A parallelised version of the native Python map function that utilises the Python multiprocessing module to divide and conquer sequence. Example:

>>> parallel_map(lambda x: x**2, [1,2,3])
[1, 4, 9]

parallel_map does not yet support multiple argument sequences.

:param function: callable function that accepts argument from iterable :param sequence: iterable sequence :param numcores: number of cores to use