Geek asks and answers

Preventing pool processes when importing __main__ and globals

I use a multiprocessor pool of workers as part of a much larger application. Since I use it to crunch a large volume of simple math, I have a shared architecture where the only variables that workers ever need are passed as arguments. Thus, I do not need working subprocesses to import any globals, my module __main__or, therefore, any of the modules that it imports. Is there a way to force this behavior and avoid a performance hit with pool spawning?

I should note that my environment is Win32, which is missing os.fork(), and workflows are generated "using a subprocess call for sys.executable (that is, it starts a new Python process), followed by serialization of all global variables and sending those who is above the pipe. " according to this post SO . Having said that, I want to make as few of them as possible so that my pool opens faster.

Any ideas?

+3
source share
1 answer

multiprocessing.forking, get_preparation_data prepare ( Win32), , reimport __main__ , - , ; if __name__ == '__main__'. .

- ( win32, , , ). main() , script :

if '__name__' == '__main__':
    from mainmodule import main
    main()

- import site. , , mp.forking , , .

+3

More articles:


All Articles