Geek asks and answers

How to saw functions / classes defined in __main__ (python)

I would like to be able to sort a function or class from __main__ with the obvious problem (mentioned in other posts) that the pickled function / class is in the __main__ namespace and the scattering in another script / module will fail.

I have the following solution that works, is there a reason why this should not be done?

The following is shown in myscript.py:

import myscript
import pickle

if __name__ == "__main__":               

    print pickle.dumps(myscript.myclass())

else:

    class myclass:
        pass

change . Unpacking will be done in a script / module that has access to myscript.py and can do it import myscript. The goal is to use a solution like parallel python to remotely call functions and the ability to write a short, stand-alone script that contains functions / classes that can be accessed remotely.

+5
source share
3 answers

If you are trying to expand something so that you can use it somewhere else, separate it from test_scriptthat which will not work, because pickle (apparently) is simply trying to load the function from the module. Here is an example:

test_script.py

def my_awesome_function(x, y, z):
    return x + y + z

picklescript.py

import pickle
import test_script
with open("awesome.pickle", "wb") as f:
    pickle.dump(test_script.my_awesome_function, f)

python picklescript.py, test_script, , . .

:

import pickle
with open("awesome.pickle", "rb") as f:
    pickle.load(f)

:

Traceback (most recent call last):
  File "load_pickle.py", line 3, in <module>
    pickle.load(f)
  File "/Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/pickle.py", line 1378, in load
    return Unpickler(file).load()
  File "/Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/pickle.py", line 858, in load
    dispatch[key](self)
  File "/Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/pickle.py", line 1090, in load_global
    klass = self.find_class(module, name)
  File "/Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/pickle.py", line 1124, in find_class
    __import__(module)
ImportError: No module named test_script
+1

, __main__ , . , dill , - python. , , __main__ , __main__ .

>>> import dill
>>> 
>>> def bar(x):
...   return foo(x) + x
... 
>>> def foo(x):
...   return x**2
... 
>>> bar(3)
12
>>> 
>>> _bar = dill.loads(dill.dumps(bar))
>>> _bar(3)
12

, pickle, , , pickle, , monkeypatching .

, , " python", .

>>> # continuing from above
>>> dill.dump_session('foobar.pkl')
>>>
>>> ^D
dude@sakurai>$ python
Python 2.7.5 (default, Sep 30 2013, 20:15:49) 
[GCC 4.2.1 (Apple Inc. build 5566)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import dill
>>> dill.load_session('foobar.pkl')
>>> _bar(3)
12

ssh , , python , .

, mpi4py. pathos ( pyina MPI), map .

>>> # continued from above
>>> from pathos.multiprocessing import ProcessingPool as Pool
>>> Pool(4).map(foo, range(10))
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
>>>
>>> from pyina.launchers import MpiPool
>>> MpiPool(4).map(foo, range(10))
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

, . pp, , __main__. . , fork github pp , __main__. , pp , , pp ... , mpi4py. dill.source, , pp, .

+1

Pickle, , main . , , :

import myscript
import __main__
__main__.myclass = myscript.myclass
#unpickle anywhere after this
-1
source

More articles:


All Articles