Geek asks and answers

Mpi4py Etching error while trying to send classes

im using mpi4py to split some calculations between multiple proc. Basically Im just calculates the volumes of some convex hull that I create with tvtk and mayavi.

only the first proc imports these libs:

...
if rank==0:
   from tvtk.api import tvtk
   from mayavi Import mlab
   ...
   mlab.figure(size=(1024,768),bgcolor=(1,1,1))
   ...

Then I try to split the mlab and tvtk objects among all the procs:

for i in range(comm.Get_size()):
    comm.send(mlab,dest=i)
    comm.send(tvtk,dest=i)
....

The following steps will look something like this:

Points=local_data
ug=tvtk.UnstructuredGrid(Points=Points)
...
dataname="Data %s " % rank
ds=mlab.pipeline.add_dataset(ug,name=dataname)
delaunay=mlab.pipeline.delaunay3d(ds,name=dataname)
... calc volume...

However, it is obvious that it is not possible to send instances / classes (or any mlab and tvtk files), since I always get the following error:

comm.send(mlab,dest=i)
File "Comm.pyx", line 753, in mpi4py.MPI.Comm.send (src/mpi4py.MPI.c:53848)
File "pickled.pxi", line 122, in mpi4py.MPI.PyMPI_send (src/mpi4py.MPI.c:20409)
 File "pickled.pxi", line 39, in mpi4py.MPI._p_Pickle.dump (src/mpi4py.MPI.c:19503)
cPickle.PicklingError: Can't pickle <type 'module'>: attribute lookup __builtin__.module failed

Is there a way to “share” instances of mlab and tvtk among all procs?

EDIT: a short example; can you get this job?

from mpi4py import MPI
comm=MPI.COMM_WORLD

size=comm.Get_size()
rank=comm.Get_rank()

if rank==0:
        from tvtk.api import tvtk
        from mayavi import mlab

if __name__=='__main__':
        if rank==0:
                for i in range(size):
                        comm.send(mlab,dest=i)
                        comm.send(tvtk,dest=i)
        else:
                local_mlab=comm.recv(mlab,source=0)
                local_tvtk=comm.recv(tvtk,source=0)
+3
source share
2 answers

I assume the following might work in your case

from mpi4py import MPI
MPI._p_pickle.dumps = dill.dumps
MPI._p_pickle.loads = dill.loads
+2

, , , , , , , , -, , .

, , - , , , , :

global_data = None

if rank == 0:
    global_data = {'value_of_interest': mlab.some_parameter}

global_data = MPI.COMM_WORLD.bcast(global_data, root=0)
0

More articles:


All Articles