Geek asks and answers

MaybeEncodingError with pyparsing.asDict when using multiprocessing.pool map ()

I am trying to use multiprocessing.poolto speed up parsing a file using pyparsing, however, I get an exception multiprocessing.pool.MaybeEncodingErrorwhenever I try to do this.

I narrowed it down to something related to returning dictionary ( ParseResults.asDict()), using the asList()error does not occur; but the input that I actually parses is quite complicated, so I would like to use asDict.

The actual parsed data is a list of tagged Erlang tags that I want to map to a python list. The grammar for this is pretty complicated, so instead I got a simplified test case ( updated to include a nested dict ):

#!/usr/bin/env python2.7
from pyparsing import *
import multiprocessing

dictionary = Forward()
key = Word(alphas)
sep   = Suppress(":")
value = ( key | dictionary )
key_val = Group( key + sep + value )
dictionary <<= Dict( Suppress('[') + delimitedList( key_val ) + Suppress(']') )

def parse_dict(s):
    p = dictionary.parseString(s).asDict()
    return p

def parse_list(s):
    return dictionary.parseString(s).asList()

# This works (list)
data = ['[ foo : [ bar : baz ] ]']
pool = multiprocessing.Pool()
pool.map(parse_list, data)

# This fails (dict)
pool.map(parse_dict, data)

Failure:

Traceback (most recent call last):
  File "lib/python/nutshell/multi_parse.py", line 19, in <module>
    pool.map(parse, data)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 250, in map
    return self.map_async(func, iterable, chunksize).get()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 554, in get
    raise self._value
multiprocessing.pool.MaybeEncodingError: Error sending result: '[{'foo': ([(['bar', 'baz'], {})], {'bar': [('baz', 0)]})}]'. Reason: 'TypeError("'str' object is not callable",)'
+3
1

. . .

, delimitedList, :

data = ['[ foo : [ bar : baz ], cat:dog ]']

, "dictionary" dict, . , delimitedList - . , parseAction:

dictionary = Forward()
key   = Word(alphas)
LP, RP, sep = map(Suppress, "[]:")
value = key | dictionary
key_val = key("key") + sep + value("val")
dictionary <<= LP + delimitedList( key_val ) + RP

def parse_key_val(x): return {x.key:x.val}
key_val.setParseAction(parse_key_val)

def parse_dict(s):
    # Yes, it a list, not a dict!
    return dictionary.parseString(s).asList()

def parse_list(s):
    return dictionary.parseString(s).asList()

:

[[{'foo': {'bar': 'baz'}}, {'cat': 'dog'}]]

: , , . , , :

def parse_dict(s):
    val = lang.parseString(s).asDict()
    print type(val["foo"])
    return val

, - <class 'pyparsing.ParseResults'>. , pp.Dict , :

value = ( Word(alphas) )
sep   = Suppress(":")
key_val = Group( value + sep + value )
lang = Dict( Suppress('[') + delimitedList( key_val ) + Suppress(']') )

pp.Dict . , , , , , .

:

pyparsing Dict()

+2

More articles:


All Articles