I am trying to implement a very simple wordcount example with MRJob. Everything works fine with ascii input, but when I mix cyrillic words into input, I get something like this as output
"\u043c\u0438\u0440" 1
"again!" 1
"hello" 2
"world" 1
As I understand it, the first line above is the encoded one-time occurrence of the Cyrillic word "world", which is the correct result in relation to my text input example. Here is the MR code
class MRWordCount(MRJob):
def mapper(self, key, line):
line = line.decode('cp1251').strip()
words = line.split()
for term in words:
yield term, 1
def reducer(self, term, howmany):
yield term, sum(howmany)
if __name__ == '__main__':
MRWordCount.run()
I am using Python 2.7 and mrjob 0.4.2 on windows. My questions:
a) how do I manage to correctly output the readable cyrillic output to the cyrillic input? b) what is the reason for this behavior - is it because of the python / MR version or is it expected that it will work differently on any windows - any hints?
python -c "print u'mir '"
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Python27\lib\encodings\cp866.py", line 12, in encode
return codecs.charmap_encode(input,errors,encoding_map)
UnicodeEncodeError: 'charmap' codec can't encode characters in position 0-2: character maps to <undefined>