Geek asks and answers

Convert latin string to unicode in python

I work o scrapy, I cleaned up some sites and saved elements from the treble page in json files, but some of them contain the following format.

l = ["Holding it Together",
     "Fowler RV Trip",
     "S\u00e9n\u00e9gal - Mali - Niger","H\u00eatres et \u00e9tang",
     "Coll\u00e8ge marsan","N\u00b0one",
     "Lines through the days 1 (Arabic) \u0633\u0637\u0648\u0631 \u0639\u0628\u0631 \u0627\u0644\u0623\u064a\u0627\u0645 1",
     "\u00cdndia, Tail\u00e2ndia & Cingapura"]

I can expect that the list consists of a different format, but I want to convert it and save the lines in the list with their original names, for example below

l = ["Holding it Together",
     "Fowler RV Trip",
     "Lines through the days 1 (Arabic) سطور عبر الأيام 1 | شمس الدين خ | Blogs"         ,
     "Índia, Tailândia & Cingapura "]

Thanks in advance...........

+3
source share
2 answers

You have byte strings containing unicode escape codes. You can convert them to unicode using the codec unicode_escape:

>>> print "H\u00eatres et \u00e9tang".decode("unicode_escape")
Hêtres et étang

And you can encode it back to byte strings:

>>> s = "H\u00eatres et \u00e9tang".decode("unicode_escape")
>>> s.encode("latin1")
'H\xeatres et \xe9tang'

You can filter and decode strings other than unicode, for example:

for s in l: 
    if not isinstance(s, unicode): 
        print s.decode('unicode_escape')
+7
source

,

JSON , , ASCII, \u. json, ensure_ascii:

>>> print json.dumps(u'Índia')
"\u00cdndia"
>>> print json.dumps(u'Índia', ensure_ascii= False)
"Índia"

, , ASCII, UnicodeError s. , JSON , Unicode (, UTF-8).

j= json.dumps(u'Índia', ensure_ascii= False)
open('file.json', 'wb').write(j.encode('utf-8'))
+1

More articles:


All Articles