Geek asks and answers

UnicodeEncodeError when writing to CSV from JSON

I get the following error when trying to write to CSV using JSON:

Traceback (most recent call last):
File "twitter_search_csv.py", line 25, in <module>
status['retweet_count'],
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2026' in position 139: ordinal not in range(128)

Here is the code I'm working with:

import requests
import urllib2
from requests_oauthlib import OAuth1
import csv

auth = OAuth1('', '', '', '')
url = 'https://api.twitter.com/1.1/search/tweets.json?q=%23OpeningCeremony'

response = requests.get(url, auth=auth)

data = response.json()['statuses']

with open('olympic_search.csv', 'wb') as csvfile:
    f = csv.writer(csvfile)
    for status in data:
        f.writerow([
            status['id'],
            status['text'],
            status['created_at'],
            status['coordinates'],
            status['user']['id_str'],
            status['retweet_count'],
        ])
+3
source share
1 answer

Explicitly encode the field. Otherwise, Python will try to encode it using ascii encoding.

>>> print u'\u2026'.encode('ascii')

Traceback (most recent call last):
  File "<pyshell#2>", line 1, in <module>
    print u'\u2026'.encode('ascii')
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2026' in position 0: ordinal not in range(128)
>>> print u'\u2026'.encode('utf-8')
…

f.writerow([
    status['id'],
    status['text'].encode('utf-8'), # <----
    status['created_at'],
    status['coordinates'],
    status['user']['id_str'],
    status['retweet_count'],
])
+6
source

More articles:


All Articles