Geek asks and answers

Streaming data from Postgres to Python

I am looking for advice on efficient ways to stream data step by step from a Postgres table in Python. I am executing an online learning algorithm, and I want to read lots of training examples from a database table into the memory to be processed. Any thoughts on good ways to increase bandwidth? Thanks for your suggestions.

+3
source share
2 answers

If you use psycopg2, you will need to use a named cursor, otherwise it will try to immediately read all the request data in memory.

cursor = conn.cursor("some_unique_name")
cursor.execute("SELECT aid FROM pgbench_accounts")
for record in cursor:
    something(record)

This will retrieve the records from the server in batches of 2000 (the default value itersize), and then upload them in a loop one at a time.

+3
source

More articles:


All Articles