Geek asks and answers

Iterate over SQLAlchemy assembly efficiently

I have an SQLAlchemy ORM model that I use to transfer certain rows (full graphs of objects) from our production database to our test and development databases. This works very well until I get a collection with many child dependencies, and I came across MemoryError. I already set up dynamic loading at key points in the hierarchy of objects and loaded children separately with different requests, but there are still collections that have a sufficient amount of child data that appear in my memory.

What is the best way to load only one collection item at a time, so I can copy individual objects (and all their child lines) at a time?

+3
source share
1 answer

Try to clear the session after each Node (with children) processed using Session.expunge . The sample code below displays the number of instances in a session:

class Node(Base):
    __tablename__ = 'node'
    id = Column(Integer, primary_key=True)
    parent_id = Column(Integer, ForeignKey('node.id'))
    name = Column(String(50))
    children = relationship("Node",
            backref=backref("parent", remote_side=[id],)
            )

def process_node(node, expunge=False, ident=1):
    print "Node: ", "-" * ident, node, " --> ", len(session.identity_map)
    for child in node.children:
        process_node(child, expunge, ident + 4)
        session.expunge(child)

roots = session.query(Node).filter(Node.parent == None)
for root in roots:
    process_node(root, True)
+1
source

More articles:


All Articles