I am completely confused about the memory usage of a particular python script. I guess I donβt know how to profile usage, despite the advice from a few SO questions / Answers .
My questions: What is the difference between memory_profilerand guppy.hpy? Why do they tell me that I use a huge amount of memory, and another tells me that I do not?
I work with a pysamlibrary for accessing bio / informatics SAM / BAM files. My main script runs out of memory quickly when converting SAM (ASCII) to BAM (Binary) and manipulating files between them.
I created a small test case to understand how much memory is allocated at each step.
import pysam
TESTFILENAME = ('/projectnb/scv/yannpaul/MAR_CEJ082/' +
'test.sam')
@profile # for memory_profiler
def samopen(filename):
samf = pysam.Samfile(filename)
pass
if __name__ == "__main__":
samopen(TESTFILENAME)
memory_profiler (python -m memory_profiler test_pysam.py) :
Filename: test_pysam.py
Line
================================================
10 @profile
11 def samopen(filename):
12 10.48 MB 0.00 MB
13 539.51 MB 529.03 MB samf = pysam.Samfile(filename)
14
15 539.51 MB 0.00 MB pass
@profile guppy, (python test_pysam.py):
Partition of a set of 3 objects. Total size = 624 bytes.
Index Count % Size % Cumulative % Kind (class / dict of class)
0 1 33 448 72 448 72 types.FrameType
1 1 33 88 14 536 86 __builtin__.weakref
2 1 33 88 14 624 100 csamtools.Samfile
13 529,03 624 . ? 'test.sam' SAM ( ASCII ) 52 . pysam, C, samtools. , Samfile, , , . python?