Python memory usage: What is the difference between memory_profiler and guppy?

I am completely confused about the memory usage of a particular python script. I guess I don’t know how to profile usage, despite the advice from a few SO questions / Answers .

My questions: What is the difference between memory_profilerand guppy.hpy? Why do they tell me that I use a huge amount of memory, and another tells me that I do not?

I work with a pysamlibrary for accessing bio / informatics SAM / BAM files. My main script runs out of memory quickly when converting SAM (ASCII) to BAM (Binary) and manipulating files between them.

I created a small test case to understand how much memory is allocated at each step.

# test_pysam.py: 

import pysam
#from guppy import hpy

TESTFILENAME = ('/projectnb/scv/yannpaul/MAR_CEJ082/' +
                'test.sam')
#H = hpy()

@profile # for memory_profiler
def samopen(filename):
#    H.setrelheap()
    samf = pysam.Samfile(filename)
#    print H.heap()
    pass


if __name__ == "__main__":
    samopen(TESTFILENAME)

memory_profiler (python -m memory_profiler test_pysam.py) :

Filename: test_pysam.py

Line #    Mem usage    Increment   Line Contents
================================================
    10                             @profile # for memory_profiler
    11                             def samopen(filename):
    12     10.48 MB      0.00 MB   #    print H.setrelheap()
    13    539.51 MB    529.03 MB       samf = pysam.Samfile(filename)
    14                             #    print H.heap()
    15    539.51 MB      0.00 MB       pass

@profile guppy, (python test_pysam.py):

Partition of a set of 3 objects. Total size = 624 bytes.
 Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
     0      1  33      448  72       448  72 types.FrameType
     1      1  33       88  14       536  86 __builtin__.weakref
     2      1  33       88  14       624 100 csamtools.Samfile

13 529,03 624 . ? 'test.sam' SAM ( ASCII ) 52 . pysam, C, samtools. , Samfile, , , . python?

+5
1

memory_profiler guppy.hpy?

? (, Python free 1 , 1 .) , : memory_profiler ; guppy .

, memory_profiler guppy doesn't - ; , . - , , memory_profiler, , , , , ; , , , .

, , , ?

, ; :

, samtools mmap . , .

, samtools pysam , . ( PyObjects ), malloc , freelist - , , , VM , , . ; , .

python?

, memory_profiler , . pysam , @profile. , C ; , , , C ( samtools samtools).

+7

All Articles