Geek asks and answers

Performance issue: large numbers array and system call

When using a system call, there is a performance problem when preallocating a large amount of memory (for example, a numpy array). The problem is growing with the amount of memory.

test.py:

import os
import sys
import time
import numpy

start = time.clock()
test = int(sys.argv[1])
a = numpy.zeros((test,500,500))
for i in range(test) :
    os.system("echo true > /dev/null")
elapsed = (time.clock() - start)
print(elapsed)

The time of each iteration increases sharply:

edouard@thorin:~/now3/code$ python test.py 100
0.64
edouard@thorin:~/now3/code$ python test.py 200
2.09
edouard@thorin:~/now3/code$ python test.py 400
14.26

This should not be related to virtual memory. Is this a known issue?

+3
source share
2 answers

You seem to have narrowed the problem down to os.system()taking more time after you have allocated a large NumPy array.

system() fork(). fork() (- copy-on-write), , .

, Linux fork(), . . :

, , . , .

system(), :

  • .
  • , system().
+5

, os.system?

:

python test.py 10   # 0.14
python test.py 100  # 1.18
python test.py 1000 # 11.77

os.system. , , , numpy ( , , ). : " (er) ?"... , .

, bash, ( )...

time for i in `seq 1 1000`; do echo true > /dev/null; done

, os.system - subprocess.Popen ... ( subprocess os.system , ...)

. numpy... numpy, . (1000 800 800) 1 . , ( ), , . .

+4

More articles:


All Articles