Geek asks and answers

Decrease in cycle productivity

I have the following code:

keywordindex = cPickle.load(open('keywordindex.p','rb'))#contains~340 thousand items
masterIndex = {}

indexes = [keywordindex]
for partialIndex in indexes:
    start = time.time()
    for key in partialIndex.iterkeys():
        if key not in masterIndex.keys():
            masterIndex[key]= partialIndex[key]
        elif key in masterIndex.keys():
            masterIndex[key].extend(partialIndex[key])
    cPickle.dump(masterIndex,open('MasterIndex.p','wb'))

    print int(time.time() - start), ' seconds'#every thousand loops

and I experience deteriorating performance as the cycle works, the first 10 thousand take about 5 seconds per thousand, but every 10 thousand or so takes another second until it takes 3 times longer. I tried to arrange the code in all possible ways, but I can not understand what causes it. Is there a reason for this? This is not a memory problem, I use only 30%

+3
source share
2 answers

This block contains two instances of terribly inefficient coding:

    if key not in masterIndex.keys():
        masterIndex[key]= partialIndex[key]
    elif key in masterIndex.keys():
        masterIndex[key].extend(partialIndex[key])

masterIndex, elif. not in , in . , :

    if key not in masterIndex.keys():
        masterIndex[key]= partialIndex[key]
    else:
        masterIndex[key].extend(partialIndex[key])

-, masterIndex dict. Dicts - ;-) ( .keys()) , dict . :

    if key not in masterIndex:
        masterIndex[key]= partialIndex[key]
    else:
        masterIndex[key].extend(partialIndex[key])

.

+8

masterIndex.keys(). , else:

if key not in masterIndex:
    ...
else:
    ...

in dict dict, - O (1) .

+3

More articles:


All Articles