Geek asks and answers

Django caches a large list

My django application runs 25 bit binaries. Each of them has about 100,000 "records" of 256 bytes.

It takes me about 7 seconds to read the binary from disk and decode it using the struct python module. I turn the data into a list of approximately 100,000 elements, where each element is a dictionary with values ​​of various types (float, string, etc.).

In my django lists, you need to search this list. Clearly 7 seconds is too long.

I tried using django's low-level caching API to cache the entire list, but this will not work because the maximum size is 1 MB for any one cached item. I tried to cache 100,000 elements of the list separately, but it takes much more than 7 seconds - most of the time is spent unfastening the elements.

Is there a convenient way to store a large list in memory between requests? Can you come up with another way to cache an object for use by my django application?

+5
source share
2 answers

edit the element size limit on 10 m (more than 1 m), add

-I 10m

in / etc / memcached.conf and restart memcached

memcached.py, /usr/lib/python 2.7/dist-packages/django/core/cache/backends, :

class MemcachedCache(BaseMemcachedCache):
"An implementation of a cache binding using python-memcached"
def __init__(self, server, params):
    import memcache
    memcache.SERVER_MAX_VALUE_LENGTH = 1024*1024*10 #added limit to accept 10mb
    super(MemcachedCache, self).__init__(server, params,
                                         library=memcache,
                                         value_not_found_exception=ValueError)
+5

, , python-memcached, , SERVER_MAX_VALUE_LENGTH .

, __init__, FizxMike, _cache . python-memcached, SERVER_MAX_VALUE_LENGTH , :

from django.core.cache.backends.memcached import BaseMemcachedCache

DEFAULT_MAX_VALUE_LENGTH = 1024 * 1024

class MemcachedCache(BaseMemcachedCache):
    def __init__(self, server, params):
        #options from the settings['CACHE'][connection]
        self._options = params.get("OPTIONS", {})
        import memcache
        memcache.SERVER_MAX_VALUE_LENGTH = self._options.get('SERVER_MAX_VALUE_LENGTH', DEFAULT_MAX_VALUE_LENGTH)

        super(MemcachedCache, self).__init__(server, params,
                                             library=memcache,
                                             value_not_found_exception=ValueError)

    @property
    def _cache(self):
        if getattr(self, '_client', None) is None:
            server_max_value_length = self._options.get("SERVER_MAX_VALUE_LENGTH", DEFAULT_MAX_VALUE_LENGTH)
            #one could optionally send more parameters here through the options settings,
            #I simplified here for brevity
            self._client = self._lib.Client(self._servers,
                server_max_value_length=server_max_value_length)

        return self._client

, BaseMemcachedCache django.

django memcached backend : https://github.com/django/django/blob/master/django/core/cache/backends/memcached.py

!

+3

More articles:


All Articles