Geek asks and answers

How to decorate a Python object with a mutex

I am new to python and am currently trying to learn threads. I'm tired of using locks to make my resources thread safe, because they are essentially not resource bound, so I have to forget to acquire and / or release them every time my code interacts with the resource. Instead, I would like to be able to "wrap" (or decorate?) The Object so that all its methods and getters / seters attributes are atomic. something like that:

state = atomicObject(dict())

# the following is atomic/thread-safe
state["some key"] = "some value"

Is it possible? If so, what is the best practice way to implement it?

EDIT: A good answer to this question is available in How to make inline containers (sets, dictations, lists) safe threads? . However; as abarnert and jsbueno have demonstrated, the solution I proposed (lock automation) is usually not a good idea, since determining the right detail for atomic operations requires some intelligence and it is probably difficult (or impossible) to automate properly.

The problem still remains that locks are in no way related to the resources they are intended to protect, so my new question is: What is a good way to associate a lock with an object?

Proposed Solution # 2: I suppose there might be a way to bind a lock to an object, so trying to access this object without first getting the lock causes an error, but I see how this can become complicated.

EDIT: The following code is not very suitable for the question. I posted it to demonstrate that I tried to solve the problem myself and got lost before posting this question.

For the record, I wrote the following code, but it does not work:

import threading    
import types
import inspect

class atomicObject(object):

    def __init__(self, obj):
        self.lock = threading.RLock()
        self.obj = obj

        # keep track of function handles for lambda functions that will be created
        self.funcs = []

        # loop through all the attributes of the passed in object
        # and create wrapped versions of each attribute
        for name in dir(self.obj):
            value = getattr(self.obj, name)
            if inspect.ismethod(value):
                # this is where things get really ugly as i try to work around the
                # limitations of lambda functions and use eval()... I'm not proud of this code
                eval("self.funcs.append(lambda self, *args, **kwargs: self.obj." + name + "(*args, **kwargs))")
                fidx = str(len(self.funcs) - 1)
                eval("self." + name + " = types.MethodType(lambda self, *args, **kwargs: self.atomize(" + fidx + ", *args, **kwargs), self)")

    def atomize(self, fidx, *args, **kwargs):
        with self.lock:
            return self.functions[fidx](*args, **kwargs)

I can create atomObject (dict ()), but when I try to add a value to the object, I get an error; "atomicObject does not support element assignment."

+6
source share
5 answers

eval, .

:

>>> d = dict()
>>> inspect.ismethod(d.__setitem__)

, ismethod:

true, , Python.

, C ( .NET, Java, .. Python) , Python.

, , callable inspect.isroutine .

, , , , eval :

self.__cmp__ = types.MethodType(lambda self, *args, **kwargs: self.atomize(0, *args, **kwargs) self)

... , .

eval . , setattr. lambda s. def; - , , lambda, , .

, , . (, , , - , .) __getattr__. ( , ... -, , , , , , . , .)

, , , , :

class atomicObject(object):

    def __init__(self, obj):
        self.lock = threading.Lock()
        self.obj = obj

    def __getattr__(self, name):
        attr = getattr(self.obj, name)
        print(attr)
        if callable(attr):
            def atomized(*args, **kwargs):
                with self.lock:
                    attr(*args, **kwargs)
            return atomized
        return attr

, . :

>>> d = atomicObject(dict())
>>> d.update({'a': 4}) # works
>>> d['b'] = 5
TypeError: 'atomicObject' object does not support item assignment

? __setitem__, :

>>> d.__setitem__
<method-wrapper '__setitem__' of dict object at 0x100706830>
>>> d.__setitem__('b', 5) # works

, , docs, , . atomicObject __setitem__.

, , __str__ __repr__ object:

>>> d
<__main__.atomicObject object at 0x100714690>
>>> print(d)
<__main__.atomicObject object at 0x100714690>
>>> d.obj #cheating
{'a': 4, 'b': 5}

, - , - , :

>>> AtomicDict = make_atomic_wrapper(dict)
>>> d = AtomicDict()

, ... , .

:

d = AtomicDict()
d['abc'] = 0
d['abc'] += 1

. __getitem__, __setitem__.

, , d . 20 , d['abc'] += 1 . , __getitem__, 0. , __setitem__, 1.

. 2000. 125.

+4

, : Atomic, - , __getattribute__ - , " ", , - - .

- Python, import this , /, , : " , "."::)

: Python - , , . - - : Python Python , - Python - "Python GIL", , - (, , , Numpy).

r- multiprocessing threading - "" - , .

+2

, Python, "" - , , .

, . , .

Python 2 3 ( @abarnet - " " .)

import threading
from functools import wraps

#see http://stackoverflow.com/questions/15960881/how-to-decorate-a-python-object-with-a-mutex/15961762#15960881

printing = False

lock = threading.Lock()
def atomize(func):
    @wraps(func)
    def wrapper(*args, **kw):
        with lock:
            if printing:
                print ("atomic")
            return func(*args, **kw)
    return wrapper

def Atomic(cls):
    new_dict = {}
    for key, value in cls.__dict__.items():
        if hasattr(value, "__get__"):
            def get_atomic_descriptor(desc):
                class Descriptor(object):
                    @atomize
                    def __get__(self, instance, owner):
                        return desc.__get__(instance, owner)
                    if hasattr(desc, "__set__"):
                        @atomize
                        def __set__(self, instance, value):
                            return desc.__set__(instance, value)
                    if hasattr(desc, "__delete__"):
                        @atomize
                        def __delete__(self, instance):
                            return desc.__delete__(instance)
                return Descriptor()
            new_dict[key] = get_atomic_descriptor(value)
        elif callable(value):
            new_dict[key] = atomize(value)
        else:
            new_dict[key] = value
    return type.__new__(cls.__class__, cls.__name__, cls.__bases__, new_dict)


if __name__ == "__main__": # demo:
    printing = True

    @atomize
    def sum(a,b):
        return a + b

    print (sum(2,3))

    @Atomic
    class MyObject(object):
        def _get_a(self):
            return self.__a

        def _set_a(self, value):
            self.__a = value +  1

        a = property(_get_a, _set_a)

        def smurf(self, b):
            return self.a + b

    x = MyObject()
    x.a = 5
    print(x.a)
    print (x.smurf(10))

    # example of atomized function call - based on
    # @abarnet code at http://pastebin.com/MrtR6Ufh
    import time, random
    printing = False
    x = 0

    def incr():
        global x
        for i in range(100):
            xx = x
            xx += 1
            time.sleep(random.uniform(0, 0.02))
            x = xx

    def do_it():
        threads = [threading.Thread(target=incr) for _ in range(20)]
        for t in threads:
            t.start()
        for t in threads:
            t.join()

    do_it()
    print("Unlocked Run: ", x)

    x = 0
    incr = atomize(incr)
    do_it()
    print("Locked Run: ", x)

NB: "eval" "exec" Python, - - . , , , eval.

0

We will return to these years later. I believe the context manager is the perfect solution to my original problem. I know that Locks supports contextual management, but you still have the problem of ensuring the connection between the lock and the locked resource. Instead, I imagine something like the following:

class Locked:
    def __init__(self, obj):
        super().__init__()
        self.__obj = obj
        self.lock = threading.RLock()

    def __enter__(self):
        self.lock.acquire()
        return self.__obj

    def __exit__(self, *args, **kwargs):
        self.lock.release()


guard = Locked(dict())

with guard as resource:
    do_things_with(resource)
0
source

More articles:


All Articles