Memory optimization with Python slots

Hi guys!

In our previous article about hashes we briefly mentioned about property __slots__ . Now it is time to overview this Python feature and why you should use it in your Python code.

__slots__ were initially designed in order to optimise memory in Python, but soon people realised that this feature can do a lot of other cool things.

The idea of __slots__ optimisation is simple. If you tell Python upfront what exactly and how many properties object can have then Python will manage to store data of your objects more efficiency, comparing with objects where you can change amount of properties and object being defined.

Let me show you an example:

class A:
    __slots__ = ['b', 'c']

ia = A()
ia.b = 5
ia.z = 6

AttributeError: 'A' object has no attribute 'z'

'z' attribute wasn't listed in __slots__ that's why we see an exception here.

Let's make a small memory test. I'll use ipython_memory_usage for testing.

import ipython_memory_usage.ipython_memory_usage as imu
imu.start_watching_memory()

used 0.0000 MiB RAM

class A:
    def __init__(self, a, b):
        self.a = a
        self.b = b

instances = list([A(1,2) for i in range(10**6)])

used 171.1797 MiB RAM

class B:
    __slots__ = ['a', 'b']
    def __init__(self, a, b):
        self.a = a
        self.b = b
instances2 = list([B(1,2) for i in range(10**6)])

used 70.5625 MiB RAM

Looks like we've saved 100MiB on 1 million objects.

Because of this optimisation instance of B doesn't have __dict__ attributes, Python has used more optimal way to store, instead of using dict.

>>> ia = A(1,2)
>>> vars(ia)
{'a': 1, 'b': 2}

>>> ib = B(1,2)
>>> vars(ib)
TypeError: vars() argument must have __dict__ attribute

Conclusion

In this very short article I wanted to show you that __slots__ for your code can be not only the memory optimisation in Python, but it can also make your program much easier to read, because you tell at the very beginning that your class will use a specific set of attributes.

P.S.

As usual, we have found a couple of cool solutions where __slots__ are being used.

A very well commented solution of URL Normalization mission from simleo and Water Jars solution from papaeye, where he uses __slots__ together with collections.namedtuple (I plan to describe it more detailed in the future article).