Python : Variables and Assignments

How does it work under the hood

Photo by Brookside Admin

Motivation — why should you care about what is going on under python’s hood?

First of all understanding what is going on under the python “hood” will make you a better engineer, I believe curiosity is a key! Python uses a different mechanism than most programing languages.

Second, it will help you find and understand bugs and make your code efficient.

And last but definitely not least is that we use variables and assignments everywhere in our code. So out of respect for the Python language you should take a deeper look into them ;)

Variables in Python are objects

In Python, every variable is an object!

This is much simpler than other languages, where some variables are native types and some are objects, also some variables are passed by value and some passed by reference.

Take a look at the following code, that shows you even an integer is a python object

>>> a = 1
>>> a.__sizeof__()
28
>>> a.__dir__()
['__repr__', '__hash__', '__getattribute__', '__lt__', '__le__', ..., '__dir__', '__class__']
>>> type(a)
<class 'int'>

We can see from the example above few things

• The variable ‘a’ has size of 28 bytes — size of an integer in python is 28 bytes (in C, C++ or Java size of integer is usually 4 bytes, int16_t is 2 bytes)
• The variable ‘a’ has a method named __dir__, that returns a list of valid attributes of that variable
• We can conclude that the variable ‘a’ is in fact an object, because it has methods, it size is 28 bytes (also hints that) and it type is of class ‘int’.

Variables in Python are always passed by reference

To anyone not knowing what do I mean by pass by reference or pass by value, I recommend seeing this link. In my opinion this is a must know for any developer.

All variables are objects and they are always passed by reference, this means that every time you pass a variable to a function or a method, that variable is passed to that function by reference, hence the pointer to that variable is passed to the function.

See the following example to truly understand that variables are passed by reference

>>> x = [1, 2, 3]
>>> y = x
>>> y.append(4)
>>> x
[1, 2, 3, 4]
>>> x = [1, 2, 3]
>>> y = x
>>> def append_4(items: List[int]) -> None:
...    items.append(4)
>>> append_4(x)
>>> x
[1, 2, 3, 4]
>>> y
[1, 2, 3, 4]

We can learn from the example above few important ideas

• ‘x’ is assigned to be a list [1, 2, 3] after assigning ‘y’ to be ‘x’, now ‘y’ is pointing to the same address as ‘x’ does, ‘y’ and ‘x’ are pointing to the same exact memory address.
• After appending 4 to the list ‘y’, also the list ‘x’ got the value 4 appended to it, this is because ‘x’ and ‘y’ are the same.
• When passing ‘x’ to the function append_4(), the value of ‘x’ changed (even when in different scope), this is because we passed a pointer of ‘x’ to the function append_4().
• Note that also the list ‘y’ changed, again because ‘y’ and ‘x’ point to the same memory address.

What is going on under the Python hood

In each scope Python creates a dictionary that keeps track of all the objects in that scope (these objects can be integers, floats, strings, functions and more). The dictionary provides a mapping from a variable name to an object.

When creating a variable a = 1, Python is adding to that dictionary the mapping a -> 1.

>>> a = 1
>>> b = 1.1
>>> c = "hello"
>>> def d():
...     return 42
...
>>> vars()
{'__name__': '__main__', '__doc__': None, ... 
'a': 1,
'b': 1.1, 
'c': 'hello', 
'd': <function d at 0x000001B400CB3E20>
}

By calling the built in vars() function you can take a look at variables dictionary of the current scope, some build in information is already there and the variables you created can be seen as well.

Common Bugs

Scopes bugs

Be aware when playing around with dictionaries that are not created in your scope.

>>> x = [1, 2, 3]
>>> def append_4(items: List[int]) -> None:
...    # some code
...    items.append(4)
...    # some additional code
>>> append_4(x)
>>> x
[1, 2, 3, 4]

Because variables are passed by reference, remember that if editing in inner scope causes changes outer scope as well.

Also you should be carful when editing __dict__, especially when using the keywords nonlocal global.

Late Binding Bugs

Example of the Late Binding bug can be seen it the following code example

>>> functions_list = []
>>> for i in range(10):
...    functions_list.append(lambda: i)
>>> functions_list[0]()
9      # expected 0 but got 9
>>> functions_list[5]()
9      # expected 5 but got 9
>>> i
9
>>> i = 99
>>> functions_list[0]()
99      # expected 0 but got 99

The functions_list is appended with lambda function that simply return an integer, when calling functions_list[0](), we expect to get 0, but we got 9. This happens because the i variable is bind late.

The values of variables are looked up at the time the inner function is called, whenever any of the retuned functions are called, the value of iis looked up in the current scope at call time, not at assignment time! By the time the loop is completed i = 9, so all the function will return 9.

Note, this has nothing to do with lambdas, the same exact behavior is exhibited with ordinary def function as well.

This is fixed usually be introducing a new scope, see below

>>> def create_lambda(i):
...    return lambda: i
>>> functions_list = []
>>> for i in range(10):
...    functions_list.append(create_lambda(i))
>>> fs[0]()
0
>>> fs[5]()
5

This fixed the problem because when introduction a new scope, the python built in dictionary of that scope will store the given input argument i and later when looking for that i it will be found in dictionary (and not the i that was incremented from the outer scope)

Conclusion

Understanding what really is going on under the hood of Python is very powerful and useful!

I hope that even more mature Python developers learned few things from this post.

Thanks for reading! I hope you found this article helpful.