Advanced Python Features

Vax
32.7K views

Open Source Your Knowledge, Become a Contributor

Technology knowledge has to be shared and made accessible for free. Join the movement.

Create Content

Python is full of awesome features and tricks, that make you think "Wow! Python is so cool!".

We've done a selection of features we particularly like. We hope you'll learn something that will make you say "Neat! I didn't know that".

The source code is on GitHub, please feel free to come up with ideas to improve it.

Generators

A generator is an object that produces a sequence of values. It can be used as an iterator, which means that you can use it with a for statement, or use the next function to get the next value. However, you can iterate over the values only once.

A generator can be created using a function that uses the yield keyword to generate a value. When a generator function is called, a generator object is created.

yield operator
1
2
3
4
5
6
7
8
9
10
11
12
def fibonacci_generator():
a, b = 0, 1
while True:
yield a
a, b = b, a + b
# Print all the numbers of the Fibonacci sequence that are lower than 1000
for i in fibonacci_generator():
if i > 1000:
break
print(i)
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

For simple cases, it is possible to create a generator using a generator expression. As opposed to a list, the values will be computed on the fly instead of being computed once and stored in memory. Learn more about list and generator expressions.

generator expressions
1
2
3
4
5
6
7
8
9
10
11
a = (x * x for x in range(100))
# a is a generator object
print(type(a))
# Sum all the numbers of the generator
print(sum(a))
# There are no elements left in the generator
print(sum(a))
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Collections Module

collections is a module in the standard library that implements alternative container datatypes.

For example, a Counter is a collection where elements are stored as dictionary keys and their counts are stored as dictionary values:

Counter
1
2
3
4
5
6
7
8
9
from collections import Counter
a = Counter('blue')
b = Counter('yellow')
print(a)
print(b)
print((a + b).most_common(3))
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

A defaultdict is a subclass of dict, which allows to pass a factory used to create automatically a new value when a key is missing.

defaultdict
1
2
3
4
5
6
7
8
from collections import defaultdict
my_dict = defaultdict(lambda: 'Default Value')
my_dict['a'] = 42
print(my_dict['a'])
print(my_dict['b'])
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

The defaultdict can be used to create a tree data structure!

Tree
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
from collections import defaultdict
import json
def tree():
"""
Factory that creates a defaultdict that also uses this factory
"""
return defaultdict(tree)
root = tree()
root['Page']['Python']['defaultdict']['Title'] = 'Using defaultdict'
root['Page']['Python']['defaultdict']['Subtitle'] = 'Create a tree'
root['Page']['Java'] = None
print(json.dumps(root, indent=4))
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Itertools Module

itertools is a module in the standard library that allows you to create iterators for efficient looping.

For example, permutations allows you to generate all the possible ways of ordering a set of things:

permutations
1
2
3
4
5
from itertools import permutations
for p in permutations([1,2,3]):
print(p)
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Similarly, combinations generates all the possible ways of selecting items from a collection, such that (unlike permutations) the order does not matter:

combinations
1
2
3
4
5
from itertools import combinations
for c in combinations([1, 2, 3, 4], 2):
print(c)
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

itertools also contains utility functions such as chain, which takes iterables and creates a new iterator that returns elements from the given iterables sequentially, as a single sequence:

chain
1
2
3
4
5
from itertools import chain
for c in chain(range(3), range(12, 15)):
print(c)
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Packing / Unpacking

The * operator, known as the unpack or splat operator allows very convenient transformations, going from lists or tuples to separate variables or arguments and conversely.

Extended Iterable Unpacking
1
2
3
4
5
a, *b, c = [2, 7, 5, 6, 3, 4, 1]
print(a)
print(b)
print(c)
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

When the arguments for your function are already in a list or in a tuple, you can unpack them using *args if it's a list, or **kwargs if that's a dict.

unpacking arguments
1
2
3
4
5
6
7
8
9
10
11
12
def repeat(count, name):
for i in range(count):
print(name)
print("Call function repeat using a list of arguments:")
args = [4, "cats"]
repeat(*args)
print("Call function repeat using a dictionary of keyword arguments:")
kwargs = {'count': 4, 'name': 'cats'}
repeat(**kwargs)
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

The opposite is also possible, you can define a function that will pack all the arguments in a single tuple and all the keyword arguments in a single dict.

keyword arguments
1
2
3
4
5
6
def f(*args, **kwargs):
print("Arguments: ", args)
print("Keyword arguments: ", kwargs)
f(3, 4, 9, foo=42, bar=7)
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Decorators

A decorator is simply a function which takes a function as a parameter and returns a function.

For example, in the following code, the cache function is used as a decorator to remember the Fibonacci numbers that have already been computed:

decorators
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
def cache(function):
cached_values = {} # Contains already computed values
def wrapping_function(*args):
if args not in cached_values:
# Call the function only if we haven't already done it for those parameters
cached_values[args] = function(*args)
return cached_values[args]
return wrapping_function
@cache
def fibonacci(n):
print('calling fibonacci(%d)' % n)
if n < 2:
return n
return fibonacci(n-1) + fibonacci(n-2)
print([fibonacci(n) for n in range(1, 9)])
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

The functools module provides a few decorators, such as lru_cache which can do what we just did: memoization. It saves recent calls to save time when a given function is called with the same arguments:

lru_cache
1
2
3
4
5
6
7
8
9
10
11
from functools import lru_cache
@lru_cache(maxsize=None)
def fibonacci(n):
print('calling fibonacci(%d)' % n)
if n < 2:
return n
return fibonacci(n-1) + fibonacci(n-2)
print([fibonacci(n) for n in range(1, 9)])
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Context Managers

Context managers are mainly used to properly manage resources. The most common use of a context manager is the opening of a file: with open('workfile', 'r') as f:. However most developers have no idea how that really works underneath nor how they can create their own.

Actually, a context manager is just a class that implements the methods __enter__ and __exit__.

Context Manager
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
from time import time
class Timer():
def __init__(self, message):
self.message = message
def __enter__(self):
self.start = time()
return None # could return anything, to be used like this: with Timer("Message") as value:
def __exit__(self, type, value, traceback):
elapsed_time = (time() - self.start) * 1000
print(self.message.format(elapsed_time))
with Timer("Elapsed time to compute some prime numbers: {}ms"):
primes = []
for x in range(2, 500):
if not any(x % p == 0 for p in primes):
primes.append(x)
print("Primes: {}".format(primes))
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

For simple use cases, it's also possible to use a generator function with a single yield, using the @contextmanager decorator.

Context Manager Using @contextmanager
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
from contextlib import contextmanager
@contextmanager
def colored_output(color):
print("\033[%sm" % color, end="") # lines before the yield associated with __enter__ method
yield
print("\033[0m", end="") # lines after the yield associated with __exit__ method
print("Hello, World!")
with colored_output(31):
print("Now in color!")
print("So cool.")
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Voilà! We hoped you enjoyed our selection of best features in Python 3, feel free to share your feedback on the forum or on our Github :)

There is a great demand for other playgrounds about async / await, unit testing or metaclasses. Create your own playground and we will add a link to it.

Open Source Your Knowledge: become a Contributor and help others learn. Create New Content