Python course / Advanced

functools and itertools

Replace hand-written loops with the tools that already do the job, and judge when they read better than a loop. A chain of itertools can be clearer than the loop it replaces, or can hide a bug where nobody thinks to look.

reduce
singledispatch
operator
accumulate
product
combinations
groupby
sliding windows

Start this module

Runs in your browser. Start straight away; create an account when you want your progress kept.

grouping and running totals

from itertools import accumulate, groupby, product
from operator import itemgetter

orders = [("tea", 2), ("tea", 1), ("jam", 3), ("jam", 1), ("tea", 4)]

for name, group in groupby(orders, key=itemgetter(0)):
    counts = [count for _, count in group]
    print(name, counts, "total", sum(counts))

print("running:", list(accumulate([2, 1, 3, 1, 4])))
print("choices:", len(list(product(["tea", "jam"], ["small", "large"]))))

tea [2, 1] total 3
jam [3, 1] total 4
tea [4] total 4
running: [2, 3, 6, 7, 11]
choices: 4

What you will learn

Use functools.reduce to combine items pairwise, with an initial value for the empty case.
Choose a function's behaviour by the type of its argument without a chain of isinstance checks.
Replace small lambdas that fetch a field or call a method with the operator module's ready-made ones.
Produce running totals, maxima and balances lazily with itertools.accumulate.
Use groupby for runs of neighbouring items, and keep each group before moving on.
Walk a stream in overlapping windows of any size, lazily, and tell windows from batches.

The lessons in this module

reduce: folding a sequence into one value
When reduce is the wrong tool
singledispatch: one function, a version per type
cmp_to_key: sorting by comparison
operator's itemgetter, attrgetter and methodcaller
accumulate: every step of a fold
product: every combination of choices
Permutations and combinations
takewhile and dropwhile
compress, filterfalse and starmap
groupby, beyond sorting
Flattening
Sliding windows
The itertools recipes
cycle and repeat
Pipelines or loops
Why the built-ins are fast
Counting with a cached recursion
Capstone: a fair rota
Module review: the week's takings

Ready when you are

Start this module

Previous: Dataclasses and the data model. Next: Async syntax and event loops. All modules are on the course page.