Python course / Advanced

Iterators and generators

Write generators that hand out values one at a time, and chain them into pipelines over data too big to hold. A generator you have already used up gives a silently wrong answer rather than an error.

iter and next
yield
laziness
endless generators
pipelines
yield from
tee
cleanup

Start this module

Runs in your browser. Start straight away; create an account when you want your progress kept.

values handed out one at a time

def readings(rows):
    for row in rows:
        yield int(row.strip())


def above(values, limit):
    for value in values:
        if value > limit:
            yield value


rows = [" 4\n", "9\n", " 12 \n", "7\n"]
pipeline = above(readings(rows), 5)

print(type(pipeline).__name__)
print(list(pipeline))
print("used up:", list(pipeline))

generator
[9, 12, 7]
used up: []

What you will learn

Know the two calls behind every for loop, and use them yourself when a loop is the wrong shape.
Write a function that hands out values one at a time, instead of building and returning a list.
Write a generator that never stops, and take from it only what you need.
Recognise the silent wrong answer a used-up generator gives, and make code fail loudly or cope.
Chain generators into stages, so a large file is processed one line at a time from start to end.
Choose a list, a set or a generator by what the caller does with the result.

The lessons in this module

What a for loop really does
Iterables and iterators
Writing an iterator class
Generator functions: yield
What calling a generator does, and does not do
Endless generators
Generator expressions, from the inside
The generator that was already used up
When you need a generator's values twice
Pipelines
yield from, and what a generator returns
Cleaning up after a generator
send(): values going the other way
Generators that remember
Iterable classes, the usual way
iter() with a sentinel
When not to use a generator
Measuring what laziness saves
Capstone: streaming the stall's log
Module review: the fridge sensor

Ready when you are

Start this module

Previous: Standard library tour. Next: Decorators. All modules are on the course page.