Generators are functions that produce their values one at a time instead of building a whole list up front. That single change turns "load everything into memory" into "stream it as you go", and it is the difference between a script that can handle a two-gigabyte log file and one that dies on it.
The previous lesson in this module covered "Decorators Step by Step", and the next lesson in this module is about "Context Managers and the with Statement".
What is the iterator protocol?
Python iterates over any object that has two methods. __iter__ returns an iterator, and __next__ returns the next item or raises StopIteration when there are no more items. Every for loop that you have written uses it, and you can implement it by hand.
class Countdown:
"""Count down from the starting number to one."""
def __init__(self, start):
self.current = start
def __iter__(self):
return self
def __next__(self):
if self.current <= 0:
raise StopIteration
value = self.current
self.current -= 1
return value
for number in Countdown(3):
print(number)
The loop calls iter() once to get the iterator, and then it calls next() again and again until StopIteration is raised. StopIteration is how an iterator says "there are no more items". Raising it inside the body of a generator, on the other hand, is a bug, because the machinery turns it into a RuntimeError.
What does yield actually do?
Calling a function that contains yield does not run the function at all. It returns a generator object that remembers where it stopped. Each call to next() resumes the function at that point, runs it until the next yield, and pauses again.
def evens(limit):
"""Yield even numbers below limit."""
number = 0
while number < limit:
yield number
number += 2
gen = evens(7)
print(type(gen).__name__)
print(next(gen))
print(next(gen))
print(list(evens(7)))
The type name is generator, the first two next calls produce 0 and 2, and list drains a fresh generator into [0, 2, 4, 6]. A generator can only be used once, because once it is exhausted it stays exhausted. That is why list(evens(7)) calls the factory again instead of reusing gen.
A generator expression is the same idea in a single line. You use parentheses instead of brackets, and the comprehension is lazy.
squares = (n * n for n in range(5))
print(squares)
print(next(squares))
print(list(squares))
drained = (n * n for n in range(3))
print(list(drained))
print(list(drained))
The first generator has already produced 1, so list(squares) returns [4, 9, 16]. The second generator is used twice, and the second list call returns an empty list, because an exhausted generator stays empty. That surprises people regularly.
Why should you care about laziness?
Think about reading a log file. Building a list of every line first means that the whole file sits in memory, while iterating over the file object directly means that you handle one line at a time. For a file that does not fit in memory, only the second approach works at all.
from itertools import islice
def naturals():
"""Yield positive integers, forever."""
number = 1
while True:
yield number
number += 1
print(list(islice(naturals(), 5)))
An infinite generator is perfectly legal. The function never finishes, but each yield hands back a value. islice takes the first five of them and then stops, and that is how you consume an endless stream safely.
| Type | Lazy | Reusable | Use it for |
|---|---|---|---|
list | No | Yes | Small collections that you need to slice, index, and pass around. |
tuple | No | Yes | Records and fixed-size groupings. |
| Generator | Yes | No | Streams, such as file lines, database rows, and pages of API results. |
| Iterator | Yes | No | Any object that you step through with next(). |
The practical test is simple. If you would write for line in open(path), then you can write a generator instead, and the memory behaviour changes from the size of the file to the size of one line.
Laziness also changes how much work gets done. The built-ins any() and all() stop pulling from a generator as soon as they know the answer, so a generator that could produce a million items may only ever produce three. The same is true of sum() over a generator of prices, and of a search loop that breaks on the first match. Building a list first throws that early exit away.
How do you send values into a generator?
yield evaluates to whatever is passed to send(). That turns a generator into a coroutine, so you can push data in and the generator decides what to do with it. The throw() method raises an exception inside the generator, and close() runs its cleanup code.
def echo():
"""Print whatever is sent in, until None arrives."""
while True:
received = yield
if received is None:
break
print('got:', received)
gen = echo()
next(gen) # start the generator up to the first yield
gen.send('hello')
gen.send('world')
gen.close()
The bare next(gen) is needed before the first send, because a generator has to reach its first yield before it has anywhere to put the value. The gen.close() call then raises GeneratorExit inside the generator, which is how a try and finally block inside a generator releases resources such as open files.
For plain iteration, yield from is the tidier way to hand the work over to another iterable.
def tagged(source):
"""Yield each item from source, unchanged."""
yield from source
print(list(tagged('abc')))
print(list(tagged([1, 2, 3])))
Which tools does the standard library give you?
Most stream problems are already solved in itertools, and using them is both faster and more readable than writing your own loops by hand.
from itertools import chain, groupby, islice, zip_longest
def numbers():
"""Yield the numbers 1 to 6."""
yield from range(1, 7)
print(list(chain([1, 2], [3, 4])))
print(list(islice(numbers(), 3)))
groups = []
for key, group in groupby('aabbcca'):
groups.append((key, len(list(group))))
print(groups)
print(list(zip_longest([1, 2, 3], ['a', 'b'], fillvalue='-')))
chain joins iterables lazily, islice takes a slice of an endless stream, groupby groups consecutive equal items, and zip_longest zips sequences of different lengths without dropping any data. There is a related module called collections.abc that holds the abstract base classes, such as Iterator, Iterable, and Generator, which make those protocols explicit in your own class definitions.
One detail about groupby is worth remembering. It groups consecutive equal items only, so "aabbcca" produces three groups and not two. If you need to group by a key no matter where it appears, sort by that key first.
Recap
forloops use__iter__and__next__, andStopIterationends the loop.- A generator function returns a generator object when you call it, and it pauses at every
yield. - Generators are lazy and can only be used once, while a list is eager and can be reused.
- Use
send,throw, andcloseto drive a generator like a coroutine. - Reach for
itertoolsbefore you write your own lazy loops.
Try it yourself: Write a generator function read_chunks(path, size) that yields the contents of a file in chunks of size characters instead of loading everything at once. Run it over a file that you create with a few thousand characters, and print the number of chunks together with the length of each one. Then convert it into a version that returns a list, and measure the memory difference with sys.getsizeof. The numbers make the argument about laziness better than any explanation could.