Generators and the Iterator Protocol

Generators are functions that produce their values one at a time instead of building a whole list up front. That single change turns "load everything into memory" into "stream it as you go", and it is the difference between a script that handles a two-gigabyte log file and one that dies on it.

What is the iterator protocol?

Python iterates over any object with two methods: __iter__ returns an iterator, and __next__ returns the next item or raises StopIteration when there are none. Every for loop you have written uses it, and you can implement it by hand.

class Countdown:
    """Count down from the starting number to one."""

    def __init__(self, start: int) -> None:
        self.current = start

    def __iter__(self) -> "Countdown":
        return self

    def __next__(self) -> int:
        if self.current <= 0:
            raise StopIteration
        value = self.current
        self.current -= 1
        return value


for number in Countdown(3):
    print(number)

The loop calls iter() once to get the iterator, then calls next() repeatedly until StopIteration is raised. StopIteration is how an iterator says "no more items" - raising it inside a generator body, by contrast, is a bug, because the machinery converts it into RuntimeError.

What does yield actually do?

Calling a function that contains yield does not run the function. It returns a generator object that remembers where it stopped. Each call to next() resumes the function at that point, runs until the next yield, and pauses again.

def evens(limit: int):
    """Yield even numbers below limit."""
    number = 0
    while number < limit:
        yield number
        number += 2


gen = evens(7)
print(type(gen).__name__)
print(next(gen))
print(next(gen))
print(list(evens(7)))

The type name is generator, the first two next calls produce 0 and 2, and list drains a fresh generator into [0, 2, 4, 6]. A generator is single-use: once it is exhausted it stays exhausted, which is why list(evens(7)) calls the factory again instead of reusing gen.

A generator expression is the same idea in one line: parentheses instead of brackets, and the comprehension is lazy.

squares = (n * n for n in range(5))
print(squares)
print(next(squares))
print(list(squares))

drained = (n * n for n in range(3))
print(list(drained))
print(list(drained))

The first generator has already produced 1, so list(squares) returns [4, 9, 16]. The second generator is used twice and the second list call returns an empty list - an exhausted generator stays empty, and that surprises people regularly.

Why should you care about laziness?

Consider reading a log file. Building a list of every line first means the whole file sits in memory; iterating over the file object directly means one line at a time. For a file that does not fit in memory, only the second approach works at all.

from itertools import islice


def naturals():
    """Yield positive integers, forever."""
    number = 1
    while True:
        yield number
        number += 1


print(list(islice(naturals(), 5)))

An infinite generator is perfectly legal - the function never finishes, but each yield hands back a value. islice takes the first five and stops, which is how you consume an unbounded stream safely.

The practical test is simple: if you would write for line in open(path), you can write a generator instead and the memory behaviour changes from O(file) to O(line).

Laziness also changes how much work gets done. The built-ins any() and all() stop pulling from a generator as soon as the answer is known, so a generator that could produce a million items may only ever produce three. The same is true of sum() over a generator of prices, and of a search loop that breaks on the first match. Building a list first throws that early exit away.

How do you send values into a generator?

yield evaluates to whatever is passed to send(). That turns a generator into a coroutine: you can push data in, and the generator decides what to do with it. throw() raises an exception inside the generator and close() runs its cleanup.

def echo():
    """Print whatever is sent in, until None arrives."""
    while True:
        received = yield
        if received is None:
            break
        print("got:", received)


gen = echo()
next(gen)          # start the generator up to the first yield
gen.send("hello")
gen.send("world")
gen.close()

The bare next(gen) is required before the first send: a generator has to reach its first yield before it has anywhere to put the value. gen.close() then raises GeneratorExit inside the generator, which is how a try/finally block in a generator releases resources such as open files.

For plain iteration, yield from is the tidier way to delegate to another iterable.

def tagged(source):
    """Yield each item from source, unchanged."""
    yield from source


print(list(tagged("abc")))
print(list(tagged([1, 2, 3])))

Which tools does the standard library give you?

Most stream problems are already solved in itertools, and using them is both faster and more readable than hand-rolling loops.

from itertools import chain, groupby, islice, zip_longest


def numbers():
    """Yield the numbers 1 to 6."""
    yield from range(1, 7)


print(list(chain([1, 2], [3, 4])))
print(list(islice(numbers(), 3)))
print([(key, len(list(group))) for key, group in groupby("aabbcca")])
print(list(zip_longest([1, 2, 3], ["a", "b"], fillvalue="-")))

chain concatenates iterables lazily, islice takes a slice of an infinite stream, groupby groups consecutive equal items, and zip_longest zips sequences of different lengths without dropping data. A related module, collections.abc, holds the abstract base classes - Iterator, Iterable, Generator - that make those protocols explicit in your own class definitions.

One detail about groupby is worth remembering: it groups consecutive equal items only, so "aabbcca" produces three groups, not two. If you need to group by a key regardless of position, sort by that key first.

Recap

• for loops use __iter__ and __next__; StopIteration ends the loop.
• A generator function returns a generator object when called and pauses at every yield.
• Generators are lazy and single-use; a list is eager and reusable.
• Use send, throw and close to drive a generator as a coroutine.
• Reach for itertools before writing your own lazy loops.

Try it yourself: Write a generator function read_chunks(path, size) that yields the file's contents in chunks of size characters instead of loading everything. Run it over a file you create with a few thousand characters and print the number of chunks and the length of each. Then convert it to a version that returns a list, and measure the memory difference with sys.getsizeof - the numbers make the laziness argument better than any explanation.