What Are Generators in Python?

A generator is a function that produces a sequence of values on demand instead of building them all up front. You write it with yield in place of return.

def counter(limit):
    for i in range(limit):
        yield i * 10


gen = counter(4)
print(type(gen).__name__)      # generator
print(next(gen))               # 0
print(next(gen))               # 10
print(list(gen))               # [20, 30] - only what was left

Calling counter does not run the body. It returns a generator object, and the body only starts when you ask for the first value. Each call to next runs the body up to the next yield, produces one value and suspends the function there, keeping its local variables alive.

The list call at the end shows the laziness clearly: it only produced the two values that had not been yielded yet. Once a generator is exhausted, calling next on it again raises StopIteration, and list on it gives an empty list.

Two details worth knowing. The generator remembers where it was, so resuming continues from the yield rather than restarting the function. And the local variables of the suspended frame are preserved, which is what makes the generator useful for loops over data that is being produced as you go.

A quick way to see the memory difference is to compare sys.getsizeof on a list of a million values with the same values produced by a generator expression. The list is tens of megabytes; the generator object is a couple of hundred bytes.

Both ways work, and the for loop is the one you will normally use.

def letters():
    yield "a"
    yield "b"
    yield "c"


for letter in letters():
    print(letter)


gen = letters()
print(next(gen))              # a
print(next(gen))              # b
print(next(gen))              # c
try:
    next(gen)
except StopIteration:
    print("StopIteration - the generator is exhausted")

The for loop calls next repeatedly and stops when StopIteration is raised, so it consumes the generator to the end without you handling the exception. The manual version takes one value at a time, which is what you want when you need to interleave two generators or stop partway through.

The exception at the end is the important part. A generator signals exhaustion by raising StopIteration, and once it is exhausted it stays exhausted — every further call to next raises again. That is why the try block is needed around the manual calls: an unhandled StopIteration would terminate the program.

Two shortcuts worth knowing. The built-in next function has a default argument, so next(gen, None) returns None instead of raising when the generator is done. And the walrus operator lets you write a for loop that captures the last value, or use it with next to read values in a loop condition.

A note on generators of generators: if you yield another generator from inside a generator, you should use yield from, which delegates iteration to it. That is simpler and faster than a manual for loop over the inner generator, and it propagates send and throw correctly.

A generator is a function that returns a generator object, which produces values one at a time when they are asked for. You write it with yield, and each time the caller asks for the next value the function runs up to the next yield and suspends there.

The difference from a list is eagerness and memory. A list is built completely, in memory, before you can touch it. A generator produces each value only when it is needed, so a generator over a million items occupies a few hundred bytes while the list of the same items takes tens of megabytes. It also means the values can be produced from a source that does not exist all at once — a file being read, a database cursor, a stream from a network.

Use a generator when the values are consumed once, in order, and the full set is either large or expensive to produce. Reading a file line by line, streaming rows from a database, and passing results through a chain of transformations are the classic cases. The pattern of a generator expression passed straight into sum, or into a function that consumes an iterable, is the idiomatic version of this.

Use a list when you need random access, when you will iterate more than once, when you need len, or when the collection is small. A generator is also single-use: once it is exhausted it stays exhausted, so if you need a second pass you have to recreate it.

Two related forms to know. The itertools module is built entirely around the same idea — count, cycle, chain, islice, groupby all produce lazy iterables. And async generators are the asyncio equivalent, using yield inside an async def function to produce values from an asynchronous source.