Generator vs List: Memory Trade-offs

The difference is dramatic, and it comes down to eagerness. A list comprehension builds every value immediately; a generator expression produces each one only when it is asked for.

import sys

data = range(1_000_000)

as_list = [n * 2 for n in data]
as_gen = (n * 2 for n in data)

print(sys.getsizeof(as_list))     # 8448728 - about 8 MB
print(sys.getsizeof(as_gen))      # 200 - a tiny generator object
print(len(list(as_gen)))          # 1000000

The list occupies roughly 8.4 megabytes just for the list of pointers, and that figure does not count the million integer objects it refers to. The generator object is 200 bytes, because all it holds is a reference to the range and a position in it.

The last line is the catch. Converting the generator to a list gives you the same million values and the same 8 megabytes — the memory was never saved, it was merely deferred. The saving is real only when you consume the values and let them go as you go.

That is the honest framing for an interview: a generator does not use less memory to hold the same data. It uses less memory because it never holds all the data at once, which only helps if you process the stream as it arrives rather than collecting it.

A generator that yields batches keeps the working set small no matter how big the input is.

def read_chunks(source, size=3):
    chunk = []
    for line in source:
        chunk.append(line.rstrip())
        if len(chunk) == size:
            yield chunk
            chunk = []
    if chunk:
        yield chunk


lines = ["a\n", "b\n", "c\n", "d\n", "e\n"]
for chunk in read_chunks(lines):
    print(chunk)

The output is two chunks: a, b, c and then d, e. The final yield handles the remainder, so a stream whose length is not a multiple of the chunk size is not truncated.

This is the shape of real streaming code. The source could be a file object, a database cursor, a network response or a subprocess pipe, and the generator reads from it one item at a time and hands you fixed-size batches. The generator holds only the current chunk, so processing a ten-gigabyte file uses the same memory as processing a ten-line file.

Two details make it robust. The chunk list is rebound to a fresh list after each yield rather than being cleared, which avoids the classic bug where the consumer holds a reference to the list you are still mutating. And the trailing yield means the last partial chunk is not lost.

The same idea scales up. itertools islice will cut a slice of any iterator without materialising it, and the standard library's csv, json and zipfile readers all stream rather than load. If you find yourself writing list(reader) anywhere, a generator is usually what you meant.

Use a generator when the values are consumed once, in order, and either the collection is large or the values are expensive to produce. The three classic cases are reading a file, streaming rows from a database, and chaining transformations over a stream — where each of those can be written as a generator that never holds more than one value at a time.

Use a list when you need random access, when you will iterate more than once, when you need len, when you need to slice, or when the collection is small enough that the memory does not matter. A list is also the better choice when you want to pass the data to code that expects a concrete sequence.

What you give up is the whole random-access interface. A generator has no len, no indexing and no slicing, because producing the tenth item might mean producing the first nine as well. You also lose the ability to iterate more than once — a generator is exhausted after one pass, and calling next on an exhausted generator raises StopIteration forever. There is also no sorting, no reverse and no in operator with the same efficiency, since each of those would have to consume the whole generator.

One practical consequence people are surprised by. If you write a generator and then convert it with list, you have thrown away the entire benefit — the values are all in memory again, and you have paid for the conversion on top. If a generator ends up being materialised, the right answer is usually to find a way to consume it as it flows instead.