A closure is a function that carries a memory of the variables that it used from the scope where it was defined. That memory is why an inner function can keep a counter running, can cache a lookup table, and can build a small state machine, all without a class anywhere in sight.
The previous lesson in this module covered "lambda and Functions as Values", and the next lesson in this module is about "Decorators Step by Step".
What is a closure, in one sentence?
When you define a function inside another function and then return it, the inner function keeps a reference to the enclosing scope instead of throwing it away when the outer function returns. The returned function is the closure, and the remembered variables are its free variables.
def make_greeter(greeting):
"""Return a function that greets people with the given greeting."""
def greet(name):
return greeting + ', ' + name + '!'
return greet
hi = make_greeter('Hi')
morning = make_greeter('Good morning')
print(hi('Ada'))
print(morning('Grace'))
print(hi.__name__)
print(len(hi.__closure__))
Two different greeters come out of the same factory, and each one holds on to its own greeting. They are independent objects, so changing one of them would not affect the other, and the local variable of the factory stayed alive after the factory returned.
You can see the machinery through __closure__, which is a tuple of cells that hold the captured values. The number of cells is the number of free variables that the function captured, and there is one of them here.
How does Python decide which variables a closure captures?
Only the names that the inner function reads and that are not defined inside it become free variables. Everything else, such as the locals of the inner function, the globals, and the built-ins, is looked up normally at call time. The __code__ object of a closure lists the captured names in co_freevars.
def make_counter(start=0):
"""Return a closure that counts up from start."""
count = start
def increment(step=1):
nonlocal count
count += step
return count
return increment
counter = make_counter()
print(counter(), counter(), counter())
print(counter.__code__.co_freevars)
The three calls print 1, 2, and 3, because count lives in the closure and not in the call. The nonlocal statement is what allows increment to rebind it. Without that keyword, count += step would create a brand new local variable, and the counter would always return 1.
Notice the difference between rebinding and mutating. The nonlocal keyword is only needed when you assign to the captured name. If you change an object instead, by appending to a list or by updating a dictionary, then no keyword is required at all, because you are changing the object and not the name.
def make_accumulator():
"""Return a closure that collects the values it is given."""
values = []
def add(value):
values.append(value)
return values
return add
acc = make_accumulator()
print(acc(1))
print(acc(2))
print(acc(1) is acc(2))
The last line prints True, because both calls return the very same list object, so the earlier values are still there. Shared mutable state inside a closure is a legitimate technique and a common source of bugs, because when two callers hold the same closure, they share the state that it carries.
Closures also keep the objects that they captured alive. If a closure captures a large data structure, then that structure cannot be cleaned up while the closure exists, even if nothing else refers to it. In a long-running server this is how memory quietly grows, because a callback that is stored on a long-lived object keeps its whole captured world alive. When a closure feels suspiciously heavy, look at what it captured with func.__closure__ and check the sizes of the objects inside those cells.
Why do all the lambdas in a loop return the same value?
This is the classic closure bug. A lambda captures the variable and not the value that the variable happened to hold when the lambda was created. Every lambda in the list shares that one variable, so by the time you call them, the loop has already finished and the variable holds its final value.
handlers = [lambda: name for name in ('ada', 'grace', 'linus')]
print([handler() for handler in handlers])
fixed = [lambda name=name: name for name in ('ada', 'grace', 'linus')]
print([handler() for handler in fixed])
The first list prints three copies of linus, and the second prints ada, grace, and linus. The only difference is the default argument, which binds the current value to a new name at the moment that the lambda is created. A closure-based equivalent is a small factory function.
def make_reader(name):
"""Return a function that reports the captured name."""
return lambda: name
readers = []
for name in ('ada', 'grace', 'linus'):
readers.append(make_reader(name))
print([reader() for reader in readers])
The same trap appears in callbacks that are registered inside a loop, in graphical interface handlers, and in asyncio tasks that are created in a loop. Whenever a callback outlives the iteration that created it, bind the value explicitly.
Should you use a closure, a class or a partial?
| Tool | Best when | Cost |
|---|---|---|
| Closure | You need a small piece of state plus a behaviour, and the state is private. | The state is invisible at the call site, and a debugger shows it as a cell variable. |
| Class | You need several related pieces of state, or comparison, or a public API. | More boilerplate, but everything is visible and easy to test. |
functools.partial | You only need to fix arguments, with no extra behaviour. | It only fixes arguments, and it cannot add any logic. |
| Class attribute or default argument | Never for mutable state. | It is shared across all calls, which is the classic mutable-default bug. |
Closures and classes are not rivals. A closure is often the right way to write a class that has one method and one piece of state, and a great deal of production code does exactly that.
Closures also show up in decorators without you noticing. Every decorator from the previous lesson, such as @announce and @repeat(3), is a closure that captured the function that it wraps. Once you recognise the shape, a whole family of Python libraries stops looking mysterious. You get callbacks that remember their configuration, validators that remember their rules, and factories that remember their settings.
What to remember
- A closure remembers the variables of the enclosing scope, and not a copy of their values.
- Use
nonlocalwhen the inner function rebinds a captured name. Mutating an object needs no keyword at all. - Bind loop values explicitly, with a default argument or with a factory function.
- Closures keep their state private and classes keep it visible. Choose based on how much state there is.
func.__closure__andfunc.__code__.co_freevarsshow you exactly what a function captured.
Try it yourself: Build a make_rate_limiter(calls_per_second) factory that returns a function which refuses to run more often than the limit, using a closure to remember the time of the last call. Give it a time.sleep of one second between two calls and check that the second call is delayed. Then convert the same idea into a class with a __call__ method, and compare which of the two versions you would rather test.