The with statement guarantees that the setup runs before a block and that the cleanup runs after it, even when the block raises an error. The open() function is the one that you meet first, but the same protocol handles locks, database transactions, temporary directories, and timing blocks.
The previous lesson in this module covered "Generators and the Iterator Protocol", and this lesson is the last one in the Python Decorators, Closures and Generators module.
What does the with statement actually do?
Given an object with __enter__ and __exit__ methods, with calls __enter__, binds its return value to the name after as, runs the block, and then calls __exit__. The exit call receives the exception information if something went wrong, and it receives nothing if everything was fine.
from time import perf_counter
class Timer:
"""Measure how long the block takes."""
def __enter__(self):
self.start = perf_counter()
return self
def __exit__(self, exc_type, exc, tb):
self.elapsed = perf_counter() - self.start
print('block took ' + str(round(self.elapsed, 6)) + 's')
return False
with Timer() as timer:
sum(range(100000))
print('elapsed recorded:', round(timer.elapsed, 6))
Two things are worth noticing here. The value that __enter__ returns is what the as name is bound to, and returning self is the convention that lets the object carry state out of the block. Also, __exit__ returning False means "do not suppress the exception", while returning True would swallow it. That is a powerful tool, and it is also a good way to hide bugs.
The three arguments that __exit__ receives are the exception type, the exception instance, and the traceback. They are exactly what sys.exc_info() would give you inside an except block. When the block finishes cleanly, all three of them are None, and that is how a manager knows that nothing went wrong.
The with block is also safe when things go wrong. If the body raises an error, __exit__ still runs, so the cleanup cannot be skipped by the error. That is exactly the guarantee that a try and finally block gives you, with less noise.
How do you write a context manager without a class?
The contextlib.contextmanager decorator turns a generator function into a context manager. The code before the yield is the setup, the yield hands the value to the with statement, and the code after it is the cleanup.
from contextlib import contextmanager
@contextmanager
def open_text(path, mode='r'):
"""Open a file and close it when the block ends."""
handle = open(path, mode, encoding='utf-8')
try:
yield handle
finally:
handle.close()
with open_text('notes.txt', 'w') as handle:
handle.write('first line\nsecond line\n')
with open_text('notes.txt') as handle:
print(handle.read().strip())
The try and finally around the yield are not optional decoration. If the block raises an error, the generator is thrown into at the yield point, and the finally clause is what closes the file anyway. Without it, the file stays open until the garbage collector gets around to it.
The pattern generalises to any resource that has an acquire step and a release step, such as a lock, a temporary directory, or a database transaction.
Which context managers does the standard library give you?
You do not have to write most of these yourself, because the contextlib module covers the common shapes and several of them are worth reaching for by name.
import io
from contextlib import nullcontext, redirect_stdout, suppress
buffer = io.StringIO()
with redirect_stdout(buffer):
print('captured output')
print('buffer holds:', buffer.getvalue().strip())
with suppress(ZeroDivisionError):
1 / 0
print('the error was suppressed')
with nullcontext('default') as value:
print('nullcontext gave:', value)
redirect_stdout captures printed output, which is how test frameworks check what a function printed. suppress swallows one named exception type and lets everything else through. nullcontext does nothing at all, which is useful when a context manager is optional and you need to branch between using one and not using one.
Two more are worth knowing. contextlib.closing turns any object with a close() method into a context manager, and contextlib.chdir changes the working directory for the length of the block, which has been available since Python 3.11.
from contextlib import closing
class Resource:
"""A resource that must be closed."""
def __init__(self, name):
self.name = name
self.closed = False
def close(self):
self.closed = True
print('closed ' + self.name)
with closing(Resource('report')) as resource:
print('using', resource.name)
print('closed already:', resource.closed)
ExitStack is the general-purpose version of this idea. You register the resources as you discover them, and the stack closes them all in reverse order when the block ends, even if one of the closes raises an error. That is how libraries that open an unknown number of files keep their cleanup correct.
How do you suppress an exception, and when should you?
There are two mechanisms for this. Returning True from __exit__ swallows any exception that matches your condition, and contextlib.suppress does the same thing for named exception types.
class IgnoreValueError:
"""Swallow ValueError and let everything else through."""
def __enter__(self):
return self
def __exit__(self, exc_type, exc, tb):
return exc_type is ValueError
with IgnoreValueError():
raise ValueError('this is ignored')
print('execution continued')
try:
with IgnoreValueError():
raise KeyError('this one is not ignored')
except KeyError as error:
print('not ignored, so it reached the caller:', error)
The first block swallows the ValueError and execution continues. The second raises a KeyError, which the same context manager does not match, so it travels out of the with statement and is caught by the surrounding try. The traceback ends with KeyError: 'this one is not ignored'.
Suppress deliberately or not at all. Swallowing an exception usually removes the evidence of a bug, and a program that continues with bad state fails later in a place that is much harder to diagnose. The good uses are narrow and specific. An optional cleanup step that is allowed to fail, or a lookup that has a sensible default.
How do you manage several resources at once?
One with statement can hold several managers, separated by commas. Python enters them from left to right and exits them from right to left, and that is the order that you want when the resources depend on each other.
from contextlib import contextmanager
@contextmanager
def open_text(path, mode='r'):
"""Open a file and close it when the block ends."""
handle = open(path, mode, encoding='utf-8')
try:
yield handle
finally:
handle.close()
with open_text('input.txt', 'w') as source, open_text('copy.txt', 'w') as target:
source.write('payload\n')
target.write('payload\n')
with open_text('copy.txt') as handle:
print('copied contents:', handle.read().strip())
For a number of resources that is not known in advance, use ExitStack and register each one with stack.enter_context(...). The stack unwinds everything for you when the block ends.
The ordering rule is worth remembering, because it appears in every language that has a with statement. Resources are acquired from left to right and released from right to left. If handle B depends on handle A, write with a as first, b as second, because the reverse order would close A while B is still using it.
What to remember
withcalls__enter__, runs the block, and then always calls__exit__.- Use
@contextmanagerwith atryandfinallyaround theyieldfor generator-based managers. contextlibshipssuppress,nullcontext,redirect_stdout,closing, andExitStack.- Returning
Truefrom__exit__suppresses the exception, so use it deliberately. - Several managers can share one
withstatement, and they exit in the reverse order.
Try it yourself: Write a @contextmanager named temporary_file(prefix) that creates a file with tempfile.NamedTemporaryFile(delete=False), yields the path, and deletes the file when the block ends, even if the block raises an error. Use it to write a report to disk and then read the file back inside the same block. Add a second context manager of your own that restores the current working directory with os.getcwd() and os.chdir(), and then combine both of them in one with statement.