Writing files in Python uses the same open() call that reading files uses, but the mode letter and the way that newlines are handled are where the surprises live. Get those two right and text output is boring in the best possible way. Get them wrong and you will produce files that look perfect on your own machine and break on somebody else's, and building CSV by hand is the fastest way to make that happen.
The previous lesson in this module covered "Reading Files: open(), Modes and with", and the next lesson in this module is about "Paths Done Right with pathlib".
For plain text, the whole contract is short. open(path, "w", encoding="utf-8") creates the file or empties it, handle.write(chunk) adds one string, and handle.writelines(chunks) adds many. Use "a" instead of "w" and nothing is truncated, so every write lands at the end of what is already in the file.
What is the difference between writing and appending?
The difference is entirely in the mode string. "w" truncates, so open a 10,000 line log with "w" and you are left with an empty file the instant the handle opens, before you have written a single character. "a" preserves, because every write() is positioned at the end of the file.
from pathlib import Path
import tempfile
with tempfile.TemporaryDirectory() as tmp:
log = Path(tmp) / 'events.log'
log.write_text('boot\n', encoding='utf-8')
with log.open('a', encoding='utf-8') as handle:
handle.write('login\n')
handle.write('logout\n')
with log.open('w', encoding='utf-8') as handle:
handle.write('truncated\n')
print(repr(log.read_text(encoding='utf-8')))
There is a third option worth knowing. "x" creates the file only if it does not already exist, and it raises FileExistsError otherwise. Use it whenever a script generates a file whose name would otherwise silently destroy earlier work, such as a dated export.
Why does newline handling matter so much?
Every text file is a sequence of bytes, and lines are separated by a line-ending convention. That convention is \n on Linux and macOS, and \r\n on classic Windows. The open() function takes a newline argument that decides what happens on the way in and on the way out.
newline | When writing | When reading |
|---|---|---|
None (default) | \n is translated to the line ending of the platform | all line endings become \n |
"" | no translation, you write exactly what you give | line endings are left untouched |
"\n" | no translation, \n stays \n | split on \n only |
On Windows, that default means handle.write("a\nb\n") puts \r\n on disk, while on Linux it puts \n. If your output has to be byte-identical on every machine, which matters for a test fixture, a checksum, or a file that another program parses, then write with newline="\n" and take control of the bytes yourself.
from pathlib import Path
import tempfile
import os
with tempfile.TemporaryDirectory() as tmp:
target = Path(tmp) / 'exact.txt'
with target.open('w', encoding='utf-8', newline='\n') as handle:
handle.write('first\n')
handle.write('second\n')
print(target.read_bytes())
other = Path(tmp) / 'default.txt'
with other.open('w', encoding='utf-8') as handle:
handle.write('first\n')
print('default mode :', other.read_bytes())
print('platform sep :', repr(os.linesep))
When you are reading, the default newline=None is usually what you want, because it normalises line endings so that line == "alpha\n" is true no matter which platform wrote the file. The csv module is the notable exception to this, and it is deliberate.
How does the csv module handle commas, quotes and newlines?
Building CSV by hand is a trap. A field that contains a comma has to be quoted, a field that contains a quote has to escape it by doubling it, and a field that contains a newline has to be quoted as well. The csv module knows all of that, so let it do the work for you.
import csv
from pathlib import Path
import tempfile
rows = [
['name', 'role'],
['Ravi, A.', 'tutor'],
['Ada "The Countess"', 'mentor'],
]
with tempfile.TemporaryDirectory() as tmp:
target = Path(tmp) / 'team.csv'
with target.open('w', encoding='utf-8', newline='') as handle:
csv.writer(handle).writerows(rows)
print(target.read_text(encoding='utf-8'))
with target.open('r', encoding='utf-8', newline='') as handle:
for record in csv.DictReader(handle):
print(record)
Two details in that snippet are not optional. The file is opened with newline="" because the csv module wants to write the row terminators itself, and on Windows the default translation would turn each \n that the writer emits into \r\n on top of the \r\n that it already wrote, which produces doubled line endings. The reader is opened with newline="" as well, so that the module can see the true end of each record instead of a pre-translated one.
| Input field | What the csv module writes |
|---|---|
Ravi | Ravi, with nothing to escape |
Ravi, A. | "Ravi, A.", quoted because of the comma |
Ada "The Countess" | "Ada ""The Countess""", quoted with doubled quotes |
How do I write dictionaries to CSV and read them back?
Real data usually arrives as dictionaries and not as lists of lists. csv.DictWriter takes a list of dictionaries plus the field names that you want in the header row, and csv.DictReader walks a file back into dictionaries, one for each row.
import csv
from pathlib import Path
import tempfile
orders = [
{'id': 1, 'customer': 'Asha', 'total': 42.5, 'notes': 'gift wrap, hurry'},
{'id': 2, 'customer': 'Ben', 'total': 17.0, 'notes': 'call me on "Friday"'},
]
with tempfile.TemporaryDirectory() as tmp:
target = Path(tmp) / 'orders.csv'
fields = ['id', 'customer', 'total', 'notes']
with target.open('w', encoding='utf-8', newline='') as handle:
writer = csv.DictWriter(handle, fieldnames=fields)
writer.writeheader()
writer.writerows(orders)
with target.open('r', encoding='utf-8', newline='') as handle:
for row in csv.DictReader(handle):
print(row['id'], row['customer'], row['total'], '|', row['notes'])
Two things catch people out with DictWriter. First, the writer needs the fieldnames before the first row is written, which is why the field list is given to the constructor. Second, if a dictionary is missing one of those keys, then DictWriter writes an empty cell, and if it has a key that is not in fieldnames, then that key is silently discarded unless you pass extrasaction="raise".
For reading, DictReader takes the header row from the first line of the file, so a file that has no header line produces a first row with meaningless keys. It is also worth knowing that reader.fieldnames gives you that header, and that a row with more values than the header has columns shows up under the restkey name as None.
There is one more decision to make on purpose, which is how numbers reach the file. The csv module writes the str() of whatever you hand it, so 42.5 becomes the text 42.5 and 42.50 becomes 42.50. It does not round, and it knows nothing about currency or percentages. If the exact formatting matters, then format the value yourself before it goes into the row.
Quick reference
"w"truncates,"a"appends, and"x"creates the file only if it is not there yet.- Pass
encoding="utf-8"on every write, so that the bytes are the same everywhere. - Use
newline="\n"when you need byte-identical output, and usenewline=""with thecsvmodule, always. - Never build CSV by joining strings with commas. Use
csv.writerandcsv.DictWriterinstead. writelines()takes an iterable of strings, and it does not add separators for you.
Try it yourself: turn a list of dictionaries into a report
Start with a list of dictionaries that describe a few orders, each with id, customer, total, and notes. Put at least one note that contains a comma and one note that contains a double quote, so that the escaping really matters.
Write that list to orders.csv with DictWriter, read the file back with DictReader, and print the parsed dictionaries. Then reopen the file in binary mode and print the raw bytes, checking whether the line terminators are \r\n or \n. Finally, change the writer's lineterminator to "\n" and compare the bytes again.