Writing Files: Text, Newlines and CSV

Writing files in Python uses the same open() call as reading, but the mode letter and the newline handling are where the surprises live. Get those two right and text output is boring in the best way; get them wrong and you will produce files that look fine on your machine and break on someone else's — and a hand-rolled CSV is the fastest way to make that happen.

For plain text, the whole contract is: open(path, "w", encoding="utf-8") creates the file or empties it, handle.write(chunk) adds a string, and handle.writelines(chunks) adds many. Add "a" instead of "w" and nothing is truncated, so every write lands at the end of what is already there.

What is the difference between writing and appending?

The distinction is entirely the mode string. "w" truncates: open a 10,000-line log with "w" and you are left with an empty file the instant the handle opens, before you have written a single character. "a" preserves: every write() is positioned at the end of the file.

from pathlib import Path
import tempfile

with tempfile.TemporaryDirectory() as tmp:
    log = Path(tmp) / "events.log"
    log.write_text("boot\n", encoding="utf-8")

    with log.open("a", encoding="utf-8") as handle:
        handle.write("login\n")
        handle.write("logout\n")

    with log.open("w", encoding="utf-8") as handle:
        handle.write("truncated\n")

    print(repr(log.read_text(encoding="utf-8")))

There is a third option worth knowing. "x" creates the file only if it does not already exist and raises FileExistsError otherwise. Use it whenever a script generates a file whose name would otherwise silently destroy earlier work, such as a dated export.

Why does newline handling matter so much?

Every text file is a sequence of bytes, and lines are separated by a line-ending convention: \n on Linux and macOS, \r\n on classic Windows. Python's open() takes a newline argument that decides what happens on the way in and on the way out.

On Windows, that default means handle.write("a\nb\n") puts \r\n on disk. On Linux it puts \n. If your output has to be byte-identical on every machine — a test fixture, a checksum, a file another program parses — write with newline="\n" and take control of the bytes yourself.

from pathlib import Path
import tempfile
import os

with tempfile.TemporaryDirectory() as tmp:
    target = Path(tmp) / "exact.txt"
    with target.open("w", encoding="utf-8", newline="\n") as handle:
        handle.write("first\n")
        handle.write("second\n")
    print(target.read_bytes())

    other = Path(tmp) / "default.txt"
    with other.open("w", encoding="utf-8") as handle:
        handle.write("first\n")
    print("default mode :", other.read_bytes())
    print("platform sep :", repr(os.linesep))

When reading, the default newline=None is usually what you want, because it normalises line endings so that line == "alpha\n" is true no matter which platform wrote the file. The csv module is the notable exception, and it is deliberate.

How does the csv module handle commas, quotes and newlines?

Hand-rolling CSV is a trap. A field containing a comma has to be quoted, a field containing a quote has to be escaped by doubling it, and a field containing a newline has to be quoted too. The csv module knows all of that, so let it do the work.

import csv
from pathlib import Path
import tempfile

rows = [["name", "role"], ["Ravi, A.", "tutor"], ['Ada "The Countess"', "mentor"]]

with tempfile.TemporaryDirectory() as tmp:
    target = Path(tmp) / "team.csv"
    with target.open("w", encoding="utf-8", newline="") as handle:
        csv.writer(handle).writerows(rows)
    print(target.read_text(encoding="utf-8"))

    with target.open("r", encoding="utf-8", newline="") as handle:
        for record in csv.DictReader(handle):
            print(record)

Two details in that snippet are not optional. The file is opened with newline="" because the csv module wants to write the row terminators itself, and on Windows the default translation would turn each \n the writer emits into \r\n on top of the \r\n it already wrote, producing doubled line endings. And the reader is opened with newline="" too, so the module can see the true end of each record rather than a pre-translated one.

How do I write dictionaries to CSV and read them back?

Real data usually arrives as dictionaries, not as lists of lists. csv.DictWriter takes a list of dicts plus the field names you want in the header row, and csv.DictReader walks a file back into dictionaries, one per row.

import csv
from pathlib import Path
import tempfile

orders = [
    {"id": 1, "customer": "Asha", "total": 42.5, "notes": "gift wrap, hurry"},
    {"id": 2, "customer": "Ben", "total": 17.0, "notes": 'call me on "Friday"'},
]

with tempfile.TemporaryDirectory() as tmp:
    target = Path(tmp) / "orders.csv"
    fields = ["id", "customer", "total", "notes"]
    with target.open("w", encoding="utf-8", newline="") as handle:
        writer = csv.DictWriter(handle, fieldnames=fields)
        writer.writeheader()
        writer.writerows(orders)

    with target.open("r", encoding="utf-8", newline="") as handle:
        for row in csv.DictReader(handle):
            print(row["id"], row["customer"], row["total"], "|", row["notes"])

Two things catch people out with DictWriter. First, writerows needs fieldnames before the first row is written, which is why the field list is passed to the constructor. Second, if a dictionary is missing one of those keys, DictWriter writes an empty cell; if it has a key that is not in fieldnames, that key is silently discarded unless you pass extrasaction="raise".

For reading, DictReader takes the header row from the first line of the file, so a file without a header line produces a first row with garbage keys. Also worth knowing: reader.fieldnames gives you that header, and a row that has more values than the header has columns shows up as None under the restkey name.

One more decision to make deliberately: how numbers reach the file. The csv module writes the str() of whatever you hand it, so 42.5 becomes the text 42.5 and 42.50 becomes 42.50. It does not round, and it does not know about currency or percentages. If the exact formatting matters, format the value yourself before it goes into the row.

Quick reference

• "w" truncates, "a" appends, "x" creates only if absent.
• Pass encoding="utf-8" on every write so the bytes are the same everywhere.
• Use newline="\n" when you need byte-identical output; use newline="" with the csv module, always.
• Never build CSV by joining strings with commas. Use csv.writer and csv.DictWriter.
• writelines() takes an iterable of strings; it does not add separators for you.

Try it yourself: turn a list of dictionaries into a report

Start with a list of dictionaries describing a few orders, each with id, customer, total and notes. Put at least one note containing a comma and one containing a double quote, so the escaping actually matters.

Write that list to orders.csv with DictWriter, read the file back with DictReader, and print the parsed dictionaries. Then reopen the file in binary mode and print the raw bytes, checking whether the line terminators are \r\n or \n. Finally, change the writer's lineterminator to "\n" and compare the bytes again.