Splitting, Joining and Parsing Text Data

Text arrives as one long string, and your program wants it as a list of fields, a dictionary, or a table. Python gives you three levels of tooling for that job: methods for splitting and joining plain text, the re module for patterns, and the json and csv modules for data that already has a structure. Knowing which level to pick is the difference between a parser that works and a parser that works until it meets a quote.

What are the different ways to split a string?

There are four, and they differ in more than their names suggest. split() without a separator splits on runs of whitespace and discards empty pieces. split(separator) splits on every occurrence of that exact separator and keeps empty pieces. The maxsplit argument limits how many splits happen, and everything after that stays in the final element. splitlines() breaks a text into lines, and rsplit() splits from the right, which is handy for pulling a file extension off the end.

import re

line = 'one two   three\tfour'
print('no sep     ->', line.split())
print('sep        ->', 'a,b,c'.split(','))
print('maxsplit   ->', 'a,b,c'.split(',', 1))
print('rsplit     ->', 'a,b,c'.rsplit(',', 1))
print('splitlines ->', 'one\ntwo\nthree'.splitlines())
print('keepends   ->', 'one\ntwo'.splitlines(keepends=True))
print('re.split   ->', re.split(r'[,;]\s*', 'a, b; c'))

try:
    'abc'.split('')
except ValueError as error:
    print('split("")  ->', error)

The split('') error is worth recognising rather than avoiding by accident. An empty separator has no meaningful definition, so Python refuses instead of guessing. If you want to split into individual characters, ask for a list instead.

The two separators produce different shapes of data. Whitespace splitting gives you clean fields with no empty values, which is what you want for a log line. Comma splitting preserves the structure exactly, including the empty field between two commas, which is what you want for a table where position matters.

How do you join pieces back together?

join() is a method on the separator, not on the list, which surprises almost everyone the first time. You call it on the string you want between the pieces, and pass the pieces as the argument.

parts = ['ada', 'grace', 'alan']

print('join       ->', ', '.join(parts))
print('numbers    ->', ', '.join(str(number) for number in (1, 2, 3)))
print('empty part ->', ', '.join(['a', '', 'b']))
print('generator  ->', '-'.join(str(index) for index in range(5)))
print('characters ->', ''.join(list('Tutorsbot')))

Two rules follow from the syntax. Every element must already be a string, which is why the second line wraps the numbers in str() before joining; and a generator works directly, so you can join a stream without materialising a list first.

Joining is also the fast way to build a long string, for the reason covered in the immutability lesson: ''.join(pieces) allocates the result once, while appending in a loop copies the whole string on every iteration. Two habits to avoid are result += piece inside a loop, and sum(pieces) with an empty string, which is a hidden way of writing the same slow loop.

How do you parse records and key-value text?

Splitting is only half of parsing. The other half is pairing each field with its name, and partition() is the tidy way to do that: it splits a string into head, separator and tail at the first occurrence of the separator, and always returns three pieces whether or not it found anything.

def parse_record(line):
    """Turn 'name=ada; age=36' into a dictionary."""
    fields = {}
    for chunk in line.split(';'):
        key, separator, value = chunk.partition('=')
        if separator:
            fields[key.strip()] = value.strip()
    return fields


print(parse_record('name=ada; age=36; city=london'))
print(parse_record('name=grace; age=45'))
print(parse_record('broken record'))

Notice the if separator check. partition() hands back empty strings when the separator is missing, so without that guard a malformed line would silently add an empty key to your dictionary. Parsers are supposed to be strict about the shape of their input.

For command-like text, the standard library has a dedicated parser. shlex.split() understands quotes and escaping, so it handles the difference between grep "def main" app.py as three arguments and not four.

import shlex

command = 'grep -n "def main" app.py'
print(shlex.split(command))
print(shlex.split('say "hello world"'))

When should you use json and csv instead?

Whenever the data has a structure that already exists. Parsing JSON by hand is a solved problem, and the standard library solves it properly, including the awkward cases: escaped quotes, unicode, nested objects and numbers that should be numbers.

import json

record = {'name': 'ada', 'tags': ['math', 'code'], 'score': 98}

print('compact ->', json.dumps(record))
print('pretty  ->')
print(json.dumps(record, indent=2))

loaded = json.loads('{"id": 7, "price": "19.90"}')
print('loaded  ->', loaded, '| types:', {key: type(value).__name__ for key, value in loaded.items()})

The gap between those two approaches is the reason to prefer the parser: the JSON parser gives you a real integer for the number and a string for the quoted price, and it tells you which is which. A hand-rolled parser gives you strings for everything and leaves you to guess.

For tabular text, the csv module handles quoting, embedded commas and newlines, which are exactly the cases where a naive split(',') falls apart.

import csv
import io

csv_text = 'name,score\nada,98\ngrace,95\n'
rows = list(csv.DictReader(io.StringIO(csv_text)))

print('rows  ->', rows)
print('total ->', sum(int(row['score']) for row in rows))

buffer = io.StringIO()
writer = csv.DictWriter(buffer, fieldnames=['name', 'score'])
writer.writeheader()
writer.writerow({'name': 'alan', 'score': 91})
print('written ->')
print(buffer.getvalue().strip())

DictReader turns each row into a dictionary keyed by the header, and DictWriter writes rows back out with a header line. Because both work on file-like objects, you can point them at a real file or, as here, at an in-memory buffer.

Two habits make parsers dependable. Validate the shape of what you parsed, usually by checking that the number of fields matches what you expected, and fail loudly rather than silently skipping rows you did not understand. And when the input is large, work a line at a time rather than reading everything into memory: the csv module is happy to take a file object and yield one row at a time, which keeps memory flat no matter how long the file is.

Your turn: summarise a mini CSV of orders

1. Write a CSV file by hand with three columns: order id, customer name and amount. Include one row where the customer name contains a comma, to see how the csv module handles it.
2. Read it with csv.DictReader and print each row as a dictionary. Confirm that the quoted name came back in one piece.
3. Total the amounts with a generator expression and print the total to two decimal places.
4. Now read the same file with a plain split(',') and print the rows. The row with the comma in the name will be split incorrectly, which is the point of the exercise.
5. Finally, parse one line of key=value text with your parse_record() function, then rebuild a single line from the dictionary in a fixed key order, and check that the rebuilt line matches the original.

Parsing recipes

• split() on whitespace is forgiving and drops empty fields; split(separator) is literal and keeps them.
• maxsplit bounds the number of splits, rsplit() splits from the right, and splitlines() handles line breaks.
• join() is called on the separator, every element must be a string, and generators are accepted.
• partition() always returns three pieces, which makes it a safe way to pair a field with its name.
• Use json and csv for structured data, and shlex.split() for command lines with quotes.