Text arrives as one long string, and your program wants it as a list of fields, a dictionary, or a table. Python gives you three levels of tooling for that job. There are methods for splitting and joining plain text, there is the re module for patterns, and there are the json and csv modules for data that already has a structure. Knowing which level to pick is the difference between a parser that works and a parser that works until it meets a quote.
The previous lesson in this module covered "Searching, Counting and Replacing in Strings", and this lesson is the last one in the Python Strings module.
What are the different ways to split a string?
There are four of them, and they differ in more than their names suggest. split() without a separator splits on runs of whitespace and throws the empty pieces away. split(separator) splits on every occurrence of that exact separator and keeps the empty pieces. The maxsplit argument limits how many splits happen, and everything after that stays in the final element. splitlines() breaks a text into lines, and rsplit() splits from the right, which is handy for pulling a file extension off the end.
import re
line = 'one two three\tfour'
print('no sep ->', line.split())
print('sep ->', 'a,b,c'.split(','))
print('maxsplit ->', 'a,b,c'.split(',', 1))
print('rsplit ->', 'a,b,c'.rsplit(',', 1))
print('splitlines ->', 'one\ntwo\nthree'.splitlines())
print('keepends ->', 'one\ntwo'.splitlines(keepends=True))
print('re.split ->', re.split(r'[,;]\s*', 'a, b; c'))
try:
'abc'.split('')
except ValueError as error:
print('split("") ->', error)
The split('') error is worth recognising rather than running into by accident. An empty separator has no meaningful definition, so Python refuses instead of guessing. If you want to split a string into its individual characters, ask for a list instead.
The two kinds of separator produce different shapes of data. Splitting on whitespace gives you clean fields with no empty values, which is what you want for a log line. Splitting on a comma preserves the structure exactly, including the empty field between two commas, which is what you want for a table where the position of each column matters.
How do you join pieces back together?
join() is a method on the separator and not on the list, which surprises almost everybody the first time. You call it on the string that you want between the pieces, and you pass the pieces in as the argument.
parts = ['ada', 'grace', 'alan']
print('join ->', ', '.join(parts))
print('empty part ->', ', '.join(['a', '', 'b']))
print('characters ->', ''.join(list('Tutorsbot')))
numbers = []
for number in (1, 2, 3):
numbers.append(str(number))
print('numbers ->', ', '.join(numbers))
Two rules follow from the syntax. Every element must already be a string, and that is why the numbers are turned into text before they are joined. A generator also works directly, so you can join a stream without building a list first.
Joining is also the fast way to build a long string, for the reason that the immutability lesson covered. ''.join(pieces) allocates the result once, while appending inside a loop copies the whole string on every single pass. Two habits to avoid are result += piece inside a loop, and sum(pieces) with an empty string, which is a hidden way of writing that same slow loop.
How do you parse records and key-value text?
Splitting is only half of parsing. The other half is pairing each field with its name, and partition() is the tidy way to do that. It splits a string into a head, a separator, and a tail at the first occurrence of the separator, and it always gives you three pieces whether or not it found anything.
def parse_record(line):
"""Turn 'name=ada; age=36' into a dictionary."""
fields = {}
for chunk in line.split(';'):
key, separator, value = chunk.partition('=')
if separator:
fields[key.strip()] = value.strip()
return fields
print(parse_record('name=ada; age=36; city=london'))
print(parse_record('name=grace; age=45'))
print(parse_record('broken record'))
Notice the if separator check. partition() hands back empty strings when the separator is missing, so without that guard a malformed line would quietly add an empty key to your dictionary. A parser is supposed to be strict about the shape of its input.
For command-like text, the standard library has a dedicated parser. shlex.split() understands quotes and escaping, so it handles the difference between grep "def main" app.py as three arguments and not four.
import shlex
command = 'grep -n "def main" app.py'
print(shlex.split(command))
print(shlex.split('say "hello world"'))
When should you use json and csv instead?
Whenever the data already has a structure. Parsing JSON by hand is a solved problem, and the standard library solves it properly, including all of the awkward cases: escaped quotes, unicode, nested objects, and numbers that should be numbers.
import json
record = {'name': 'ada', 'tags': ['math', 'code'], 'score': 98}
print('compact ->', json.dumps(record))
print('pretty ->')
print(json.dumps(record, indent=2))
loaded = json.loads('{"id": 7, "price": "19.90"}')
print('loaded ->', loaded)
for key, value in loaded.items():
print(' ', key, 'is a', type(value).__name__)
The gap between those two approaches is the reason to prefer the real parser. The JSON parser gives you a real integer for the number and a string for the quoted price, and it tells you which one is which. A hand-written parser gives you strings for everything and leaves you to guess.
For tabular text, the csv module handles quoting, embedded commas, and newlines, which are exactly the cases where a naive split(',') falls apart.
import csv
import io
csv_text = 'name,score\nada,98\ngrace,95\n'
rows = list(csv.DictReader(io.StringIO(csv_text)))
print('rows ->', rows)
total = 0
for row in rows:
total += int(row['score'])
print('total ->', total)
buffer = io.StringIO()
writer = csv.DictWriter(buffer, fieldnames=['name', 'score'])
writer.writeheader()
writer.writerow({'name': 'alan', 'score': 91})
print('written ->')
print(buffer.getvalue().strip())
DictReader turns each row into a dictionary that is keyed by the header, and DictWriter writes rows back out with a header line. Because both of them work on file-like objects, you can point them at a real file or, as here, at an in-memory buffer.
Two habits make parsers dependable. Validate the shape of what you parsed, usually by checking that the number of fields matches what you expected, and fail loudly instead of quietly skipping rows that you did not understand. And when the input is large, work a line at a time rather than reading everything into memory. The csv module is happy to take a file object and give you one row at a time, which keeps memory flat no matter how long the file is.
Your turn: summarise a mini CSV of orders
- Write a CSV file by hand with three columns: order id, customer name, and amount. Include one row where the customer name contains a comma, so that you can see how the
csvmodule handles it. - Read it with
csv.DictReaderand print each row as a dictionary. Confirm that the quoted name came back in one piece. - Total the amounts and print the total to two decimal places.
- Now read the same file with a plain
split(',')and print the rows. The row with the comma inside the name will be split incorrectly, and that is the point of the exercise. - Finally, parse one line of
key=valuetext with yourparse_record()function, then rebuild a single line from the dictionary in a fixed key order, and check that the rebuilt line matches the original.
Parsing recipes
split()on whitespace is forgiving and drops empty fields, whilesplit(separator)is literal and keeps them.maxsplitbounds the number of splits,rsplit()splits from the right, andsplitlines()handles line breaks.join()is called on the separator, every element must be a string, and generators are accepted.partition()always returns three pieces, which makes it a safe way to pair a field with its name.- Use
jsonandcsvfor structured data, andshlex.split()for command lines that contain quotes.