How Do You Remove Duplicates from a List?

There are three common approaches to removing duplicates from a list, and they differ mainly in whether they keep the original order. The question before this one covered how lists work, so this answer builds on that and focuses on the choice between the three methods.

items = [3, 1, 3, 2, 1, 4]

# 1. With a set. This is the fastest, but the order is lost.
unique_set = list(set(items))
print(len(unique_set))   # 4

# 2. With dict.fromkeys. This keeps the order and it is still fast.
unique_dict = list(dict.fromkeys(items))
print(unique_dict)       # [3, 1, 2, 4]

# 3. With a plain loop. This keeps the order and it is the clearest.
seen = set()
unique_loop = []
for item in items:
    if item not in seen:
        seen.add(item)
        unique_loop.append(item)
print(unique_loop)       # [3, 1, 2, 4]

The set version is the shortest and the fastest of the three, but the order of the result is whatever the hash table happens to produce. You cannot rely on that order, and it looks random to anyone who reads the code later.

The dict.fromkeys version is the one to reach for in most situations. Since Python 3.7, a dictionary keeps its keys in the order they were added. So dict.fromkeys(items) builds a dictionary whose keys are the first appearance of each item, in the order the items first appeared, and whose values are all None. Turning that back into a list with list() gives you the list without duplicates and with the original order intact. It runs at the same speed as the set version, and it is only one line.

The plain loop is the most readable of the three, and it makes the intent obvious to anybody who reads it. This is also the version to write when you also need to count how many times each item appears, because you can swap the set of seen items for a counter and get a frequency table instead.

There are two limits to keep in mind. All three approaches need the items to be hashable, so a list that contains another list cannot be handled this way at all. And if you need the order of the last appearance rather than the first, you can reverse the list, remove the duplicates, and reverse the result again. That last request shows up often in interview questions.

The standard answer is to turn the list into a set and then back into a list, because a set holds each value only once. The expression list(set(items)) removes the duplicates, but it also loses the original order, because a set has no defined order of its own.

If the order matters, and in most real problems it does, the best one-liner is list(dict.fromkeys(items)). It keeps the first appearance of each value, in the order the values first appeared, and it runs at the same speed as the set version because a dictionary is a hash table as well.

Two other approaches are worth knowing. If you also want the counts, then collections.Counter(items) gives you a mapping of each value to how often it appears in a single line, and its most_common method ranks the values for you. If you need the order of the last appearance instead of the first, you can reverse the list, remove the duplicates, and reverse the result again.

There are two constraints to know about. The items must be hashable, so lists and dictionaries inside the list are a problem. For unhashable items you have to fall back to a loop that compares the items one by one, which is slower, or you can convert them to tuples first. Also, removing the duplicates keeps the original objects themselves, so if you change an item in the result you are also changing the item that was in the input list.

For an interview, the short summary is this. Use a set when you want speed and the order does not matter, use dict.fromkeys when the order does matter, and use Counter when you want the counts as well.