In this article, we will discuss different ways to remove duplicate items from a Python List with some examples.
What is Python List?
A list is one of the built-in data types in Python. It is used to store multiple values in a single variable.
A python list is ordered which means every item in a list has a defined order and it cannot be changed. A list also allows us to have items with the same values too. i.e duplicates are allowed.
We can create a List in Python by using the square brackets [] or the list().
However, while working with lists, we might sometimes want to get rid of all the duplicate items and create a list unique.
The different methods to remove identical items and create a unique list are:
- Using for loop
- Using set method
- Using list comprehension and enumerate
- Using unique method from Panda module
- Using OrderedDict.fromkeys() from collections module
le’s see each method with some examples to understand it better.
Using for loop to remove duplicates from Python List
To remove duplicate items from a python list, we can use the for loop to iterate through each item and store the first occurrence of each item in a new list.
While looping through the items we will filter out the common or already existing items using the not operator.
Example:
my_list = [1,2,3,2,5,6,2]
no_dup_list =[]
for item in my_list:
if item not in no_dup_list:
no_dup_list.append(item)
print(no_dup_list) Output:
[1, 2, 3, 5, 6]In the above example, we have a python list called my_list which has numbers with duplicates, and an empty list called no_dup_list, where we will store the item after removing the duplicates.
So first, we loop through each item in my_list and on each iteration we check if the item is present or not in the no_dup_list using the not operator, if not, then we append the item to it.
We can also remove duplicate items using list comprehension, which is writing the same for loop code in a shorthand way.
Example:
my_list = [1,2,3,2,5,6,2]
unique_list = []
[unique_list.append(item) for item in my_list if item not in unique_list]
print(unique_list)
Output:
[1, 2, 3, 5, 6]Using set() method to remove duplicates
Using set() is probably the easiest way to remove any multiple occurring items from a Python List.
A set is a built-in data type in python that store unordered, unchangeable, and unindexed values in it.
We can easily remove duplicates using set() because it does not allow to have two items with the same value in it.
Example:
my_list = [1,2,3,2,5,6,2]
set_list = set(my_list)
unique_list = list(set_list)
print(unique_list)Output:
[1, 2, 3, 5, 6]Here, we have converted our list with duplicate items to a set using set(mylist) which removed all the identical items from it and stores the value in the set_list.
Next, we again converted the set items into a List using list(set_list), which then returns a list with only unique items in it.
If you want to remove duplicates from a list without using the set method, you can follow the methods below.
Using list comprehension and enumerate() method
We can use the list comprehensive and the enumerate() method together to remove duplicate items from a Python list.
my_list = [1,2,7,3,2,6,3]
for i,n in list(enumerate(my_list)):
if n not in my_list[:i]:
print(n)The above code gives us the distinct items from the given list.
We can write the above code using list comprehension to get a list with only unique elements.
[n for i,n in list(enumerate(my_list)) if n not in my_list[:i]]Output:
[1, 2, 7, 3, 6]This method keeps the order of the items the same and the already occurred items are skipped from the list.
Using OrderedDict.fromkeys() method to remove duplicates from Python List
To remove any duplicate items from a list in Python we can also use the in-built collections modules in Python.
We can import the OrderedDict.fromkeys() method which removes any identical values from the list and returns a dictionary.
Example:
from collections import OrderedDict
my_list = [1,2,7,3,2,6,3]
unique_list = list(OrderedDict.fromkeys(my_list))
print(unique_list)Output:
[1, 2, 7, 3, 6]Once we get the OrderedDict values we then convert it back to a list by wrapping it inside the List() function.
This is the fastest method and it also maintains the order of the items in the List.
Remove duplicates using unique() method from Panda modules
The unique() method in the Panda modules is used to remove all duplicate items and return the unique items from a given list.
Example:
import pandas as pd
my_list = [1,2,7,3,2,6,3]
unique_list = pd.unique(my_list).tolist()
print(unique_list)Output:
[1, 2, 7, 3, 6]Here, we got all the unique items from the list using pd.unique() and then we used the tolist() method to convert it to a Python List.
Conclusion: In this article, we have learned about python lists and all the different methods we can use to remove duplicate items from the list and create a list with only unique values.
Related Topics:
Get the Index or Position of Item in List in Python
How to split a list into multiple list using python
Prepend List in Python (Append at the beginning)
How to flatten nested list in python (5 ways)
Python – Insert an element at specific index in a list
