Python Remove List Duplicates

Last modified on: October 19, 2024

<div class="flex-shrink-0 flex flex-col relative items-end">
<div>
<div class="pt-0">
<div class="gizmo-bot-avatar flex h-8 w-8 items-center justify-center overflow-hidden rounded-full">
<div class="relative p-1 rounded-sm flex items-center justify-center bg-token-main-surface-primary text-token-text-primary h-8 w-8">&nbsp;</div>
</div>
</div>
</div>
</div>
<div class="group/conversation-turn relative flex w-full min-w-0 flex-col agent-turn">
<div class="flex-col gap-1 md:gap-3">
<div class="flex max-w-full flex-col flex-grow">
<div class="min-h-8 text-message flex w-full flex-col items-end gap-2 whitespace-normal break-words [.text-message+&amp;]:mt-5" dir="auto" data-message-author-role="assistant" data-message-id="4139df41-8e30-4172-af3b-b8ca300544cf" data-message-model-slug="gpt-4o">
<div class="flex w-full flex-col gap-1 empty:hidden first:pt-[3px]">
<div class="markdown prose w-full break-words dark:prose-invert light">
<h3>How to Remove Duplicates from a List in Python</h3>
<p>Removing duplicates from a list is a common task when working with data. Python provides multiple ways to achieve this, depending on whether you need to preserve the order of elements or not. Below are several methods to remove duplicates from a list.</p>
<hr />
<h3>Method 1: Using <code>set()</code> to Remove Duplicates (Unordered)</h3>
<p>The easiest and fastest way to remove duplicates from a list is by converting the list to a <code>set</code>, as sets do not allow duplicate elements. However, this method does <strong>not</strong> preserve the order of the original list.</p>
<div class="contain-inline-size rounded-md border-[0.5px] border-token-border-medium relative bg-token-sidebar-surface-primary dark:bg-gray-950">
<div class="flex items-center text-token-text-secondary px-4 py-2 text-xs font-sans justify-between rounded-t-md h-9 bg-token-sidebar-surface-primary dark:bg-token-main-surface-secondary">python</div>
<div class="sticky top-9 md:top-[5.75rem]">
<div class="absolute bottom-0 right-2 flex h-9 items-center">
<div class="flex items-center rounded bg-token-sidebar-surface-primary px-2 font-sans text-xs text-token-text-secondary dark:bg-token-main-surface-secondary">
<pre class="language-python"><code># Original list with duplicates
my_list = [1, 2, 2, 3, 4, 4, 5]

# Remove duplicates by converting the list to a set
unique_list = list(set(my_list))

print(unique_list)  # Output: [1, 2, 3, 4, 5]
</code></pre>
</div>
</div>
</div>
</div>
<hr />
<h3>Method 2: Using a <code>for</code> Loop to Preserve Order</h3>
<p>If you need to preserve the order of the list, you can iterate over the list and add elements to a new list only if they haven't been added before.</p>
<div class="contain-inline-size rounded-md border-[0.5px] border-token-border-medium relative bg-token-sidebar-surface-primary dark:bg-gray-950">
<div class="flex items-center text-token-text-secondary px-4 py-2 text-xs font-sans justify-between rounded-t-md h-9 bg-token-sidebar-surface-primary dark:bg-token-main-surface-secondary">python</div>
<div class="sticky top-9 md:top-[5.75rem]">
<div class="absolute bottom-0 right-2 flex h-9 items-center">
<div class="flex items-center rounded bg-token-sidebar-surface-primary px-2 font-sans text-xs text-token-text-secondary dark:bg-token-main-surface-secondary">
<pre class="language-python"><code># Original list with duplicates
my_list = [1, 2, 2, 3, 4, 4, 5]

# Initialize an empty list to store unique elements
unique_list = []
for item in my_list:
    if item not in unique_list:
        unique_list.append(item)

print(unique_list)  # Output: [1, 2, 3, 4, 5]
</code></pre>
</div>
</div>
</div>
</div>
<hr />
<h3>Method 3: Using Dictionary <code>fromkeys()</code> to Preserve Order</h3>
<p>A more Pythonic way to preserve the order of the elements while removing duplicates is to use the <code>dict.fromkeys()</code> method, which retains only unique values while maintaining their order.</p>
<div class="contain-inline-size rounded-md border-[0.5px] border-token-border-medium relative bg-token-sidebar-surface-primary dark:bg-gray-950">
<div class="flex items-center text-token-text-secondary px-4 py-2 text-xs font-sans justify-between rounded-t-md h-9 bg-token-sidebar-surface-primary dark:bg-token-main-surface-secondary">python</div>
<div class="sticky top-9 md:top-[5.75rem]">
<div class="absolute bottom-0 right-2 flex h-9 items-center">
<div class="flex items-center rounded bg-token-sidebar-surface-primary px-2 font-sans text-xs text-token-text-secondary dark:bg-token-main-surface-secondary">
<pre class="language-python"><code># Original list with duplicates
my_list = [1, 2, 2, 3, 4, 4, 5]

# Remove duplicates and preserve order
unique_list = list(dict.fromkeys(my_list))

print(unique_list)  # Output: [1, 2, 3, 4, 5]
</code></pre>
</div>
</div>
</div>
</div>
<hr />
<h3>Method 4: Using List Comprehension and <code>set()</code> to Preserve Order</h3>
<p>Another approach that uses a <code>set</code> to track seen elements while iterating through the list:</p>
<div class="contain-inline-size rounded-md border-[0.5px] border-token-border-medium relative bg-token-sidebar-surface-primary dark:bg-gray-950">
<div class="flex items-center text-token-text-secondary px-4 py-2 text-xs font-sans justify-between rounded-t-md h-9 bg-token-sidebar-surface-primary dark:bg-token-main-surface-secondary">python</div>
<div class="sticky top-9 md:top-[5.75rem]">
<div class="absolute bottom-0 right-2 flex h-9 items-center">
<div class="flex items-center rounded bg-token-sidebar-surface-primary px-2 font-sans text-xs text-token-text-secondary dark:bg-token-main-surface-secondary">
<pre class="language-python"><code># Original list with duplicates
my_list = [1, 2, 2, 3, 4, 4, 5]

# Using list comprehension with a set to preserve order
seen = set()
unique_list = [x for x in my_list if not (x in seen or seen.add(x))]

print(unique_list)  # Output: [1, 2, 3, 4, 5]
</code></pre>
</div>
</div>
</div>
</div>
<hr />
<h3>Method 5: Using <code>pandas</code> (When Working with Large Datasets)</h3>
<p>If you are working with large datasets, using the <code>pandas</code> library can be a fast and convenient way to remove duplicates.</p>
<div class="contain-inline-size rounded-md border-[0.5px] border-token-border-medium relative bg-token-sidebar-surface-primary dark:bg-gray-950">
<div class="flex items-center text-token-text-secondary px-4 py-2 text-xs font-sans justify-between rounded-t-md h-9 bg-token-sidebar-surface-primary dark:bg-token-main-surface-secondary">python</div>
<div class="sticky top-9 md:top-[5.75rem]">
<div class="absolute bottom-0 right-2 flex h-9 items-center">
<div class="flex items-center rounded bg-token-sidebar-surface-primary px-2 font-sans text-xs text-token-text-secondary dark:bg-token-main-surface-secondary">
<pre class="language-python"><code>import pandas as pd

# Original list with duplicates
my_list = [1, 2, 2, 3, 4, 4, 5]

# Convert the list to a pandas Series and drop duplicates
unique_list = pd.Series(my_list).drop_duplicates().tolist()

print(unique_list)  # Output: [1, 2, 3, 4, 5]
</code></pre>
</div>
</div>
</div>
</div>
<hr />
<h3>Summary</h3>
<ul>
<li><strong>Using <code>set()</code></strong> is the quickest way but does not preserve order.</li>
<li><strong>Using a <code>for</code> loop</strong> or <strong><code>dict.fromkeys()</code></strong> preserves the order of the list.</li>
<li><strong>Using list comprehension</strong> with a <code>set</code> is a more efficient and concise approach to maintaining order.</li>
<li><strong>Using <code>pandas</code></strong> is great for large datasets when working with data frames.</li>
</ul>
<hr />
<h3>&nbsp;</h3>
</div>
</div>
</div>
</div>
</div>
</div>

Web Technology View More