Remove duplicate lines

Check what one line represents before deleting it

This tool keeps the first occurrence of each matching nonblank line. A second appearance anywhere later in the same input can be removed. It does not sort the text, choose a preferred spelling or compare parts of a line. Start by asking whether a row represents a unique item or a separate event. A repeated destination in a list of places to visit may be redundant; a repeated amount in a payment log may record two different transactions.

Paste the smallest representative part of your data first. Include an exact repeat, one entry that looks similar but should stay and any blank lines that carry structure. Read the result next to the original before applying it to the input. If the task is to check which values differ between two exports, keep the two lists separate and use Compare two lists instead. Joining both lists and deduplicating them would lose the information about which side contained each value.

Use a separate copy when the original is a historical record. A deduplicated view can be useful for a checklist or review, but it should not overwrite the source if repeated rows have meaning.

Try an exact repeat beside a case variant

Paste these lines with Remove duplicate lines checked and leave Trim extra spaces off. The second Lisbon disappears, but Madrid and madrid stay as distinct entries. The relative order of every retained entry remains the same.

Lisbon
    Madrid
    Lisbon
    madrid
    Rome
    Madrid

The expected result is Lisbon, Madrid, madrid and Rome, in that order, with one item per line. Notice that the repeat does not need to sit next to its original. In a list assembled from several notes or files, the second copy can be far from the first one. Keeping the first appearance preserves the position at which an item was originally introduced.

If a colleague expects all capitalization variants to collapse, agree on that rule before changing the text. Readme and README could be interchangeable labels for people but different identifiers in some systems. If you make an intentionally lowercased working copy with the case converter, its output changes spelling as well as which items match. Preserve the source if the original capitalization matters.

Try the list in the editor

Check spaces, tabs and indentation separately

Two rows can look identical while one contains an extra space at the end or two spaces between words. Leave trimming off if you need strictly exact comparison. Turn Trim extra spaces on when you want to remove trailing spaces and collapse runs of horizontal spaces between words before comparing. This is a transformation of the kept text as well as a matching choice, so check the result for any wording or spacing you wanted to preserve.

Leading indentation made from spaces or tabs is preserved. That means a line inside an indented checklist can remain distinct from the same words at the outer level. Do not flatten indentation just to force a match unless the list is truly flat. When leading and trailing spaces should be removed explicitly, inspect the available options in Clean whitespace and apply only the operations that fit your input.

If a repeat still survives, compare the two lines in the invisible character inspector. This helps find a nonbreaking space or zero-width mark that ordinary rendering can hide. Remove such a mark only after checking whether it is part of meaningful text or an identifier.

Keep paragraph breaks and fenced examples intact

Blank or whitespace-only lines are not treated as repeated items by this utility. Multiple blank separators can remain in the result. That is useful in a document organized into groups, where removing a blank line could join two sections visually. If you need to remove blank lines in addition to duplicates, use the line sorter and cleaner, set it to keep the original order and choose the cleanup options you need.

Recognized Markdown-style fenced blocks are protected here. Identical lines inside a block may therefore stay even while identical ordinary lines outside it are removed. Do not assume that every indentation pattern or every structured format gets the same protection. If you only want to clean one portion of a mixed document, select that list by itself, deduplicate it and return it to the document after review.

Also note that when cleanup is active, Windows-style line endings are normalized to line feeds in the output. That usually does not change how a pasted list looks. If exact file bytes or a particular line-ending convention matter to the receiving system, check the exported file with that system before replacing the original.

Do not deduplicate raw rows when the data needs a key

Consider two rows that start with the same product code but have different descriptions. Whole-line comparison keeps both because the full strings differ. It cannot tell whether those descriptions are historical versions, duplicate records with different fields or two different entities that happen to share a code. Pick the correct record key before deciding what to keep.

Use the CSV and TSV converter to extract a key column from a small sample. Then inspect which keys are repeated and examine the original records side by side. A CSV field can contain an embedded line break, so running a line deduplicator across a raw multiline CSV file can remove text inside a field or otherwise damage its structure. Parse or import structured records before comparing a single column.

For a spreadsheet, the right workflow might be grouping records by key and reviewing each group. “Keep the first” is only safe if the order of the export actually expresses your preference. A list sorted oldest first and a list sorted newest first can produce different surviving records when deduplicating by a key elsewhere.

Use a different tool when order or frequency is the question

Remove duplicate lines gives one retained copy of each matching line without rearranging the list. If you want an alphabetical or natural-number order, use Sort and clean lines. Keep the original version when order carries meaning, such as agenda priority or the sequence in which records were received. A sorted copy can help a reviewer scan a catalog, but it no longer shows the order in which the entries appeared.

If you need to know how many times each value occurred, save the original before removing anything. The duplicate count in the editor describes how many rows were removed, not the count per individual value. A word-frequency report counts word forms, not whole records, so it cannot replace a frequency table grouped by line or by an identifier column.

Use the text diff checker if you need to see the removed entries and the surrounding context after cleaning. Paste the saved original and the cleaned copy separately. Line comparison may be especially helpful when you are reviewing a list in which you expected only a few repeated entries to disappear.

Review a small result before applying it to the original

First, identify two lines you expect to match and one that should remain distinct. Test them in the editor. Next, check the result counts and read the first few retained values. A large drop may be expected if multiple source lists overlap, but it can also be a sign that repeated lines were meaningful or that a whitespace change merged entries you meant to keep.

When cleaning names or addresses, inspect a pair with punctuation, accents or extra spaces. Do not globally simplify punctuation just to increase the number of matches. An apostrophe, hyphen or accent can be part of a real name or exact address. For labels copied from a document, Unicode normalization can make equivalent spellings consistent, but check its options and preserve the original values when exact forms matter.

After verifying the cleaned list, copy or download it as a separate result. If the next operation is to number it, use the list formatter after deduplicating. Numbering first would make each line different even when the value after the number was the same.

Write down the cleanup rule when sharing the result

A useful handoff includes the source version and the matching rule: for example, “Exact repeated lines removed, first occurrence kept, no case changes.” If you enabled trimming, mention it. If repeated records were reviewed by key in another tool, describe that separately. This lets someone reproduce the result without assuming that the exported list is an unmodified record of every occurrence.

Reopen the result where it will be used and check the first and last entry, any blank group separator and one item that intentionally differs only in case. If the destination changes capitalization, spacing or line endings on paste, that is a separate transformation from deduplication. Keep both the source and the cleaned result until you no longer need to explain the changes.

For a fuller workflow involving one-item-per-line lists and delimited tables, use the guide to preparing pasted lists and tables. It explains when line cleanup is appropriate and when the fields of a structured record need to be preserved instead.