Remove Duplicate Lines

Duplicate lines are a constant nuisance in data work — exported CSV files, email lists, log files, and database dumps often contain repeated entries that need to be cleaned before analysis. This tool removes duplicate lines from any text, preserving the original order of first appearances. Options include case-sensitive or case-insensitive matching and whitespace trimming, so 'John Smith' and 'john smith' or lines with trailing spaces are caught. Paste your list, and the deduplicated version appears instantly.

Real-world jobs this handles well: a marketer merging two email signup lists where a few subscribers registered twice under slightly different casings ('Jane@Example.com' vs 'jane@example.com'); a support engineer consolidating error-message lines from three log files to identify which unique errors are actually occurring; a writer compiling a research bibliography and removing citations that appeared in multiple source documents. Each use case benefits from slightly different settings — the email case wants case-insensitive and whitespace-trimmed, the log case usually wants exact (because trailing spaces can be a signal), and the citations want case-insensitive to catch Title Case vs sentence case variants.

Under the hood, deduplication uses a JavaScript Set to track lines already seen, which gives O(n) performance: 100,000 lines deduplicate in under 20 milliseconds on a modern laptop. The case-insensitive mode normalizes each key to lowercase before the Set lookup. Whitespace-trim mode calls .trim() on each line, which removes leading/trailing space, tab, and any Unicode whitespace character. The output preserves the original line that appeared first, not a normalized version — so if 'Apple' appears before 'APPLE', the output contains 'Apple'.

Compared to running `sort -u` in the terminal, this tool preserves original order (sort -u does not, unless you add 'sort | awk "!seen[$0]++"'), and works on any device without a shell. Compared to Excel's Remove Duplicates feature, it handles inputs larger than ~1M rows (Excel's row limit) and does not silently trim trailing zeros from numeric-looking entries.