Normalize Unicode text

Identify equivalent-looking strings before converting them

A visible letter can be represented by more than one sequence of Unicode code points. For example, é can be a single precomposed character or an e followed by a combining acute accent. The two forms may look the same in a browser but fail an exact string comparison. Normalization gives canonically equivalent sequences a consistent representation; it is not a spellchecker, translation system or way to remove all accents.

Start with the smallest pair that fails to match. Inspect it in the character-level text diff or paste both versions separately into a system that exposes code points. If one value is actually written as café, it may be an encoding error rather than a normalization difference. Try the broken accents tool on a representative sample or retrieve a clean copy from the source. Do not run every cleanup option just because two strings look unexpected.

Once you know the difference is representation, decide what the receiving application expects. A search workflow may benefit from canonical equivalence, while a byte-for-byte comparison should preserve the original. Save the unmodified values so you can distinguish the normalization step from later edits. A visually correct result does not tell you which underlying form a different application will use.

Choose NFC or NFD for canonical equivalence

NFC composes canonically equivalent sequences where a composed form exists. It is often a practical choice when you want names copied from different applications to compare consistently. NFD decomposes canonically equivalent characters into their component sequences where defined. Use it when an existing workflow explicitly expects decomposed text, or when you want to inspect those components with another tool.

Try two visually identical accented labels, one containing a precomposed é and the other using e plus a combining acute accent. Normalize both with NFC, then compare them again. They should have a consistent canonical representation if that was the only difference. Try NFD on a copy and notice that the visible letter can remain accented even though its underlying representation changes. Neither option is a general accent remover.

Normalization can also order combining marks in a canonical sequence. Do not judge the result only by counting visible glyphs or selecting a position in a line. A grapheme may contain more than one code point. For a downstream rule with strict character or byte limits, use the text counter for an initial check and confirm the unit used by the receiving application.

Use NFKC or NFKD only when compatibility folding is intended

Compatibility forms cover more than canonically equivalent accents. A superscript, a ligature or a full-width letter can become a more ordinary sequence. That can make a text search or import more consistent if those visual distinctions are not meaningful in that workflow. It can also remove information a reader or machine relies on. An exponent written as a superscript can be part of a formula, not decoration to discard.

Try a sample containing full-width letters and a superscript digit. Compare NFC with NFKC on separate copies. NFC does not deliberately fold those compatibility distinctions; the compatibility form can change them. Inspect each change before applying the same operation to a full document. If a sample contains a unit, formula, typographic example or product code, review those separately. The fact that a representation is easier to match does not mean it preserves the original intent.

NFKD also uses compatibility mappings but leaves the result decomposed where applicable. Choose it only when your workflow specifically needs that form. If the task is simply to make accented names compare consistently, start with a canonical form instead. A broader transformation is not more correct just because it catches more visible variations.

Make punctuation choices independently of normalization

The editor lets you simplify certain quotes, dashes and ellipses separately. Those operations address punctuation conventions, not canonical equivalence. A curly apostrophe may be expected in polished prose but rejected by an exact-match system using an ASCII apostrophe. A long dash may be part of the meaning or style of a heading. Turn on only the option required by the destination and test on a short excerpt that contains the relevant punctuation.

Use the option to leave the Unicode form unchanged if you want only a supported punctuation transformation. This is useful when the characters are already in the expected form and you have identified a specific punctuation difference. Compare the result with the original and confirm that a name, citation or example was not modified unintentionally. Do not use broad normalization as a substitute for a single known replacement.

If your goal is to delete punctuation entirely, use Remove characters and check whether deleting a mark would join words. If the problem is a visually blank separator or a hidden mark, use the invisible character inspector first. Unicode normalization here is not a general tool for removing zero-width characters.

Normalize a comparison copy, not necessarily the source record

When matching two lists of names, decide whether you want exact storage equality or canonical equality. For an ordinary review, making a canonical copy of each list can prevent visually equivalent names from being reported as different. For an audit of what the system actually stores, compare the raw values first and note any differences. These two reports answer different questions.

Apply the same normalization form and punctuation choices to both copies. Then use Compare two lists for membership or Remove duplicate lines for repeated values in one list. Use a stable identifier when names are not unique. A normalized name does not identify a person or record on its own, and different names can coincide after broader transformations. Keep the original text alongside the derived key if the original spelling is important.

Before updating a system, compare a few entries that should match and a few that must stay separate. An overly aggressive compatibility or case conversion may collapse values you need to distinguish. Inspect the exact behavior of the receiving system rather than assuming your local comparison result will be interpreted identically by its database or import process.

Distinguish typographic variants from deliberate styled characters

Some styles copied from a plain-text generator are separate Unicode characters, not actual font formatting. Compatibility normalization may map some such characters to ordinary letters, but that is broader than reversing a known transformation. The Unicode text styles tool can restore the specific styles it supports. For a safe conversion, test one letter, one digit and an accented word that may have remained unstyled.

Underline or strikethrough can also be represented using combining marks. An apparent difference in styling does not automatically mean a canonical normalization will remove that style. Use the documented restoration operation for the supported marks and inspect the result. If you need an accessible heading in a document that supports formatting, use normal text with the destination's actual formatting controls rather than changing the stored letters.

The guide to encoding and invisible characters shows how to tell a formatting problem from a normalization or decoding problem. Keep the distinction clear when reporting an issue: “canonical variants did not match” is more actionable than “Unicode broke.”

Verify the smallest transformation that solves the problem

Write down the chosen form and whether punctuation settings were enabled. Test one string that failed, one that already worked and one value whose appearance or exact identifier must stay unchanged. Compare the result in the actual receiving application. A normalization step can be locally consistent while a different application applies extra case, whitespace or type conversion on import.

If the original and normalized versions differ only in a few records, keep a small example of each. This gives a future maintainer evidence about why the conversion was applied. Do not replace the original export unless that is explicitly part of the workflow. A reproducible normalization step should make the matching rule clear while leaving you able to recover the unmodified source.

Check an accent-sensitive lookup with two examples

Test a pair that should match, then test a second pair that differs by an actual accent rather than only its representation. Canonical normalization can make the first pair equivalent without making the second identical. This confirms that you solved the specific representation problem without switching to a more destructive accent-removal rule.