PasteCleanly / Guides

Diagnose broken accents and invisible characters

Broken accents, invisible marks, and different Unicode forms can look like the same problem. They need different fixes.

In this guide

Classify the symptom before changing the text

There are three common reasons a pasted label does not behave the way it looks. Garbled characters are visible: a name that should contain an accent may contain a short run of unexpected letters and symbols. Invisible characters are present but not obvious: a zero-width mark can sit between letters, or a nonbreaking space can look like an ordinary space. Unicode representation differences are subtler: a letter with an accent can be stored as one code point or as a base letter followed by a combining mark.

These problems need different tools. Repairing encoding does not remove zero-width marks. Replacing spaces does not compose an accented character. Normalizing Unicode does not reverse a wrong byte decoding. Start with a short copy of the suspect text and a version you know is correct. Keep both until the result works in the app where the problem occurred. The goal is a verified value, not merely one that looks tidy in the preview.

Open the invisible character inspector

Use the obvious visual clue for garbled accents

When you see something like café instead of café, the sequence is a useful clue. Bytes originally written as UTF-8 may have been read using a Windows-1252 interpretation. The resulting characters are valid text, but they represent the wrong decoding of the source. The same mistake can affect punctuation: It’s can be a misread form of It’s.

Try a short example in Fix broken accents. Compare each proposed change against the expected spelling or a trustworthy source. The preview attempts to reverse a specific family of mistaken conversions; it does not detect or translate every possible encoding. If a sequence was changed several times, one pass may not restore it. Test one pass first. Run another only when the result still shows a recognizable pattern and you can verify the expected text. Do not apply repair simply because a name contains a letter you do not recognize.

Recognize lost information rather than guessing

The replacement character � often signals that a decoder could not interpret some bytes. Once a copy contains only that replacement, the missing bytes are not present in the text you pasted. A question mark can also have been inserted by an earlier export, though it may just be ordinary punctuation. In both cases, guessing from a name or sentence is not the same as recovering the source.

Look for the original file, a fresh export, or another copy made before the decoding error. If you control the export, select the intended encoding there and open the result using the same encoding. Compare a small set of accented names and curly punctuation before replacing a full list. If the source cannot be recovered, keep the unmodified copy and document any manual correction. This is especially important for names, identifiers, and quoted language. Repeatedly encoding and decoding the damaged text will not reconstruct bytes that were discarded in a previous step.

Find an invisible mark by its code and position

When two labels look alike but only one matches, paste the suspect label into the inspector. Turn on Show markers in output and open the character report. In a value such as Invoice[NBSP]042, the bracketed marker represents a nonbreaking space. The report includes the Unicode code, a count and the first position of each recognized mark. Compare that position against the original text to see whether the mark is between words or inside an identifier.

The markers are only a diagnostic display. They are not the characters that were in the text and should not be pasted into a destination that expects the original value. After deciding what to keep, turn markers off to copy a cleaned result. If several kinds of mark appear, try one cleanup option at a time. That makes it clear whether a difference came from spacing, from a discretionary hyphen, or from a character you intended to preserve.

Inspect the code points in your pasted label

Distinguish spaces from hidden formatting controls

A nonbreaking space can be entirely appropriate in a display label such as a number followed by a unit. It keeps related pieces together when a line wraps. A thin space may be part of a typographic convention. A tab can be a field separator rather than a gap that should be collapsed. Check what the destination expects before treating every nonstandard space as a defect.

The inspector can replace certain nonbreaking spaces with ordinary spaces. This can help when pasting into a system that compares label values literally. It should not be used automatically on typography where line-breaking behavior is intentional. If you are copying spreadsheet rows separated by tabs, keep the tabs and use the CSV and TSV converter when you need a different separator. Do not use whitespace cleanup to convert a table. Collapsing two tab-separated fields into one gap changes the structure even if the pasted line still looks readable.

Treat joiners and directional marks as meaningful until checked

Some marks affect how characters combine or display. A zero-width joiner may be part of an emoji sequence or important to a writing system. A directional mark can influence the order in which mixed-direction text appears. The report is useful because it distinguishes these controls from a zero-width space. Presence alone does not mean the mark is unwanted.

PasteCleanly's narrow cleanup option for zero-width spaces, soft hyphens, and BOM characters does not remove the joiners or directional marks listed in its report. That distinction matters when copying names, messages with mixed scripts, or emoji. If the issue occurs only in one app, compare what that app actually receives. Avoid a blanket rule such as “delete every non-ASCII character.” It can remove part of a name, split a visible sequence, or change the reading behavior of a sentence. Check one complete example in the destination before applying a broad change to a batch.

Review invisible marks without changing the original

Normalize equivalent accents when the letters look correct

Two visually identical accented strings may use different sequences. For example, an accented letter can be stored as one code point or as a base letter followed by a combining accent. A display can render both the same way while a literal comparison treats them differently. This is a good case for Normalize Unicode, not for accent removal. Normalization keeps the accent while changing its representation.

Use NFC on both the reference and the pasted value before matching them if the receiving system allows that change. NFC composes canonically equivalent sequences where a composed form exists. NFD takes the decomposed approach. Choose the form required by the destination if it specifies one. When neither tool reports a visible change, that is not proof that the original code points were identical. Check the comparison after applying the same transformation to both sides, rather than relying on a visual glance.

Use compatibility forms only when the distinction can be lost

NFKC and NFKD can do more than compose or decompose equivalent accents. They can also fold compatibility characters. A full-width form, a ligature, or a superscript may become a more ordinary representation. That can help when the destination explicitly treats those distinctions as interchangeable. It can also change what a reader sees or merge labels that the original document intentionally distinguishes.

Test a sample before normalizing an entire table or catalog. Include entries with accents, full-width forms, punctuation and codes that should be untouched. If identifiers are assigned by another system, preserve that system's exact values until you know how it handles normalization. A human-readable display label can often be cleaned independently of an identifier. If the only issue is curly quotes or an em dash, use a specific punctuation option rather than a compatibility form. Narrow operations are easier to explain and to undo when the result is not what you intended.

Decorative Unicode letters are another distinct case: they may be readable but not match the ordinary word in a search. Use Restore plain letters on a short excerpt if it uses one of the supported styles, and compare the result with the intended spelling. If the issue is repeated ordinary spaces rather than character representation, use Clean whitespace with only the relevant option enabled.

Tell entity references and URL escapes apart from mojibake

Literal text like é is a character reference, not a garbled accent. Decode it with the HTML entity decoder when the destination needs the visible letter. The same idea applies to numeric references. This changes the text representation; it does not interpret pasted markup as a page. If the result still contains another reference, look at it before deciding whether to run another pass.

A value such as Caf%C3%A9 is an example of URL percent encoding. Choose the URL component decoder in Encode and decode text instead. A plus can mean a literal plus or a space depending on the format. If you are unsure which representation you have, compare against how the source produced it: an HTML example, a URL query value, and a text file all have different rules. None of these operations repairs a selection with text already omitted or lost.

Repeat the same preparation on both sides of a comparison

If a comparison should ignore a specific representation difference, prepare both inputs the same way. Inspect a few problematic labels from each list, decide whether nonbreaking spaces should become ordinary spaces, and apply NFC when canonical equivalence matters. Then paste the two lists into Compare lists. Do not normalize one side and leave the other side in a different representation unless you are deliberately testing the effect.

Review the result before treating every remaining mismatch as an error. One value may actually differ by punctuation, an extra digit, or a different spelling. A duplicate-removal tool should keep those values distinct unless your use case says otherwise. If matching rules must be strict, save the original columns and write the normalized values to new columns in your destination. That leaves a trace of which changes made a match possible and avoids silently replacing a record's official form.

Open the inspector for a final comparison check

Keep a small test set and verify the destination

For a recurring import, build a short test set with a known good accent, a visibly damaged accent, a nonbreaking space, a decomposed accent, and a sequence that intentionally uses a joiner. Write down the expected result for each one. Run the same series of steps on these examples when you change the source, the target app, or a cleanup option. Check both what changed and what remained untouched.

Finally, paste the cleaned example into the actual destination and repeat the original search, comparison, or import. A preview can show the characters correctly without proving that another app compares them in the same way. If it still fails, keep the before-and-after values and check the destination's expected encoding and matching rules. Use the smallest correction that makes the documented example work. If a value is meant to stay byte for byte identical, prefer inspection over changing it and resolve the mismatch at the system that owns the value.

Return to the Unicode character report

Compare the same sample at two points in the workflow

When the same problem keeps coming back, capture the text immediately before and after the step that changes it. For example, compare a French name in the application that owns the record with the same name after downloading and opening an export. Then compare what appears after copying it into the destination. The first point at which the strings differ tells you more than another guess about the final display. If you can export a small record with invented names, use that to investigate without including customer data.

Keep the sample small but varied. Use one character with an accent, one curly apostrophe and a word without any special marks. Add a name from a non-Latin script if the workflow handles it. If only the accented letter changes, the problem may be tied to a conversion step. If accents remain visually intact but search stops finding them, compare the actual code points: composed and decomposed versions can look identical. If every special character became a question mark, the original values may no longer be recoverable from that copy.

When you find the first differing step, change the exporter or import settings there if you control them. Record the expected encoding or Unicode form next to the sample. A future export can then be checked against the same reference, instead of relying on a manual repair after each import.

Compare the exact characters before and after