"Special characters" can mean a few different things. Here is how to remove the ones that cause problems without stripping the characters you actually need.
What people mean by special characters
Usually one of these:
- Invisible characters: zero-width spaces, the byte-order mark, direction marks. These are pure noise and should go.
- Exotic spaces: non-breaking and other spaces that look normal but behave differently.
- Typographic characters: smart (curly) quotes, em and en dashes, and the ellipsis glyph, which cause problems in code and plain-text fields.
- Genuine symbols: currency, math, and accented letters, which you usually want to keep.
The goal is rarely to strip every non-letter. It is to remove the characters that break things while keeping the ones that carry meaning.
The targeted way
Paste your text into a cleaner that removes the problem characters by category. textscrubr:
- strips invisible characters and the byte-order mark
- normalizes non-breaking and exotic spaces to regular spaces
- straightens smart quotes and converts the ellipsis glyph to three dots
- replaces em and en dashes with a comma, hyphen, or nothing
It leaves your letters, numbers, normal punctuation, and structure alone, so you remove the noise without mangling the text.
If you need to strip everything non-standard
Sometimes you do want only plain ASCII, for example for a strict system or a legacy database. In that case you want a stricter pass that removes or transliterates anything outside a basic set. That is a heavier operation and can change accented names, so use it only when the destination truly requires it.
Why manual removal fails
Many special characters are invisible or look identical to normal ones, so you cannot reliably find them by eye. A cleaner targets them by their actual code point, catching every instance and showing you a count of what it removed.
Quick approach
For most cases, remove the noise (invisible characters, exotic spaces, smart quotes, em dashes) and keep everything meaningful. A structure-aware cleaner does this in one pass and preserves your formatting.