Precompiled Pipelines¶

Canonicalize text for security-sensitive comparison.

For cleaning untrusted input before comparison, this is the entry point. It does not make text safe to emit; encode at the sink.

It is half of a spoof-resistance answer, not all of it (#882). This reports what two strings collapse to; find_confusables reports what looks like something else. Measured over confusable-bench.v1, the six key reducers together catch 72 of 120 malicious identifiers and find_confusables catches 66 — but either one firing catches 108, at 0 false positives on the 20 benign controls for each alone and for the pair. The reducers take the evasion class (42/54) that the detector cannot see, and the detector takes the composability class (31/31) the reducers cannot. A registry needs both questions asked.

Two steps introduce ASCII, not one (#719). The leading NFKC is the obvious one; the confusable fold is the second and reaches characters NFKC leaves alone. U+2236 RATIO has no decomposition at all and becomes :, U+2044 FRACTION SLASH becomes /, U+2216 SET MINUS becomes \. 232 code points reach ASCII by the fold alone, 76 of them producing one of : = % & ? # / \. A string that carried no delimiter can leave here carrying one — encode at the sink.

inspect_anomalies reports it as confusable when the word also carries an ASCII letter, which is #633's gate and what keeps Привет from firing. A delimiter-only string such as ∶∶∶ folds to ::: and is not reported.

Pipeline: resolve deletions → [digit-policy pre-fold] → NFKC → strip bidi/format → strip invisible classes (#413) → strip_control → strip_zero_width → collapse_whitespace → drop repeated marks → cap combining marks at 3 (anti-zalgo,

429) → NFC → confusables and NFC to a fixed point → drop repeated marks → cap¶

combining marks again (the confusable fold is iterated with NFC so TR39 skeletoning is normalization-stable and the preset is idempotent — #416/#434). PRESETS lists the steps exactly.

Collapses fullwidth bypasses, neutralizes homoglyph spoofing, strips dangerous bidi overrides and soft hyphens, then normalizes whitespace (collapsing runs, stripping control chars and zero-width injections).

Scoped to identifiers, not body text (#624). The confusable fold runs toward Latin, so it rewrites non-Latin text that has a Latin lookalike — Arabic alef becomes l, Hebrew yod becomes ', and Ελληνικά comes back as Eλλnvikά. It also removes U+200C, which Persian orthography requires. Nothing flags this first, because ordinary Persian is not an anomaly. Use it on usernames, hostnames, filenames and log lines; see Limitations (docs/limitations.md) before pointing it at a sentence.

The fold is not a romanization (#907). It runs without transliterating first, so a Cyrillic word reaches the Latin table as shapes rather than as sounds: Москва comes back Mockba, where search_key and catalog_key give the romanization moskva. Mockba is not a spelling of anything. It is what the confusable table does to letters that no transliteration step has already handled, and catalog_key's own step list states the rule: transliterate first, so non-Latin scripts are romanized before confusables.

Here that order is a decision rather than an omission, because the collapse is the point: an attacker's Latin Mockba and a Cyrillic Москва are meant to meet, and the romanizing surfaces cannot make them — find_key_collisions(["Москва", "Mockba"], key="search_key") is empty. Pick by what you need. normalize applies Unicode normalization and nothing else, so the script survives — but it is not an identity function, and NFC still recomposes e followed by U+0301 into é. The key builders give a romanization, and this gives the collapse.

Examples:

>>> canonicalize("Ηello Ꮤorld")  # Greek Η + Cherokee Ꮤ → Latin
'Hello World'

The output can be the empty string (#728).

Measured at Unicode 15.0.0, 137,955 single characters reduce to "" here (487 excluding the Private Use Area), and so does every string built from them. A caller keying a table on this has all of them, plus "no value", competing for one slot.

There is no on_empty here: this returns text rather than a key. The four key builders take one.

ML/NLP text normalization pipeline.

Produces clean, accent-free text suitable for tokenizers, embeddings, and feature extraction. Emoji are expanded to their CLDR short-name descriptions.

"Emoji" means the Unicode property, not the CLDR table. The annotation data also names 326 code points that carry neither Emoji nor Extended_Pictographic — the curly quotes, the dashes, the currency signs, the math operators — and naming those inserts words into ordinary prose: film’s came back as film right apostrophe s, one token to four with the possessive gone. They pass through unchanged since 0.15.0. demojize called directly still names them.

Case folding is on by default. Turn it off for a cased downstream model: folding is destructive and cannot be undone later in the chain, and an uncased evaluation harness cannot measure what it costs. This is the one preset where folding is a side effect rather than the point — catalog_key, search_key, and sort_key fold because a key has to collide, so they have no such switch.

fold_case=False does not mean "preserve the input untouched": strip_accents still runs, so José becomes Jose with the capital kept. Use normalize_confusables when diacritics must survive as well.

Examples:

>>> ml_normalize("Café RÉSUMÉ")
'cafe resume'
>>> ml_normalize("München", lang="de")
'muenchen'
>>> ml_normalize("José Martínez", fold_case=False)
'Jose Martinez'

The output can be the empty string (#728).

Measured at Unicode 15.0.0, 2,735 single characters reduce to "" here (2,735 excluding the Private Use Area), and so do most strings built from them — but not all: each regional indicator reduces to "" alone, and two together name a flag (U+1F1FA U+1F1F8 is flag: united states). A caller keying a table on this has all of them, plus "no value", competing for one slot.

There is no on_empty here: this returns text rather than a key. The four key builders take one.

Library catalog key generation pipeline.

Pipeline: resolve deletions → [digit-policy pre-fold] → NFKC → strip_bidi → strip invisibles → fold_case → (transliterate → confusables → strip_accents, to a fixed point) → fold_case → strip_control → strip_zero_width → collapse_whitespace → NFC

Produces a canonical deduplication key for bibliographic titles.

раураl  → raural     not "paypal"  (Cyrillic а, р, у)
аррlе   → arrle      not "apple"
В       → v          looks like B
Ѕ       → dz         looks like S
Η       → i          Greek Eta, looks like H

Examples:

>>> catalog_key("  Café  RÉSUMÉ  ")
'cafe resume'
>>> catalog_key("ΩMEGA  café")
'omega cafe'

The output can be the empty string (#728).

Measured at Unicode 15.0.0, 139,867 single characters reduce to "" here (2,399 excluding the Private Use Area), and so does every string built from them. A caller keying a table on this has all of them, plus "no value", competing for one slot.

on_empty reserves a sentinel for that case — the fix sanitize_filename already made with _ (#485). It applies only when the input was non-empty, so absence keeps its own key.

Strip bidi/format and invisible-injection vectors from rendered content.

Lightweight cleanup for user-submitted content destined for rendering. Strips bidirectional overrides (which can visually reorder text to hide malicious content), soft hyphens, control characters, and zero-width injections, then collapses runs of whitespace to single spaces.

Examples:

>>> strip_format("hello\x00world\u200b!")
'helloworld!'
>>> strip_format("  spaced   out  ")
'spaced out'

Search index key generation pipeline.

Pipeline: resolve deletions → [digit-policy pre-fold] → NFKC → strip_bidi → strip invisibles → fold_case → transliterate → strip_accents → fold_case → strip_control → strip_zero_width → collapse_whitespace → NFC

Produces a case-insensitive, accent-insensitive, script-insensitive lookup key. Like catalog_key but without confusable normalization — lighter and faster for search indexes.

Examples:

>>> search_key("  Café  RÉSUMÉ  ")
'cafe resume'
>>> search_key("Москва")
'moskva'
>>> search_key("Über allen Gipfeln")
'uber allen gipfeln'

The output can be the empty string (#728).

Measured at Unicode 15.0.0, 139,870 single characters reduce to "" here (2,402 excluding the Private Use Area), and so does every string built from them. A caller keying a table on this has all of them, plus "no value", competing for one slot.

on_empty reserves a sentinel for that case — the fix sanitize_filename already made with _ (#485). It applies only when the input was non-empty, so absence keeps its own key.

A spoof key: the TR39 skeleton plus the prototype classes disarm keeps apart.

Pipeline: resolve deletions → strip_bidi → strip invisibles → strip_control → strip_zero_width → NFKC → confusables → prototype fold → fixed-point(fold_case → confusables → NFKC) → collapse_whitespace

The confusable fold runs twice, and the second pass is not redundant. The table's entry for a homoglyph is often on the lowercase form, so a capital the first pass cannot match becomes matchable the moment case is folded: Ω (U+2126 OHM SIGN) reaches the fold as Ω, folds to ω, and only then to w. With a single pass skeleton_key("Ω") returned ω while skeleton_key("ω") returned w — and a key that is not a fixed point is not a key. A second pass rather than a reorder: the first has to see cased text or the prototype fold has nothing to work with.

NFKC runs inside the loop too, because the case fold and the confusable fold can each leave the text decomposed: U+0390 case-folds to three code points, and U+00A5 + U+0300 folds to Y beside a grave it composes with. The strip steps run first, ahead of NFKC: a character they remove that sits between a base and its mark would otherwise block the composition, and the key would see the bare base. Both were found by the Lean model in formal/lean/Confusables, and before them the key was not always a fixed point.

TR39 puts I, l and 1 in one equivalence class and O/0 in another. disarm's table stops short of both — every member of the capital-I family folds to I and stops there — so paypaI survives every other surface intact. This closes it.

Why a separate builder. The letter half costs six collision groups in the 235,976 entries of /usr/share/dict/words, and Ione/lone is the only ordinary-word merge among them. That price holds only on cased text: after a case fold, I ≡ l is i ≡ l, and the same class costs 264 groups of ordinary vocabulary — boiling/bolling, doit/dolt, ail/all. A factor of 44. No existing key builder runs a confusable fold before folding case, and catalog_key cannot be reordered to (#419).

Not for display. The output is a key, and it is more destructive than any preset that forwards text — the same split canonicalize and canonicalize_strict already make.

Examples:

>>> skeleton_key("paypaI")  # the class catalog_key cannot reach
'paypal'
>>> skeleton_key("paypal") == skeleton_key("paypaI")
True
>>> skeleton_key("SKU-1O0")  # digits kept apart by default
'sku-1o0'
>>> skeleton_key("SKU-1O0", digit_policy="tr39")  # ...and merged on request
'sku-loo'

The digit half is destructive by design. Under "tr39" every one of SKU-100, SKU-1O0, SKU-IOO and SKU-l00 is one key, as are v1.0.1, vI.O.I and vl.o.l. For a spoof detector that is the point; for a deduplication key over anything carrying a part number, a version or an ISBN it destroys the field.

The output can be the empty string (#728).

Measured at Unicode 15.0.0, 137,955 single characters reduce to "" here (487 excluding the Private Use Area), and so does every string built from them. A caller keying a table on this has all of them, plus "no value", competing for one slot.

on_empty reserves a sentinel for that case — the fix sanitize_filename already made with _ (#485). It applies only when the input was non-empty, so absence keeps its own key.

Sort key generation pipeline.

Pipeline: resolve deletions → [digit-policy pre-fold] → NFKC → strip_bidi → strip invisibles → fold_case → transliterate-non-Latin → fold_case → strip_control → strip_zero_width → collapse_whitespace → drop repeated marks → cap marks at 3 → NFC

A case-insensitive collation key that, unlike search_key, preserves base accented characters rather than folding them away. It keeps the accent so accented and unaccented forms stay distinct ("Über" folds to "über", not "uber") and the accent survives for a locale-aware collator. Non-Latin scripts are still folded to a consistent Latin form ("Война" → "voyna") so cross-script titles interfile. This is the collation counterpart to search_key, which folds accents away for exact-match lookup — the two are deliberately not interchangeable for accented Latin input.

Note: the result is a normalized string, not a UCA collation-weight key, so comparing keys with plain codepoint ordering will not interfile über with ASCII u… words. Pass the key to a Unicode/locale collator when linguistically-correct order matters; the value here is that the accent is preserved for it rather than folded away.

Because Latin letters are preserved verbatim, lang only affects transliteration of non-Latin runs; an accented Latin letter is never expanded by a language profile here (e.g. sort_key("Über", lang="de") is "über", whereas search_key("Über", lang="de") is "ueber").

Examples:

>>> sort_key("Война и мир")
'voyna i mir'
>>> sort_key("Über allen Gipfeln")
'über allen gipfeln'
>>> sort_key("  Café  ")
'café'

The output can be the empty string (#728).

Measured at Unicode 15.0.0, 138,404 single characters reduce to "" here (936 excluding the Private Use Area), and so does every string built from them. A caller keying a table on this has all of them, plus "no value", competing for one slot.

on_empty reserves a sentinel for that case — the fix sanitize_filename already made with _ (#485). It applies only when the input was non-empty, so absence keeps its own key.

Strict Unicode canonicalization of user input — not an injection defense.

Scoped to identifiers, not body text (#624). The confusable fold runs toward Latin, so it rewrites non-Latin text that has a Latin lookalike — Arabic alef becomes l, Hebrew yod becomes ', and Ελληνικά comes back as Eλλnvikά. It also removes U+200C, which Persian orthography requires. Nothing flags this first, because ordinary Persian is not an anomaly. Use it on usernames, hostnames, filenames and log lines; see Limitations (docs/limitations.md) before pointing it at a sentence.

The fold is not a romanization (#907). Москва comes back Mockba, where the key builders give moskva; the confusable table runs without transliterating first, so a non-Latin word reaches it as shapes rather than as sounds. See canonicalize for why that order is deliberate and what to use instead.

Runs no transliteration step while neutralizing Unicode-level attack vectors: zalgo stacking, homoglyph spoofing, bidi overrides, zero-width injections, and control characters. It used to say it "preserves the original script", which the paragraph above disproves: the confusable fold rewrites individual letters, and only the romanization step is absent (#907).

Pipeline: resolve deletions → [digit-policy pre-fold] → NFKC → strip_bidi → strip_zero_width → strip_control → strip invisible classes (#413) → (confusables and NFC, then the cross-script mark strip, to a fixed point) → drop repeated marks → strip_zalgo → collapse_whitespace → NFC (invisibles are stripped before zalgo-capping so they cannot split combining-mark runs, the cap follows the cross-script mark strip for the same reason (#862), and the terminal NFC recomposes any base+mark left adjacent by a stripped invisible — keeping the output idempotent, #416/#413)

Examples:

>>> canonicalize_strict("Hello, world!")
'Hello, world!'
>>> canonicalize_strict("p\u0430ypal")  # Cyrillic а → Latin a
'paypal'
>>> canonicalize_strict("admin\u202euser")  # RLO stripped
'adminuser'

The output can be the empty string (#728).

Measured at Unicode 15.0.0, 137,955 single characters reduce to "" here (487 excluding the Private Use Area), and so does every string built from them. A caller keying a table on this has all of them, plus "no value", competing for one slot.

There is no on_empty here: this returns text rather than a key. The four key builders take one.

Maximum-strength text deobfuscation.

Neutralizes homoglyph spoofing, zalgo abuse, invisible character injection, and bidi attacks. Uses TR39 confusable mapping (visual similarity) — Cyrillic р→p, с→c, В→B — not phonetic transliteration.

Does not transliterate. Non-Latin scripts that have no Latin confusable equivalent pass through unchanged. Chain with transliterate() explicitly if you also need romanization.

Read the exclusion in that sentence literally: the scripts that do have a Latin confusable equivalent are rewritten. 22 Arabic code points, 12 Hebrew and 65 Greek fold to ASCII, so wholly non-Latin text comes back with Latin letters in it. This preset also strips combining marks, which takes Indic vowel signs along with Latin accents — বাংলা becomes বল, which is not a word.

Scoped to identifiers, not body text (#624). The confusable fold runs toward Latin, so it rewrites non-Latin text that has a Latin lookalike — Arabic alef becomes l, Hebrew yod becomes ', and Ελληνικά comes back as Eλλnvikά. It also removes U+200C, which Persian orthography requires. Nothing flags this first, because ordinary Persian is not an anomaly. Use it on usernames, hostnames, filenames and log lines; see Limitations (docs/limitations.md) before pointing it at a sentence.

The fold is not a romanization (#907). Москва comes back Mockba, where the key builders give moskva; the confusable table runs without transliterating first, so a non-Latin word reaches it as shapes rather than as sounds. See canonicalize for why that order is deliberate and what to use instead.

Preserves case. Case is not deception — proper nouns, acronyms, and sentence boundaries are meaningful. Chain with fold_case() if lowercasing is also needed.

Pipeline: resolve deletions → [digit-policy pre-fold] → NFKC → strip_zalgo(max_marks=0) → strip_bidi → strip_zero_width → strip invisibles → confusables → strip_accents → strip_control → collapse_whitespace → NFC. There is no demojize step (#910): an emoji is left where it stands, because a comparison surface must not write attacker-chosen words into the value being compared. The terminal NFC recomposes two characters a stripped control had kept apart, keeping the output idempotent.

Examples:

>>> strip_obfuscation("P\u0430yP\u0430l")  # Cyrillic а → Latin a
'PayPal'
>>> strip_obfuscation("\u0420rodu\u0441t")  # Cyrillic Р→P, с→c
'Product'
>>> strip_obfuscation("H\u0338a\u0338t\u0338e\u0338 speech")
'Hate speech'

The output can be the empty string (#728).

Measured at Unicode 15.0.0, 140,200 single characters reduce to "" here (2,732 excluding the Private Use Area), and so does every string built from them. A caller keying a table on this has all of them, plus "no value", competing for one slot.

There is no on_empty here: this returns text rather than a key. The four key builders take one.