Precompiled Pipelines¶

Canonicalize text for security-sensitive comparison.

Pipeline: NFKC → strip bidi/format → strip invisible classes (#413) → strip_control → strip_zero_width → collapse_whitespace → cap combining marks (anti-zalgo, #429) → NFC → confusables → NFC (the confusable fold is sandwiched between two NFC passes so TR39 skeletoning is normalization-stable and the preset is idempotent — #416)

Collapses fullwidth bypasses, neutralizes homoglyph spoofing, strips dangerous bidi overrides and soft hyphens, then normalizes whitespace (collapsing runs, stripping control chars and zero-width injections).

.. warning:: Canonicalizes Unicode for comparison; it is not an output sanitizer and provides no XSS/HTML/SQL/injection protection. The NFKC step maps fullwidth lookalikes to live ASCII metacharacters by design (＜ → <), so the output may be more important to context-encode on the way out, not less. Encode at the sink; never emit this result into markup or a query unescaped.

Examples:

>>> canonicalize("Ηello Ꮤorld")  # Greek Η + Cherokee Ꮤ → Latin
'Hello World'

ML/NLP text normalization pipeline.

Produces clean, accent-free, lowercased text suitable for tokenizers, embeddings, and feature extraction. Emoji are expanded to their CLDR short-name descriptions.

Examples:

>>> ml_normalize("Café RÉSUMÉ")
'cafe resume'
>>> ml_normalize("München", lang="de")
'muenchen'

Library catalog key generation pipeline.

Produces a canonical deduplication key for bibliographic titles.

Examples:

>>> catalog_key("  Café  RÉSUMÉ  ")
'cafe resume'
>>> catalog_key("ΩMEGA  café")
'omega cafe'

Strip bidi/format and invisible-injection vectors from rendered content.

Lightweight cleanup for user-submitted content destined for rendering. Strips bidirectional overrides (which can visually reorder text to hide malicious content), soft hyphens, control characters, and zero-width injections, then collapses runs of whitespace to single spaces.

.. warning:: "Display-safe" means visual hygiene (no bidi reordering, no invisible injections) — not markup-safe. This does no HTML escaping and does not strip <, >, &. When rendering into HTML, still escape at the template/output layer; disarm is not an XSS defense.

Examples:

>>> strip_format("hello\x00world\u200b!")
'helloworld!'
>>> strip_format("  spaced   out  ")
'spaced out'

Search index key generation pipeline.

Produces a case-insensitive, accent-insensitive, script-insensitive lookup key. Like :func:catalog_key but without confusable normalization — lighter and faster for search indexes.

Examples:

>>> search_key("  Café  RÉSUMÉ  ")
'cafe resume'
>>> search_key("Москва")
'moskva'
>>> search_key("Über allen Gipfeln")
'uber allen gipfeln'

Sort key generation pipeline.

Pipeline: NFKC → strip_bidi → fold_case → transliterate-non-Latin → fold_case → collapse_whitespace

A case-insensitive collation key that, unlike :func:search_key, preserves base accented characters rather than folding them away. It keeps the accent so accented and unaccented forms stay distinct ("Über" folds to "über", not "uber") and the accent survives for a locale-aware collator. Non-Latin scripts are still folded to a consistent Latin form ("Война" → "voyna") so cross-script titles interfile. This is the collation counterpart to :func:search_key, which folds accents away for exact-match lookup — the two are deliberately not interchangeable for accented Latin input.

Note: the result is a normalized string, not a UCA collation-weight key, so comparing keys with plain codepoint ordering will not interfile über with ASCII u… words. Pass the key to a Unicode/locale collator when linguistically-correct order matters; the value here is that the accent is preserved for it rather than folded away.

Because Latin letters are preserved verbatim, lang only affects transliteration of non-Latin runs; an accented Latin letter is never expanded by a language profile here (e.g. sort_key("Über", lang="de") is "über", whereas search_key("Über", lang="de") is "ueber").

Examples:

>>> sort_key("Война и мир")
'voyna i mir'
>>> sort_key("Über allen Gipfeln")
'über allen gipfeln'
>>> sort_key("  Café  ")
'café'

Strict Unicode canonicalization of user input — not an injection defense.

.. warning:: This normalizes Unicode; it does not make text safe to emit into HTML, JS, URLs, SQL, or shells. It performs no escaping and does not strip <, >, & — <script>alert(1)</script> passes through unchanged, and the NFKC step can surface ASCII metacharacters from fullwidth lookalikes (＜script＞ → <script>). This is not XSS or injection protection: encode at the output sink (framework auto-escaping, DOMPurify, parameterized queries). Run this before that encoder, never instead of it. The name predates this clarification.

Preserves the original script (no transliteration) while neutralizing Unicode-level attack vectors: zalgo stacking, homoglyph spoofing, bidi overrides, zero-width injections, and control characters.

Pipeline: NFKC → strip_bidi → strip_zero_width → strip_control → strip invisible classes (#413) → strip_zalgo → confusables → collapse_whitespace → NFC (invisibles are stripped before zalgo-capping so they cannot split combining-mark runs, and the terminal NFC recomposes any base+mark left adjacent by a stripped invisible — keeping the output idempotent, #416/#413)

Examples:

>>> canonicalize_strict("Hello, world!")
'Hello, world!'
>>> canonicalize_strict("p\u0430ypal")  # Cyrillic а → Latin a
'paypal'
>>> canonicalize_strict("admin\u202euser")  # RLO stripped
'adminuser'

Maximum-strength text deobfuscation.

Neutralizes homoglyph spoofing, zalgo abuse, invisible character injection, and bidi attacks. Uses TR39 confusable mapping (visual similarity) — Cyrillic р→p, с→c, В→B — not phonetic transliteration.

Not an output sanitizer. Resolves Unicode obfuscation only; performs no HTML/JS/SQL escaping and does not strip <, >, &. Encode at the output sink — this is not XSS or injection protection.

Does not transliterate. Non-Latin scripts that have no Latin confusable equivalent pass through unchanged. Chain with transliterate() explicitly if you also need romanization.

Preserves case. Case is not deception — proper nouns, acronyms, and sentence boundaries are meaningful. Chain with fold_case() if lowercasing is also needed.

Pipeline: NFKC → strip_zalgo(max_marks=0) → strip_bidi → strip_zero_width → demojize → confusables → strip_accents → collapse_whitespace (confusables runs after demojize so typographic punctuation in emoji names is folded too, keeping the output idempotent)

Examples:

>>> strip_obfuscation("P\u0430yP\u0430l")  # Cyrillic а → Latin a
'PayPal'
>>> strip_obfuscation("\u0420rodu\u0441t")  # Cyrillic Р→P, с→c
'Product'
>>> strip_obfuscation("H\u0338a\u0338t\u0338e\u0338 speech")
'Hate speech'