Precompiled Pipelines¶

Security-focused text canonicalization.

Pipeline: NFKC → confusables → strip bidi/format → collapse_whitespace → (path-separator neutralization)

Collapses fullwidth bypasses, neutralizes homoglyph spoofing, strips dangerous bidi overrides and soft hyphens, then normalizes whitespace (collapsing runs, stripping control chars and zero-width injections).

.. warning:: Canonicalizes Unicode for comparison; it is not an output sanitizer and provides no XSS/HTML/SQL/injection protection. The NFKC step maps fullwidth lookalikes to live ASCII metacharacters by design (＜ → <), so the output may be more important to context-encode on the way out, not less. Encode at the sink; never emit this result into markup or a query unescaped.

Examples:

>>> security_clean("Ηello Ꮤorld")  # Greek Η + Cherokee Ꮤ → Latin
'Hello World'

ML/NLP text normalization pipeline.

Produces clean, accent-free, lowercased text suitable for tokenizers, embeddings, and feature extraction. Emoji are expanded to their CLDR short-name descriptions.

Examples:

>>> ml_normalize("Café RÉSUMÉ")
'cafe resume'
>>> ml_normalize("München", lang="de")
'muenchen'

Library catalog key generation pipeline.

Produces a canonical deduplication key for bibliographic titles.

Examples:

>>> catalog_key("  Café  RÉSUMÉ  ")
'cafe resume'
>>> catalog_key("ΩMEGA  café")
'omega cafe'

Display-safe text cleaning pipeline.

Pipeline: strip bidi/format → collapse_whitespace (strip control + strip zero-width)

Lightweight cleanup for user-submitted content destined for rendering. Strips bidirectional overrides (which can visually reorder text to hide malicious content), soft hyphens, control characters, and zero-width injections, then collapses runs of whitespace to single spaces.

.. warning:: "Display-safe" means visual hygiene (no bidi reordering, no invisible injections) — not markup-safe. This does no HTML escaping and does not strip <, >, &. When rendering into HTML, still escape at the template/output layer; disarm is not an XSS defense.

Examples:

>>> display_clean("hello\x00world\u200b!")
'helloworld!'
>>> display_clean("  spaced   out  ")
'spaced out'

Search index key generation pipeline.

Produces a case-insensitive, accent-insensitive, script-insensitive lookup key. Like :func:catalog_key but without confusable normalization — lighter and faster for search indexes.

Examples:

>>> search_key("  Café  RÉSUMÉ  ")
'cafe resume'
>>> search_key("Москва")
'moskva'
>>> search_key("Über allen Gipfeln")
'uber allen gipfeln'

Sort key generation pipeline.

Pipeline: NFKC → strip_bidi → transliterate → fold_case → collapse_whitespace

Produces a case-insensitive ASCII key for alphabetical ordering. Transliteration folds accented characters to their ASCII base (é → e, ü → u), so the result is accent-folded, not accent-preserving.

.. note:: In practice this currently produces the same output as :func:search_key: search_key adds an explicit accent-strip pass, but transliteration has already removed accents by that point, so the two keys coincide for typical input. Use whichever name documents intent at the call site. (Distinct accent-preserving ordering is tracked for a future release.)

Examples:

>>> sort_key("Война и мир")
'voyna i mir'
>>> sort_key("Über allen Gipfeln")
'uber allen gipfeln'
>>> sort_key("  Café  ")
'cafe'

Unicode hygiene for user-submitted input — not an injection defense.

.. warning:: This normalizes Unicode; it does not make text safe to emit into HTML, JS, URLs, SQL, or shells. It performs no escaping and does not strip <, >, & — <script>alert(1)</script> passes through unchanged, and the NFKC step can surface ASCII metacharacters from fullwidth lookalikes (＜script＞ → <script>). This is not XSS or injection protection: encode at the output sink (framework auto-escaping, DOMPurify, parameterized queries). Run this before that encoder, never instead of it. The name predates this clarification.

Preserves the original script (no transliteration) while neutralizing Unicode-level attack vectors: zalgo stacking, homoglyph spoofing, bidi overrides, zero-width injections, and control characters.

Pipeline: NFKC → strip_bidi → strip_zero_width → strip_control → strip_zalgo → confusables → collapse_whitespace → (path-separator neutralization) (invisibles are stripped before zalgo-capping so they cannot split combining-mark runs, keeping the output idempotent)

Examples:

>>> normalize_user_input("Hello, world!")
'Hello, world!'
>>> normalize_user_input("p\u0430ypal")  # Cyrillic а → Latin a
'paypal'
>>> normalize_user_input("admin\u202euser")  # RLO stripped
'adminuser'