Classes¶

Stateful objects and builders for repeated or specialized text processing.

Text¶

>>> from disarm import Text
>>> Text("Straße").fold_case().value
'strasse'
>>> Text("  hello   world  ").collapse_whitespace().value
'hello world'
>>> str(Text("café").strip_accents())
'cafe'

value: str

normalize(*, form: NormalizationForm = 'NFC') -> Text

normalize_confusables(*, target_script: str | Script = 'latin', digit_policy: str = 'numeric') -> Text

strip_accents() -> Text

transliterate(*, lang: str | None = None, target: str | None = None, errors: ErrorMode = 'replace', replace_with: str = '[?]', strict_iso9: bool = False, gost7034: bool = False, tones: bool = False, context: bool = False) -> Text

fold_case() -> Text

collapse_whitespace() -> Text

strip_control_chars() -> Text

strip_zero_width_chars() -> Text

slugify(*, separator: str = '-', lowercase: bool = True, max_length: int = 0, word_boundary: bool = False, save_order: bool = False, stopwords: Iterable[str] = (), regex_pattern: str | None = None, replacements: Iterable[tuple[str, str]] = (), allow_unicode: bool = False, lang: str | None = None, entities: bool = True, decimal: bool = True, hexadecimal: bool = True, default: str | None = None) -> Text

sanitize_filename(*, separator: str = '_', max_length: int = 255, platform: Platform = 'universal', lang: str | None = None, preserve_extension: bool = True) -> Text

demojize(*, strip_modifiers: bool = False, errors: ErrorMode = 'replace', replace_with: str = '[?]', provider: EmojiProvider | None = None) -> Text

strip_bidi() -> Text

canonicalize() -> Text

security_clean() -> Text

ml_normalize(*, lang: str | None = None, emoji: str = 'cldr', fold_case: bool = True) -> Text

strip_format() -> Text

display_clean() -> Text

canonicalize_strict() -> Text

normalize_user_input() -> Text

strip_obfuscation() -> Text

is_ascii() -> bool

is_case_fold_stable() -> bool

is_canonical(*, preset: str = 'canonicalize') -> bool

is_normalized(*, form: NormalizationForm = 'NFC') -> bool

is_confusable(*, target_script: str | Script = 'latin') -> bool

is_mixed_script() -> bool

has_bidi_conflict() -> bool

has_bidi_control() -> bool

detect_scripts() -> list[Script]

grapheme_len() -> int

grapheme_split() -> list[str]

grapheme_truncate(max_graphemes: int) -> Text

catalog_key(*, lang: str | None = None, strict_iso9: bool = False) -> Text

Usage¶

from disarm import Text

result = (
    Text("Ünïcödé Café ☕")
    .normalize(form="NFKC")
    .demojize()
    .transliterate()
    .strip_accents()
    .fold_case()
    .value
)
assert result == "unicode cafe hot beverage"

Each transform method returns a new Text instance (immutable semantics, matching Python str). Predicates return their native type (bool, list) and do not chain.

Chainable transforms¶

All core transforms are available as methods:

Non-chaining predicates¶

Result extraction¶

Use .value or str() to extract the underlying string:

from disarm import Text

text = Text("café").strip_accents()
assert text.value == "cafe"
assert str(text) == "cafe"

Text supports ==, hash(), len(), and bool() — comparing against the underlying string value.

Slugifier¶

>>> s = Slugifier(separator="_", lang="de")
>>> s("Ärger im Büro")
'aerger_im_buero'

Usage¶

from disarm import Slugifier

slug = Slugifier(separator="_", lang="de", max_length=50)
assert slug("Ärger im Büro") == "aerger_im_buero"
assert slug("Über den Wolken") == "ueber_den_wolken"

# Auto-detect language from script
auto_slug = Slugifier(lang="auto")
assert auto_slug("Москва") == "moskva"

Accepts all the same parameters as slugify(). Construct once, call many times.

UniqueSlugifier¶

>>> u = UniqueSlugifier()
>>> u("My Post")
'my-post'
>>> u("My Post")
'my-post-1'

reset() -> None

Usage¶

from disarm import UniqueSlugifier

unique = UniqueSlugifier()
assert unique("My Post") == "my-post"
assert unique("My Post") == "my-post-1"
assert unique("My Post") == "my-post-2"

unique.reset()  # clear seen slugs
assert unique("My Post") == "my-post"

External uniqueness check¶

def exists_in_db(slug: str) -> bool:
    return db.slugs.filter(slug=slug).exists()


unique = UniqueSlugifier(check=exists_in_db)

The check callback is called for each candidate slug. If it returns True, the slugifier increments the suffix and tries again. An empty slug (an input with nothing sluggable) is returned as it is, without calling check.

TextPipeline¶

>>> pipe = TextPipeline(normalize="NFC", fold_case=True, collapse_whitespace=True)
>>> pipe("  Héllo  WÖRLD  ")
'héllo wörld'
>>> TextPipeline(confusables=True, digit_policy="tr39")("g੦ogle")
'google'

purpose: str | None

>>> get_pipeline("rag_ingest").purpose
'Normalizing retrieved documents for a RAG index, romanizing legitimate non-Latin text rather than folding homoglyphs onto Latin.'
>>> TextPipeline(fold_case=True).purpose is None
True

>>> {p: get_pipeline(p).purpose for p in list_profiles()}
{'code_context': ...}

steps: list[tuple[str, str | None]]

>>> pipe = TextPipeline(normalize="NFC", fold_case=True)
>>> pipe.steps
[('normalize', 'NFC'), ('fold_case', None)]

explain() -> str

>>> pipe = TextPipeline(normalize="NFC", fold_case=True)
>>> print(pipe.explain())
TextPipeline with 2 steps:
  1. normalize (NFC)
  2. fold_case

Usage¶

from disarm import TextPipeline

pipe = TextPipeline(
    normalize="NFC",
    confusables=True,
    strip_accents=True,
    fold_case=True,
    collapse_whitespace=True,
)

assert pipe("  Héllo Wörld  ") == "hello world"

strip_zalgo is a cap, and 0 is not "off"¶

Every other step here is switched with a boolean. strip_zalgo takes the maximum number of combining marks to allow per base character, so 0 allows none and removes every diacritic in the text — including from words that were never obfuscated. Off is None, which leaves the step out of the compiled pipeline altogether (#958):

from disarm import TextPipeline

text = "Čeština, naïve café"

# None — the default. The step is not compiled in at all.
off = TextPipeline(strip_zalgo=None)
assert off.steps == []
assert off(text) == text

# 0 — a cap of zero marks. Every diacritic goes.
assert TextPipeline(strip_zalgo=0)(text) == "Cestina, naive cafe"

# A small positive cap runs the step and still leaves ordinary accents alone.
assert TextPipeline(strip_zalgo=3)(text) == text

The same literal reads the other way in PRESETS, where ("strip_zalgo", None) names a step that does run, at its default cap.

Execution order¶

Operations execute in this fixed order regardless of construction order, which is the order steps reports:

1. Resolve deletions → 2. Normalize → 3. Strip zalgo → 4. Strip bidi → 5. Strip Plane 14 → 6. Demojize → 7. Strip accents → 8. Transliterate → 9. Confusables → 10. Fold case →
2. Confusables again and 12. fold case again (only when both are on) → 13. Strip control → 14. Strip zero-width → 15. Strip PUA → 16. Collapse whitespace

A TextPipeline runs its steps once. A named profile from get_pipeline runs them again until the output stops changing, so the two agree wherever one pass is already a fixed point (see get_pipeline).

Performance¶

The pipeline is pre-compiled at construction. Enabled steps are stored as a bitflag set — only enabled steps execute at call time.

Compatibility aliases (awesome-slugify)¶

These classes provide drop-in replacements for awesome-slugify's Slugify and UniqueSlugify. They accept awesome-slugify's parameter names and map them to native disarm parameters.

See the migration guide for full details.

Slugify¶

from disarm import Slugify
custom = Slugify(to_lower=True)
custom("Hello World")  # => "hello-world"

from disarm import Slugify

# Same API as awesome-slugify
custom = Slugify(to_lower=True)
assert custom("Hello World") == "hello-world"

# Attribute-style configuration (awesome-slugify pattern)
s = Slugify()
s.to_lower = True
s.stop_words = ("the", "a")
s.max_length = 200
assert s("The Big Fox") == "big-fox"

Accepts both awesome-slugify parameter names (to_lower, stop_words, safe_chars, capitalize, pretranslate) and native disarm names (lowercase, stopwords, replacements).

Defaults to to_lower=False (matching awesome-slugify). For python-slugify compatibility (which defaults to lowercase=True), use the native Slugifier class or the slugify() function.

UniqueSlugify¶

from disarm import UniqueSlugify
unique = UniqueSlugify()
unique("My Post")   # => "My-Post"
unique("My Post")   # => "My-Post-1"

reset() -> None

from disarm import UniqueSlugify

unique = UniqueSlugify(to_lower=True)
assert unique("My Post") == "my-post"
assert unique("My Post") == "my-post-1"

unique.reset()
assert unique("My Post") == "my-post"

Extends Slugify with uniqueness tracking. Accepts uids and unique_check parameters from awesome-slugify.

Preconfigured instances¶

Drop-in replacements for awesome-slugify's preconfigured slugifiers:

from disarm import (
    slugify_url,  # lowercase, strips articles, max 200 chars
    slugify_filename,  # underscore separator, preserves -., max 255 chars
    slugify_unicode,  # keeps non-ASCII letters
    slugify_ru,  # Russian transliteration
    slugify_de,  # German transliteration (ä→ae, ö→oe, ü→ue)
    slugify_el,  # Greek transliteration
)

assert slugify_url("The Big Fox") == "big-fox"
assert slugify_de("Ärger im Büro") == "Aerger-im-Buero"
assert slugify_filename("My Report.pdf") == "My_Report.pdf"