tokenizer

Fetched 2026-10-26 (cache).

Synonyms: tokenized, tokenize, tokenizes, tokenise, tokenised, tokenises

Antonyms: tokenized, tokenize, tokenizes, tokenise, tokenised, tokenises

Related words from Datamuse; suggestions, not definitions.

Noun

1. {{lb|en|computing}} A [[system]] that [[parse]]s an [[input]] [[stream]] into its component [[token]]s.

Source: Wiktionary · CC BY-SA 4.0 (Wiktionary dump) · cached copy, not a rewrite.