[Home]  [Edit this page]  [Recent Changes]  [Special Pages]  [Help] 

---

tokenizer

---

What is a tokenizer?

A tokenizer is a programming device that parses a string and divides it into tokens, small subsets of the original string. For example:

Some people like "Wheel of Fortune."

could be parsed by the space character (" ") into the following:

1: Some
2: people
3: like
4: "Wheel
5: of
6: Fortune."

or parsed by the letter m into:

1: So
2: e people like "Wheel of Fortune."

(Note that the delimiting character, m, is discarded.) A tokenizer is quite useful for strings that represent arrays; without a tokenizer, one would have to go through the arduous process of finding all instances of a character in the string, and dividing the string into subsets marked off by those characters. Tokenizers are put into programming languages (Java, for example) to keep this function from being unnecessarily rewritten every time it is needed.
[Edit this page]  [Page history]  [What links here]  [Discuss this topic]  [Printer Friendly]