Tokenization
The Script Manager provides a way to take programming-language text in an arbitrary script system and break it into tokens: language-independent symbols that can be used as input to a parser. TheIntlTokenizefunction uses information in a script system's tokens ('itl4') resource to convert text to tokens that stand for names, symbols, comments, and quoted literals.
Subtopics
- IntlTokenize