resources Interfaces, Classes, Traits and Enums Tokenizer Korean specific tokenization code. Typically, tokenizer.php either contains a stemmer for the language in question or it specifies how many characters in a char gram