resources Interfaces, Classes, Traits and Enums Tokenizer Spanish specific tokenization code. Typically, tokenizer.php either contains a stemmer for the language in question or it specifies how many characters in a char gram