Fixes to TokenTool and ContextTaggers after yesterday's refactor. Adds a tests/ZhTokenizerTest.php to unit test this code, a=chris

Author Chris Pollett <chris@pollett.org>
Author date 2020-07-11 22:Jul:th
Author local date 2020-07-11 15:Jul:th -0700
Committer Chris Pollett <chris@pollett.org>
Committer date 2020-07-11 22:Jul:th
Committer local date 2020-07-11 15:Jul:th -0700
Commit 9bd0ce8b6c70a82612bcb8c71caa986c0caf1999
Tree ec9b91d0b9274c73b6ba5016fc252abf8513e378
Parent fe626fd18628fbe59d8faa0a1af330c85d910ffa
Fixes to TokenTool and ContextTaggers after yesterday's refactor. Adds a tests/ZhTokenizerTest.php to unit test this code, a=chris
Affected files:
src/configs/TokenTool.php
src/library/ContextTagger.php
src/library/NamedEntityContextTagger.php
src/library/PartOfSpeechContextTagger.php
src/library/StochasticTermSegmenter.php
src/locale/zh_CN/resources/Tokenizer.php
src/locale/zh_CN/resources/nect_weights.txt.gz
tests/ZhTokenizerTest.php
ViewGit