Added more features in Chinese Segmenter;Moved partial code from StochasticTermSegmenter class to zh_CN\Tokenizer class for readability;Changed the default segmenter for Chinese from reverseMaximalMatch to StochasticTermSegmenter; r=chris

Author	Forrest Sun <sxh19911230@gmail.com>
Author date	2019-12-07 05:Dec:th
Author local date	2019-12-06 21:Dec:th -0800
Committer	Chris Pollett <chris@pollett.org>
Committer date	2019-12-10 22:Dec:th
Committer local date	2019-12-10 14:Dec:th -0800
Commit	9b049e44348bbdb7bde16ca6e99b3ae67b2c3c99
Tree	b33d1f39cbdd76620ee1776457236cc3d38e8650
Parent	7999b03fa940f1406021ae87ffd8c535b3b3734c

Added more features in Chinese Segmenter;Moved partial code from StochasticTermSegmenter class to zh_CN\Tokenizer class for readability;Changed the default segmenter for Chinese from reverseMaximalMatch to StochasticTermSegmenter; r=chris

Signed-off-by: Chris Pollett <chris@pollett.org>

Affected files:
src/library/StochasticTermSegmenter.php
src/locale/zh_CN/resources/Tokenizer.php

ViewGit