Tokenizer
in package
Indonesian specific tokenization code. Typically, tokenizer.php either contains a stemmer for the language in question or it specifies how many characters in a char gram
Tags
Table of Contents
- $char_gram_len : int
- How many characters in a char gram for this locale
- $stop_words : mixed
- A list of frequently occurring terms for this locale which should be excluded from certain kinds of queries. This is also used for language detection
- stopwordsRemover() : mixed
- Removes the stop words from the page (used for Word Cloud generation and language detection)
Properties
$char_gram_len
How many characters in a char gram for this locale
public
static int
$char_gram_len
= 5
$stop_words
A list of frequently occurring terms for this locale which should be excluded from certain kinds of queries. This is also used for language detection
public
static mixed
$stop_words
= ['sebagai', 'saya', 'nya', 'bahwa', 'dia', 'adalah', 'untuk', 'pada', 'adalah', 'dengan', 'mereka', 'menjadi', 'di', 'satu', 'memiliki', 'ini', 'dari', 'oleh', 'hot', 'kata', 'tapi', 'apa', 'beberapa', 'adalah', 'itu', 'anda', 'atau', 'memiliki', 'itu', 'dari', 'untuk', 'dan', 'sebuah', 'di', 'kami', 'bisa', 'out', 'lainnya', 'yang', 'yang', 'melakukan', 'mereka', 'waktu', 'jika', 'akan', 'bagaimana', 'kata', 'an', 'masing-masing', 'memberitahu', 'tidak', 'Kumpulan', 'tiga', 'ingin', 'udara', 'baik', 'juga', 'bermain', 'kecil', 'end', 'menempatkan', 'rumah', 'baca', 'tangan', 'pelabuhan', 'besar', 'mantra', 'tambahkan', 'bahkan', 'tanah', 'di sini', 'harus', 'besar', 'tinggi', 'seperti', 'ikuti', 'tindakan', 'mengapa', 'bertanya', 'laki-laki', 'perubahan', 'pergi', 'cahaya', 'jenis', 'off', 'perlu', 'rumah', 'gambar', 'coba', 'kami', 'lagi', 'hewan', 'titik', 'ibu', 'dunia', 'dekat', 'membangun', 'diri', 'bumi', 'ayah']
Tags
Methods
stopwordsRemover()
Removes the stop words from the page (used for Word Cloud generation and language detection)
public
static stopwordsRemover(mixed $data) : mixed
Parameters
- $data : mixed
-
either a string or an array of string to remove stop words from
Return values
mixed —$data with no stop words