Yioop_V9.5_Source_Code_Documentation

Tokenizer

in package

• Application

This class has a collection of methods for English locale specific tokenization. In particular, it has a stemmer, a stop word remover (for use mainly in word cloud creation), and a part of speech tagger (for question answering). The stemmer is my stab at implementing the Porter Stemmer algorithm presented http://tartarus.org/~martin/PorterStemmer/def.txt The code is based on the non-thread safe C version given by Martin Porter.

Tags

author

Chris Pollett

Table of Contents

$adjective_type  : mixed

List of adjective-like parts of speech that might appear in lexicon file

$adverb_type  : mixed

List of adverb-like parts of speech that might appear in lexicon file

$conjunction_type  : mixed

List of conjunction-like parts of speech that might appear in lexicon file

$determiner_type  : mixed

List of determiner-like parts of speech that might appear in lexicon file

$no_stem_list  : array<string|int, mixed>

Words we don't want to be stemmed

$noun_type  : mixed

List of noun-like parts of speech that might appear in lexicon file

$question_token  : mixed

Any unique identifier corresponding to the component of a triplet which can be answered using a question answer list

$semantic_rewrites  : array<string|int, mixed>

Phrases we would like yioop to rewrite before performing a query

$stop_words  : mixed

A list of frequently occurring terms for this locale which should be excluded from certain kinds of queries

$verb_type  : mixed

List of verb-like parts of speech that might appear in lexicon file

$buffer  : string

storage used in computing the stem

$j  : int

Index to start of the suffix of the word being considered for manipulation

$k  : int

Index of the current end of the word at the current state of computing its stem

__construct()  : mixed

Do any global set up for tokenizer (none in the case of en-US)

canonicalizePunctuatedTerms()  : mixed

This methods tries to handle punctuation in terms specific to the English language such as abbreviations.

compressSentence()  : the

Take in a sentence and try to compress it to a smaller version that "retains the most important information and remains grammatically correct" (Jing 2000).

extractDeepestSpeechPartPhrase()  : string

Takes phrase tree $tree and a part-of-speech $pos returns the deepest $pos only path in tree.

extractObjectParseTree()  : array<string|int, mixed>

Takes a parse tree of a phrase or statement and returns an array with two fields CONCISE and RAW the former having the object of the original phrase (as a string) the latter having the importart parts of the object

extractPredicateParseTree()  : array<string|int, mixed>

Takes a parse tree of a phrase or statement and returns an array with two fields CONCISE and RAW the former having the predicate of the original phrase (as a string) the latter having the importart parts of the predicate

extractSubjectParseTree()  : array<string|int, mixed>

Takes a parse tree of a phrase or statement and returns an array with two fields CONCISE and RAW the former having the subject of the original phrase (as a string) the latter having the importart parts of the subject

extractTripletByType()  : array<string|int, mixed>

Takes a triplets array with subject, predicate, object fields with CONCISE, RAW subfields and produces a triplits with $type subfield (where $type is one of CONCISE and RAW) and with subject, predicate, object, and QUESTION_ANSWER_LIST subfields

extractTripletsParseTree()  : array<string|int, mixed>

Takes a parse tree of a phrase and computes subject, predicate, and object arrays. Each of these array consists of two components CONCISE and RAW, CONCISE corresponding to something more similar to the words in the original phrase and RAW to the case where extraneous words have been removed

extractTripletsPhrases()  : array<string|int, mixed>

Scans a word list for phrases. For phrases found generate a list of question and answer pairs at two levels of granularity: CONCISE (using all terms in original phrase) and RAW (removing (adjectives, etc).

isQuestion()  : bool

Takes a phrase query entered by user and return true if it is question and false if not

parseAdjective()  : array<string|int, mixed>

Takes a part-of-speech tagged phrase and pre-tree with a parse-from position and builds a parse tree for an adjective if possible

parseAuxClause()  : array<string|int, mixed>

Takes a part-of-speech tagged phrase and pre-tree with a parse-from position and builds a parse tree for a auxiliary clause if possible

parseDeterminer()  : array<string|int, mixed>

Takes a part-of-speech tagged phrase and pre-tree with a parse-from position and builds a parse tree for a determiner if possible

parseNoun()  : array<string|int, mixed>

Takes a part-of-speech tagged phrase and pre-tree with a parse-from position and builds a parse tree for a noun if possible

parseNounPhrase()  : array<string|int, mixed>

Takes a part-of-speech tagged phrase and pre-tree with a parse-from position and builds a parse tree for a noun phrase if possible

parsePrepositionalPhrases()  : array<string|int, mixed>

Takes a part-of-speech tagged phrase and pre-tree with a parse-from position and builds a parse tree for a sequence of prepositional phrases if possible

parseTypeList()  : string

Starting at the $cur_node in a $tagged_phrase parse tree for an English sentence, create a phrase string for each of the next nodes which belong to part of speech group $type.

parseVerb()  : array<string|int, mixed>

Takes a part-of-speech tagged phrase and pre-tree with a parse-from position and builds a parse tree for a verb if possible

parseVerbPhrase()  : array<string|int, mixed>

Takes a part-of-speech tagged phrase and pre-tree with a parse-from position and builds a parse tree for a verb phrase if possible

parseWholePhrase()  : array<string|int, mixed>

Given a part-of-speeech tagged phrase array generates a parse tree for the phrase using a recursive descent parser.

parseWhoQuestion()  : array<string|int, mixed>

Takes tagged question string starts with Who and returns question triplet from the question string

parseWHPlusQuestion()  : array<string|int, mixed>

Takes tagged question string starts with Wh+ except Who and returns question triplet from the question string Unlike the WHO case, here we assume there is an auxliary verb followed by a noun phrase then the rest of the verb phrase. For example, Where is soccer played?

questionParser()  : array<string|int, mixed>

Takes any question started with WH question and returns the triplet from the question

rearrangeTripletsByType()  : array<string|int, mixed>

Takes a triplets array with subject, predicate, object fields with CONCISE and RAW subfields and rearranges it to have two fields CONCISE and RAW with subject, predicate, object, and QUESTION_ANSWER_LIST subfields

segment()  : string

Stub function which could be used for a word segmenter.

stem()  : string

Computes the stem of an English word

stopwordsRemover()  : mixed

Removes the stop words from the page (used for Word Cloud generation)

tagPartsOfSpeechPhrase()  : string

Takes a phrase and tags each term in it with its part of speech.

tagTokenizePartOfSpeech()  : array<string|int, mixed>

Split input text into terms and output an array with one element per term, that element consisting of array with the term token and the part of speech tag.

cons()  : if

Checks to see if the ith character in the buffer is a consonant

cvc()  : bool

Checks whether the letters at the indices $i-2, $i-1, $i in the buffer have the form consonant - vowel - consonant and also if the second c is not w,x or y. this is used when trying to restore an e at the end of a short word. e.g.

doublec()  : bool

Checks if $j,($j-1) contain a double consonant.

ends()  : bool

Checks if the buffer currently ends with the string $s

m()  : mixed

m() measures the number of consonant sequences between 0 and j. if c is a consonant sequence and v a vowel sequence, and [.] indicates arbitrary presence, <pre> [c][v] gives 0 [c]vc[v] gives 1 [c]vcvc[v] gives 2 [c]vcvcvc[v] gives 3 .... </pre>

r()  : mixed

Sets the ending in the buffer to $s if the number of consonant sequences between $k and $j is positive.

setto()  : mixed

setto($s) sets (j+1),...k to the characters in the string $s, readjusting k.

stemPhrase()  : string

Given an English phrase produces a phrase where each of the terms has been stemmed

step1ab()  : mixed

step1ab() gets rid of plurals and -ed or -ing. e.g.

step1c()  : mixed

step1c() turns terminal y to i when there is another vowel in the stem.

step2()  : mixed

step2() maps double suffices to single ones. so -ization ( = -ize plus -ation) maps to -ize etc.Note that the string before the suffix must give m() > 0.

step3()  : mixed

step3() deals with -ic-, -full, -ness etc. similar strategy to step2.

step4()  : mixed

step4() takes off -ant, -ence etc., in context <c>vcvc<v>.

step5()  : mixed

step5() removes a final -e if m() > 1, and changes -ll to -l if m() > 1.

taggedPartOfSpeechTokensToString()  : string

Takes an array of pairs (token, tag) that came from phrase and builds a new phrase where terms look like token~tag.

vowelinstem()  : bool

Checks if 0,...$j contains a vowel

  cav(e), lov(e), hop(e), crim(e), but
  snow, box, tray.

   caresses  ->  caress
   ponies    ->  poni
   ties      ->  ti
   caress    ->  caress
   cats      ->  cat

   feed      ->  feed
   agreed    ->  agree
   disabled  ->  disable

   matting   ->  mat
   mating    ->  mate
   meeting   ->  meet
   milling   ->  mill
   messing   ->  mess

   meetings  ->  meet