Yioop_V9.5_Source_Code_Documentation

__construct()

Makes or initializes an FeedArchiveBundle with the provided parameters

Parameters

$dir_name : string

folder name to store this bundle

$db : mixed

$read_only_archive : bool = true

whether to open archive only for reading or reading and writing

$description : string = null

a text name/serialized info about this IndexDocumentBundle

$num_docs_per_partition : int = CNUM_DOCS_PER_PARTITION

the number of pages to be stored in a single shard

Return values

addFilters()

Adds the key (often GUID) of a feed item to the bloom filter pair associated with this archive. This always adds to filter a, if filter a is more than half full it adds to filter b. If filter a is full it is deletedand filter b is renamed filter a and te process continues where a new filter b is created when this becomee half full.

Parameters

$key : string

unique identifier of a feed item

Return values

addPages()

Add the array of $pages to the documents PartitionDocumentBundle

Parameters

$pages : array<string|int, mixed>

data to store

$visited_urls_count : int

number to add to the count of visited urls (visited urls is a smaller number than the total count of objects stored in the index).

Return values

addPagesAndSeenKeys()

Adds pages of feed items to document bundle and adds their unique hashes (guids)) to bloom filters so they are not reindexed

Parameters

$pages : array<string|int, mixed>

array of feed items

$visited_urls_count : int

number of feed items

Return values

addPartitionPostingsDictionary()

Adds the previously constructed inverted index $partition to the inverted index of the whole bundle

Parameters

$partition : int = -1

which partitions inverted index to add, by default the current save partition

$taking_too_long_touch : string = null

a filename of a file to touch so its last modified time becomes the current time. In a typical Yioop crawl this is done for the CrawlConstants::crawl_status_file file to prevent Yioop's web interface from stopping the crawl because it has seen no recent progress activity on a crawl.

Return values

addScoresDocMap()

Used to add a doci_id => doc_record to the current partition's document map ($this->doc_map). A doc record records the number of words in the document, an overall length of the document, the length of its title, scores for each of the sentences included into the summary for the documents, and classifier scores for each classifier that was used by the crawl.

Parameters

$doc_id : string

new document id to add a record for

$num_words : int

number of terms in the document associated with the doc-id

$score : float

overall score for the important of this document

$host_keywords_end_pos : int

end of the portion of the document summary containing terms coming from the hostname

$title_end_pos : int

end of the portion of the document summary containing terms in the title

$path_keywords_end_pos : int

length of the portion of the document summary containing terms in the url path

$description_scores : array<string|int, mixed>

pairs of the form (length of summary portion, score for that portion)

$user_ranks : array<string|int, mixed>

for each user defined classifier for this crawl the float score of the classifier on this document

Return values

addTermCountsTrendingTable()

Updates TRENDING_TERM, hourly, daily, and weekly top term occurrences.

Parameters

$term_counts : array<string|int, mixed>

for the most recent update of the feed index, it should be an array [$lang => [$term => $occurrences]] for the top NUM_TRENDING terms per language

Return values

addTermPostingLists()

Adds posting records associated to a document to the posting lists for a partition.

Parameters

$position_offset : int

number of header bytes that might be used before including any position data in the file that positions will eventually be stored.

$doc_length : int

length of document in terms for the document for which we are adding posting data.

$word_lists : array<string|int, mixed>

term => positions within current document of that term for the document whose posting data we are adding

$meta_ids : array<string|int, mixed>

meta terms associated with the document we are adding. An example, meta term might be "media:news"

$doc_map_index : int

which document within the partition is the one we are adding. I.e., 5 would mean there were 5 earlier documents whose postings we have already added.

Return values

buildInvertedIndexPartition()

Copies all feeds items newer than $age to a new shard, then deletes old index shard and database entries older than $age. Finally sets copied shard to be active. If this method is going to take max_execution_time/2 it returns false, so an additional job can be schedules; otherwise it returns true

Parameters

$partition : int = -1

bundle partition to build inverted index for

$taking_too_long_touch : string = null

name of file to touch if building inverted index takes too long (whether SCHEDULES_DIR/ . "/{$this->channel}-" . CrawmConstants::crawl_status_file has been recently modified) is used in crawling to see if have run out of new data and the crawl can stopped.

$just_stats : bool = false

whether to just compute stats on the inverted or to actually save the results

Return values

calculateMetas()

Used to calculate the meta words for RSS feed items

Parameters

$lang : string

the locale_tag of the feed item

$pubdate : int

UNIX timestamp publication date of item

$source_name : string

the name of the feed

$guid : string

the guid of the item

$media_category : string = "news"

determines what media: metas to inject. Default is news.

Return values

contains()

Whether the active filter for this feed contain thee feed item of thee supplied key

Parameters

$key : string

the feed item id to check if in archive

Return values

deDeltaPostingsSumFrequencies()

Within postings DOC_MAP_INDEX and POSITION_OFFSETS to position lists are stored as delta lists (difference over previous values), this method undoes the delta list to restore the actual DELTA_DOC_MAP_INDEX and POSITION_OFFSETS values. It also computes the of the frequencies of items within the list of postings. This method is current only used for active partition in an index (the one whose terms haven't yet been added to the B+-tree).

Parameters

$postings : array<string|int, mixed>

a reference to an array of posting lists for a term (this will be changed by this method)

Return values

forceSave()

Forces the current shard to be saved

Return values

getCachePage()

Given the $doc_id of a document and a $partition to look for it in return's the cached page of the document if present and [] otherwise

Parameters

$doc_id : string

of document to look up

$partition : int

to look for document in

Return values

getPartitionBaseFolder()

Gets the file path corresponding to the partition with index $partition

Parameters

$partition : int

desired partition index

Return values

getPostingsString()

Get the postings stored in the postings file in a partition from $offset to $offset+len remove the 255 encoding.

Parameters

$partition : int

partition to retrieve posting from

$offset : int

byte offset int partition/postings file to look for them

$len : int

length of the posting list to retrieve.

Return values

getSummary()

Given the $doc_id of a document and a $partition to look for it in return's the document summary info if present and [] otherwise.

Parameters

$doc_id : string

of document to look up

$partition : int

to look for document in

Return values

getWordInfo()

Gets an array of posting list positions for each shard in the bundle $index_name for the word id $term_id

Parameters

$term_id : string

id of phrase or word to look up in bundle dictionary

$threshold : int = -1

after the number of results exceeds this amount stop looking for more dictionary entries.

$offset : mixed

$num_partitions : mixed = -1

$with_remaining_total : bool = false

whether to total number of postings found as well or not

Return values

invertOneSite()

Used to create inverted index for one site and add its information to the current partition.

Parameters

$site : array<string|int, mixed>

site to invert

$url_info : array<string|int, mixed>

collection of url and hash's of documents which map to the same document

$link_cnt : int

current count of number of links discovered so far

Return values

prepareIndexMap()

As pre-step to calculating the inverted index information for a partition this method groups documents and links to documents into single objects.

Parameters

$partition : int

index of partition to do deduplication for in the case that test index is empty

$test_index : array<string|int, mixed> = []

is non-null only when doing testing of what this method does. In which case, it should consist of an array of $doc_id => string represent a possible record for that doc. As deduplication is done entirely based on component of the doc_id (hash_url, doc_type, hash_doc, hash_host) the string doesn't matter too much.

Return values

stopIndexing()

Used when a crawl stops to perform final dictionary operations to produce a working stand-alone index.

Return values

unpackPostings()

Given the postings as a string for a partition for a term, unpacks them into an array of postings, doing de-delta of doc_map_indices and de-delta of positions. Each posting represents occurrence of a term in a documents, so the frequency component is the number of occurrences of the term in the document. This method also computes the sum of these frequencies over all postings in partition.

Parameters

$postings_string : string

compress string representation of a set of postings for a term

Return values

updateDictionary()

For every partition between next partition and save partition, adds the posting list information to the dictionary BPlusTree. At the end of this process next partition and save partition should be the same

Parameters

$taking_too_long_touch : string = null

a filename of a file to touch so its last modified time becomes the current time. In a typical Yioop crawl this is done for the CrawlConstants::crawl_status_file file to prevent Yioop's web interface from stopping the crawl because it has seen no recent progress activity on a crawl.

$till_equal : bool = true

is set to true will keep adding each partition up till the save partition if set to false, oln;y adds one partition

Return values

updateTrendingTermCounts()

Updates trending term counts based on the string from the current feed item.

Parameters

$term_counts : array<string|int, mixed>

lang => [term => occurrences]

$source_phrase : string

original non-stemmed phrase from feed item to adjust $term_counts with. Used to remember non-stemmed terms. We assume we have already extracted position lists from

$word_or_phrase_list : array<string|int, mixed>

associate array of stemmed_word_or_phrase => positions in feed item of where occurs

$media_category : string

of feed source the item case from. We trending counts grouped by media category

$source_name : string

of feed source the item case from. We exclude from counts the name of the feed source

$lang : string

locale_tag for this feed item

$pubdate : int

timestamp when string was published (used in weighting)

$source_stop_regex : string = ""

a regex to remove terms which occur frequently for this particular source

Return values