Yioop_V9.5_Source_Code_Documentation

__construct()

Sets up the database manager that will be used and name of the search engine database

Parameters

$db_name : string = CDB_NAME

the name of the database for the search engine

$connect : bool = true

whether to connect to the database by default after making the datasource class

Return values

aggregateCrawlList()

When @see getCrawlList() is used in a multi-queue server this method used to integrate the crawl lists received by the different machines

Parameters

$list_strings : array<string|int, mixed>

serialized crawl list data from different queue servers

$data_field : string = null

field of $list_strings to use for data

Return values

aggregateStalled()

When @see crawlStalled() is used in a multi-queue server this method used to integrate the stalled information received by the different machines

Parameters

$stall_statuses : array<string|int, mixed>

contains web encoded serialized data one one field of which has the boolean data concerning stalled statis

$data_field : string = null

field of $stall_statuses to use for data if null then each element of $stall_statuses is a web encoded serialized boolean

Return values

aggregateStatuses()

When @see crawlStatus() is used in a multi-queue server this method used to integrate the status information received by the different machines

Parameters

$status_strings : array<string|int, mixed>

$data_field : string = null

field of $status_strings to use for data

Return values

appendSuggestSites()

Add new distinct urls to those already saved in the suggest_url_file If the supplied url is not new or the file size exceeds MAX_SUGGEST_URL_FILE_SIZE then it is not added.

Parameters

$url : string

to add

Return values

boldKeywords()

Given a string, wraps in bold html tags a set of key words it contains.

Parameters

$text : string

haystack string to look for the key words

$words : array<string|int, mixed>

an array of words to bold face

Return values

clearCrawlCaches()

Clears several memory and file caches related to crawls and networking.

Return values

clearQuerySavePoint()

A save point is used to store to disk a sequence generation-doc-offset pairs of a particular mix query when doing an archive crawl of a crawl mix. This is used so that the mix can remember where it was the next time it is invoked by the web app on the machine in question.

Parameters

$save_timestamp : int

timestamp of save point to delete

$machine_urls : array<string|int, mixed> = null

machines on which to try to delete savepoint

Return values

clearSuggestSites()

Resets the suggest_url_file to be the empty file

Return values

combinedCrawlInfo()

This method is used to reduce the number of network requests needed by the crawlStatus method of admin_controller. It returns an array containing the results of the @see crawlStalled

Parameters

$machine_urls : array<string|int, mixed> = null

an array of urls of yioop queue servers

$use_cache : bool = false

whether to try to use a cached version of the the crawl info or to always recompute it.

Tags

see

crawlStatus

and @see getCrawlList methods

Return values

crawlStalled()

Determines if the length of time since any of the fetchers has spoken with any of the queue servers has exceeded CRAWL_TIMEOUT. If so, typically the caller of this method would do something such as officially stop the crawl.

Parameters

$machine_urls : array<string|int, mixed> = null

an array of urls of yioop queue servers

Return values

crawlStatus()

Returns data about current crawl such as DESCRIPTION, TIMESTAMP, peak memory of various processes, most recent fetcher, most recent urls, urls seen, urls visited, etc.

Parameters

$machine_urls : array<string|int, mixed> = null

an array of urls of yioop queue servers on which the crawl is being conducted

Return values

createIfNecessaryDirectory()

Creates a directory and sets it to world permission if it doesn't already exist

Parameters

$directory : string

name of directory to create

Return values

deleteCrawl()

Deletes the crawl with the supplied timestamp if it exists. Also deletes any crawl mixes making use of this crawl

Parameters

$timestamp : string

a Unix timestamp

$machine_urls : array<string|int, mixed> = null

an array of urls of yioop queue servers

Return values

deleteCrawlMix()

Deletes from the DB the crawl mix ans its associated components and fragments

Parameters

$timestamp : int

of the mix to delete

Return values

deleteCrawlMixIteratorState()

Deletes the archive iterator and savepoint files created during the process of iterating through a crawl mix.

Parameters

$timestamp : int

The timestamp of the crawl mix

Return values

execMachines()

This method is invoked by other ParallelModel (@see CrawlModel for examples) methods when they want to have their method performed on an array of other Yioop instances. The results returned can then be aggregated. The invocation sequence is crawlModelMethodA invokes execMachine with a list of urls of other Yioop instances. execMachine makes REST requests of those instances of the given command and optional arguments This request would be handled by a CrawlController which in turn calls crawlModelMethodA on the given Yioop instance, serializes the result and gives it back to execMachine and then back to the originally calling function.

Parameters

$command : string

the ParallelModel method to invoke on the remote Yioop instances

$machine_urls : array<string|int, mixed>

machines to invoke this command on

$arg : string = null

additional arguments to be passed to the remote machine

$num_machines : int

the integer to be used in calculating partition

$send_specs : bool = false

whether to send the queue_server, num fetcher info for given machine

$fetcher_queue_server_ratio : int = 1

maximum of the number 1 and the number of active fetchers running across all yioop instances currently divided by the number of queue servers

Return values

fileGetContents()

Either a wrapper for file_get_contents, or if a WebSite object is being used to serve pages, it reads it in using blocking I/O file_get_contents() and caches it before return its string contents.

Parameters

$filename : string

name of file to get contents of

$force_read : bool = false

whether to force the file to be read from persistent storage rather than the cache

Return values

filePutContents()

Either a wrapper for file_put_contents, or if a WebSite object is being used to serve pages, writes $data to the persistent file with name $filename. Saves a copy in the RAM cache if there is a copy already there.

Parameters

$filename : string

name of file to write to persistent storages

$data : string

string of data to store in file

Return values

formatSinglePageResult()

Given a page summary, extracts snippets which are related to a set of search words. For each snippet, bold faces the search terms, and then creates a new summary array.

Parameters

$page : array<string|int, mixed>

a single search result summary

$words : array<string|int, mixed> = null

keywords (typically what was searched on)

$description_length : int = self::DEFAULT_DESCRIPTION_LENGTH

length of the description

Return values

fromCallback()

{@inheritDoc}

Parameters

$args : mixed = null

any additional arguments which should be used to determine these tables (in this case none)

Return values

getChannel()

Gets the channel of the crawl with the given timestamp

Parameters

$timestamp : int

of crawl to get channel for

Return values

getCrawlItem()

Get a summary of a document based on its url, the active machines and the idnex we want to look up in.

Parameters

$url : string

of summary we are trying to look-up

$machine_urls : array<string|int, mixed> = null

an array of urls of yioop queue servers

$index_name : string = ""

timestamp of the index to do the lookup in

Return values

getCrawlItems()

Gets summaries for a set of document by their url, or by group of 5-tuples of the form (machine, key, index, generation, offset).

Parameters

$lookups : string

things whose summaries we are trying to look up

$machine_urls : array<string|int, mixed> = null

an array of urls of yioop queue servers

$exclude_fields : array<string|int, mixed> = []

an array of fields which might be int the crawlItem but which should be excluded from the result. This will make the result smaller and so hopefully faster to transmit

$format_words : array<string|int, mixed> = null

words which should be highlighted in search snippets returned

$description_length : int = self::DEFAULT_DESCRIPTION_LENGTH

length of snippets to be returned for each search result

Return values

getCrawlList()

Gets a list of all index archives of crawls that have been conducted

Parameters

$return_arc_bundles : bool = false

whether index bundles used for indexing arc or other archive bundles should be included in the list

$return_recrawls : bool = false

whether index archive bundles generated as a result of recrawling should be included in the result

$machine_urls : array<string|int, mixed> = null

an array of urls of yioop queue servers

$cache : bool = false

whether to try to get/set the data to a cache file

Return values

getCrawlMix()

Retrieves the weighting component of the requested crawl mix

Parameters

$timestamp : string

of the requested crawl mix

$just_components : bool = false

says whether to find the mix name or just the components array.

Return values

getCrawlMixTimestamp()

Returns the timestamp associated with a mix name;

Parameters

$mix_name : string

name to lookup

Return values

getCrawlSeedInfo()

Returns the crawl parameters that were used during a given crawl

Parameters

$timestamp : string

timestamp of the crawl to load the crawl parameters of

$machine_urls : array<string|int, mixed> = null

an array of urls of yioop queue servers

Return values

getCurrentIndexDatabaseName()

Gets the name (aka timestamp) of the current index archive to be used to handle search queries

Return values

getDbmsList()

Gets a list of all DBMS that work with the search engine

Return values

getDeltaFileInfo()

Returns all the files in $dir or its subdirectories with modified times more recent than timestamp. The file which have in their path or name a string in the $excludes array will be exclude

Parameters

$dir : string

a directory to traverse

$timestamp : int

used to check modified times against

$excludes : array<string|int, mixed>

an array of path substrings tot exclude

Return values

getInfoTimestamp()

Get a description associated with a Web Crawl or Crawl Mix

Parameters

$timestamp : int

of crawl or mix in question

$machine_urls : array<string|int, mixed> = null

an array of urls of yioop queue servers

Return values

getMixList()

Gets a list of all mixes of available crawls

Parameters

$user_id : int

user that we are getting a list of mixes for We have disabled mix sharing so for now this is all mixes

$with_components : bool = false

if false then don't load the factors that make up the crawl mix, just load the name of the mixes and their timestamps; otherwise, if true loads everything

Return values

getRows()

Gets a range of rows which match the provided search criteria from $th provided table

Parameters

$limit : int

starting row from the potential results to return

$num : int

number of rows after start row to return

$total : int

gets set with the total number of rows that can be returned by the given database query

$search_array : array<string|int, mixed> = []

each element of this is a quadruple name of a field, what comparison to perform, a value to check, and an order (ascending/descending) to sort by

$args : array<string|int, mixed> = null

additional values which may be used to get rows (what these are will typically depend on the subclass implementation)

Return values

getSeedInfo()

Returns the initial sites that a new crawl will start with along with crawl parameters such as crawl order, allowed and disallowed crawl sites

Parameters

$use_default : bool = false

whether or not to use the Yioop! default crawl.ini file rather than the one created by the user.

Return values

getSnippets()

Given a string, extracts a snippets of text related to a given set of key words. For a given word a snippet is a window of characters to its left and right that is less than a maximum total number of characters.

Parameters

$text : string

haystack to extract snippet from

$words : array<string|int, mixed>

keywords used to make look in haystack

$description_length : string

length of the description desired

Return values

getSuggestSites()

Returns an array of urls which were stored via the suggest-a-url form in suggest_view.php

Return values

getUserId()

Get the user_id associated with a given username (In base class as used as an internal method in both signin and user models)

Parameters

$username : string

the username to look up

Return values

injectUrlsCurrentCrawl()

Add the provided urls to the schedule directory of URLs that will be crawled

Parameters

$timestamp : string

Unix timestamp of crawl to add to schedule of

$inject_urls : array<string|int, mixed>

urls to be added to the schedule of the active crawl

$machine_urls : array<string|int, mixed> = null

an array of urls of yioop queue servers

Return values

isCrawlMix()

Returns whether the supplied timestamp corresponds to a crawl mix

Parameters

$timestamp : string

of the requested crawl mix

Return values

isMixOwner()

Returns whether there is a mix with the given $timestamp that $user_id owns. Currently mmix ownership is ignored and this is set to always return true;

Parameters

$timestamp : string

to see if exists

$user_id : string

id of would be owner

Return values

isSingleLocalhost()

Used to determine if an action involves just one yioop instance on the current local machine or not

Parameters

$machine_urls : array<string|int, mixed>

urls of yioop instances to which the action applies

$index_timestamp : string = -1

if timestamp exists checks if the index has declared itself to be a no network index.

Return values

loginDbms()

Returns whether the provided dbms needs a login and password or not (sqlite or sqlite3)

Parameters

$dbms : string

the name of a database management system

Return values

lookupSummaryOffsetGeneration()

Determines the offset into the summaries WebArchiveBundle and generation of the provided url (or hash_url) so that the info:url (info:base64_hash_url) summary can be retrieved. This assumes of course that the info:url meta word has been stored.

Parameters

$url_or_key : string

either info:base64_hash_url or just a url to lookup

$index_name : string = ""

index into which to do the lookup

$is_key : bool = false

whether the string is info:base64_hash_url or just a url

Return values

networkGetCrawlItems()

In a multiple queue server setting, gets summaries for a set of document by their url, or by group of 5-tuples of the form (machine, key, index, generation, offset). This makes an execMachines call to make a network request to the CrawlController's on each machine which in turn calls getCrawlItems (and thence nonNetworkGetCrawlItems) on each machine. The results are then sent back to networkGetCrawlItems and aggregated.

Parameters

$lookups : string

things whose summaries we are trying to look up

$machine_urls : array<string|int, mixed>

an array of urls of yioop queue servers

$exclude_fields : array<string|int, mixed> = []

an array of fields which might be int the crawlItem but which should be excluded from the result. This will make the result smaller and so hopefully faster to transmit

$format_words : array<string|int, mixed> = null

words which should be highlighted in search snippets returned

$description_length : int = self::DEFAULT_DESCRIPTION_LENGTH

length of snippets to be returned for each search result

Return values

nonNetworkGetCrawlItems()

Gets summaries on a particular machine for a set of document by their url, or by group of 5-tuples of the form (machine, key, index, generation, offset) This may be used in either the single queue_server setting or it may be called indirectly by a particular machine's CrawlController as part of fufilling a network-based getCrawlItems request. $lookups contains items which are to be grouped (as came from same url or site with the same cache). So this function aggregates their descriptions.

Parameters

$lookups : string

things whose summaries we are trying to look up

$exclude_fields : array<string|int, mixed> = []

an array of fields which might be in the crawlItem but which should be excluded from the result. This will make the result smaller and so hopefully faster to transmit

$format_words : array<string|int, mixed> = null

words which should be highlighted in search snippets returned

$description_length : int = self::DEFAULT_DESCRIPTION_LENGTH

length of snippets to be returned for each search result

Return values

postQueryCallback()

Called after getRows has retrieved all the rows that it would retrieve but before they are returned to give one last place where they could be further manipulated. For example, in MachineModel this callback is used to make parallel network calls to get the status of each machine returned by getRows. The default for this method is to leave the rows that would be returned unchanged

Parameters

$rows : array<string|int, mixed>

that have been calculated so far by getRows

Return values

rowCallback()

{@inheritDoc}

Parameters

$row : array<string|int, mixed>

row as retrieved from database query

$args : mixed

additional arguments that might be used by this callback. In this case, should be a boolean flag that says whether or not to add information about the components of the crawl mix

Return values

searchArrayToWhereOrderClauses()

Creates the WHERE and ORDER BY clauses for a query of a Yioop table such as USERS, ROLE, GROUP, which have associated search web forms. Searches are case insensitive

Parameters

$search_array : array<string|int, mixed>

each element of this is a quadruple name of a field, what comparison to perform, a value to check, and an order (ascending/descending) to sort by

$any_fields : array<string|int, mixed> = ['status']

these fields if present in search array but with value "-1" will be skipped as part of the where clause but will be used for order by clause

Return values

selectCallback()

Controls which columns and the names of those columns from the tables underlying the given model should be return from a getRows call.

Parameters

$args : mixed = null

any additional arguments which should be used to determine the columns

Return values

sendStartCrawlMessage()

Used to send a message to the queue servers to start a crawl

Parameters

$crawl_params : array<string|int, mixed>

has info like the time of the crawl, whether starting a new crawl or resuming an old one, etc.

$seed_info : array<string|int, mixed> = null

what urls to crawl, etc as from the crawl.ini file

$machine_urls : array<string|int, mixed> = null

an array of urls of yioop queue servers

$num_fetchers : int

number of fetchers on machine to start. This parameter and $channel are used to start the daemons running on the machines if they aren't already running

$fetcher_queue_server_ratio : int = 1

maximum of the number 1 and the number of active fetchers running across all yioop instances currently divided by the number of queue servers

Return values

sendStopCrawlMessage()

Used to send a message to the queue servers to stop a crawl

Parameters

$channel :

of crawl to stop

$machine_urls : array<string|int, mixed> = null

an array of urls of yioop queue servers

Return values

setCrawlMix()

Stores in DB the supplied crawl mix object

Parameters

$mix : array<string|int, mixed>

an associative array representing the crawl mix object

Return values

setCrawlSeedInfo()

Changes the crawl parameters of an existing crawl (can be while crawling) Not all fields are allowed to be updated

Parameters

$timestamp : string

timestamp of the crawl to change

$new_info : array<string|int, mixed>

the new parameters

$machine_urls : array<string|int, mixed> = null

an array of urls of yioop queue servers

Return values

setCurrentIndexDatabaseName()

Sets the IndexArchive that will be used for search results

Parameters

$timestamp :

the timestamp of the index archive. The timestamp is when the crawl was started. Currently, the timestamp appears as substring of the index archives directory name

Return values

setSeedInfo()

Writes a crawl.ini file with the provided data to the user's WORK_DIRECTORY

Parameters

$info : array<string|int, mixed>

an array containing information about the crawl

Return values

startQueueServerFetchers()

Used to start QueueServers and Fetchers on current machine when it is detected that someone tried to start a crawl but hadn't started any queue servers or fetchers.

Parameters

$channel : int

channel of crawl to start

$num_fetchers : int

the number of fetchers on the current machine

Return values

translateDb()

Used to get the translation of a string_id stored in the database to the given locale.

Parameters

$string_id : string

id to translate

$locale_tag : string

to translate to

Return values

whereCallback()

Controls the WHERE clause of the SQL query that underlies the given model and should be used in a getRows call.

Parameters

$args : mixed = null

additional arguments that might be used to construct the WHERE clause.

Return values