Yioop_V9.5_Source_Code_Documentation

QueueServer

in package

• Application

implements CrawlConstants

Command line program responsible for managing Yioop crawls.

Tags

author

Chris Pollett

Interfaces, Classes, Traits and Enums

CrawlConstants

Shared constants and enums used by components that are involved in the crawling process

Table of Contents

$all_file_types  : array<string|int, mixed>

List of all known file extensions including those not used for crawl

$allow_disallow_cache_time  : int

Microtime used to look up cache $allowed_sites and $disallowed_sites filtering data structures

$allowed_sites  : array<string|int, mixed>

Web-sites that crawler can crawl. If used, ONLY these will be crawled

$archive_modified_time  : int

This keeps track of the time the current archive info was last modified This way the queue server knows if the user has changed the crawl parameters during the crawl.

$cache_pages  : bool

Used in schedules to tell the fetcher whether or not to cache pages

$channel  : int

Channel that queue server listens to messages for

$crawl_index  : string

If the crawl_type is self::ARCHIVE_CRAWL, then crawl_index is the timestamp of the existing archive to crawl

$crawl_order  : string

Constant saying the method used to order the priority queue for the crawl

$crawl_queue  : object

Holds the CrawlQueueBundle for the crawl. This bundle encapsulates the queue of urls that specifies what to crawl next

$crawl_status_file_name  : string

name of file used to hold statistic about current crawl

$crawl_time  : int

The timestamp of the current active crawl

$crawl_type  : string

Indicates the kind of crawl being performed: self::WEB_CRAWL indicates a new crawl of the web; self::ARCHIVE_CRAWL indicates a crawl of an existing web archive

$db  : object

Reference to a database object. Used since has directory manipulation functions

$debug  : string

Holds the value of a debug message that might have been sent from the command line during the current execution of loop();

$disallowed_sites  : array<string|int, mixed>

Web-sites that the crawler must not crawl

$hourly_crawl_data  : array<string|int, mixed>

This is a list of hourly (timestamp, number_of_urls_crawled) statistics

$index_archive  : object

Holds the IndexDocumentBundle for the current crawl. This encapsulates the inverted index word-->documents for the crawls as well as document summaries of each document.

$index_dirty  : int

flags for whether the index has data to be written to disk

$indexed_file_types  : array<string|int, mixed>

List of file extensions supported for the crawl

$indexing_plugins  : array<string|int, mixed>

This is a list of indexing_plugins which might do post processing after the crawl. The plugins postProcessing function is called if it is selected in the crawl options page.

$indexing_plugins_data  : array<string|int, mixed>

This is a array of crawl parameters for indexing_plugins which might do post processing after the crawl.

$info_parameter_map  : array<string|int, mixed>

A mapping between class field names and parameters which might be sent to a queue server via an info associative array.

$last_index_save_time  : int

Last time index was saved to disk

$last_next_partition_to_add  : int

Holds the int value of the previous partition in index

$max_depth  : string

Constant saying the depth from the seeds crawl can go to

$max_description_len  : int

Max number of chars to extract for description from a page to index.

$max_links_to_extract  : int

Maximum number of urls to extract from a single document

$messages_bundle  : object

Hold the MessagesBundle to be used for the crawl. This bundle is used to store data that is sent between the QueueServer and Fetcher that has yet to be processed.

$most_recent_fetcher  : string

IP address as a string of the fetcher that most recently spoke with the queue server.

$page_range_request  : int

Maximum number of bytes to download of a webpage

$page_recrawl_frequency  : int

Number of days between resets of the page url filter If nonpositive, then never reset filter

$page_rules  : array<string|int, mixed>

Used to add page rules to be applied to downloaded pages to schedules that the fetcher will use (and hence apply the page )

$process_name  : string

String used for naming log files and for naming the processes which run related to the queue server

$quota_clear_time  : int

Timestamp of lst time download from site quotas were cleared

$quota_sites  : array<string|int, mixed>

Web-sites that have an hourly crawl quota

$quota_sites_keys  : array<string|int, mixed>

Cache of array_keys of $quota_sites

$repeat_type  : int

Controls whether a repeating crawl (negative man no) is being done and if so its frequency in second

$restrict_sites_by_url  : bool

Says whether the $allowed_sites array is being used or not

$robots_txt  : int

One of a fixed set of values which are used to control to what extent Yioop follows robots.txt files: ALWAYS_FOLLOW_ROBOTS, ALLOW_LANDING_ROBOTS, IGNORE_ROBOTS

$server_name  : string

String used to describe this kind of queue server (Indexer, Scheduler, etc. in the log files.

$server_type  : mixed

Used to say what kind of queue server this is (one of BOTH, INDEXER, SCHEDULER)

$sleep_duration  : string

If a crawl quiescent period is being used with the crawl, then this sproperty will be positive and indicate the number of seconds duration for the quiescent period.

$sleep_start  : string

If a crawl quiescent period is being used with the crawl, then this stores the time of day at which that period starts

$start_dictionary_time  : int

Keeps track of the time needed for the dictionary updater to add the current partition contents to index

$summarizer_option  : string

Stores the name of the summarizer used for crawling.

$waiting_hosts  : array<string|int, mixed>

This is a list of hosts whose robots.txt file had a Crawl-delay directive and which we have produced a schedule with urls for, but we have not heard back from the fetcher who was processing those urls. Hosts on this list will not be scheduled for more downloads until the fetcher with earlier urls has gotten back to the queue server.

$window_size  : int

Maximum number of to_crawl schedules that can waiting to be returned in the sequence they were sent out. I.e, if crawl results for fetch batch x have not been returned then fetch batch x + $window_size cannot be created from the queue.

__construct()  : mixed

Creates a Queue Server Daemon

allowedToCrawlSite()  : bool

Checks if url belongs to a list of sites that are allowed to be crawled and that the file type is crawlable

calculateScheduleMetaInfo()  : string

Used to create encode a string representing with meta info for a fetcher schedule.

checkBothProcessesRunning()  : mixed

Checks to make sure both the indexer process and the scheduler processes are running and if not restart the stopped process

checkProcessRunning()  : mixed

Checks to make sure the given process (either Indexer or Scheduler) is running.

checkRepeatingCrawlSwap()  : bool

Check for a repeating crawl whether it is time to swap between the active and search crawls.

checkUpdateCrawlParameters()  : mixed

Checks to see if the parameters by which the active crawl are being conducted have been modified since the last time the values were put into queue server field variables. If so, it updates the values to to their new values

deleteOrphanedBundles()  : mixed

Delete all the queue bundles and schedules that don't have an associated index bundle as this means that crawl has been deleted.

disallowedToCrawlSite()  : bool

Checks if url belongs to a list of sites that aren't supposed to be crawled

dumpBigScheduleToSmall()  : mixed

Used to split a large schedule of to crawl sites into small ones (which are written to disk) and which can be handled by processToCrawlUrls

getEarliestSlot()  : int

Gets the first unfilled schedule slot after $index in $arr

handleAdminMessages()  : array<string|int, mixed>

Handles messages passed via files to the QueueServer.

indexSave()  : mixed

Builds inverted index and saves active partition

initializeCrawlQueue()  : mixed

This method sets up a CrawlQueueBundle according to the current crawl order so that it can receive urls and prioritize them.

initializeIndexBundle()  : mixed

Function used to set up an indexer's IndexDocumentBundle or DoubleIndexBundle according to the current crawl parameters or the values stored in an existing bundle.

isAIndexer()  : bool

Used to check if the current queue server process is acting a indexer of data coming from fetchers

isAScheduler()  : bool

Used to check if the current queue server process is acting a url scheduler for fetchers

isOnlyIndexer()  : bool

Used to check if the current queue server process is acting only as a indexer of data coming from fetchers (and not some other activity like scheduler as well)

isOnlyScheduler()  : bool

Used to check if the current queue server process is acting only as a indexer of data coming from fetchers (and not some other activity like indexer as well)

loop()  : mixed

Main runtime loop of the queue server.

processCrawlData()  : mixed

Main body of queue server loop where indexing, scheduling, robot file processing is done.

processEtagExpires()  : mixed

Process cache page validation data files sent by Fetcher

processEtagExpiresArchive()  : mixed

Processes a cache page validation data. Extracts key-value pairs from data and inserts into the LinearHashTable used for storing cache page validation data.

processIndexArchive()  : mixed

Adds the summary and index data in $file to summary bundle and word index

processIndexData()  : mixed

Sets up the directory to look for a file of unprocessed index archive data from fetchers then calls the function processDataFile to process the oldest file found

processReceivedRobotTxtUrls()  : mixed

This method is used to send urls that are in the waiting hosts folder for hosts listed in $this->crawl_queue->robot_notify_hosts to be received to be moved to the queue because host membership in $this->crawl_queue->robot_notify_hosts indicates that a robots.txt file has just been received for the particular domain.

processRecrawlDataArchive()  : mixed

Processes fetcher data file information during a recrawl

processRecrawlRobotUrls()  : mixed

Even during a recrawl the fetcher may send robot data to the queue server. This function prints a log message and calls another function to delete this useless robot file.

processRobotArchive()  : mixed

Reads in $sites of robot data host and associated robots.txt allowed/disallowed paths, crawl delay info, and dns info.

processRobotUrls()  : mixed

Checks how old the oldest robot data is and dumps if older then a threshold, then sets up the path to the robot schedule directory and tries to process a file of robots.txt robot paths data from there

processToCrawlArchive()  : mixed

Process to-crawl urls adding to or adjusting the weight in the PriorityQueue of those which have not been seen. Also updates the queue with seen url info

processToCrawlUrls()  : mixed

Checks for a new crawl file or a schedule data for the current crawl and if such a exists then processes its contents adding the relevant urls to the priority queue

produceFetchBatch()  : mixed

Produces a schedule.txt file of url data for a fetcher to crawl next.

runPostProcessingPlugins()  : mixed

During crawl shutdown this is called to run any post processing plugins

shutdownDictionary()  : mixed

During crawl shutdown, this function is called to do a final save and merge of the crawl dictionary, so that it is ready to serve queries.

start()  : mixed

This is the function that should be called to get the queue server to start. Calls init to handle the command line arguments then enters the queue server's main loop

startCrawl()  : mixed

Begins crawling base on time, order, restricted site $info Setting up a crawl involves creating a queue bundle and an index archive bundle

stopCrawl()  : mixed

Used to stop the currently running crawl gracefully so that it can be restarted. This involved writing the queue's contents back to schedules, making the crawl's dictionary all the same tier and running any indexing_plugins.

updateDisallowedQuotaSites()  : mixed

This is called whenever the crawl options are modified to parse from the disallowed sites, those sites of the format: site#quota where quota is the number of urls allowed to be downloaded in an hour from the site. These sites are then deleted from disallowed_sites and added to $this->quota sites. An entry in $this->quota_sites has the format: $quota_site => [$quota, $num_urls_downloaded_this_hr]

updateMostRecentFetcher()  : mixed

Determines the most recent fetcher that has spoken with the web server of this queue server and stored the result in the field variable most_recent_fetcher

withinQuota()  : bool

Checks if the $url is from a site which has an hourly quota to download.

writeAdminMessage()  : mixed

Used to write an admin crawl status message during a start or stop crawl.

writeArchiveCrawlInfo()  : mixed

Used to write info about the current recrawl to file as well as to process any recrawl data files received

writeCrawlStatus()  : mixed

Writes status information about the current crawl so that the webserver app can use it for its display.