Yioop_V9.5_Source_Code_Documentation

Fetcher

in package

• Application

implements CrawlConstants

This class is responsible for fetching web pages for the SeekQuarry/Yioop search engine

Tags

author

Chris Pollett

Interfaces, Classes, Traits and Enums

CrawlConstants

Shared constants and enums used by components that are involved in the crawling process

Table of Contents

DEFAULT_POST_MAX_SIZE  = 2000000

Before receiving any data from a queue server's web app this is the default assumed post_max_size in bytes

DOMAIN_FILTER_GLOB  = \seekquarry\yioop\configs\DATA_DIR . "/domain_filters/*.ftr"

Domain Filter Glob pattern for Bloom filters used to specify allowed domains. (Only specifies if some filter file of this type exists)

GIT_URL_CONTINUE  = '@@@@'

constant indicating Git repository

HEX_NULL_CHARACTER  = "\x00"

A indicator to represent next position after the access code in Git tree object

INDICATOR_NONE  = 'none'

An indicator to tell no actions to be taken

REPOSITORY_GIT  = 'git'

constant indicating Git repository

$active_classifiers  : array<string|int, mixed>

Contains which classifiers to use for the current crawl Classifiers can be used to label web documents with a meta word if the classifiers threshold is met

$active_rankers  : array<string|int, mixed>

Contains which classifiers to use for the current crawl that are being used to rank web documents. The score that the classifier gives to a document is used for this ranking purposes

$all_file_types  : array<string|int, mixed>

List of all known file extensions including those not used for crawl

$all_git_urls  : array<string|int, mixed>

To store all the internal git urls fetched

$allow_disallow_cache_time  : int

Microtime used to look up cache $allowed_sites and $disallowed_sites filtering data structures

$allowed_sites  : array<string|int, mixed>

Web-sites that crawler can crawl. If used, ONLY these will be crawled

$arc_dir  : string

For a non-web archive crawl, holds the path to the directory that contains the archive files and their description (web archives have a different structure and are already distributed across machines and fetchers)

$arc_type  : string

For an archive crawl, holds the name of the type of archive being iterated over (this is the class name of the iterator, without the word 'Iterator')

$archive_iterator  : object

If an web archive crawl (i.e. a re-crawl) is active then this field holds the iterator object used to iterate over the archive

$cache_pages  : bool

Whether to cache pages or just the summaries

$channel  : int

Channel that queue server listens to messages for

$check_crawl_time  : int

The last time the name server was checked for a crawl time

$crawl_index  : string

If the crawl_type is self::ARCHIVE_CRAWL, then crawl_index is the timestamp of the existing archive to crawl

$crawl_order  : string

Stores the name of the ordering used to crawl pages. This is used in a switch/case when computing weights of urls to be crawled before sending these new urls back to a queue_server.

$crawl_stat_filename  : string

Name of file used to store fetcher statistics for the current crawl

$crawl_stat_info  : string

Fetcher statistics for the current crawl

$crawl_time  : int

Timestamp of the current crawl

$crawl_type  : string

Indicates the kind of crawl being performed: self::WEB_CRAWL indicates a new crawl of the web; self::ARCHIVE_CRAWL indicates a crawl of an existing web archive

$current_server  : int

Index into $queue_servers of the server get schedule from (or last one we got the schedule from)

$db  : object

Reference to a database object. Used since has directory manipulation functions

$debug  : string

Holds the value of a debug message that might have been sent from the command line during the current execution of loop();

$disallowed_sites  : array<string|int, mixed>

Web-sites that the crawler must not crawl

$domain_filters  : array<string|int, mixed>

An array of Bloom filters which if non empty will be used to restrict discovered url links. Namely, if the domain of a url is not in any of the filters and is not of thhe form of a company level domain (cld) or www.cld then it will be pruned. Here a company level domain is of the form some_name.tld where tld is a top level domain or the form some_name.country_level_tld.tld if the the tld is that of a country. So for site.somewhere.jp, somewhere.jp would be the cld; for site.somewhere.co.jp, somewhere.co.jp would be the cld.

$fetcher_num  : string

Which fetcher instance we are (if fetcher run as a job and more that one)

$found_sites  : array<string|int, mixed>

Summary information for visited sites that the fetcher hasn't sent to a queue_server yet

$hosts_with_errors  : array<string|int, mixed>

An array to keep track of hosts which have had a lot of http errors

$indexed_file_types  : array<string|int, mixed>

List of file extensions supported for the crawl

$max_depth  : int

Maximum depth fetcher should extract need seed urls to

$max_description_len  : int

Max number of chars to extract for description from a page to index.

$max_links_to_extract  : int

Maximum number of urls to extract from a single document

$minimum_fetch_loop_time  : int

Fetcher must wait at least this long between multi-curl requests.

$name_server  : array<string|int, mixed>

Urls or IP address of the web_server used to administer this instance of yioop. Used to figure out available queue_servers to contact for crawling data

$no_process_links  : bool

When processing recrawl data this says to assume the data has already had its inks extracted into a field and so this doesn't have to be done in a separate step

$num_download_attempts  : mixed

Number of attempts to download urls in current fetch batch

$num_multi_curl  : int

For a web crawl only the number of web pages to download in one go.

$page_processors  : array<string|int, mixed>

An associative array of (mimetype => name of processor class to handle) pairs.

$page_range_request  : int

Maximum number of bytes to download of a webpage

$page_rule_parser  : array<string|int, mixed>

Holds the parsed page rules which will be applied to document summaries before finally storing and indexing them

$plugin_hash  : string

Hash used to keep track of whether $plugin_processors info needs to be changed

$plugin_processors  : array<string|int, mixed>

An associative array of (page processor => array of indexing plugin name associated with the page processor). It is used to determine after a page is processed which plugins' pageProcessing($page, $url) method should be called

$post_max_size  : int

Maximum number of bytes which can be uploaded to the current queue server's web app in one go

$processors  : array<string|int, mixed>

Page prcoessors used by this fetcher

$programming_language_extension  : array<string|int, mixed>

To map programming languages with their extensions

$proxy_servers  : array<string|int, mixed>

an array of proxy servers to use rather than to directly download web pages from the current machine. If is the empty array, then we just directly download from the current machine

$queue_servers  : array<string|int, mixed>

Array of Urls or IP addresses of the queue_servers to get sites to crawl from

$recrawl_check_scheduler  : bool

Keeps track of whether during the recrawl we should notify a queue_server scheduler about our progress in mini-indexing documents in the archive

$restrict_sites_by_url  : bool

Says whether the $allowed_sites array is being used or not

$robots_txt  : int

One of a fixed set of values which are used to control to what extent Yioop follows robots.txt files: ALWAYS_FOLLOW_ROBOTS, ALLOW_LANDING_ROBOTS, IGNORE_ROBOTS

$schedule_time  : int

Timestamp from a queue_server of the current schedule of sites to download. This is sent back to the server once this schedule is completed to help the queue server implement crawl-delay if needed.

$scrapers  : array<string|int, mixed>

Contains an array of scrapers used to extract the import content from particular kind of HTML pages, for example, pages generated by a particular content management system.

$sequence_number  : int

Holds the sequence number of the current schedule received from queue server

$sleep_duration  : string

If a crawl quiescent period is being used with the crawl, then this sproperty will be positive and indicate the number of seconds duration for the quiescent period.

$sleep_start  : string

If a crawl quiescent period is being used with the crawl, then this stores the time of day at which that period starts

$summarizer_option  : string

Stores the name of the summarizer used for crawling.

$to_crawl  : array<string|int, mixed>

Contains the list of web pages to crawl from a queue_server

$to_crawl_again  : array<string|int, mixed>

Contains the list of web pages to crawl that failed on first attempt (we give them one more try before bailing on them)

$tor_proxy  : string

If this is not null and a .onion url is detected then this url will used as a proxy server to download the .onion url

$total_git_urls  : int

To keep track of total number of Git internal urls

__construct()  : mixed

Sets up the field variables so that crawling can begin

addToCrawlSites()  : mixed

Used to add a set of links from a web page to the array of sites which need to be crawled.

allowedToCrawlSite()  : bool

Checks if url belongs to a list of sites that are allowed to be crawled and that the file type is crawlable

checkArchiveScheduler()  : array<string|int, mixed>

During an archive crawl this method is used to get from the name server a collection of pages to process. The fetcher will later process these and send summaries to various queue_servers.

checkCrawlTime()  : bool

Makes a request of the name server machine to get the timestamp of the currently running crawl to see if it changed

checkScheduler()  : mixed

Get status, current crawl, crawl order, and new site information from the queue_server.

compressAndUnsetSeenUrls()  : string

Computes a string of compressed urls from the seen urls and extracted links destined for the current queue server. Then unsets these values from $this->found_sites

copySiteFields()  : mixed

Copies fields from the array of site data to the $i indexed element of the $summarized_site_pages. This will flatten info in a DOC_INFO subarray. This method is both useful to prepare downloaded site info and to prepare meta info for sub docs that might be produced by an indexing puglin.

cullNoncrawlableSites()  : mixed

Used to remove from the to_crawl urls those that are no longer crawlable because the allowed and disallowed sites have changed.

deleteOldCrawls()  : mixed

Deletes any crawl web archive bundles not in the provided array of crawls

disallowedToCrawlSite()  : bool

Checks if url belongs to a list of sites that aren't supposed to be crawled

downloadPagesArchiveCrawl()  : array<string|int, mixed>

Extracts NUM_MULTI_CURL_PAGES from the current Archive Bundle that is being recrawled.

downloadPagesWebCrawl()  : array<string|int, mixed>

Get a list of urls from the current fetch batch provided by the queue server. Then downloads these pages. Finally, reschedules, if possible, pages that did not successfully get downloaded.

exceedMemoryThreshold()  : bool

Function to check if memory for this fetcher instance is getting low relative to what the system will allow.

getFetchSites()  : array<string|int, mixed>

Prepare an array of up to NUM_MULTI_CURL_PAGES' worth of sites to be downloaded in one go using the to_crawl array. Delete these sites from the to_crawl array.

getPageThumbs()  : mixed

Adds thumbs for websites with a self::THUMB_URL field by downloading the linked to images and making a thumb from it.

loop()  : mixed

Main loop for the fetcher.

pageProcessor()  : object

Return the fetcher's copy of a page processor for the given mimetype.

processFetchPages()  : array<string|int, mixed>

Processes an array of downloaded web pages with the appropriate page processor.

processSubdocs()  : mixed

The pageProcessing method of an IndexingPlugin generates a self::SUBDOCS array of additional "micro-documents" that might have been in the page. This methods adds these documents to the summaried_size_pages and stored_site_pages arrays constructed during the execution of processFetchPages()

pruneLinks()  : mixed

This method attempts to cull from the doc_info struct the best $this->max_links_to_extract. Currently, this is done by first removing links of filetype or sites the crawler is forbidden from crawl.

reschedulePages()  : an

Sorts out pages for which no content was downloaded so that they can be scheduled to be crawled again.

selectCurrentServerAndUpdateIfNeeded()  : mixed

At least once, and while memory is low selects next server and send any fetcher data we have to it.

setCrawlParamsFromArray()  : mixed

Sets parameters for fetching based on provided info struct ($info typically would come from the queue server)

start()  : mixed

This is the function that should be called to get the fetcher to start fetching. Calls init to handle the command-line arguments then enters the fetcher's main loop

updateDomainFilters()  : mixed

Updates the array of domain filters currently loaded into memory based on which BloomFilterFiles are present in WORK_DIRECTORY/data/domain_filters and if they have changed since the current in-memory filters were loaded

updateFoundSites()  : mixed

Updates the $this->found_sites array with data from the most recently downloaded sites. This means updating the following sub arrays: the self::ROBOT_PATHS, self::TO_CRAWL. It checks if there are still more urls to crawl. If so, a mini index is built and, the queue server is called with the data.

updateScheduler()  : mixed

Updates the queue_server about sites that have been crawled.

uploadCrawlData()  : mixed

Sends to crawl, robot, and index data to the current queue server.