FetchController
extends Controller
in package
implements
CrawlConstants
This class handles data coming to a queue_server from a fetcher Basically, it receives the data from the fetcher and saves it into various files for later processing by the queue server.
This class can also be used by a fetcher to get status information.
Tags
Interfaces, Classes, Traits and Enums
- CrawlConstants
- Shared constants and enums used by components that are involved in the crawling process
Table of Contents
- CRON_INTERVAL = 300
- Number of seconds that must elapse after last call before doing cron activities (mainly check liveness of fetchers which should be alive)
- $activities : array<string|int, mixed>
- These are the activities supported by this controller
- $activity_component : array<string|int, mixed>
- Associative array of activity => component activity is on, used by @see Controller::call method to actually invoke a given activity on a given component
- $component_activities : array<string|int, mixed>
- Associative array of $components activities for this controller Components are collections of activities (a little like traits) which can be reused.
- $component_instances : array<string|int, mixed>
- Array of instances of components used by this controller
- $crawl_status_file_name : string
- File used to store info about the status of a queue server's active crawl. Default to channel 0 but might change in
- $model_instances : array<string|int, mixed>
- Array of instances of models used by this controller
- $plugin_instances : array<string|int, mixed>
- Array of instances of indexing_plugins used by this controller
- $view_instances : array<string|int, mixed>
- Array of instances of views used by this controller
- $web_site : WebSite
- Stores a reference to the web server when Yioop runs in CLI mode, it acts as request router in non-CLI mode.
- __construct() : mixed
- Sets up component activities, instance array, and plugins.
- addDifferentialPrivacy() : int
- Adds to an integer, $actual_value, epsilon-noise taken from an L_1 gaussian source to centered at $actual_value to get a epsilon private, integer value.
- archiveSchedule() : mixed
- Checks to see whether there are more pages to extract from the current archive, and if so returns the next batch to the requesting fetcher. The iteration progress is automatically saved on each call to nextPages, so that the next fetcher will get the next batch of pages. If there is no current archive to iterate over, or the iterator has reached the end of the archive then indicate that there is no more data by setting the status to NO_DATA_STATE.
- call() : mixed
- Used to invoke an activity method of the current controller or one its components
- checkCSRFTime() : bool
- Checks if the timestamp in $_REQUEST[$token_name] matches the timestamp of the last CSRF token accessed by this user for the kind of activity for which there might be a conflict.
- checkCSRFToken() : bool
- Checks if the form CSRF (cross-site request forgery preventing) token matches the given user and has not expired (1 hour till expires)
- checkRequest() : bool
- Checks the request if a request is for a valid activity and if it uses the correct authorization key
- clean() : string
- Used to clean strings that might be tainted as originate from the user
- component() : mixed
- Dynamic loader for Component objects which might live on the current Component
- convertArrayLines() : string
- Converts an array of lines of strings into a single string with proper newlines, each line having been trimmed and potentially cleaned
- convertStringCleanArray() : array<string|int, mixed>
- Cleans a string consisting of lines, typically of urls into an array of clean lines. This is used in handling data from the crawl options text areas. # is treated as a comment
- crawlTime() : mixed
- Checks for the crawl time according either to crawl_status.txt or to NetworkStatus.txt, and presents it to the requesting fetcher, along with a list of available queue servers.
- displayView() : mixed
- Send the provided view to output, drawing it with the given data variable, using the current locale for translation, and writing mode
- doFetcherCronTasks() : mixed
- Used to do periodic maintenance tasks for the Name Server.
- generateCSRFToken() : string
- Generates a cross site request forgery preventing token based on the provided user name, the current time and the hidden AUTH_KEY
- getAccessModifiers() : array<string|int, mixed>
- Returns an array of the possible modifiers to the access to the activity in question.
- getChannel() : int
- Returns the channel used by the given uploaded data
- getCrawlTimes() : array<string|int, mixed>
- Gets a list of all the timestamps of previously stored crawls
- getCSRFTime() : int
- Used to return just the timestamp portion of the CSRF token
- getIndexingPluginList() : mixed
- Used to get a list of all available indexing plugins for this Yioop instance.
- handleUploadedData() : string
- After robot, schedule, and index data have been uploaded and reassembled as one big data file/string, this function splits that string into each of these data types and then save the result into the appropriate schedule sub-folder of the $messages_bundle. Any temporary files used during uploading are then deleted.
- initializeAdFields() : mixed
- If external source advertisements are present in the output of this controller this function can be used to initialize the field variables used to write the appropriate Javascripts
- model() : mixed
- Dynamic loader for Model objects which might live on the current Controller
- pagingLogic() : mixed
- When an activity involves displaying tabular data (such as rows of users, groups, etc), this method might be called to set up $data fields for next, prev, and page links, it also makes the call to the model to get the row data sorted and restricted as desired. For some data sources, rather than directly make a call to the model to get the data it might be passed directly to this method.
- parsePageHeadVars() : array<string|int, mixed>
- Used to parse head meta variables out of a data string provided either from a wiki page or a static page. Meta data is stored in lines before the first occurrence of END_HEAD_VARS. Head variables are name=value pairs. An example of head variable might be: title = This web page's title Anything after a semi-colon on a line in the head section is treated as a comment
- parsePageHeadVarsView() : mixed
- Used to set up the head variables for and page_data of a wiki or static page associated with a view.
- plugin() : mixed
- Dynamic loader for Plugin objects which might live on the current Controller
- processRequest() : mixed
- Checks that the request seems to be coming from a legitimate fetcher then determines which activity the fetcher is requesting and calls that activity for processing.
- recordViewSession() : mixed
- Used to store in a session which media list items have been viewed so we can put an indicator by them when the media list is rendered
- redirectLocation() : mixed
- Method to perform a 301 redirect to $location in both under web server and CLI setting
- redirectWithMessage() : mixed
- Does a 301 redirect to the given location, sets a session variable to display a message when get there.
- schedule() : mixed
- Checks if there is a schedule of sites to crawl available and if so present it to the requesting fetcher, and then delete it.
- setupGraphicalCaptchaViewData() : mixed
- Sets up the graphical captcha view Draws the string for graphical captcha
- update() : mixed
- Processes Robot, To Crawl, and Index data sent from a fetcher Acknowledge to the fetcher if this data was received okay.
- view() : mixed
- Dynamic loader for View objects which might live on the current Controller
Constants
CRON_INTERVAL
Number of seconds that must elapse after last call before doing cron activities (mainly check liveness of fetchers which should be alive)
public
mixed
CRON_INTERVAL
= 300
Properties
$activities
These are the activities supported by this controller
public
array<string|int, mixed>
$activities
= ["schedule", "archiveSchedule", "update", "crawlTime"]
$activity_component
Associative array of activity => component activity is on, used by @see Controller::call method to actually invoke a given activity on a given component
public
array<string|int, mixed>
$activity_component
= []
$component_activities
Associative array of $components activities for this controller Components are collections of activities (a little like traits) which can be reused.
public
static array<string|int, mixed>
$component_activities
= []
$component_instances
Array of instances of components used by this controller
public
array<string|int, mixed>
$component_instances
$crawl_status_file_name
File used to store info about the status of a queue server's active crawl. Default to channel 0 but might change in
public
string
$crawl_status_file_name
= \seekquarry\yioop\configs\SCHEDULES_DIR . "/0-" . self::crawl_status_file
Tags
$model_instances
Array of instances of models used by this controller
public
array<string|int, mixed>
$model_instances
$plugin_instances
Array of instances of indexing_plugins used by this controller
public
array<string|int, mixed>
$plugin_instances
$view_instances
Array of instances of views used by this controller
public
array<string|int, mixed>
$view_instances
= []
$web_site
Stores a reference to the web server when Yioop runs in CLI mode, it acts as request router in non-CLI mode.
public
WebSite
$web_site
In CLI, mode it is useful for caching files in RAM as they are read
Methods
__construct()
Sets up component activities, instance array, and plugins.
public
__construct([WebSite $web_site = null ]) : mixed
Parameters
- $web_site : WebSite = null
-
is the web server when Yioop runs in CLI mode, it acts as request router in non-CLI mode. In CLI, mode it is useful for caching files in RAM as they are read
Return values
mixed —addDifferentialPrivacy()
Adds to an integer, $actual_value, epsilon-noise taken from an L_1 gaussian source to centered at $actual_value to get a epsilon private, integer value.
public
addDifferentialPrivacy(int $actual_value) : int
Parameters
- $actual_value : int
-
number want to make private
Return values
int —$fuzzy_value number after noise added
archiveSchedule()
Checks to see whether there are more pages to extract from the current archive, and if so returns the next batch to the requesting fetcher. The iteration progress is automatically saved on each call to nextPages, so that the next fetcher will get the next batch of pages. If there is no current archive to iterate over, or the iterator has reached the end of the archive then indicate that there is no more data by setting the status to NO_DATA_STATE.
public
archiveSchedule() : mixed
Return values
mixed —call()
Used to invoke an activity method of the current controller or one its components
public
call(string $activity[, string $modifiers = [] ]) : mixed
Parameters
- $activity : string
-
method to invoke
- $modifiers : string = []
-
access modifiers to executing this method
Return values
mixed —checkCSRFTime()
Checks if the timestamp in $_REQUEST[$token_name] matches the timestamp of the last CSRF token accessed by this user for the kind of activity for which there might be a conflict.
public
checkCSRFTime(string $token_name[, string $action = "" ]) : bool
This is to avoid accidental replays of postings etc if the back button used.
Parameters
- $token_name : string
-
name of a $_REQUEST field used to hold a CSRF_TOKEN
- $action : string = ""
-
name of current action to check for conflicts
Return values
bool —whether a conflicting action has occurred.
checkCSRFToken()
Checks if the form CSRF (cross-site request forgery preventing) token matches the given user and has not expired (1 hour till expires)
public
checkCSRFToken(string $token_name, string $user_id[, bool $use_name_as_passed = false ]) : bool
Parameters
- $token_name : string
-
attribute of $_REQUEST containing CSRFToken
- $user_id : string
-
user id of the user to check the token for
- $use_name_as_passed : bool = false
-
whether to use $token_name as the token (if true) or to use $_REQUEST[$token_name]
Return values
bool —whether the CSRF token was valid
checkRequest()
Checks the request if a request is for a valid activity and if it uses the correct authorization key
public
checkRequest() : bool
Return values
bool —whether the request was valid or not
clean()
Used to clean strings that might be tainted as originate from the user
public
clean(mixed $value, mixed $type[, mixed $default = null ]) : string
Parameters
- $value : mixed
-
tainted data
- $type : mixed
-
type of data in value can be one of the following strings: bool, color, double, float, int, hash, or string, web-url; or it can be an array listing allowed values. If the latter, then if the value is not in the array the cleaned value will be first element of the array if $default is null
- $default : mixed = null
-
if $value is not set default value is returned, this isn't used much since if the error_reporting is E_ALL or -1 you would still get a Notice.
Return values
string —the clean input matching the type provided
component()
Dynamic loader for Component objects which might live on the current Component
public
component(string $component) : mixed
Parameters
- $component : string
-
name of model to return
Return values
mixed —convertArrayLines()
Converts an array of lines of strings into a single string with proper newlines, each line having been trimmed and potentially cleaned
public
convertArrayLines(array<string|int, mixed> $arr[, string $endline_string = "
" ][, bool $clean = false ]) : string
Parameters
- $arr : array<string|int, mixed>
-
the array of lines to be process
- $endline_string : string = " "
-
what string should be used to indicate the end of a line
- $clean : bool = false
-
whether to clean each line
Return values
string —a concatenated string of cleaned lines
convertStringCleanArray()
Cleans a string consisting of lines, typically of urls into an array of clean lines. This is used in handling data from the crawl options text areas. # is treated as a comment
public
convertStringCleanArray(string $str[, string $line_type = "url" ]) : array<string|int, mixed>
Parameters
- $str : string
-
contains the url data
- $line_type : string = "url"
-
does additional cleaning depending on the type of the lines. For instance, if is "url" then a line not beginning with a url scheme will have http:// prepended.
Return values
array<string|int, mixed> —$lines an array of clean lines
crawlTime()
Checks for the crawl time according either to crawl_status.txt or to NetworkStatus.txt, and presents it to the requesting fetcher, along with a list of available queue servers.
public
crawlTime() : mixed
Return values
mixed —displayView()
Send the provided view to output, drawing it with the given data variable, using the current locale for translation, and writing mode
public
displayView(string $view, array<string|int, mixed> $data) : mixed
Parameters
- $view : string
-
the name of the view to draw
- $data : array<string|int, mixed>
-
an array of values to use in drawing the view
Return values
mixed —doFetcherCronTasks()
Used to do periodic maintenance tasks for the Name Server.
public
doFetcherCronTasks() : mixed
For now, just checks if any fetchers which the user turned on have crashed and if so restarts them
Return values
mixed —generateCSRFToken()
Generates a cross site request forgery preventing token based on the provided user name, the current time and the hidden AUTH_KEY
public
generateCSRFToken(string $user) : string
Parameters
- $user : string
-
username to use to generate token
Return values
string —a csrf token
getAccessModifiers()
Returns an array of the possible modifiers to the access to the activity in question.
public
getAccessModifiers(string $activity) : array<string|int, mixed>
Parameters
- $activity : string
-
method to get access modifier list for
Return values
array<string|int, mixed> —of string names => translated names of the access modifiers for the method in question (if any exist).
getChannel()
Returns the channel used by the given uploaded data
public
getChannel() : int
Return values
int —channel used
getCrawlTimes()
Gets a list of all the timestamps of previously stored crawls
public
getCrawlTimes() : array<string|int, mixed>
This could probably be moved to crawl model. It is a little lighter than getCrawlList and should be only used with a name server so leaving it here so it won't be confused.
Return values
array<string|int, mixed> —list of timestamps
getCSRFTime()
Used to return just the timestamp portion of the CSRF token
public
getCSRFTime(string $token_name) : int
Parameters
- $token_name : string
-
name of a $_REQUEST field used to hold a CSRF_TOKEN
Return values
int —the timestamp portion of the CSRF_TOKEN
getIndexingPluginList()
Used to get a list of all available indexing plugins for this Yioop instance.
public
getIndexingPluginList() : mixed
Return values
mixed —handleUploadedData()
After robot, schedule, and index data have been uploaded and reassembled as one big data file/string, this function splits that string into each of these data types and then save the result into the appropriate schedule sub-folder of the $messages_bundle. Any temporary files used during uploading are then deleted.
public
handleUploadedData(mixed $byte_counts, mixed &$uploaded, MessagesBundle $messages_bundle) : string
Parameters
- $byte_counts : mixed
- $uploaded : mixed
- $messages_bundle : MessagesBundle
-
used to store the messages contained in the file $filename so that they can be later processed
Return values
string —$logging diagnostic info to be sent to fetcher about what was done
initializeAdFields()
If external source advertisements are present in the output of this controller this function can be used to initialize the field variables used to write the appropriate Javascripts
public
initializeAdFields(array<string|int, mixed> &$data[, bool $ads_off = false ]) : mixed
Parameters
- $data : array<string|int, mixed>
-
data to be used in drawing the view
- $ads_off : bool = false
-
whether or not ads are turned off so that this method should do nothing
Return values
mixed —model()
Dynamic loader for Model objects which might live on the current Controller
public
model(string $model) : mixed
Parameters
- $model : string
-
name of model to return
Return values
mixed —pagingLogic()
When an activity involves displaying tabular data (such as rows of users, groups, etc), this method might be called to set up $data fields for next, prev, and page links, it also makes the call to the model to get the row data sorted and restricted as desired. For some data sources, rather than directly make a call to the model to get the data it might be passed directly to this method.
public
pagingLogic(array<string|int, mixed> &$data, mixed $field_or_model, string $output_field, int $default_show[, array<string|int, mixed> $search_array = [] ][, string $var_prefix = "" ][, array<string|int, mixed> $args = null ]) : mixed
Parameters
- $data : array<string|int, mixed>
-
used to send data to the view will be updated by this method with row and paging data
- $field_or_model : mixed
-
if an object, this is assumed to be a model and so the getRows method of this model is called to get row data, sorted and restricted according to $search_array; if a string then the row data is assumed to be in $data[$field_or_model] and pagingLogic itself does the sorting and restricting.
- $output_field : string
-
output rows for the view will be stored in $data[$output_field]
- $default_show : int
-
if not specified by $_REQUEST, then this will be used to determine the maximum number of rows that will be written to $data[$output_field]
- $search_array : array<string|int, mixed> = []
-
used to sort and restrict in the getRows call or the data from $data[$field_or_model]. Each element of this is a quadruple name of a field, what comparison to perform, a value to check, and an order (ascending/descending) to sort by
- $var_prefix : string = ""
-
if there are multiple uses of pagingLogic presented on the same view then $var_prefix can be prepended to to the $data field variables like num_show, start_row, end_row to distinguish between them
- $args : array<string|int, mixed> = null
-
additional arguments that are passed to getRows and in turn to selectCallback, fromCallback, and whereCallback that might provide user_id, etc to further control which rows are returned
Return values
mixed —parsePageHeadVars()
Used to parse head meta variables out of a data string provided either from a wiki page or a static page. Meta data is stored in lines before the first occurrence of END_HEAD_VARS. Head variables are name=value pairs. An example of head variable might be: title = This web page's title Anything after a semi-colon on a line in the head section is treated as a comment
public
parsePageHeadVars(string $page_data[, mixed $with_body = false ]) : array<string|int, mixed>
Parameters
- $page_data : string
-
this is the actual content of a wiki or static page
- $with_body : mixed = false
Return values
array<string|int, mixed> —the associative array of head variables or pair [head vars, page body]
parsePageHeadVarsView()
Used to set up the head variables for and page_data of a wiki or static page associated with a view.
public
parsePageHeadVarsView(object $view, string $page_name, string $page_data) : mixed
Parameters
- $view : object
-
View on which page data will be rendered
- $page_name : string
-
a string name/id to associate with page. For example, might have 404 for a page about 404 errors
- $page_data : string
-
this is the actual content of a wiki or static page
Return values
mixed —plugin()
Dynamic loader for Plugin objects which might live on the current Controller
public
plugin(string $plugin) : mixed
Parameters
- $plugin : string
-
name of Plugin to return
Return values
mixed —processRequest()
Checks that the request seems to be coming from a legitimate fetcher then determines which activity the fetcher is requesting and calls that activity for processing.
public
processRequest() : mixed
Return values
mixed —recordViewSession()
Used to store in a session which media list items have been viewed so we can put an indicator by them when the media list is rendered
public
recordViewSession(int $page_id, string $sub_path, string $media_name) : mixed
Parameters
- $page_id : int
-
the id of page with media list
- $sub_path : string
-
the resource folder on that page
- $media_name : string
-
item to store indiicator into session for
Return values
mixed —redirectLocation()
Method to perform a 301 redirect to $location in both under web server and CLI setting
public
redirectLocation(string $location) : mixed
Parameters
- $location : string
-
url to redirect to
Return values
mixed —redirectWithMessage()
Does a 301 redirect to the given location, sets a session variable to display a message when get there.
public
redirectWithMessage(string $message[, string $copy_fields = false ][, bool $restart = false ][, bool $use_base_url = false ]) : mixed
Parameters
- $message : string
-
message to write
- $copy_fields : string = false
-
$_REQUEST fields to copy for redirect
- $restart : bool = false
-
if yioop is being run as its own server rather than under apache whether to restart this server.
- $use_base_url : bool = false
-
set true if the base_url be included in the redirect
Return values
mixed —schedule()
Checks if there is a schedule of sites to crawl available and if so present it to the requesting fetcher, and then delete it.
public
schedule() : mixed
Return values
mixed —setupGraphicalCaptchaViewData()
Sets up the graphical captcha view Draws the string for graphical captcha
public
setupGraphicalCaptchaViewData(array<string|int, mixed> &$data) : mixed
Parameters
- $data : array<string|int, mixed>
-
used by view to draw any dynamic content in this case we append a field "CAPTCHA_IMAGE" with a data url of the captcha to draw.
Return values
mixed —update()
Processes Robot, To Crawl, and Index data sent from a fetcher Acknowledge to the fetcher if this data was received okay.
public
update() : mixed
Return values
mixed —view()
Dynamic loader for View objects which might live on the current Controller
public
view(string $view) : mixed
Parameters
- $view : string
-
name of view to return