Yioop_V9.5_Source_Code_Documentation

XlsxProcessor extends TextProcessor
in package

Used to create crawl summary information for xlsx files

Tags
author

Tarun Ramaswamy

Table of Contents

$image_types  : array<string|int, mixed>
Array filetypes which should be considered images.
$indexed_file_types  : array<string|int, mixed>
Array of file extensions which can be handled by the search engine, other extensions will be ignored.
$max_description_len  : int
Max number of chars to extract for description from a page to index.
$max_links_to_extract  : int
Maximum number of urls to extract from a single document
$mime_processor  : array<string|int, mixed>
Associative array of mime_type => (page processor name that can process that type) Sub-classes add to this array with the types they handle
$plugin_instances  : array<string|int, mixed>
indexing_plugins which might be used with the current processor
$summarizer  : object
Stores the summarizer object used by this instance of page processor to be used in generating a summary
$summarizer_option  : string
Stores the name of the summarizer used for crawling.
$text_data  : bool
Whether the current processor is for text data (i.e., text, html, xml, etc) or for some other format (gif, png, etc)
__construct()  : mixed
Set-ups the any indexing plugins associated with this page processor
calculateLang()  : string
Tries to determine the language of the document by looking at the $sample_text and $url provided the language
closeDanglingTags()  : mixed
If an end of file is reached before closed tags are seen, this methods closes these tags in the correct order.
createThumb()  : mixed
Used to create an thumbnail file to a thumb folder from an epub,html, or text file provided the image magick command convert exists and the calibre command epub-convert exists.
description()  : string
Returns descriptive text concerning a xlsx file based on its document object
dom()  : object
Return a document object based on a string containing the contents of a xml file
extractHttpHttpsUrls()  : array<string|int, mixed>
Tries to extract http or https links from a string of text.
getBetweenTags()  : array<string|int, mixed>
Gets the text between two tags in a document starting at the current position.
handle()  : array<string|int, mixed>
Method used to handle processing data for a web page. It makes a summary for the page (via the process() function which should be subclassed) as well as runs any plugins that are associated with the processors to create sub-documents
initializeIndexedFileTypes()  : mixed
Get processors for different file types. constructing them will populate the self::$indexed_file_types, self::$image_types, and self::$mime_processor arrays
links()  : array<string|int, mixed>
Returns up to MAX_LINK_PER_PAGE many links from the supplied dom object where links have been canonicalized according to the supplied $site information.
process()  : array<string|int, mixed>
Used to extract the title, description and links from a xlsx file.
sheetCount()  : int
Returns the count of worksheets in the xlsx file
title()  : string
Returns title of a xlsx file from each worksheet

Properties

$image_types

Array filetypes which should be considered images.

public static array<string|int, mixed> $image_types = []

Sub-classes add to this array with the types they handle

$indexed_file_types

Array of file extensions which can be handled by the search engine, other extensions will be ignored.

public static array<string|int, mixed> $indexed_file_types = ["unknown"]

Sub-classes add to this array with the types they handle

$max_description_len

Max number of chars to extract for description from a page to index.

public static int $max_description_len

Only words in the description are indexed.

$max_links_to_extract

Maximum number of urls to extract from a single document

public static int $max_links_to_extract

$mime_processor

Associative array of mime_type => (page processor name that can process that type) Sub-classes add to this array with the types they handle

public static array<string|int, mixed> $mime_processor = []

$plugin_instances

indexing_plugins which might be used with the current processor

public array<string|int, mixed> $plugin_instances

$summarizer

Stores the summarizer object used by this instance of page processor to be used in generating a summary

public object $summarizer

$summarizer_option

Stores the name of the summarizer used for crawling.

public string $summarizer_option

Possible values are self::BASIC, self::GRAPH_BASED_SUMMARIZER, self::CENTROID_SUMMARIZER and self::CENTROID_WEIGHTED_SUMMARIZER

$text_data

Whether the current processor is for text data (i.e., text, html, xml, etc) or for some other format (gif, png, etc)

public bool $text_data

Methods

__construct()

Set-ups the any indexing plugins associated with this page processor

public __construct([array<string|int, mixed> $plugins = [] ][, int $max_description_len = null ][, int $max_links_to_extract = null ][, string $summarizer_option = self::BASIC_SUMMARIZER ]) : mixed
Parameters
$plugins : array<string|int, mixed> = []

an array of indexing plugins which might do further processing on the data handles by this page processor

$max_description_len : int = null

maximal length of a page summary

$max_links_to_extract : int = null

maximum number of links to extract from a single document

$summarizer_option : string = self::BASIC_SUMMARIZER

CRAWL_CONSTANT specifying what kind of summarizer to use self::BASIC_SUMMARIZER, self::GRAPH_BASED_SUMMARIZER and self::CENTROID_SUMMARIZER self::CENTROID_SUMMARIZER

Return values
mixed —

calculateLang()

Tries to determine the language of the document by looking at the $sample_text and $url provided the language

public static calculateLang([string $sample_text = null ][, string $url = null ]) : string
Parameters
$sample_text : string = null

sample text to try guess the language from

$url : string = null

url of web-page as a fallback look at the country to figure out language

Return values
string —

language tag for guessed language

closeDanglingTags()

If an end of file is reached before closed tags are seen, this methods closes these tags in the correct order.

public static closeDanglingTags(string &$page) : mixed
Parameters
$page : string

a reference to an xml or html document

Return values
mixed —

createThumb()

Used to create an thumbnail file to a thumb folder from an epub,html, or text file provided the image magick command convert exists and the calibre command epub-convert exists.

public static createThumb(string $folder, string $thumb_folder, string $file_name[, int $width = CTHUMB_DIM ][, int $height = CTHUMB_DIM ]) : mixed

For this method to do anything the constant IMAGE_MAGICK must be set to the path to the "convert" command and the constant CALIBRE must bet set to the path of "ebook-convert". This is very brute force and slow as currently implemented as it creates a PDF from file and then extracts the first page to make a thumb

Parameters
$folder : string

with file in it

$thumb_folder : string

folder to generate

$file_name : string

of file file in $folder

$width : int = CTHUMB_DIM

= width in pixels of thumb

$height : int = CTHUMB_DIM

= height in pixels of thumb

Return values
mixed —

description()

Returns descriptive text concerning a xlsx file based on its document object

public static description(object $dom) : string
Parameters
$dom : object

a document object to extract a description from.

Return values
string —

a description of the slide

dom()

Return a document object based on a string containing the contents of a xml file

public static dom(string $page) : object
Parameters
$page : string

xml document

Return values
object —

document object

extractHttpHttpsUrls()

Tries to extract http or https links from a string of text.

public static extractHttpHttpsUrls(string $page) : array<string|int, mixed>

Does this by a very approximate regular expression.

Parameters
$page : string

text string of a document

Return values
array<string|int, mixed> —

a set of http or https links that were extracted from the document

getBetweenTags()

Gets the text between two tags in a document starting at the current position.

public static getBetweenTags(string $string, int $cur_pos, string $start_tag, string $end_tag) : array<string|int, mixed>
Parameters
$string : string

document to extract text from

$cur_pos : int

current location to look if can extract text

$start_tag : string

starting tag that we want to extract after

$end_tag : string

ending tag that we want to extract until

Return values
array<string|int, mixed> —

pair consisting of when in the document we are after the end tag, together with the data between the two tags

handle()

Method used to handle processing data for a web page. It makes a summary for the page (via the process() function which should be subclassed) as well as runs any plugins that are associated with the processors to create sub-documents

public handle(string $page, string $url) : array<string|int, mixed>
Parameters
$page : string

string of a web document

$url : string

location the document came from

Return values
array<string|int, mixed> —

a summary of (title, description,links, and content) of the information in $page also has a subdocs array containing any subdocuments returned from a plugin. A subdocuments might be things like recipes that appeared in a page or tweets, etc.

initializeIndexedFileTypes()

Get processors for different file types. constructing them will populate the self::$indexed_file_types, self::$image_types, and self::$mime_processor arrays

public static initializeIndexedFileTypes() : mixed
Return values
mixed —

links()

Returns up to MAX_LINK_PER_PAGE many links from the supplied dom object where links have been canonicalized according to the supplied $site information.

public static links(object $dom, string $site) : array<string|int, mixed>
Parameters
$dom : object

a document object with links on it

$site : string

a string containing a url

Return values
array<string|int, mixed> —

links from the $dom object

process()

Used to extract the title, description and links from a xlsx file.

public process(string $page, string $url) : array<string|int, mixed>
Parameters
$page : string

contents of xlsx file in zip format

$url : string

the url where the page contents came from, used to canonicalize relative links

Return values
array<string|int, mixed> —

a summary of the contents of the page

sheetCount()

Returns the count of worksheets in the xlsx file

public static sheetCount(object $dom) : int
Parameters
$dom : object

a document object to extract a title from.

Return values
int —

number of worksheets in the xlsx file

title()

Returns title of a xlsx file from each worksheet

public static title(object $dom) : string
Parameters
$dom : object

a document object to extract a title from.

Return values
string —

a title of the xlsx file


        

        
    




    

    

                

    

        

            
Search results