EpubProcessor
extends TextProcessor
in package
Used to create crawl summary information for XML files (those served as application/epub+zip)
Tags
Table of Contents
- MAX_DOM_LEVEL = 15
- The constant represents the number of child levels at which the data is present in the content.opf file.
- $attributes : string
- The attribute of the tag element in an xml document
- $children : string
- The child tag element of a tag element.
- $content : string
- The content of the tag element or attribute, used to extract the fields like title, creator, language of the document
- $image_types : array<string|int, mixed>
- Array filetypes which should be considered images.
- $indexed_file_types : array<string|int, mixed>
- Array of file extensions which can be handled by the search engine, other extensions will be ignored.
- $max_description_len : int
- Max number of chars to extract for description from a page to index.
- $max_links_to_extract : int
- Maximum number of urls to extract from a single document
- $mime_processor : array<string|int, mixed>
- Associative array of mime_type => (page processor name that can process that type) Sub-classes add to this array with the types they handle
- $name : string
- The name of the tag element in an xml document
- $plugin_instances : array<string|int, mixed>
- indexing_plugins which might be used with the current processor
- $summarizer : object
- Stores the summarizer object used by this instance of page processor to be used in generating a summary
- $summarizer_option : string
- Stores the name of the summarizer used for crawling.
- $text_data : bool
- Whether the current processor is for text data (i.e., text, html, xml, etc) or for some other format (gif, png, etc)
- __construct() : mixed
- Set-ups the any indexing plugins associated with this page processor
- calculateLang() : string
- Tries to determine the language of the document by looking at the $sample_text and $url provided the language
- closeDanglingTags() : mixed
- If an end of file is reached before closed tags are seen, this methods closes these tags in the correct order.
- createThumb() : mixed
- Used to create an thumbnail file to a thumb folder from an epub,html, or text file provided the image magick command convert exists and the calibre command epub-convert exists.
- dom() : object
- Return a document object based on a string containing the contents of a web page
- extractHttpHttpsUrls() : array<string|int, mixed>
- Tries to extract http or https links from a string of text.
- getBetweenTags() : array<string|int, mixed>
- Gets the text between two tags in a document starting at the current position.
- handle() : array<string|int, mixed>
- Method used to handle processing data for a web page. It makes a summary for the page (via the process() function which should be subclassed) as well as runs any plugins that are associated with the processors to create sub-documents
- initializeIndexedFileTypes() : mixed
- Get processors for different file types. constructing them will populate the self::$indexed_file_types, self::$image_types, and self::$mime_processor arrays
- process() : array<string|int, mixed>
- Used to extract the title, description and links from a string consisting of ebook publication data.
- xmlToObject() : array<string|int, mixed>
- Used to extract the DOM tree containing the information about the epub file such as title, author, language, unique identifier of the book from a string consisting of ebook publication content OPF file.
Constants
MAX_DOM_LEVEL
The constant represents the number of child levels at which the data is present in the content.opf file.
public
mixed
MAX_DOM_LEVEL
= 15
Properties
$attributes
The attribute of the tag element in an xml document
public
string
$attributes
attributes
$children
The child tag element of a tag element.
public
string
$children
children
$content
The content of the tag element or attribute, used to extract the fields like title, creator, language of the document
public
string
$content
content
$image_types
Array filetypes which should be considered images.
public
static array<string|int, mixed>
$image_types
= []
Sub-classes add to this array with the types they handle
$indexed_file_types
Array of file extensions which can be handled by the search engine, other extensions will be ignored.
public
static array<string|int, mixed>
$indexed_file_types
= ["unknown"]
Sub-classes add to this array with the types they handle
$max_description_len
Max number of chars to extract for description from a page to index.
public
static int
$max_description_len
Only words in the description are indexed.
$max_links_to_extract
Maximum number of urls to extract from a single document
public
static int
$max_links_to_extract
$mime_processor
Associative array of mime_type => (page processor name that can process that type) Sub-classes add to this array with the types they handle
public
static array<string|int, mixed>
$mime_processor
= []
$name
The name of the tag element in an xml document
public
string
$name
name
$plugin_instances
indexing_plugins which might be used with the current processor
public
array<string|int, mixed>
$plugin_instances
$summarizer
Stores the summarizer object used by this instance of page processor to be used in generating a summary
public
object
$summarizer
$summarizer_option
Stores the name of the summarizer used for crawling.
public
string
$summarizer_option
Possible values are self::BASIC, self::GRAPH_BASED_SUMMARIZER, self::CENTROID_SUMMARIZER and self::CENTROID_WEIGHTED_SUMMARIZER
$text_data
Whether the current processor is for text data (i.e., text, html, xml, etc) or for some other format (gif, png, etc)
public
bool
$text_data
Methods
__construct()
Set-ups the any indexing plugins associated with this page processor
public
__construct([array<string|int, mixed> $plugins = [] ][, int $max_description_len = null ][, int $max_links_to_extract = null ][, string $summarizer_option = self::BASIC_SUMMARIZER ]) : mixed
Parameters
- $plugins : array<string|int, mixed> = []
-
an array of indexing plugins which might do further processing on the data handles by this page processor
- $max_description_len : int = null
-
maximal length of a page summary
- $max_links_to_extract : int = null
-
maximum number of links to extract from a single document
- $summarizer_option : string = self::BASIC_SUMMARIZER
-
CRAWL_CONSTANT specifying what kind of summarizer to use self::BASIC_SUMMARIZER, self::GRAPH_BASED_SUMMARIZER and self::CENTROID_SUMMARIZER self::CENTROID_SUMMARIZER
Return values
mixed —calculateLang()
Tries to determine the language of the document by looking at the $sample_text and $url provided the language
public
static calculateLang([string $sample_text = null ][, string $url = null ]) : string
Parameters
- $sample_text : string = null
-
sample text to try guess the language from
- $url : string = null
-
url of web-page as a fallback look at the country to figure out language
Return values
string —language tag for guessed language
closeDanglingTags()
If an end of file is reached before closed tags are seen, this methods closes these tags in the correct order.
public
static closeDanglingTags(string &$page) : mixed
Parameters
- $page : string
-
a reference to an xml or html document
Return values
mixed —createThumb()
Used to create an thumbnail file to a thumb folder from an epub,html, or text file provided the image magick command convert exists and the calibre command epub-convert exists.
public
static createThumb(string $folder, string $thumb_folder, string $file_name[, int $width = CTHUMB_DIM ][, int $height = CTHUMB_DIM ]) : mixed
For this method to do anything the constant IMAGE_MAGICK must be set to the path to the "convert" command and the constant CALIBRE must bet set to the path of "ebook-convert". This is very brute force and slow as currently implemented as it creates a PDF from file and then extracts the first page to make a thumb
Parameters
- $folder : string
-
with file in it
- $thumb_folder : string
-
folder to generate
- $file_name : string
-
of file file in $folder
- $width : int = CTHUMB_DIM
-
= width in pixels of thumb
- $height : int = CTHUMB_DIM
-
= height in pixels of thumb
Return values
mixed —dom()
Return a document object based on a string containing the contents of a web page
public
static dom(string $page) : object
Parameters
- $page : string
-
a web page
Return values
object —document object
extractHttpHttpsUrls()
Tries to extract http or https links from a string of text.
public
static extractHttpHttpsUrls(string $page) : array<string|int, mixed>
Does this by a very approximate regular expression.
Parameters
- $page : string
-
text string of a document
Return values
array<string|int, mixed> —a set of http or https links that were extracted from the document
getBetweenTags()
Gets the text between two tags in a document starting at the current position.
public
static getBetweenTags(string $string, int $cur_pos, string $start_tag, string $end_tag) : array<string|int, mixed>
Parameters
- $string : string
-
document to extract text from
- $cur_pos : int
-
current location to look if can extract text
- $start_tag : string
-
starting tag that we want to extract after
- $end_tag : string
-
ending tag that we want to extract until
Return values
array<string|int, mixed> —pair consisting of when in the document we are after the end tag, together with the data between the two tags
handle()
Method used to handle processing data for a web page. It makes a summary for the page (via the process() function which should be subclassed) as well as runs any plugins that are associated with the processors to create sub-documents
public
handle(string $page, string $url) : array<string|int, mixed>
Parameters
- $page : string
-
string of a web document
- $url : string
-
location the document came from
Return values
array<string|int, mixed> —a summary of (title, description,links, and content) of the information in $page also has a subdocs array containing any subdocuments returned from a plugin. A subdocuments might be things like recipes that appeared in a page or tweets, etc.
initializeIndexedFileTypes()
Get processors for different file types. constructing them will populate the self::$indexed_file_types, self::$image_types, and self::$mime_processor arrays
public
static initializeIndexedFileTypes() : mixed
Return values
mixed —process()
Used to extract the title, description and links from a string consisting of ebook publication data.
public
process(string $page, string $url) : array<string|int, mixed>
Parameters
- $page : string
-
epub contents
- $url : string
-
the url where the page contents came from, used to canonicalize relative links
Return values
array<string|int, mixed> —a summary of the contents of the page
xmlToObject()
Used to extract the DOM tree containing the information about the epub file such as title, author, language, unique identifier of the book from a string consisting of ebook publication content OPF file.
public
xmlToObject(string $xml) : array<string|int, mixed>
Parameters
- $xml : string
-
page contents
Return values
array<string|int, mixed> —an information about the contents of the page