Changelog

Changes in Version 10.0

(2026)

• Crawler and Search Engine
  • Crawls are safe against crashes. Each write to the index goes through an undo log that is synced to disk, so after a power cut, or a crawl stopped part way through a write, the index rolls back to its last checkpoint. Before, an index could be left with a partition holding 1 of its 99 documents.
  • Crawls run faster: on the test site a crawl went from about 60,000 to 90,000 pages an hour. Each batch of urls goes to a single fetcher, host names rather than urls are counted when looking up addresses, and a kind of document that takes 100 times longer than a web page to process is queued behind the others.
  • A long PDF is read by OCR (optical character recognition) only until there is enough text for its summary, and pictures with nothing like print in them are skipped.
  • The crawler keeps cookies between requests to a site, as a browser does, which some sites need before they show a page. It drops tracking parameters from links, so the same page is not fetched again under another address.
  • A new crawl starts from an updated list of seed sites for 2026.
  • The first pages of a crawl can be searched within about a minute of its start. A fetcher sends its results after its first 100 downloads, then after 200, 400 and so on up to 1,000.
  • More of each page's text is kept: HTML entities, structured data, the text inside PDF streams, and text set in PDF subset fonts.
  • A page's declared language now wins when Yioop's language guess is a tie, so pages are less often filed under the wrong language.
  • "Are you a robot?" check pages are dropped by the word filter's factory settings rather than indexed as content, and six more kinds of site have built-in scrapers.
  • Searching for a site's name finds the site first. A name typed as separate words, such as tik tok, also looks for the joined host name, tiktok.com, and a company's main site ranks above its subdomains, so www.yahoo.com comes before sports.yahoo.com.
  • When several copies of a page have the same content, the copy with the shortest url keeps the page's words, so results show the clean address.
  • Searches for rare words take about a third of the time: 0.027 seconds against 0.075 on a 75-partition test crawl.
  • A search for several words no longer misses pages stored after the first partition of a large index.
  • A running site sees an index changed by another process within a second, and a new default index or debug setting applies without restarting the web server.
  • A search that runs over five seconds is ended by the web server rather than holding a worker, and the crawl keeps a worker of its own, so a crawl and heavy search traffic do not stall each other.
  • A crawl keeps fetching while the dictionary of words is being merged. Before, while one fetcher merged, an hour on a crawl of about 1,550 partitions, the other fetchers got no work.
  • Clicking Stop shows "Shutting Down Queue" at once, then each stopping step, with a count of the partitions still to be merged.
  • Scrapers that send each request from a new IP address are slowed down by counting requests for each page asked for, rather than for each address.
  • A new synthetic crawl option builds a made-up web of a chosen size and shape, so an administrator can test a Yioop installation without crawling real sites.
  • News thumbnails are cropped to the middle of the picture to fill their box, instead of being stretched.
  • Unchecking Cache in Page Options now stops cached copies of pages from being served; before, it only hid the link to them.

• Indexing and Library Functionality
  • An index can be rebuilt from its saved pages by a pool of builder processes and one merger that adds the partitions in order. Before, a rebuild of a 336-partition crawl could leave queries answering nothing.
  • Yioop records and plays back voice messages, and makes video thumbnails and running times, without needing ffmpeg installed. It reads Vorbis, Opus and FLAC audio in pure PHP.
  • Where ffmpeg or tesseract is installed, Yioop finds and uses it with no setting to fill in.
  • PDF thumbnails look like the PDF's first page. Six faults that scrambled or blanked magazine covers were fixed, and JPEG 2000 photos, pictures in CMYK (print) colors, soft masks and turned text are drawn.
  • A scanned PDF page of 33 million pixels decodes in under 5 seconds and 100 megabytes, against 60 seconds and over a gigabyte before.
  • The wiki's page reader was rewritten to read text with a scanner and recursive descent rather than regular expressions, so markup crafted to inject scripts or to tie up the server with slow patterns is drawn safely.
  • Markdown pages, such as a git repository's README, support more of GitHub's Markdown, including its tables of contents.
  • Yioop runs under Apache with PHP-FPM as well as under its own web server: a request no longer rewrites the settings file, and only one request at a time runs an upgrade. It needs PHP 8.1 or later and runs without warnings on PHP 8.5.
  • A site's Public and Help wiki pages, with their dates, comments and edit history, can be exported to a file and imported into another Yioop site, with git issue pages, comments or history left out if wanted.
  • Any group's wiki pages and discussions can be exported and imported the same way from the command line with GroupWikiTool, and importing a file twice does not repeat its discussions.
  • A new Yioop install can be set up entirely from the command line: ConfigureTool sets the work directory, root password, locale, debug and search settings one command at a time.

• Admin, Group, Wiki, and Yioop Interface
  • New git repository wiki page type: a wiki page can hold a git repository that people clone, pull and push to with ordinary git over HTTP. Yioop reads the repository with its own PHP code and never runs git. Anyone who can read the group can clone, and anyone who can edit can push. The page shows a file browser, branches, a rendered README and commit statistics, and its discussion is an issue tracker with waiting reports and bans. seekquarry.com's own code now lives on such a page in place of viewgit and MantisBT, and GroupWikiTool imports MantisBT issues.
  • The wiki editor shows a live preview next to the text being edited. It is drawn in the browser as you type, with no request to the server, and picking words in either pane marks them in the other. Toolbar buttons insert wiki markup at the cursor, and the undo key takes an insertion back.
  • A wiki page's attached files are shown as a grid of cards that reflows to the window's width. Files can be uploaded, renamed and moved there, several can be picked at once and acted on together, and nothing is sent to the server until an action is chosen.
  • Wiki pages can be filed under categories with {{category|...}}, and {{category-list|...}} collects what is filed under a name. A front page can be laid out by picking one of four shapes, a lead story with headlines, a row of cards, a grid of four or stacked sections, and naming what fills each place, so a group can run as a news site.
  • Secret ballots: each witness takes part by following a signed link mailed to them, rather than all witnesses signing in on one form at one moment. The server still never sees a witness password.
  • Yioop's pages are laid out by the width of the screen they are shown on rather than by guessing the device from its browser name: the viewport tag is always sent and 108 places in the code that served different markup to phones were replaced by CSS media queries.
  • A site can follow the visitor's dark mode setting with a checkbox in Appearance, and right-to-left languages are laid out with CSS logical properties instead of a second copy of each rule.
  • Sizes are in rem units, so the layout grows with the reader's chosen text size. Animation stops for readers who ask for reduced motion, keyboard focus is shown on the controls, and text meets WCAG 2.2 AA (Web Content Accessibility Guidelines) contrast in both light and dark.
  • One Yioop installation can serve several web sites. Each domain can have its own colors, logos, favicon, background image, theme, landing page and search bar, set from a domain picker in Appearance.
  • Each domain has its own robots.txt, built from choices in the Web Server settings such as a crawl delay and which of search, feeds and wiki pages may be crawled, and its own humans.txt. A group owner can keep the public and crawlers out of the group's wiki page source, edit history and page list.
  • Server settings are split onto a Servers screen and a Users, Roles and Groups screen. A setting saved there takes effect at once, without restarting the server.
  • Each role can be given limits on how many groups its members may own and how many members, wiki pages, files and posts those groups may hold; a member with several roles gets the largest limit. Moderation of flagged posts can be turned on or off, and a new Developer role gives access to feeds and wikis only.
  • The web server's status and log are shown in the Web Server settings. From a shell, php index.php restart restarts the server and names the services it started again, and php index.php status reports on the web server and the background programs.
  • Signing up no longer shows a picture captcha, which machines solve and blind users cannot. Instead the browser does a fraction of a second of computing that costs a bot that much on every attempt, invisible checks catch simple bots, and the email address is confirmed.
  • A member who forgets their password can sign in with a short, single-use code mailed to their address, which replaces the old security questions. An offline install that keeps a question lets each member type their own.
  • An administrator chooses Locally Stored Passwords or LDAP (Lightweight Directory Access Protocol) sign-in in a new Authentication field set, sets password rules (minimum length, and whether lower case, upper case, digits and symbols are required), and fills in the LDAP settings there. The site keeps using local passwords until LDAP can sign in the root account, so a wrong directory setting cannot lock the administrator out.

• Mail, Messaging, and Calling
  • Yioop includes its own mail server, MailSite, for SMTP (sending) and IMAP (reading) mail for the site's domains. It is started, stopped and its log read from the Manage Machines screen, and the host name it gives in its greeting is set by the Banner field.
  • Each user can read and send email inside Yioop: folders, a message list, reading, writing and replying, flags, and moving and deleting messages. The same screen reads outside IMAP accounts such as Gmail, with their passwords stored encrypted, and moves a message from one account to another, or copies a whole outside account into MailSite.
  • MailSite sends mail itself, looking up each domain's mail server and trying port 25, then 587, with encryption when the other server offers it. Mail that cannot be delivered right away is tried again for days while the other recipients still get it, and mail to a domain with no MX (mail exchanger) record is tried at its Microsoft 365 or Google Workspace server.
  • Mailing-list email is accepted by Gmail and Yahoo: it carries the headers they require, a one-click unsubscribe link (RFC 8058, the one-click unsubscribe standard), and a DKIM (DomainKeys Identified Mail) signature.
  • MailTool imports existing mailboxes in Maildir or mbox form from another mail server into MailSite, can resume an import that stopped, and checks the result; pollett.org's mail was moved this way.
  • Chat messages arrive as soon as they are sent, over a WebSocket or the page's own HTTP/2 connection, instead of the page asking the server every few seconds. Each update is about 700 bytes rather than 600 kilobytes.
  • The chat box is one rounded field holding an emoji button, a text box that grows as you type, and a microphone that turns into a send button once there is text; Enter sends and Shift+Enter starts a new line. Your own messages are colored and the other person's are neutral, and the per-conversation translation control is gone.
  • Two users can make audio or video calls from the messages screen. Yioop includes its own relay, TurnSite, which lets two browsers behind home routers reach each other; turning it on in Manage Machines is all a site owner does. Contacts who are signed in have a green border, and an incoming call pulses with buttons to take it or turn it down.
  • Voice messages can be recorded and played back even on a site without ffmpeg, as Yioop decodes the browser's recording format itself.

• Web Server and Security
  • Yioop's own web server no longer freezes for everyone while one connection is slow: accepting a connection, its TLS (Transport Layer Security) handshake and reading its request no longer wait in turn, and a request waiting on a mail server, a disk or a network lookup runs in a PHP fiber so the server takes other requests meanwhile.
  • HTTPS certificates from Let's Encrypt renew themselves when the certificate authority's suggested renewal window opens, or a month before they expire, and the new certificate is used without a restart.
  • Large files over HTTP/2 no longer stall in Firefox: they are sent in pieces no larger than the browser said it would take. During a 200 megabyte download another page now answers in under 5 milliseconds, against 20 seconds before, and a 300 megabyte video no longer stops the web server.
  • The web server can run several worker processes, set by Web Server Workers in the Web Server settings, to handle more visitors at once. A stopped worker is started again by the next request.
  • A web server that gave up root after taking ports 80 and 443 can still start the mail server and restart itself, since the process that started it stays as root for that one job. Restart Server no longer leaves the site down when the old server is slow to let go of its ports.
  • A newly installed copy can be started on port 443 alone with a self-made certificate, and set up from the browser or from the command line.
  • A security review moved security tokens, such as the form token, to keyed SHA-256 hashes compared in constant time, removed old encryption fallbacks, and set the session cookie to HttpOnly, SameSite and, over HTTPS, Secure. Uploads can no longer name a folder outside their page with "..".
  • A bot that keeps asking for a wiki page's history or source, which it may not see, is refused before the server does any work for it.

• Localization
  • The interface strings and the public and help pages were translated again from the English in 24 languages, replacing earlier machine output: Arabic, Bengali, Chinese, Dutch, Farsi, French, German, Greek, Hebrew, Hindi, Indonesian, Italian, Japanese, Kannada, Korean, Polish, Portuguese, Russian, Spanish, Tagalog, Telugu, Thai, Turkish and Vietnamese.
  • The French and Spanish stemmers, which reduce a word to its stem so a search matches its other forms, now give the same stems as the standard Snowball stemmers on all 20,403 French and 28,390 Spanish test words.
  • A site's own changes to translations are kept in its work directory, apart from the translations Yioop ships, and take their place; Yioop never writes into its shipped locale folder.

• Monetization
  • A role can be put on sale with a price in credits and a billing period of once, monthly, every six months or yearly. Users buy roles from their account page and can cancel a subscription there, and a role whose period has lapsed stops giving its access.

Changes in Version 9.4

(Aug 30, 2023)

• Crawler and Search Engine
  • Sliding window used to handle messages between Fetcher and QueueServer.
  • Fixes and Expect header bug in downloading webpages
• Admin, Group, Wiki, and Yioop Interface
  • Add a new moderation group and the ability to flag posts
  • Adds a mechanism for the owner of a group to move posts between threads
  • Improves UI for change passwords, password recovery
  • Fixes a bug in how the autologout mechanisms was implemented
  • Fixes PHP 8.2 deprecations

Changes in Version 9.3

(Dec 21, 2022)

• Crawler and Search Engine
  • Improve efficiency of query impression collection.

• Admin, Group, Wiki, and Yioop Interface
  • Add new description source to search sources allowing one to use wiki folder or resource names to be used to download from the internet a description of the resource. Associated to this resource is a new DescriptionUpdateJob used by the media updater.
  • Fixes a bug where public wiki pages could be reverted to earlier versions of the page without authorization. (Security)
  • Improves recommendation job by allowing recommendations of wiki resources. It also switches the earlier pure TF-IDF approach to recommendations to a Hash2Vec approach.
  • Fixes to filenames produced by PodcastDownloaderJob and fix notices in TrendingHighlightJob.
  • Add differential privacy so as to allow for the masking of the exact number of users of a group.
  • Fixes issues where WeatherBot had ceased to work.
  • Try to speed up queries on count of groups, messages in groups, etc.

Changes in Version 9.2

(Sep 17, 2022)

• Crawler and Search Engine
  • Fixes bug that prevented MediaUpdater from working correctly.
• Admin, Group, Wiki, and Yioop Interface
  • In appearance activity changed auxiliary.css textarea into a more general UI for creating and editing CSS themes. These themes can also be selected on a page-by-page basis for wiki pages.
  • Add the ability to create wiki pages with forms elements on them. Form data is written to a form_data.csv file in page's resources.
  • Adds badges for crawl control and social control buttons. Crawl badges indicate total number of ongoing versus total number of crawls, number of machines, number of crawl mixes. Social badges indicate number of unread messages, number of unread posts to threads, and total number of groups a user belongs to.
  • Update Wiki Syntax page.

Changes in Version 9.1

(August 8, 2022)

• Crawler and Search Engine
  • Improves Results editor by allowing the Admin role to directly edit results off a search engine result page., rather than go through the menu system.
  • Fixes bugs related to crawling on Windows
  • Improvements to the clarity of the search engine result scoring mechanism, adds additional scoring conrols under Page Options
• Indexing and Library Functionality
  • Improvements to the layout of search indexes which should hopefully make indexing faster.

Changes in Version 9.0

(July 17, 2022)

• Crawler and Search Engine
  • Yioop's queueing system changed from being OPIC-based to a logarithmic tiers, company-level-domain, host budgeting mechanism.
  • Search Engine result look-and-feel modernized, favicons for sites, if present, downloaded and appear next to results.
  • Hides additional facts about a result accessible by a ... icon.
  • Image Results page layout updated, results show dimensions of base image. Now possible to search filtering by image color.
  • Video Results page layout updates, videos indicate duration, and video resolution. Possible to search filtering by video resolution.
  • Add links to make it easier for a site admin to edit knowledge wiki callouts on result pages.
  • News result layout updated
  • Code modernization to eliminate PHP 8.1 Notices.

• Indexing and Library Functionality
  • Now includes ngram entity detection filters for all locales.
  • Tokentool now updated to support Wikimedia's newer data dump formats for page counts.
  • Greek locale support added including a Greek Stemmer.
  • Enhances WordFilterPlugin by adding support for terms of form FILTER_TERM_ and FILTER_LIST_ coming from a Web Scraper.
  • Updates Arctool to support new indexing methods introduced in Version 8.
  • Work to reduce memory leaks in code, separated out dictionary operations into own process.
  • Optimizations for posting unpacking.
  • Profiling tooling of indexing methods to make it easier to do future optimizations.
  • Add additional links to UI to make it easier to switch between crawl and manage machine activities, other links added to make it faster to go from editing a scraper to testing its effects on a page.
  • Resolves some conjunctive query and quoted query bugs.

• Admin, Group, Wiki, and Yioop Interface
  • Updated UI with icon based links to make it easier to navigate between activities.
  • Updated UI for activities related to creating, finding, and joining groups.
  • Messaging between users now supported.
  • Emoji picker for messages added.
  • Image resources on pages now lazy loaded.
  • Wiki media gallery pages of video resources output Open Graph info about the video.
  • Add PDF thumbnail generation support for wiki resources if ImageMagick installed.
  • Add epub, html thumbnail generation support for wiki resources if Calibre installed.
  • Updates credit system so now possible to redeem credit back to cash.
  • Accessibility color scheme improvements.
  • Improvements to feed and wiki group bar bread crumbs.
  • Hamburger menu animation improved.
  • Removes direct invocation of PDFJS and EPUBJS code, to just use the pdfjs viewer and epubjs-reader instead.
  • Several database backend warnings fixed for users using Mysql or Postgres backends.

Changes in Version 8.0

(September 21, 2021)

• Crawler and Search Engine
  • Crawler now sends Referer header
  • Generate animated GIF thumbnails for video if using FFMPEG
  • Fix crash issue related to etag processing
  • Improve UI for Search Sources to make it easier to add and test new sources

• Indexing and Library Functionality
  • New PackedTableTools class to efficiently store integer records for several data structures
  • New LinearHashTable implementation using PackedTableTools records
  • Btree class replaced with BPlusTree using PackedTableTools records
  • WebArchiveBundle class replaced with PartitionDocumentBundle class using PackedTableTools records
  • IndexArchiveBundle format for crawls replaced with new IndexDocumentBundle format based on BPlusTree for inverted index, and PartitionDocumentBundle for storage
  • Arctool updated to new indexing sytem and has a migrate option for migrating old indexes.
  • Fixes Mysql issue because Mysql now reserves GROUPS keyword

• Admin, Group, Wiki, and Yioop Interface
  • Improvements to hamburger menu, more use of icons.
  • Increase accessibility functionality, improve how works in lynx
  • Revamps UI for media list wiki pages, supporting list, grid, and details views as well as allowing control of sort order.
  • New clipboard functionality for editing Wiki Resources.
  • Add new url shortener and page wall wiki page types
  • Support qr codes in Wiki pages
  • Modification to allow code to work in PHP 8
  • Fix security hole that allowed Public Group to post
  • Bug fixes to Yioop if redirects not on
  • Fixes XSS vulnerability in query string

Changes in Version 7.1

(August 13, 2020)

• Crawler and Search Engine
  • Make News RSS feeds discoverable via link tag.
  • Fixes a bug in how fetcher selects next queue server when run in multi-queue server mode.

• Indexing and Library Functionality
  • Adds a normalize function to make sure equivalent simplified and traditional characters handled the same.
  • translate-locale option of TokenTool enhanced to support translating Public and Help wiki pages.

• Admin, Group, Wiki, and Yioop Interface
  • Reorganization of menus related to groups and wikis.
  • Continuous scroll for group discussions added.
  • Bug fix for the display of PDFs.
  • Rework UI so largely works when Javascript disabled and when Javascript and CSS are disabled.

Changes in Version 7

(June 26, 2020)

• Crawler and Search Engine
  • Search results can be presented as continuous scroll or as paged results.
  • For privacy tell browser not to display referrer query when clicking on search results.
  • Hamburger menu now adds quick access to narrow search by time, language, video duration, etc.
  • Landing page can be configured to show trending and news highlights from admin specified subsearches.
  • Knowledge Wiki callouts on search results and tool to manually make callouts and another tool to generate them off of Wikipedia dumps.
  • Improved crawl result editor that makes it easier to filter results from searches, edit search snippets, and pin urls at top of search results.
  • Better display of trending news items, query and group statistics. Better charting of Trending News Items.
  • Speed up image loading in news item search results.
  • Improved link farm detection.
  • Which media updater jobs are running can now be controlled from the UI.
  • Adds Tesseract support for OCRing images in PDFs.
  • Improved processing of m3u8 for feed podcast downloads
  • Add slow start parameter which can be used to make sure get a good copy of seed sites before starting general crawl.
  • Fixes critical bug in IP address handling introduced by changes to cURL library. Bug was causing many pages not to get crawled after cURL version changed.
  • Improved logging and log file rotation for crawling jobs.

• Indexing and Library Functionality
  • Feed item storage moved out of database into more scalable FeedArchiveBundle class.
  • Adds a notion of direction to index bundle iterators so now can scan through posting lists in both a forward (what 6.0.4 had) and backward direction.
  • Improved Trending Calculations
  • New segmentator, named-entity recognizer, part-of-speech taggers for Chinese. Improved Chinese stop words.
  • Chinese language question answering implemented.
  • Fixes issues with Tor and proxy crawling.

• Admin, Group, Wiki, and Yioop Interface
  • Add configurable cookie consent feature to UI for GDPR.
  • Clean up UI for groups so less weird toggling between group and admin view.
  • Tool to use Yandex translate to quickly add localizing strings for languages other than English.
  • Allow browsers to better cache page content by sending better Cache and Not Modified headers.
  • Reduce mixture of url rewrite in .htaccess versus php. now all urls routed through main index.php and PHP does the rewrite.
  • Database query optimizations to speed up queries needed for display of Manage Accounts, and group and feed pages.
  • More humanly readable simplifications URL links between pages.
  • Removed Sharing and User Specified Crawl Mixes -- now crawl mixes restricted to admin account
  • Remove ZKP authentication.

Changes in Version 6

(June 13, 2019)

• Crawler and Search Engine
  • Trending keywords now available under More and Tools link.
  • Support for multiple simultaneous crawls by assigning machines to channels and then scheduling crawls to those channels.
  • Support for general repeating crawls. These crawls have a repeat frequency and two indexes: one for searching for crawling and Yioop automatically switches between the two every repeat period.
  • Support for crawling to some fix depth directly rather than using a regex in allowable sites to crawl.
  • Dropdown to allow admins to control how Yioop should follow robots.txt files.
  • Under Page Options can now test how pages will be processed by URI, File Upload, or Direct Input.
  • Safe search check box added to Settings and enabled by default.
  • Fixes issues with HTTP/2 crawling on Linux.
  • Improves Mirror server handling.
  • Removes Memcache support as cache option for search results

• Indexing and Library Functionality
  • Width, Height, EXIF, and XMP meta data now indexed for images and media:image-small, media:image-medium, media:image-large meta words added.
  • Improved language and safe website detection. Now also supports mul locale tag.
  • Adds stopWordsRemover method to all supported locales' Tokenizer class.
  • New LinearAlgebra class added to make it easier to do term vector manipulations both for summarizers and in using Yioop as a Library under Composer.
  • All summarizers rewritten. Each sentence for each summarizer now gets a score before being added to summary. This score is also used in ranking search results.
  • A Test link for Search Sources added to allow easy testing if source being correctly downloaded.
  • Adds new Scrape Podcast search source to allow downloading of podcasts to wiki pages.
  • Web Scraper order of application now determined by a priority field.
  • Web Scrapers now enhanced so can now extract fields like THUMB_URL or other meta words, such as for video duration. I.e., replaces functionality that previously only poorly served by video search sources.
  • Removes video search sources from search sources.
  • Add Library class with init method to make it easier to initialize Yioop when used with Composer.
  • Under Page Options have a toggle to control whether phrase extraction rather than just term extraction always done. In most circumstances, not using phrase extraction gives faster and better indexing.
  • Remove two copies of dictionary info, one in IndexShard and one in IndexDictionary, thus, making for smaller indexes.
  • Cache pages now stored with summary in same object allowing more compression if keeping cache of whole pages
  • Removes materialized metas and largely unused thesaurus functionality.

• Group and Wiki System
  • Adds a seen media indicator in media lists, which can be user reset.
  • Improved inter-group links.
  • If wiki url has 360 in path, checks for 360 images and adds an enter VR button to view them.
  • Media updater now has a job that allows periodic downloading of podcasts to a wiki page.
  • Time zone, Cookie name, and Session token now set under Security rather than Appearance, time before autologout now controllable by admin using dropdown.

Changes in Version 5

(May 30, 2018)

• Crawler and Search Engine
  • Now runs without the need of a separate web server.
  • Improved robots.txt handling
  • Curl flags for HTTP/2
  • Start crawls has been simplified so auto-starts queue server and fetchers

• Monetization
  • Credits can be used to buy access to wiki groups in addition to keyword ads
  • Group creators can charge credits for joining wiki groups.

• Group and Wiki System
  • Group and thread recommendation system added
  • Analytic system supports differential privacy
  • Support for encrypted groups
  • Simplified Chat Bot support
  • Can display CSV files as spreadsheets with equation support
  • Can displays embed CSV cells in wiki page as tables or charts
  • Supports data url resources in Wiki pages
  • Video subtitling support enabled using VTT files

Changes in Version 4

(Feb 28, 2016)

• User-defined Web Scrapers: These can be used to detect web pages of a certain type, for example, coming from Wordpress, and then when generating a summary, focus only on the standard area in such a document where content is found.
• New Summarizers: Two new summarizers available to pick what content on a page should be indexed: A weighted-centroid approach and a page-rank-like, sentence-graph approach.
• Media Sources Enhancements: media sources can be used to say what content should be crawled periodically. Before these were aggregated under a news search source and allowed for rss, atom, and scraped html feed types. A media keyword field has been added so one can aggregate these data into user-defined search sources besides news, and a new regex scraper feed type has been added.
• Image Thumbnail Caching System: allows news feeds results to be completely over SSL.
• Question Answer System for English: for some question queries Yioop will now list a possible question answer in the returned search results. This relies on a improved part-of-speech tagger for English. A Hindi part-of-speech is now also part of Yioop.
• General Stability Enhancements for Fetchers and QueueServers
• Analytics: a subsystem to give information about query counts as well as wiki page, and discussion group views has now been added.
• Differential Privacy: a module which can be turned on or off to use differential privacy techniques to add randomness to aggregate statistics from the analytics system to reduce the risk that individual user privacy is compromised.
• Better Management of Users and groups
• Wiki Page Enhancements: Adds two new page types: Templates: Wiki-feed pages. New dropdown-based navigation. Allows for user-defined relationships between pages and has a simple graphical way to explore relationships. Gallery pages can now display in browse mode: Epub and PDF files in addition to images, audio and video files. Tools to move pages media resources around have also been improved.
• Group Discussion Enhancements: can import RSS feeds of discussions from other discussion boards.
• Chat Bots: A first implementation of a chat bot api for the group discussion feeds. Weatherbot example provided.

Changes in Version 3.1

(Sep. 24, 2015)

• Adds support for Keyword Advertising and its own unique ad keyword pricing model. Findcan.ca demonstrates this in action and now supports sign up for advertisements.
• The keyword advertising system integrates with a payment processing script available for download for a fee. This script uses Stripe.com to handle credit card transactions.
• Yioop has been rewritten to work with the popular PHP package manager known as Composer and Yioop is available from the composer package repository https://packagist.org. This should make it easier for people to develop projects using Yioop's natural language processing facilities.
• Yioop's MediaUpdater process has been rewritten so that it can run in a distributed fashion and now supports recoding to mp4 videos uploaded to the wiki system and group feed system. It also supports sending out notification emails. The latter had been done exclusively by the web app.
• In addition to the centroid-based and ad-hoc web page summarizers, there is new a new graph-based summarizer that can be used during crawling.
• Arabic, Dutch, Hindi, Persian, and Portuguese stemmers have been added.

Changes in Version 2.1

(Mar 1, 2015)

• Fixes some security issues in Version 2.0 with regard to checking allowed activities of a user.
• Improves the accuracy of how Yioop counts the number of documents containing a word or phrase
• Improves email notifications from group feeds.
• Adds number of groups column to manage user lists
• Adds number of users info to manage groups lists
• Fixes a number of places where the Yioop code was generating Notices.

Changes in Version 2

(Jan. 25, 2015)

• General
  • New integrated wiki help system throughout software
• Search and Crawling
  • Adds Docx support. Now for zipped formats like Office, Yioop can use a partial Zip extractor to extract content even if whole file not downloaded.
  • Adds support for rel canonical meta tag
  • Adds French, Spanish, German, Russian stemmers
  • Adds support for Gopher protocol
  • Word filter plugin can apply domain and url specific rules
  • Improved scheduling of page download based on number of DNS lookups
  • Improved handling of robots.txt files when site in question is congested
  • arc_tool supports count recalculation and url suggestion injection
• News/Media Updater
  • Updater now has a scraper for HTML pages with news
  • Updater can now extract images from news feeds.
  • Updater can now auto-convert video files to mp4 and webm
• Feeds and Wikis
  • Adds ability to drag and drop images, video, and other documents in posts and wiki pages
  • Besides standard wiki pages, Yioop now supports slide presentation pages, media gallery pages, and page aliases
  • +/- Voting available on Group posts
  • Can configure so that posts expire after a certain amount of time
  • Improved mail messages associated with posting
  • Can set meta tag info for wiki pages as well as common header and footers
  • Can embed search bars into wiki pages
  • Math Mode works in posts and wiki pages
• Web Sites
  • Can use Configure activity to set sites look and feel from icons to background to timezone, etc.
  • Can Configure to use wiki system for default landing page
  • GUI for adding Ad Server scripts

Version 1

• Version appeared Jun 14, 2014

Origins Of Yioop

• July 10, 2011, git repository started.
• Software was first released publicly (non-repo based), August, 2010.
• November, 2009, project begun.