<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Phase 9 — Crawl and Index Health, and
Enhanced Testing</title>
<style>
body { font-family: system-ui, sans-serif; max-width: 760px;
margin: 1.5em auto; padding: 0 1em; line-height: 1.5; color: #1a1a1a; }
h1 { font-size: 1.4em; }
h2 { font-size: 1.1em; margin-top: 1.4em; }
ol { margin: 0.3em 0; } li { margin: 0.25em 0; }
ol.outline { list-style-type: upper-roman; }
ol.outline ol { list-style-type: upper-alpha; }
ol.outline ol ol { list-style-type: decimal; }
ol.outline ol ol ol { list-style-type: lower-alpha; }
.check { color: #1a7f37; font-weight: bold; }
.next { color: #b35900; font-weight: bold; }
.review { color: #0969da; font-weight: bold; }
.working { color: #b35900; font-weight: bold; }
.nope { color: #82071e; font-weight: bold; }
.q { color: #8250df; font-weight: bold; }
code { background: #f2f2f2; padding: 0 0.25em; border-radius: 3px; }
em { color: #555; }
</style>
</head>
<body>
<h1>Phase 9 — Crawl and Index Health, and Enhanced Testing</h1>
<p><em>Markers: <span class="next">▶</span> being worked on,
<span class="review">⚑</span> implemented and waiting to be
reviewed,
<span class="check">✓</span> done,
<span class="check">✓</span><span class="nope">✗</span> closed
by deciding not to do it, <span class="q">?</span> open decision,
unmarked = planned.</em></p>
<ol class="outline">
<li><strong>Crawling using synthetic data.</strong> A crawl of any size
can be run on one machine without fetching anything from the web. The
pages are made up as the crawl asks for them, so a run repeats exactly
and costs nobody bandwidth.
<ol>
<li><span class="next">▶</span> <strong>A synthetic crawl tab is
added to the crawl options.</strong> Manage Crawl, then Edit Crawl
Options, gains a tab beside the web crawl one. It keeps the settings that
still mean something for a made-up web and replaces the ones that do not.
<ol>
<li><strong>The settings about reaching the live web are
dropped.</strong>
Repeat type, crawl sleep, robots.txt, restrict sites by url, disallowed
sites and seed sites all describe fetching from real hosts. A made-up web
has no hosts to be polite to and no addresses to restrict.</li>
<li><strong>The shape and size of the made-up web are chosen.</strong> A
graph type says how the pages link to one another. A graph size says how
many pages the web holds. A third setting says which pages the crawl
starts from, taking the place of seed sites.</li>
<li><strong>What a page is and what it says are chosen by
chance.</strong> One model gives the chance a page is html, pdf or an
image. Another gives the words that appear on it. The same seed gives the
same web every run, so a result can be compared with the one before
it.</li>
<li><strong>The settings suit both a small check and a large
load.</strong> A small web makes it possible to ask whether every page
indexed can be recalled and whether results come back in the order
expected. A large one puts the web server, the queue server and the
fetchers under load worth measuring.</li>
</ol></li>
<li><strong>A synthetic crawl makes its own pages rather than fetching
them.</strong> Where that tab is the one chosen, the fetcher takes its
pages from a function that generates them in place of
<code>FetchUrl::getPages</code>. Everything after the fetch runs
unchanged, so what is measured is the real crawl.</li>
</ol></li>
<li><strong>The slowdown over a long crawl is found and
fixed.</strong> A synthetic crawl is run long enough for the time per
page to climb. Where it starts climbing says which stage is at fault,
and what that reading names is what gets fixed.</li>
<li><strong>An index is never left corrupt by a crawl that stops part
way through.</strong> Phase 4 added guards that keep a corrupt index
from taking the server down. Those are guards rather than prevention.
This item is the prevention.
<ol>
<li><strong>Every point that could leave a half-written entry is
named.</strong> The walk covers the partition append, the dictionary
merge and the postings write. What is looked for is a write that a
later read takes as a garbage length or offset.</li>
<li><strong>A write is either finished or undone, never left part
way.</strong> A write goes to a new name and is put in place at the
end, or a small mark says a step was begun. On starting again the mark
says whether to finish the step or drop it.</li>
<li><strong>The recovery is shown by stopping a crawl on
purpose.</strong> A synthetic crawl is driven and the queue server is
stopped at chosen points in the write path. The index is read back
afterward and must come back clean every time.</li>
</ol></li>
<li><strong>Other work, grouped by what it touched.</strong> Each is a
deliverable of its own that came up during the arc. They are grouped by
the part of Yioop they concern rather than listed one by one. <ol>
<li><strong>A large download no longer holds the web server or its
memory.</strong> One reader pulling a large video stalled every other
request for as long as the download took. A three hundred megabyte video
ended the process outright.
<ol>
<li><strong>A large file is drained by the event loop rather than by
the request.</strong> A resource request with no byte range is now
deferred, and the whole file case names the file instead of reading it
out. Measured on a two hundred megabyte file: pages answered in twenty
seconds before and under five milliseconds after.</li>
<li><strong>A request keeps its streaming context while it
waits.</strong> The context saying which protocol a reply goes out on
was not saved with the rest of a waiting request. A resumed request
therefore gathered its whole body in memory. It is saved and restored
with the other request state.</li>
<li><strong>One block of IP addresses keeps one share of the
link.</strong> Files going to the same block divide one turn of the
write loop between them. Several downloads from one place cannot take
several shares. Traffic between Yioop machines keeps a whole turn each.</li>
</ol></li>
</ol></li>
</ol>
</body>
</html>
(c) 2026 Seekquarry - Open Source Search Engine Software