Better detection for when a url is a sitemap so will queued appropriately. Fixes to delayed host handling

Author Chris Pollett <chris@pollett.org>
Author date 2023-12-04 09:Dec:th
Author local date 2023-12-04 01:Dec:th -0800
Committer Chris Pollett <chris@pollett.org>
Committer date 2023-12-04 09:Dec:th
Committer local date 2023-12-04 01:Dec:th -0800
Commit d96a61e32fc333047e0d9a9082bf6ed910b9b773
Tree 41c2994f6a7addb9e2be6f561922c4c57beb2186
Parent 4a21d3f43e09e5d622134233b51a8dcd16f25e18
Better detection for when a url is a sitemap so will queued appropriately. Fixes to delayed host handling
Affected files:
src/configs/Config.php
src/executables/Fetcher.php
src/executables/QueueServer.php
src/library/CrawlQueueBundle.php
src/library/processors/RobotProcessor.php
ViewGit