Meet CiteSurgeAuditBot.

The CiteSurge AI Readiness Check requests robots.txt first. If robots.txt permits the homepage for CiteSurgeAuditBot, the scan may request the public homepage, sitemap documents without crawling their listed page URLs, selected public AI and agent discovery files, and at most ten normalized public off-origin entity links declared by the site.

The check running today will not request private or credential-protected resources, follow arbitrary page links or run a browser. Its anonymous result is specified for the homepage only.

If the site's bot protection refuses CiteSurgeAuditBot on the homepage, with a 401, 403 or 451 answer or a challenge page, the scan retries the homepage presenting the user agents below, in order, and keeps the first one the site serves for the rest of its requests to that site. That is at most 3 extra homepage requests. A user agent whose robots.txt group disallows the homepage is skipped. These requests are never signed and never sent to other sites. A robots.txt block, a 429 answer or a timeout never starts a retry, and every robots.txt decision stays the one for CiteSurgeAuditBot. The result then says which user agent the pages were served to, never what the real crawler sees:

When one of a site's declared off-origin entity links fails with an HTTP status, or answers with a redirect to another host that the scan declines to follow, the scan may send one additional request to that destination: a control request for a path that cannot exist, made by replacing the link's last path segment with the fixed token citesurge-audit-bot-control-probe. If the destination answers the control the same way it answered the real link, its responses carry no information about that link and the check is recorded as not assessable rather than as a failure. At most one control path is probed per destination host per scan, as a HEAD request retried as a GET request only where the destination does not support HEAD, and it always uses the declared CiteSurgeAuditBot user agent.

A deeper anonymous readiness check, v10, is disclosed here before it sends any request; until it is enabled, the pattern above still applies. A v10 scan reads 4 pages in full, including the homepage and any about, contact and offering pages it can resolve. It identifies the site's privacy link but never fetches that page, and makes 12 partial reads: 8 sitemap URLs and 4 links found on the pages it reads, each capped at 1 MiB. The site supplies candidates through its sitemap and links. The scan selects among them using a keyed hash the site cannot predict before a scan. A URL the scan cannot read for its own reasons is replaced by the next in keyed order, at most 4 times.

Its fixed requests start with robots.txt and also cover up to 6 sitemap documents, /llms.txt then /.well-known/llms.txt with at most 5 declared link checks on the same registrable domain at 128 KiB each, a check of the site's alternate host and of plain http://, and one request for a path that cannot exist. This not-found probe tells a real missing-page answer apart from a catch-all that answers every path. Up to 10 declared off-origin entity links may still be checked, each with at most one control request for a path that cannot exist, disclosed above.

Within the same allowance a v10 scan may then read the public discovery and convention files the site advertises. It can send read-only POST requests to endpoints the site itself advertises. These include GraphQL introspection, Model Context Protocol discovery, list and read calls, and NLWeb /ask carrying the question What is this website about?. The POST requests are sent without credentials, never mutating anything. The scan can also send a GET to an advertised API endpoint and look the site up on a fixed list of public registries and reference sites: registry.npmjs.org, pypi.org, proxy.golang.org, rubygems.org, crates.io, api.github.com, github.com, smithery.ai, mcp.so, skills.sh, chatgpt.com, www.wikidata.org, *.wikipedia.org. These requests come last and are dropped, reported as not checked, when the request allowance runs out.

A small fixed set of homepage requests presents AI crawler user agents, plus its own CiteSurgeAuditBot and one browser baseline request for comparison. That makes 6 requests in all. These probes measure what a client presenting that user agent receives. A probe is sent whether or not robots.txt names that crawler. Those rules govern only the scan's own CiteSurgeAuditBot requests. The probes never request any other page, never try to pass a challenge, and a blocked or challenged probe never lowers the site's score. Each result is labelled as served to a client presenting that user agent, never as what the real crawler sees:

Each v10 scan permits no more than 160 HTTP requests and 160 DNS lookups in one scan, downloads at most 8 MiB per response and 24 MiB in total, and closes the scan after 45 seconds. A robots.txt crawl-delay can extend that to at most 75 seconds. Response headers are expected within 8 seconds and the response within 10 seconds. At most 2 requests run at once to one origin, 200 ms apart, and fetching from an origin stops at the first 429 or challenge answer. Anonymous scans make at most 300 requests per registrable domain in any 24-hour window.

When a signing key is configured, every v10 scan request except the crawler user-agent probes and the identity-fallback requests above is signed with Web Bot Auth message signatures, and falls back to an unsigned request if signing fails. The Signature-Agent header names https://citesurge.com, whose public key directory is at https://citesurge.com/.well-known/http-message-signatures-directory, where an edge or origin can fetch the keys and verify a request before trusting it.

To allow a v10 scan, permit CiteSurgeAuditBot in robots.txt for the pages it reads, do not block the declared user agent or its signatures at the edge, and keep rate limiting gentle. The scan stops at the first 429. The robots.txt block, the no-scan list and the removal request below apply to the v10 scan in exactly the same way as today's check.

Before the public check switches to v10, CiteSurge may run v10 scans itself: a bounded research sample over a fixed list of third-party domains, and a one-time rescan of domains that already have a published readiness result. These runs follow the same pattern, limits and controls described above. They also follow the robots.txt block, the no-scan list and removal requests. A run over a host starts only after any live scan of that registrable domain has finished, at least 60 seconds later.

The AI Readiness Check includes an account-free no-scan list. A domain controller proves control with a short-lived DNS challenge. Once verified, the exact host will be checked against the no-scan list before DNS resolution or any outbound readiness request. Use the no-scan control or request removal. For another crawler or privacy concern, use the CiteSurge contact page or write to privacy@citesurge.com.

Review the Terms and Privacy Policy for the planned readiness-check publication and retention boundaries. Send crawler, abuse, or access questions through /contact.