Meet CiteSurgeAuditBot.

CiteSurgeAuditBot/0.1 (+https://citesurge.com/bot)

What an AI Readiness Check requests

The CiteSurge AI Readiness Check requests robots.txt first. If robots.txt permits the homepage for CiteSurgeAuditBot, the scan may request the public homepage, sitemap documents without crawling their listed page URLs, selected public AI and agent discovery files, and at most ten normalized public off-origin entity links declared by the site.

It will not request private or credential-protected resources, follow arbitrary page links, run a browser, or use another company's crawler identity. The anonymous result is specified for the homepage only.

When one of a site's declared off-origin entity links fails with an HTTP status, or answers with a redirect to another host that the scan declines to follow, the scan may send one additional request to that destination: a control request for a path that cannot exist, made by replacing the link's last path segment with the fixed token citesurge-audit-bot-control-probe. If the destination answers the control the same way it answered the real link, its responses carry no information about that link and the check is recorded as not assessable rather than as a failure. At most one control path is probed per destination host per scan, as a HEAD request retried as a GET request only where the destination does not support HEAD, and it uses the same declared user agent as every other request.

Request volume and pacing

The anonymous fetch set is about 18 requests before redirects. Its broker permits no more than 64 HTTP requests in one scan and closes the scan after 30 seconds. Redirect hops count against that request budget. A single request waits at most 8 seconds for a response before it is abandoned, and once an origin has gone silent the rest of that scan stops asking. An anonymous scan is an on-demand bounded run, not a continuous crawler or background recrawl.

How to block an anonymous readiness scan

Apply a robots.txt rule to the product token. A rule disallowing the homepage makes the anonymous readiness collector stop before it requests the homepage or any other readiness resource. The check publishes no number and no findings from that attempt.

User-agent: CiteSurgeAuditBot
Disallow: /

More specific Allow and Disallow rules are evaluated under the same robots.txt matching rules. A site can also block the declared user agent at its edge.

Account-free no-scan control

The AI Readiness Check includes an account-free no-scan list. A domain controller proves control with a short-lived DNS challenge. Once verified, the exact host will be checked against the no-scan list before DNS resolution or any outbound readiness request. Use the no-scan control or request removal. For another crawler or privacy concern, use the CiteSurge contact page or write to privacy@citesurge.com.

Paid crawler divergence

The separate crawler-divergence check is not an anonymous scan. It requires paid client access, active same-domain control, and explicit opt-in. It requests the homepage at most once per crawler identity, skips every crawler that the site's robots.txt disallows, and compares declared robots access with the response delivered at the edge. It does not change the headline AI readiness result. The probe itself is available only inside the authorized paid workflow.

Policies and contact

Review the Terms and Privacy Policy for the planned readiness-check publication and retention boundaries. Send crawler, abuse, or access questions through /contact.