Agent Readiness Is Not AI Visibility

Agent readiness does not show whether AI mentions your brand

A site can allow AI crawlers, serve every public page, and still remain absent from the answer a buyer sees. It can also publish clear, useful evidence while a firewall or bot rule keeps a crawler out. Both situations get called an "AI visibility problem," but they require different work.

Cloudflare's 17 April 2026 Agent Readiness score checks four areas of a website: discoverability, content, bot access control, and capabilities. It does not report whether a brand appeared in a particular AI answer.

CiteSurge therefore separates three questions. Did a test request reach the page? Did the page clearly answer the buyer's question and back up important facts? What did the AI products in the audit return?

One score cannot answer all three. A content rewrite will not repair a firewall rule. A robots.txt change will not make a vague page clearer. Passing a readiness check does not prove that an AI product will mention the brand, cite the page, or recommend anything.

The free CiteSurge AI Readiness Check measures the first technical layer on one public homepage. Its seven fixed checks cover live-answer crawler access, origin response health, server-rendered response, structured data, heading structure, sitemap discovery, and off-origin entity links. The result is public and reproducible, but it is not an AI-visibility, citation, traffic, or revenue measure.

Check access before rewriting the page

Start with the first question the evidence can answer:

1. Check whether the crawler is allowed to request the page and whether the page actually loads for that request.
2. If the page loads, check whether it clearly identifies the company, explains the offering, answers the buyer question, and backs up important facts with current sources.
3. Then compare what the same AI products returned for the same buyer questions, markets, and dates.

Each result points to different work. If robots.txt allows a page but the live site blocks the request, web infrastructure or security should investigate. If the page loads but its answer is unclear or hard to verify, content, product, legal, or a subject-matter expert may need to revise it. If the buyer questions do not fit the business, the project owner should correct them before anyone interprets the results.

The CiteSurge agent-readiness guide explains the crawler-access checks in detail. This order prevents a team from rewriting a useful page when the real problem is a firewall. It also prevents a robots.txt change when the page itself is unclear.

For companies that need a crawler policy and live-delivery plan, see Agent Access Posture.

The four access results and what each means

Robots.txt states which pages a crawler is allowed to request. It does not show whether a CDN, firewall, bot rule, challenge, or origin server delivered the page. RFC 9309 defines how crawlers match their names and page paths against allow and disallow rules. CiteSurge checks that policy and the live response separately.

Four crawler-access outcomes and the team that should investigate each one first.

What robots.txt says	What the site returned	What it means	First action
Allows the page	Page delivered	The test request reached the page. This does not show whether the brand appeared in an AI answer.	Check whether the page answers the buyer question clearly, then measure what the AI products return.
Allows the page	Blocked, challenged, or rate-limited	The site's live controls blocked a request that robots.txt allowed.	Web infrastructure or security should review the controls applied to that page and crawler.
Allows the page	No reliable result	The test did not confirm whether the page was reachable.	Repeat the request before assigning content or measurement work.
Disallows the page	Any live result	The crawler rule is the first confirmed barrier. One delivered test request does not override that rule.	The site owner should confirm whether the restriction is intentional.

The table is not another score. It shows what to check first: crawler rules, live delivery, page content, or the questions being measured.

What a user-agent test can prove

A user-agent test sends a request using a crawler's registered name. It shows how the site handled that request at that moment. It does not confirm that the request came from the provider's network or predict how every future request will behave.

Crawler purpose also matters. OpenAI's publisher guidance distinguishes OAI-SearchBot, which supports search discovery, from GPTBot, which is associated with potential training. Anthropic's crawler guidance documents separate crawler identities for search, user-requested retrieval, and training. A rule for one crawler should not be treated as a rule for every purpose.

If the page loads, check whether it answers the buyer's question

Access removes one possible barrier. It does not make the page useful, specific, or convincing.

Review the page as a buyer would see it. Is the company clearly named? Is the product or service easy to understand? Does the page answer the question? Can a reader verify important facts from visible, current sources?

This is where content work begins. Improve the answer and the sources shown on the page. Do not add a new protocol merely because it is visible or easy to implement.

Compare the same buyer questions over time

Once the page is accessible and clear, measure what the selected AI products return for a defined set of buyer questions. The useful rule is simple: compare like with like.

"Which platforms measure AI visibility?" is a category-discovery question. "How does CiteSurge compare with alternatives?" is a comparison question. "Why is a brand absent from AI answers?" begins with a problem to solve. "What does CiteSurge measure?" checks direct brand understanding. Combining those questions into one unexplained score hides the buyer situation behind the number.

CiteSurge builds the question set from the brand, its offerings, audiences, markets, competitors, existing questions, and latest audit. The public buyer-question guide covers category discovery, comparisons, problem-led questions, and direct brand checks. The project owner can review, edit, activate, and retire questions within the active project.

Every result stays connected to its question, market, date, and AI product. If an AI product was unavailable, CiteSurge records it as unavailable instead of scoring it as if the brand were absent. Compare results only when the question and market match.

Presence Share of Voice, Mention Share of Voice, and Citation Share of Voice answer different questions. Presence asks which brands appear, either by name or through an attributed citation. Mention asks which brands are named. Citation asks whose sources are cited. These measurements do not represent traffic, reach, search volume, demand, or market share. CiteSurge does not monitor private conversations with AI assistants or build its question sets from those conversations.

A change over time does not prove cause and effect. Models, retrieval systems, available sources, competitors, interfaces, and provider settings can change. A later mention or citation is useful evidence, but timing alone does not prove that the latest site edit caused it.

Most sites do not need every AI-related standard

The newest machine-facing file is rarely the first thing to fix. A site can publish every optional convention and still have a blocked page, an unclear explanation, or facts that are hard to verify.

• Markdown negotiation is useful when the alternate version is maintained, easier for software to consume, and faithful to the main page.
• An llms.txt file is useful when it provides a maintained map of important public resources. It does not replace internal links, sitemaps, or working pages.
• An API catalog is relevant when a validated public API exists. RFC 9727 defines how software can discover published APIs; it does not create an API that is not there.
• MCP, OAuth, agent skills, and authenticated actions matter when an agent is meant to perform an action. A brochure site does not need an action layer merely because agents exist.

Google's guidance for generative AI features states that no special schema markup is required for its generative search features.

CiteSurge does not mark a site down simply because an optional file is missing. A page that should be public but cannot be reached is a confirmed barrier. A missing optional file is not automatically a failure.

Limitations

These checks can identify where the problem starts. They cannot represent every crawler, network path, AI product, or future answer. A request made with a crawler's registered name does not confirm that it came from the provider's network. One successful request does not predict every future delivery path.

A buyer-question set is a sample, not the entire market. Different vendors may define Share of Voice differently. Compare their values only when the questions, markets, dates, included AI products, availability rules, and measurement definitions match.

If the page is blocked, fix access. If it loads, check whether it answers the buyer question clearly and backs up important facts, then compare the same questions over time. None of these checks guarantees a mention, citation, ranking, recommendation, traffic, or revenue.