Sundar Pichai has said publicly that AI agents will reach websites via two distinct routes: browsing rendered HTML pages, exactly as a human does, and calling structured protocols or APIs directly. He also noted that the protocol route will scale better over time. That single statement rewrites the definition of “crawlable” for every technical SEO and site architect working today. A site optimised only for human-readable pages is already half-invisible to the agentic layer being built on top of Google Search, ChatGPT, Perplexity, and Gemini. This piece maps the gap between the two modes and gives you a prioritised checklist for closing it.

What the Two-Path Model Actually Means

The two-path model for ai agent site architecture pages protocols works like this. In the first path, an agent spins up a browser session, loads your URL, parses the rendered DOM, and extracts what it needs, product details, pricing, availability, answers. This is the “pages” route. It is flexible and requires no cooperation from the site owner, but it is slow, computationally expensive, and brittle: a layout change or a JavaScript-heavy render can break the agent’s extraction entirely.

In the second path, the agent calls a documented endpoint, a REST API, a structured feed, or an emerging protocol layer, and receives clean, typed data in return. No rendering, no parsing ambiguity, no dependency on CSS selectors. This is the “protocols” route. It is faster, more reliable at scale, and far easier for an agent to reason over. Pichai’s point about protocols scaling better is not a prediction about a distant future: it is a description of why every serious agentic system will prefer the protocol path whenever one exists.

Five Questions Technical Teams Are Asking Right Now

Question: Does optimising for the pages path still matter if protocols are the future?

Answer: Yes, and for a long time. The majority of agents operating today, including those powering Google AI Overviews, Perplexity, and ChatGPT browsing, still rely primarily on rendered HTML. Protocol-based access requires deliberate investment from site owners. Until that investment is widespread, agents will fall back to the pages path by default. Abandoning HTML optimisation now would be like removing your sitemap because voice search exists.

Question: What makes a page “agent-readable” on the pages path?

Answer: Three things above all others. First, the critical content must be present in the raw HTML served to the crawler, not injected by JavaScript after load. Agents that do not execute JS will see a blank or partial page. Second, the information must be structured into discrete, self-contained chunks: short paragraphs, labelled sections, lists, and tables that an agent can extract without understanding the surrounding context. Third, schema.org markup in JSON-LD must anchor the entity (product, organisation, article) so the agent can classify what it has found without guessing.

Question: What does the protocol path require from a site owner?

Answer: At minimum, a documented, stable API or structured data feed that exposes the same information available on your pages (product catalogue, pricing, availability, content) in a machine-readable format. Beyond that, the protocol layer needs authentication that agents can handle programmatically, versioning so agents do not break when you update the schema, and rate-limiting policies that distinguish agent traffic from scraper abuse. This is not a content task; it is an infrastructure task that requires engineering resource.

Question: How do ChatGPT, Gemini, Perplexity, and Google AI Overviews differ in which path they use?

Answer: All four currently rely heavily on the pages path for general web content. Google AI Overviews and Gemini have the deepest integration with Google’s own crawl infrastructure, meaning Googlebot’s ability to render your page directly affects whether your content enters the generative layer. ChatGPT’s browsing tool fetches live pages on demand. Perplexity runs its own crawler. None of these systems has a universal protocol standard yet, though Google’s push toward structured protocols signals the direction the ecosystem is moving. Anthropic’s Claude similarly accesses web content via rendered pages when given browsing capability.

Question: If my site has good SEO rankings, am I already covered for agent access?

Answer: Not automatically. A high ranking means Googlebot can crawl and index your pages, which is a necessary condition but not a sufficient one. Agent legibility adds further requirements: server-side rendering of key content, entity-level schema markup, chunk-friendly structure, and, for the protocol path, API availability. A site can rank on page one and still be largely opaque to an agent trying to extract structured data from it.

Prioritised Checklist: Making Your Site Legible to Both Agent Types

Step 1 โ€” Audit server-side rendering coverage

Fetch your key commercial and content pages using a tool that disables JavaScript, curl or a headless request with JS off, and compare what you see against the rendered version. Any content that disappears is invisible to agents on the pages path that do not execute JS. Prioritise moving product names, prices, descriptions, and FAQ content into the server-rendered HTML. This is the single highest-leverage fix for pages-path legibility.

Step 2 โ€” Implement entity-anchoring schema markup

Add JSON-LD blocks for Organisation, WebSite, and Product (or Article, depending on page type) to every template. The Organisation block should include your canonical brand name, URL, and a brief description that matches how you define yourself in your content. Without these signals, an agent extracting data from your page has no reliable way to associate that data with your brand entity in its knowledge representation, it is reading words without a label.

Step 3 โ€” Restructure content into extractable chunks

Review your highest-traffic pages and break long prose into labelled sections with clear H2 headings, short paragraphs of two to four sentences, and lists wherever you are enumerating items. Add a FAQ or Q&A block to any page that answers a common question, these are the structures that generative engines extract most reliably. Each chunk should be able to stand alone: an agent reading only that paragraph should understand the point without needing the surrounding context.

Step 4 โ€” Document and expose a structured data endpoint

For e-commerce and product-led sites, publish a product feed or API endpoint that mirrors your catalogue in structured JSON. For content sites, an RSS or Atom feed with full-text entries and metadata is a starting point. Document the schema publicly. This is the first step onto the protocol path and the one that will compound in value as agentic systems mature. Coordinate with your engineering team early: this is not a CMS configuration, it is a development project.

Step 5 โ€” Set explicit agent access policies in robots.txt and response headers

Review your robots.txt to ensure you are not inadvertently blocking known agent crawlers. Consider adding an AI-agents section that explicitly permits the crawlers you want to allow. In parallel, set cache-control headers that give agents fresh data without hammering your origin server. Ambiguous access policies cause agents to either skip your site or cache stale versions of your content.

Step 6 โ€” Monitor which path agents are actually using

Check your server logs for known agent user-agent strings from GPTBot, Google-Extended, PerplexityBot, and ClaudeBot. Segment these from standard Googlebot traffic. If you see agent crawlers hitting your pages but your brand is absent from AI-generated answers about your category, the problem is likely in extraction quality โ€” the agent can reach your page but cannot reliably parse what it finds. That gap is where structured markup and chunk-friendly content do their work.

Pages Path vs Protocol Path: What Each Demands

DimensionPages Path (HTML browsing)Protocol Path (API / structured feed)
Who controls accessAgent decides; no opt-in neededSite owner must build and expose endpoint
Content renderingServer-side HTML required for JS-blind agentsNot applicable; data served directly
Structured markupJSON-LD schema critical for entity classificationSchema embedded in API response format
Scalability for agentsLow โ€” rendering is expensive per pageHigh โ€” typed data, no parsing overhead
Implementation ownerSEO and content teamsEngineering team, with SEO input on schema
Time to implementWeeks (content and markup changes)Months (API design, documentation, auth)
Risk of breakageHigh โ€” layout or JS changes can break extractionLower โ€” versioned schema, explicit contracts
Measurability todayLog analysis, prompt monitoring, citation auditsAPI call logs, structured response validation

A Realistic Picture of the Work Involved

A mid-sized e-commerce brand โ€” several hundred product pages, a blog, and a standard CMS setup โ€” ran an internal audit after noticing their products were absent from Perplexity and ChatGPT answers in their category, despite strong Google rankings. The audit phase took the majority of the project time. Identifying which pages had JS-dependent content, which schema blocks were missing or malformed, and which chunks were too long for reliable extraction required cross-functional input from SEO, development, and content. The fixes themselves โ€” adding JSON-LD templates, restructuring three key category pages, moving price data server-side โ€” were implemented in a fraction of the time once the diagnosis was clear. The lesson the team drew: the bottleneck is never the implementation. It is knowing precisely where the agent loses the thread.

That ratio โ€” diagnosis heavy, implementation relatively light, holds across almost every agentic legibility project. And it points to the real ongoing cost: the diagnosis is not a one-time event. Crawler behaviour changes. Models update their extraction logic. A CMS upgrade reintroduces JS-dependent rendering. The six steps above, run once, give you a baseline. Run quarterly across all your key pages and all four major agent systems, the arithmetic compounds quickly: six steps across four crawlers across even fifty pages, four times a year, is a substantial audit programme. That is before you add protocol-path monitoring as your API layer matures.

What This Means Going Forward

Pichai’s two-path model is not a roadmap for a future state, it describes the architecture that is being built right now, by Google, OpenAI, Anthropic, and Perplexity simultaneously. The pages path is the present; the protocol path is where the leverage will concentrate. Sites that treat these as sequential priorities, “we’ll sort HTML first, then worry about APIs later”, will find themselves perpetually catching up, because the agents evaluating their site architecture are not waiting. The structural challenge is that the work spans SEO, content, engineering, and infrastructure simultaneously, and the results are not visible in standard analytics: a site can be fully agent-invisible while its Google rankings remain intact. That is precisely the gap that aisearchaudit.ai is built to surface, auditing how your site is actually perceived and cited across ChatGPT, Gemini, Perplexity, and Google AI Overviews, and mapping the specific gaps in your pages-path and protocol-path legibility so your teams know where to act first.


Run your first AI Search Audit report for free

If you want to see exactly where your site stands across the four major AI systems, aisearchaudit.ai runs a full citation audit and returns a structured report with the specific gaps to fix first. Check our plans or contact us for a walkthrough.

Featured image: photo by Shuxuan Cao on Pexels.