The AI Visibility Tech Stack

WHAT IT IS

The requirement that a page's substantive text be present in the HTML the server returns in its first response, rather than assembled in the visitor's browser by scripts.

HOW IT WORKS

A human's browser executes JavaScript and completes the page. Most AI crawlers make a single HTTP request and parse what arrives. Measurements across GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Meta-ExternalAgent, Bytespider, and CCBot show that they download scripts but do not execute them: GPTBot fetched script files in roughly 11.5 percent of requests and ClaudeBot in 23.8, and neither ran them. Two render: Gemini via Googlebot infrastructure and Applebot via its own browser-based crawler [1].

A DATING CAVEAT THAT MATTERS FOR INTERPRETATION

Nearly everything published on this subject, including the percentages above, traces back to a single Vercel and MERJ study published in December 2024. No independent re-verification at comparable scale has been published since. Crawler infrastructure may have changed, so the correct phrasing is "according to the latest published data" rather than an assertion about a fixed property of these systems.

WHICH JAVASCRIPT IS SAFE AND WHICH HIDES CONTENT

The distinction is not the number of scripts but whether a script creates text.

Safe, because the text is already in the HTML and the script only adds behavior: AJAX form submission, menu toggles, sliders, counters, chat widgets, consent banners, analytics.

Hides content, because the text is not in the source HTML: single-page applications in React or Vue with client-side rendering, "load more" pagination, tabs that fetch content on click, reviews and prices pulled in by a third-party widget after load, and consent screens that block content until a button is pressed.

On forms specifically: the fact that a form is submitted by script has no effect on citation, because a reading system does not fill in forms. It matters only for browser agents, and there what counts is the conventionality of the field markup, not the submission technology.

HOW THIS WORKS IN TYPICAL SYSTEMS

An ordinary WordPress site with a PHP theme returns finished HTML from the server by default; nothing needs configuring. The same holds for most classic content management systems. The problems come from what is layered on top, and they arise in three typical cases: a headless configuration where the system serves only as a data source while the interface is assembled on the client; page builders that render blocks as the user scrolls; and plugins that load reviews, prices, or schedules after the page has loaded.

DO STYLESHEET SIZE AND DOM WEIGHT MATTER

They do, but not in the way usually assumed. A crawler does not render the page and largely does not parse stylesheets, so CSS weight in itself is nearly irrelevant. What matters is HTML structure, for two distinct reasons, neither of which is about parsing time.

First: extraction quality. The system separates the substantive part of the page from navigation, footer, and boilerplate markup. Deeply nested containers, large menus, inline vector graphics in the markup, and repeating boilerplate blocks degrade that separation, raising the chance that junk ends up in the extracted fragment while something useful is lost.

Second: token cost. Tools that feed raw HTML into a model's context window spend that window on markup. This is where the multiple-fold differences arise that Markdown exports exist to address [2].

Plus a general factor: crawlers operate on a crawl budget, and slow server responses with heavy pages consume it.

HOW TO CHECK

Open an address of the form view-source:https://yoursite.com in a browser and search the page for a distinctive phrase from the visible text. That is the server response without script execution: if the phrase is not there, a significant share of AI systems will not see it.

One caution about tooling: URL Inspection in Search Console is not suitable for this task. It shows the result of Googlebot's rendering, not what a crawler receives without executing scripts.

WHAT IT GIVES THE BUSINESS

This is not an improvement, it is a precondition. Everything else in this document only makes sense once it is in place.

WHAT IT IS

A text file at the site root declaring where automated visitors may go.

WHAT HAPPENS BY DEFAULT

The absence of a prohibition means permission. If there is no file at all, or nothing in it is closed off, every automated visitor, including the training crawlers of AI companies, treats the site as open. No permissions need to be issued: an Allow line adds nothing to the absence of a Disallow line. For the overwhelming majority of sites the correct robots.txt is a short file closing off administrative paths and pointing to the sitemap. That is a complete option, not a simplified one.

HOW THE FILE IS READ

A robot looks for groups whose User-agent line names it. The RFC 9309 rule has two parts, and the second is often lost:

• If several groups name that same robot, their rules are combined into one.
• The general User-agent: * group applies only when no group naming it specifically exists at all.

The key consequence: a specific group does not inherit rules from the general one. The appearance of a line naming a robot switches off the entire User-agent: * block for it.

Hence a typical mistake. A site owner wants to explicitly welcome AI crawlers and writes:

User-agent: *
Disallow: /wp-admin/

User-agent: GPTBot
Allow: /

The result is the opposite of the intent. GPTBot now operates under its own group, where the administrative path is not disallowed, and gains access to exactly what is closed to everyone else. To do this correctly, every prohibition would have to be duplicated inside each named group.

WHEN NAMED GROUPS ARE NONETHELESS JUSTIFIED

Only when different crawlers need to be treated differently. That need arises because AI companies split their crawlers by purpose: at OpenAI, GPTBot collects training data, OAI-SearchBot serves search results, and ChatGPT-User handles user-initiated link fetches; at Anthropic the equivalents are ClaudeBot, Claude-SearchBot, and Claude-User. Blocking a training crawler does not impede citation; blocking a search crawler impedes it directly. If you do not need that distinction, it is better not to create named groups: there is no gain and the risk of error is real.

WHAT IT GIVES THE BUSINESS

It does not improve visibility by itself. A mistake in it can nullify everything else.

WHAT IT IS

An XML file listing the pages of a site that should be indexed.

HOW IT WORKS

The crawler gets a complete list of addresses without relying on link traversal. It speeds up discovery of new pages and deeply nested sections.

HOW IT IS CREATED

Automatically, by any modern CMS or SEO plugin. Referenced by a Sitemap line in robots.txt.

AN IMPORTANT NOTE ON THE SPECIFICATION

The sitemap protocol is intended for indexable pages. Machine-readable resources such as JSON endpoints should not be placed there: crawlers do not discover them that way, and the Search Console report starts producing errors.

WHAT IT GIVES THE BUSINESS

Coverage completeness. Most noticeable on sites with many pages and weak internal linking.

WHAT IT IS

The practical embodiment, at the level of markup and editing, of what the source document calls atomic knowledge units.

HOW IT WORKS, AND THE STANDING OF THAT CLAIM

The generalized retrieval mechanism is described like this: the system does not read the whole site but splits texts into fragments, computes their vector representations, finds those closest to the question, and passes only those to the model. Hence the conclusion that the unit of optimization is a self-contained fragment.

That description is characteristic of systems built on retrieval-augmented generation. It is not a universal law governing all AI systems, and the largest platform says so directly.

Google's official guidance of 15 May 2026 states, in its mythbusting section, that there is no requirement to break content into tiny pieces for AI to understand it better, that Google's systems are able to understand the nuance of multiple topics on a page and show the relevant part to the user, that there is no ideal page length, and that pages should be made for the audience rather than for generative search. The same list of unnecessary tactics includes llms.txt, special markup, and rewriting text for AI [19].

It does not follow that the principles below are useless. It follows that they should be defended on grounds of clarity for the reader, rather than by an assertion about how the parser works. And separately: Google's statement concerns Google Search and does not describe the pipelines of ChatGPT, Claude, or Perplexity, which genuinely are built on passage retrieval.

The correct standing of this section: a cross-platform editorial heuristic, useful both to people and to retrieval systems, but not a requirement of AI search systems. Specific chunking strategies differ between systems and are in most cases unpublished.

HOW IT IS IMPLEMENTED ON A SITE

This is a discipline of writing and markup, not a technology. The first two points hold under any architecture; the rest are recommendations.

• The subject is named explicitly in the first sentence of a block. A paragraph beginning "This allows..." is useless after extraction in any system.
• Real h2 and h3 tags rather than containers styled to look like headings: this is markup, not styling, and everything reads it.
• One second-level heading, one complete thought, the answer inside that same section. Useful but not required: multi-topic pages are handled fine.
• The answer before the explanation. A section beginning with a definition is extracted more readily than one beginning "many companies today face...". This is a consideration of clarity, not a requirement.
• Be careful with references of the form "as noted above". Google states that its systems pick up the relationship between a claim near the top of a page and a conclusion much further down. But a fragment built entirely on a reference loses its meaning when extracted.
• Facts meant to be reused (price, timeline, coverage, credentials) are stated explicitly rather than dissolved into descriptive prose.

WHAT IT GIVES THE BUSINESS

The cheapest work in this entire document. It requires no new technology and no development budget, and it pays off with readers regardless of how AI systems behave. That is the ground on which it should be defended.

WHAT IT IS

Material that exists nowhere else: your own data, results from your own practice, a named methodology, observations and conclusions formed inside the company.

HOW IT WORKS

The mechanism is twofold.

The first concerns fragment selection. Retrieval picks the passage that answers the question, and it does so by vector proximity. If your text restates what fifty other pages say, the system has no reason to pick you over a more authoritative source with the same content. Your own numbers, results from specific cases, an observation from practice, all create fragments that exist nowhere else. When a query touches them, you have no competitors.

The second concerns the external channel. First-hand data gets cited by others, and third-party coverage is what supplies the bulk of AI citations [12].

HOW IT IS IMPLEMENTED

This is editorial work, not technical: publishing your own statistics, writing up real cases, articulating methodologies, taking a reasoned position on contested questions in your field. A formally measurable test: can a verbatim equivalent of your claim be found in another source.

DO AI MODELS KEEP AN ORIGINALITY SCORE

No. Models maintain no separate assessment of your page's uniqueness. Indirect mechanisms do the work: vector-proximity search means near-duplicate content competes against more authoritative copies of the same meaning, and Google's long-standing deduplication and content-usefulness mechanisms feed its AI features too, since Google confirms these run on the same ranking systems as ordinary search [19].

WHAT IT GIVES THE BUSINESS

The part of the work most resistant to copying. A competitor can reproduce your markup and structure in a week. Your own data they cannot reproduce.

WHAT IT IS

A questions-and-answers section where the answer text is present in the HTML immediately rather than being expanded on click by a script.

HOW IT WORKS

A question-and-answer pair structurally matches the shape of an extractable fragment: it is self-contained, its boundaries are obvious, and it answers a stated question. This is an argument about the form of the content, and it does not depend on markup.

A CRITERION THAT IS OFTEN STATED INCORRECTLY

The requirement is not that the answer be visually expanded, but that the answer text be physically present in the delivered HTML. If the markup looks like this:

<div class="answer" hidden>answer text</div>

and the script only toggles visibility, the text is in the document and a reading system receives it. The problem arises in a different case: when a click triggers a network request and the text is created only afterwards. Then it is absent for anyone who does not execute scripts.

In other words, a visual accordion is not in itself a violation. Deferred loading of the content is.

HOW IT IS IMPLEMENTED

A third-level heading with the question, a paragraph with the answer, all the text in the source server response.

ON FAQPAGE MARKUP SEPARATELY, WHERE THE PICTURE HAS CHANGED

As of 7 May 2026 Google stopped showing FAQ rich results in Search, and support in Search Console reports and the Rich Results Test is being withdrawn in stages in June and August 2026 [46]. This was the final stage of a three-year withdrawal: back in August 2023 eligibility had already been restricted to a narrow set of government and health sites.

The practical consequence: the one proven application of FAQPage markup has disappeared. The type remains a valid schema.org vocabulary term, it can be left on pages, Google confirms that unused structured data causes no problems, and other crawlers continue to parse it. But there is no basis for promising a visibility gain from it. The correct category for it today is optional semantic markup: add it if convenient, skip it if it creates maintenance overhead.

It is important not to fuse two separate questions. What lost its value is the markup. The content format, meaning an open question-and-answer pair in the page text, lost nothing at all, and the argument for it rests entirely on text structure rather than on schema.

WHAT IT GIVES THE BUSINESS

A direct channel: the phrasing of a question in an FAQ matches the phrasing of a user's query, and the answer is extracted whole.

WHAT IT IS

The correspondence of what is published to the current state of affairs. Two cases need distinguishing, because the requirements differ.

WHAT IS KNOWN, AND WITH WHAT CONFIDENCE

Two levels of claim are worth separating here, because they are frequently mixed.

The observation: pages not updated within a quarter lose citations three times more often, and roughly 30 percent of brands retain visibility across consecutive AI answers [9]. This is the result of an observational study on a large sample, and as an observation it is robust.

What does not follow from it: that the signals listed below carry precisely those weights in the algorithms. Ranking weights for ChatGPT, Claude, Perplexity, and Google AI Mode are unpublished, and reconstructing them from correlation is not possible. Moreover, regularly updated pages differ systematically on other dimensions too: they more often cover competitive topics, they are maintained by stronger editorial teams, they simultaneously accumulate links and improve their structure. Any of these factors could explain the difference.

What is known reliably is the set of signals these systems can observe at all: the Last-Modified response header, the lastmod field in the sitemap, the datePublished and dateModified properties in markup, the visible date on the page, and the actual difference in content between crawls. The last of these is the only one that cannot be faked, and that is a common-sense observation rather than a measured quantity.

DIFFERENT REQUIREMENTS FOR DIFFERENT PAGES

The quarterly rule applies to content whose answer changes over time: prices, service descriptions and scope, comparisons, guides, analytical material, anything answering the question "how do things stand now".

Permanent pages such as contacts, the about section, or a founder's biography do not need updating in that sense. What matters there is not freshness but accuracy and the absence of contradictions with external sources. A contacts page unchanged for two years is entirely fine if the phone number is correct and matches the directory profiles.

HOW IT IS IMPLEMENTED

A review calendar assigned separately by page type rather than applied across the board. Real change to content, not a cosmetic edit.

A SEPARATE WARNING

Updating the modification date without changing the content is a recognizable tactic and works against you. Many content management systems update dateModified on any save, including a typo fix or a plugin action, which devalues the signal. The date should reflect a substantive update.

WHAT IT GIVES THE BUSINESS

The only factor listed here that requires ongoing work rather than a one-time implementation. That same property makes it the natural basis for a recurring engagement.

WHAT IT IS, AND WHY THESE ARE TWO DIFFERENT THINGS

They are often spoken of as one, but they are separate layers.

schema.org is a vocabulary: an agreed list of entity types and their properties. Organization, Person, Product, LocalBusiness, FAQPage, and inside them properties such as openingHours, priceRange, hasCredential.

JSON-LD is a notation, one of three ways to express that vocabulary on a page (alongside Microdata and RDFa) and the one recommended today. The full correct phrasing: markup in JSON-LD format using the schema.org vocabulary.

HOW IT WORKS

A <script type="application/ld+json"> element holds an object in which each fact is labeled with a standard property name. The system does not have to guess that one string on the page is a phone number and another is a founding year.

HOW IT APPEARS ON A SITE

Via an SEO plugin, a theme template, or by hand. Placement is valid in both <head> and <body>; both positions are supported, and the settled convention favors <head>.

ON INJECTION VIA TAG MANAGERS AND SCRIPTS, WHERE PRECISION MATTERS

Google officially documents two methods of generating structured data dynamically, through Google Tag Manager and through custom JavaScript, and processes markup that appears in the DOM after rendering [45]. So for Google, the claim that "dynamic JSON-LD is invisible" is incorrect.

However, systems that do not execute scripts will not receive such markup, and they are the majority. The correct recommendation reads: for maximum cross-platform availability, structured data should be delivered in the source server response. Google will process script-injected markup, but a significant share of AI crawlers will not see it. Google's own recommendation also favors server-side delivery, since rendering adds a separate stage and a delay.

HOW MUCH THIS AFFECTS CITATION

One widely circulated figure is worth unpacking, because it is usually quoted without its second half.

Ahrefs analyzed 6 million URLs and found markup present on 53 percent of AI-cited pages against roughly 18 percent of non-cited ones, nearly three times as often [8]. That figure is cited everywhere as proof that markup raises visibility. The same team then tested whether the relationship was causal: they tracked 1,885 pages that added JSON-LD between August 2025 and March 2026, matched them against 4,000 control pages with comparable citation levels, and measured the change. There was no meaningful lift on any platform [10].

The explanation the authors themselves give for the gap: markup is found on better-maintained and technically more mature sites, and such sites simultaneously publish stronger content, accumulate more links, and rank better. The correlation reflects site quality, not the effect of markup [10].

Other measurements give a softer but uneven picture. AirOps together with Kevin Indig, across 16,851 queries and 353,799 pages, found a citation rate of 38.5 percent with JSON-LD present against 32.0 without it [42]. Otterly.AI presented at BrightonSEO in April 2026 the results of a sitewide rollout across more than 2,000 URLs: growth in Google AI Overviews and AI Mode citations alongside declines in ChatGPT, Gemini, and Copilot, and no effect in Perplexity [43]. The effect is platform-dependent, and the Google ecosystem responds noticeably more than the rest.

Both major search companies affirm the usefulness directly: Fabrice Canel of Microsoft stated in March 2025 that markup helps Microsoft's language models understand content for Copilot, and Google's structured data engineer Ryan Levering has said much the same [11].

A THIRD MEASUREMENT, THE MOST METHODOLOGICALLY RIGOROUS, AND IT NEEDS QUOTING IN FULL

A study published in February 2026 tested whether the presence of markup predicts citation, using 730 citations from ChatGPT and Gemini across 75 commercial queries with a control group drawn from Google organic results, 1,006 pages in total [41].

The headline result is null. After model correction, the presence of markup does not predict citation (OR = 0.678, p = 0.296), nor does entity richness (OR = 1.001) or schema-to-query alignment (OR = 1.068). Moreover, within Google's own results the share of pages carrying markup among cited and non-cited pages was statistically indistinguishable, 43.1 against 44.8 percent.

The dominant predictor turned out to be something else entirely: position in ordinary organic search results (OR = 0.762 per position, p < 0.001). Pages at position one were cited in 43 percent of queries; at position seven, in 5 percent. The authors' conclusion: citation is substantially mediated by the search backend ranking that precedes content evaluation at the AI level.

One exception within that null result: pages carrying Product or Review schema with populated concrete attributes (price, aggregate rating, specifications) were cited in 61.7 percent of cases against 41.6 for pages with generic types such as Article, Organization, and BreadcrumbList (p = 0.012). The advantage was most pronounced among lower-authority domains, which the authors read as factual payload in structured data partially compensating for weak authority signals.

HOW TO READ THIS CORRECTLY

One cannot conclude from it that rich markup raises citation. There is an obvious confounder: a page whose markup carries a price, rating, and specifications almost certainly contains those same concrete facts in its visible text as well. It is quite possible the system is selecting facts and the markup merely correlates with their presence. The authors themselves phrase the conclusion cautiously: markup with extractable factual content may confer a modest advantage on lower-authority domains, whereas generic markup confers none, and the primary lever remains organic ranking position.

A caveat on the status of the source: this is a preprint that has not been peer reviewed, the author represents an agency in this same field, the sample is small, and a note discloses AI assistance in research design and manuscript preparation.

THE PRACTICAL CONCLUSION THAT DOES NONETHELESS FOLLOW FROM ALL OF THIS

It is not about a citation lift but about whether the work is meaningful: if you are going to add markup, it should carry verifiable facts rather than marketing copy. An Organization type carrying nothing but a name and a logo gives the system nothing under any scenario. The same type with an address, opening hours, a price range, external profile identifiers, and a license at least carries an extractable claim. This is an argument that empty markup is pointless, not that full markup has proven benefits.

THE CORRESPONDENCE REQUIREMENT, AND ITS EXACT BOUNDARIES

Google's rule is that structured data must represent the main visible content of the page and must not be applied to content hidden from the user [50]. A discrepancy here is classified as a policy violation, not a technical defect. In practice: do not mark up a price that is not on the page, do not mark up reviews the user cannot see, do not mark up questions whose text differs from what is visible.

WHAT THAT REQUIREMENT DOES NOT MEAN

It concerns factual claims addressed to the user, not all markup indiscriminately. Machine-oriented properties are not required to appear as text and are not hidden claims: @id, sameAs, canonical identifiers, machine-readable URLs, organization identifiers, technical relationships between entities. Their presence in markup while absent from visible text is normal and useful.

The distinction matters practically: an audit conducted against an over-broad reading of the rule leads to the removal of sameAs and @id, precisely the properties responsible for binding to external anchors.

WHAT IT GIVES THE BUSINESS

It is cheap, it is set up once, it pins down entity identity and removes ambiguity in how facts are understood. Some types still unlock rich results in ordinary search, but the list of such types is shrinking: FAQ was removed from it in May 2026 and HowTo earlier [46]. There is no basis for treating markup as a lever for AI citation growth. There is a basis for treating it as inexpensive hygiene that removes ambiguity.

WHAT IT IS

Records about your organization or about you in sources you do not control and that external systems already trust.

HOW IT WORKS

A system building knowledge about the world needs to determine that mentions under different names refer to the same object. That process is called entity resolution. An anchor is a record to which such mentions attach. Its defining property is that it has passed through someone else's process: moderation, verification, licensing, editorial rules.

SOURCES DIFFER GREATLY IN EVIDENTIAL WEIGHT, AND PUTTING THEM IN ONE LIST IS INCORRECT

Below they are separated by tier.

What all of these share is that the record lives outside your control and participates in stitching mentions together. What differs is how far it constitutes evidence.

HOW EACH IS CREATED

Each by its own rules: a claim and verification in Google Business Profile, editing Wikidata under community rules, maintaining an entry in a professional registry, completing a profile in an industry directory. The work is organizational, not technical.

WHO READS THEM

Every system building a knowledge graph, and every retrieval system indirectly, through the sources that rely on those records.

WHAT IT GIVES THE BUSINESS

It raises external systems' confidence that mentions under different names refer to the same object, and it adds corroboration wherever the source genuinely verifies. A simple test of the distinction: if you can change the record tomorrow without anyone's agreement, that is self-declaration in someone else's environment, not independent corroboration. Your own site is neither.

The phrase "this turns a claim into a confirmed fact" should be applied only to the top tier of the table. For the rest it is more accurate to speak of independent corroboration and confidence in entity resolution.

WHAT IT IS

Articles, reviews, ratings, discussions, and profiles where someone else talks about you.

HOW IT WORKS

A retrieval system answers on the basis of what it found for the query, and what it finds is predominantly not your site. An analysis of more than 25 million links found that earned third-party coverage accounts for 84 percent of AI citations, with the figure holding between 82 and 89 percent across three editions of the study [12]. By other data roughly 85 percent of brand mentions originate on external domains, making citation via third-party coverage 6.5 times more likely than via owned properties [13].

AN IMPORTANT CAVEAT AGAINST OVERGENERALIZING

The picture is platform-dependent. In Google AI Overviews the share of brand-owned sites is the highest across platforms, around 59.8 percent of citations [13]. The direction is common; the proportions are not.

HOW IT IS CREATED

Publications in industry outlets, expert commentary, presence in reviews and rankings, customer reviews, participation in domain-specific discussion.

WHAT IT GIVES THE BUSINESS

Judging by converging measurements, the weightiest contribution to the probability of being named, and at the same time the only element that can be neither coded nor configured.

A caveat on the strength of that claim: this is a robust observation across large samples, not an established law of ranking. Behavior differs between platforms and between query classes, as the Google AI Overviews difference above already shows.

WHAT IT IS

Your own database in which every fact about the business exists in one place, in structured form, with its source, verification date, and publication level recorded.

HOW IT WORKS

It solves the problem of data drifting apart. A company's phone number appears in the site header, the footer, the contacts page, the markup, the Google profile, three directories, and a newsletter signature. When it changes, half those places get updated.

The practical form of the registry looks like this:

HOW IT IS CREATED

A spreadsheet in Airtable or Google Sheets is enough to start. In mature form it becomes a database from which text, markup, and directory feeds are generated.

WHO READS IT

Nobody externally. It is an operational tool, and there is nothing wrong with that: it delivers three things otherwise unattainable. A single source for generating everything else. A reference against which live pages can be compared. Control over what is permitted to be published at all, which is critical in regulated industries.

AN IMPORTANT TERMINOLOGICAL CLARIFICATION

The word "canonical" here carries an operational sense, not an authoritative one. The registry is canonical for you: it determines which version of a fact is correct inside the organization. External systems do not treat it as a source of truth, because any organization can write anything about itself on its own domain. Authority arises in the anchors from the previous section. The precise formulation of the relationship: the registry is the input, the anchors are the output, and consistency between them is what reads as credibility.

WHAT IT IS

Three related but distinct concepts from the source document's final layer.

• Staleness. A fact has not been confirmed for longer than its expected interval. This is about the absence of change.
• Drift. The registry says one thing; a live page or an external profile says another. The price was updated in the system but the site still shows the old one. The phone was changed on the site but not in the directory.
• Synchronization. The mechanism for eliminating drift, on a schedule or on an event.

HOW IT IS IMPLEMENTED

Technically this is the simplest part of the entire architecture. A scheduler fetches live pages, extracts markup and key text fragments, compares them against the registry, and sends a notification on mismatch. An extension: the same reconciliation against external profiles via their APIs.

WHO READS IT

Your own monitoring. There is no external consumer, and none is needed.

WHAT IT GIVES THE BUSINESS

It rests on the measurable effect of currency [9] and eliminates contradictions between sources, which directly reduce the chance that a system will repeat your version of a fact. Beyond that, it is the only element of the architecture requiring ongoing work, which makes it the natural basis for a recurring engagement.

WHAT IT IS

A Markdown text file at the site root: a heading, a short project description, and sections with links to key material and one-line notes on each.

HOW IT WORKS

When an AI tool fetches an ordinary page it receives HTML together with navigation, consent banners, advertising scripts, and a footer. All that structural noise competes for a limited context window against the content that is actually needed. Companies serving Markdown instead of HTML have reported token consumption falling by up to a factor of ten [2]. The file provides a map: here are the sections, here is what is in each.

HOW IT IS CREATED

By hand or with a generator, placed at the site root.

WHO READS IT TODAY

Precision matters here, because a great deal of contradictory material circulates about this file.

AI search crawlers barely request it, and five independent measurements of differing kinds confirm this. Otterly.AI: 84 requests out of more than 62,100 AI bot visits over 90 days, or 0.1 percent, worse than an average page of the site [14]. Limy.AI: 408 requests out of 515 million events over 90 days, filtered to GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, and Google-Extended [15]. Ahrefs across the logs of 137,000 domains: 97 percent of files received zero requests [16]. Weekerp: zero requests out of 68,759 AI bot requests over a month, and the most frequent requesters of the file were not AI systems but site analysis tools [17]. Trakkr across 37,894 AI-cited domains: zero advantage [18].

Google's position is stated plainly: in its updated generative search guidance of 15 May 2026, llms.txt is placed among the markup not required for AI features [19], while earlier Gary Illyes confirmed the absence of support and John Mueller compared the file to the obsolete keywords meta tag, on the grounds that a self-declared signal cannot be verified [20].

Real consumers of the file do exist, they are simply different: agents in development environments (Cursor, Claude Code, Copilot, Windsurf), MCP servers for documentation access, and assistants embedded in SaaS products working over their own documentation. The list of who publishes the file is telling: OpenAI, Anthropic, Stripe, Cloudflare, Mastercard, Vercel, Microsoft Teams SDK [2]. These are platforms whose buyer is a developer.

Since May 2026 the file is also checked automatically: Lighthouse 13.3 on 7 May added the Agentic Browsing category to the default configuration, and PageSpeed Insights inherits it within about two weeks [21]. What is checked is not only presence but format: a first-level heading, sufficient length, the presence of links [22]. An important detail: when the file is absent the audit returns not-applicable rather than a failure, since providing the file remains optional. So the existence of the check in the tool should not be read as Google expecting the file.

WHAT IT GIVES THE BUSINESS

For products with documentation and APIs, real value for the buyer's agents. For local services and most B2B service businesses there is no practical benefit, but the cost is close to zero, and with the arrival of the Lighthouse audit the file's absence became visible to anyone running a check.

WHAT IT IS

The extended variant: not a map of links but the entire documentation text collected into one file.

HOW IT WORKS

A tool receives all the content in a single request instead of dozens of page fetches.

HOW IT IS CREATED

Generated from documentation sources, updated when they change.

WHO READS IT

The same audience as llms.txt.

A REAL RISK, STATED PRECISELY

A common approach is to create separate Markdown copies of every page on a site. At scale this is usually pointless and creates surplus URLs that consume crawling and generate noise in canonical selection.

A caveat against overstating it: duplicate content is not in itself a spam policy violation for Google; the engine groups duplicates and selects a canonical version [51]. So this is about inefficiency and canonicalization noise, not ranking suppression.

The practical conclusion: if such representations are needed by agents, define a canonicalization and indexing policy for them and do not spawn uncontrolled alternative URLs. There is no universal requirement to block such a file from indexing.

WHAT IT IS

The accessibility tree is a simplified representation of a page's structure that the browser builds for assistive technologies. Layout stability is measured by the CLS metric.

HOW IT WORKS

In its guidance on building agent-friendly sites, Google notes that agents work with content in three formats: screenshots, HTML source, and the accessibility tree [22]. The tree is processed more efficiently than full HTML and far more efficiently than a screenshot. Stability matters because agents that take screenshots get confused when the layout keeps shifting.

HOW IT IS IMPLEMENTED

Ordinary accessibility work: correct roles, labels, field labels, a meaningful heading structure, reserved space for images and embeds.

WHO READS IT

Browser agents and screen readers. This is a rare case where a single piece of work serves human accessibility, agent readiness, and ordinary quality metrics at once.

WHAT IT GIVES THE BUSINESS

The most rational entry point into the agent topic: the work pays off regardless of whether agent forecasts come true, because people need it and search engines account for it.

WHAT IT IS

A way of serving data to programs rather than to a browser. A JSON profile is the simplest form: a single static document at a fixed address holding facts about the organization.

HOW IT WORKS

Instead of parsing a page, a program requests an address and receives a structured response: {"name": "...", "founded": "...", "phone": "..."}. It does not have to guess where on the page the phone number sits.

HOW IT IS CREATED

A dedicated route in the application, or a static file. WordPress ships with a REST API.

WHO READS IT TODAY

Only those explicitly given the address. There is no mechanism by which a crawler would discover and call your endpoints on its own: it makes one request for HTML and leaves. This is the key difference from JSON-LD markup, which sits inside the page and therefore reaches a reading system automatically.

WHAT IT GIVES THE BUSINESS

Useful where data changes often and there is a specific consumer: partners, your own applications, integrations. As a visibility instrument it does not work today.

WHAT IT IS

A formal machine-readable description of an API: which addresses exist, what parameters they take, what they return, how to authenticate. Formerly called Swagger.

HOW IT WORKS

A single YAML or JSON file replaces documentation written for humans. Clients, test harnesses, and agent tool definitions are generated from it.

HOW IT IS CREATED

Written by hand or generated from code. Many frameworks do this automatically.

WHO READS IT

Developers and agent frameworks to which the API has already been connected. This is precisely why both agents.json and AI Manifest build on top of OpenAPI: it is a ready-made contract that does not need reinventing [23].

WHAT IT GIVES THE BUSINESS

Any interface intended to be called externally needs a machine-readable contract. OpenAPI is the standard for HTTP and REST; GraphQL, gRPC, and event-driven interfaces have their own forms of schema description, and substituting OpenAPI there is unnecessary. The requirement should be phrased as: an interface must have a machine-readable contract, and for REST that contract is OpenAPI. It is not a visibility signal in any form.

WHAT IT IS

A protocol standardizing how a language model connects to external tools and data.

HOW IT WORKS

A client (Claude, Cursor, an IDE) connects to a server. The server declares a list of available tools with typed parameters. The model picks the one it needs and calls it. The integration is written once against the protocol rather than again for each model-service pair.

HOW IT IS CREATED

By writing an MCP server that wraps your business logic. This is development, not configuration.

WHO READS IT

MCP clients, on explicit connection. Two levels need separating here, and they are easily conflated.

The protocol itself contains no mechanism by which an agent arriving at an arbitrary site would discover that site's MCP server: it was designed on the assumption that the server being connected is already known [24]. Hence the attempts to close that gap with the manifests in the next group.

The ecosystem has already acquired such a layer. The official MCP Registry launched in preview on 8 September 2025 as an open catalog and API for discovering publicly available MCP servers, with participation from Anthropic, GitHub, PulseMCP, and Microsoft; the catalog specification is open, which allows compatible sub-registries to be built, and in October 2025 the API was frozen at v0.1 [47]. Third-party catalogs and marketplaces exist alongside it.

So the claim that "discovery does not exist" is too broad today. Discovery exists, but it is catalog-based rather than site-based: an agent finds your server in a registry, not on your domain.

WHAT IT GIVES THE BUSINESS

It becomes meaningful if you have a product with actions or data someone will want to automate: SaaS, an API service, a platform, a data source.

An important clarification about entering someone else's workflows. Suppose a site provides weather forecasts and its owner wants the source used in other people's agentic solutions. A technical interface is necessary here but not sufficient: nobody will discover your interface on their own. A developer chooses a source by documentation, reliability, price, and license terms, and finds it in specific places: MCP server catalogs, package managers, repositories, API aggregators, reviews and comparisons. So the work splits in two: the interface itself with its specification, and distribution. Distribution for an MCP server has a concrete address today: publication in the official MCP Registry and in relevant third-party catalogs [47], plus package managers and documentation. This is the one scenario in which the whole of Group 3 is not preparation for the future but the product itself.

WHAT IT IS

A Microsoft project presented at Build 2025. Its author is R.V. Guha, who was involved in creating RSS, RDF, and Schema.org.

HOW IT WORKS

A site stands up two endpoints. The first takes a natural-language question and returns a structured answer in schema.org terms. The second exposes the same functionality as an MCP server so agents can list tools and call them [25]. Under the hood is search over the site's own content with answer generation. Microsoft frames the ambition this way: NLWeb could play a role for the agentic web analogous to the role of HTML.

HOW IT IS CREATED

By deploying the open-source project and indexing your own content. Cloudflare offers a managed variant that crawls the site, indexes it, and stands up the endpoints, but in their documentation it is marked as public preview for experimentation, with production use directed to contacting them directly [26].

WHO READS IT

Experimental deployments. Named early participants: Shopify, Snowflake, O'Reilly, Tripadvisor, Eventbrite, Hearst.

WHAT IT GIVES THE BUSINESS

Most meaningful for catalogs, large content libraries, and e-commerce, meaning wherever there is a lot of content and it is structured.

WHAT IT IS

The same idea as MCP but inside the browser and without a server component.

HOW IT WORKS

A page registers typed functions via navigator.modelContext; a declarative variant using HTML form markup also exists. A browser agent receives a list of what can be done and the exact parameters, instead of taking a screenshot and guessing where to click. Control is inverted: rather than the agent parsing the interface, the site declares its capabilities. The saving is substantial: a structured tool call consumes on the order of 20 to 100 tokens against more than 2,000 for a single screenshot [27].

HOW IT IS CREATED

Two paths.

Declarative, if the form already exists: attributes on the existing markup, from which the browser synthesizes a field schema.

<form toolname="book_consultation" tooldescription="Book a consultation">
  <input name="full_name" toolparamdescription="First and last name">
  <input name="phone" toolparamdescription="Contact phone number">
  <input name="preferred_date" toolparamdescription="Preferred date, format YYYY-MM-DD">
</form>

Imperative, when control over the logic is needed:

if (window.isSecureContext && navigator.modelContext) {
  navigator.modelContext.registerTool({
    name: "book_consultation",
    description: "Book an initial consultation",
    inputSchema: {
      type: "object",
      properties: {
        full_name: { type: "string" },
        phone:     { type: "string" },
        preferred_date: { type: "string" }
      },
      required: ["full_name", "phone"]
    },
    execute: async (args) => { /* booking logic */ }
  });
}

Implementation constraints: secure context only, top-level browsing context only. The interface surface changes between revisions: the provideContext and clearContext methods were removed in the March 2026 revision, leaving registerTool and unregisterTool; the attribute names of the declarative variant are given differently across sources because the draft is still being edited [44]. Writing code today is possible, but on the expectation of rewriting it.

ON SENSITIVE DATA SEPARATELY

Technically nothing prevents declaring a tool that collects information about health, financial circumstances, or other protected categories. It should not be done, and the reason is not convenience.

The agent is a separate party in the chain. What a person tells an agent enters its context window and may be transmitted to the operator of the AI system. For an organization this is an additional data transmission channel that did not previously exist.

Whether such a channel is permissible is not universal and depends on the industry, jurisdiction, contractual relationship with the model operator, the presence of a data processing agreement, enterprise plan configuration, and who in the specific arrangement acts as controller and who as processor. No general conclusion of "this is not allowed" or "this is allowed" follows from that, and drawing one in a document of this kind would be wrong.

The engineering principle holds regardless of the legal characterization: data minimization. An agent-facing tool collects the minimum fields sufficient for the action, meaning contact and time, and everything else is collected through your own channel after identity confirmation. For protected data categories, the permissibility of the channel is assessed separately and before launch.

Separately and without qualification: do not expose tools that read existing records, or the agent gains the ability to enumerate other people's data. This is a security question, not a compliance one.

WHO READS IT TODAY, AND PRECISION IS NEEDED HERE

Firm facts: the specification is being developed in a W3C Community Group, with editors from Google and Microsoft, announced on 10 February 2026, with a public origin trial in Chrome from version 149 and a dedicated audit in Lighthouse [21][27]. Vendor statements: Google named Expedia, Booking.com, Shopify, Credit Karma, TurboTax, Redfin, Etsy, Instacart, and Target among those experimenting, but that is an origin trial announcement, not confirmed production deployment. An independent assessment as of July 2026: the standard has everything except users, deployment on real sites is close to zero, and mainstream agents do not call these tools [28]. The first news from the consumer side came on 26 August 2026, when OpenAI reported WebMCP support in the ChatGPT desktop browser [29].

WHAT IT GIVES THE BUSINESS

A bet on the coming years for anyone with transactional scenarios: purchase, reservation, booking, application submission. For everyone else it is premature.

WHAT IT IS

An open specification from Wildcard AI describing API-to-agent interaction contracts on top of the OpenAPI standard. Current version 0.1.0 [23].

HOW IT WORKS

OpenAPI describes individual endpoints but does not describe that solving a practical task requires several calls in a particular order. An example from the specification's own documentation: the Gmail API has separate endpoints for searching threads, listing messages in a thread, and sending a base64-encoded reply, and the agent needs to understand their sequence. agents.json adds the notions of flows (call chains) and links (substituting the result of one call into a parameter of the next). Its principles: build on OpenAPI, optimize the schema for models rather than for humans, leave orchestration to the calling agent, require minimal changes to existing APIs [23].

HOW IT IS CREATED

A JSON file referencing an existing OpenAPI specification.

WHO READS IT

Agents specifically configured to work with this specification.

A COMMON ATTRIBUTION ERROR

This specification is sometimes described online as an Anthropic standard. That is incorrect; the author is Wildcard AI.

WHAT IT IS

A Community Draft at version 0.1: an attempt to give a site a single machine-readable business card for agents. It lives at the standard path /.well-known/ai.json. The .well-known directory is a long-established convention: addresses within it are reserved for service files, so an agent does not have to guess where to look and simply requests a fixed address [32].

The manifest itself describes what the organization is, which machine interfaces it has, and where they are located. All detailed interface description is delegated outward to a separate OpenAPI or JSON Schema specification that the manifest references: the author did not reinvent interface description.

WHAT THE KEYS AND SIGNATURES ARE

The third component of the bundle addresses the question of whether the domain owner really did publish this. It uses a commonly accepted set of formats collectively known as JOSE, with public keys published in a separate file at /.well-known/jwks.json. It works like this: the organization signs the manifest with its private key, and anyone can take the public key from the published directory and verify the signature. There is also /.well-known/ai-crl.json, a revocation list: if a key is compromised, an entry is added and signatures made with that key cease to be considered valid.

A practical analogy: the manifest is a business card, the key directory is a signature specimen posted in plain view, and the revocation list is a notice that the old specimen is no longer valid.

HOW IT WORKS OVERALL

Its claimed properties: discoverability through standard placement, interoperability through reliance on OpenAPI, MCP, and agents.json, and verifiability through the key directory.

HOW IT IS CREATED

A JSON file at the stated path. Validators and a ready-made continuous integration workflow exist.

WHO READS IT

Nobody by default. The registry is voluntary and the specification has not passed through IETF or W3C.

WHAT IT IS

A format in the spirit of robots.txt, plain text, renamed from agents.txt in March 2026.

HOW IT WORKS

It declares permissions by usage type (training, retrieval-augmented generation, actions), a preferred interface, the address of an OpenAPI specification, the address of an MCP server, plus blocks for individual agents with authentication requirements [33].

WHY THE AUTHORS BELIEVE IT WILL BE HONORED

Two arguments. The first is rational: when a site declares an MCP server or an API, a well-built agent is better off using them, since that is faster and more reliable than parsing pages. The second is legal: a published machine-readable policy materially strengthens a position in disputes over unauthorized access when it has been ignored [33]. The second argument is non-trivial and carries independent value for content rights holders.

WHO IT APPLIES TO

Media, databases, publishers, meaning anyone for whom unauthorized content collection is a commercial problem.

WHAT IT IS

The reverse direction from all the preceding. This is a manifest of the agent itself rather than of a site: an agent publishes its identity, capabilities, and addresses at /.well-known/ai-agent.json so other agents can discover it [30]. The specification was published on 28 March 2026.

Alongside it exist ai-me.json (site identity) and /.well-known/ai-catalog.json, which lists a publisher's agentic resources: MCP servers, agent cards, skill sets, and ordinary HTTP interfaces.

WHO IT APPLIES TO

Agent developers, not owners of ordinary websites.

WHAT IT IS

Four fields accompanying every fact in the registry.

• Provenance. Where the fact came from. "License No. 123, source: state registry, link."
• Verification date. When the fact was last checked. Not when the page was edited, but when the content was reconciled.
• Version history. Previous values. The price was 200 until March, then 250. This lets you explain discrepancies with copies cached by external systems.
• Verification status. A flag for "confirmed by evidence" against "asserted". The difference between "500 clients per audit" and "500 clients per our own account".

HOW IT IS IMPLEMENTED

Additional columns in the registry from Group 2. Technically trivial, organizationally a matter of discipline.

WHO READS IT

The internal loop. External systems do not request these fields.

WHAT IT GIVES THE BUSINESS

In regulated industries this is not bureaucracy but a requirement: the ability to show where every public claim came from and who is accountable for keeping it current. Plus the basis for staleness detection.

WHAT IT IS

Two different mechanisms that are frequently confused.

A hash is a short string computed from content. Change a single character and the string becomes entirely different. It answers the question "has the content changed".

A signature is the output of a signature algorithm that takes a message and a private key as input and produces a value verifiable by anyone holding the corresponding public key. Verification answers two questions at once: whether the message has changed, and whether the signature corresponds to that specific key.

The common explanation that "a signature is a hash encrypted with a private key" describes the particular case of classic RSA and is incorrect as a general definition: modern algorithms such as Ed25519 work differently and perform no encryption of a hash at all. The practical meaning is the same; the internal mechanics differ.

HOW IT IS IMPLEMENTED

Hashing with standard libraries. Signatures require key management: generation, storage, rotation, revocation. The formats are described in AI Manifest via JOSE and JWKS.

WHAT THIS DOES NOT SOLVE, AND THE SOURCE DOCUMENT NOTES IT CORRECTLY

A signature proves origin and integrity, not truth. A signed false claim remains false; it is simply now known who signed it.

WHO READS IT

There is no external consumer for signed business data today. No AI crawler verifies a publisher's signature.

WHAT IT IS

The idea from the source document: an AI system that has encountered a claim about a company somewhere queries the company's endpoint and receives an answer as to whether the claim matches reality.

WHAT PROBLEM IT SOLVES

Publishing facts on your own site does not correct outdated or distorted information about you that has already spread across aggregators, directories, and discussions. Reverse verification in theory closes exactly that gap: instead of you publishing and hoping to be read, the consumer comes to check before repeating.

CURRENT STATE

There is no consumer. Three reasons: the system does not know the endpoint exists; it has no grounds for accepting an answer from an interested party; and this is an additional network round trip in a chain everyone is optimizing for speed. The second reason is fundamental: it is the same self-declaration defect that has historically kept self-declared signals from taking hold.

WHAT EXISTS NEARBY, AND IN WHAT CONDITION

Google's ClaimReview schema and Fact Check Tools API. ClaimReview is a markup system with which publishers tag verified claims, developed through an open process involving the Duke Reporters' Lab, the fact-checking community, Google, Bing, and Jigsaw [34]. The API provides search over verified claims and an interface for managing the markup [35]. The markup is submitted to DataCommons.org, with the publisher first authorized through Search Console [34].

The precise status today: ClaimReview rich results were removed from Google Search in June 2025, in a batch alongside six other markup types [49]. So it is no longer a search visibility mechanism. Fact Check Explorer and the Fact Check Tools API continue to operate, meaning the instrument lives inside the specialized fact-checking ecosystem rather than in general search.

It should not be read as a visibility instrument. Its value here is different and architectural.

The fundamental difference from the reverse verification idea: there the claim is checked by a third party rather than by the subject of the claim. It is precisely the externality of the verifier that creates trust. This is the direction in which the source document's idea could be developed into something workable.

WHAT IT IS

An industry standard for attesting the provenance of media files.

HOW IT WORKS

A signed manifest recording creation and edit history is attached to the file. Three stages: signing, embedding, verification, with verification requiring no network call back to the signer, since all necessary certificates travel inside the manifest [36]. Version 2.3 was released in January 2026.

WHO SUPPORTS IT

Adobe, OpenAI, Google in Gemini, Search, and Chrome, Samsung in the Galaxy S25 line, Canon. Regulatory pressure is real: Article 50 of the European AI Act and California's SB 942 require machine-readable disclosure of generated content, and the standard supplies the cryptographic layer for that [36].

A KNOWN LIMITATION

A tool without support for the standard can save an image while silently stripping the manifest container. The absence of a manifest does not prove forgery; it means only the absence of verifiable provenance [36].

WHO IT APPLIES TO

Media, stock photography, newsrooms, generative services. Not applicable to business facts: the standard concerns images and video.

WHAT IT IS

A mechanism by which a bot cryptographically proves its identity on every request.

HOW IT WORKS

Request signing per the HTTP Message Signatures standard (RFC 9421), an Ed25519 key, a Signature-Agent header, and a published directory of public keys. This replaces the spoofable user-agent string and brittle IP lists with provable identity [37]. In February 2026 Cloudflare, together with Amazon Bedrock AgentCore, introduced an open registry format in which each entry carries the agent's name, operator details, expected request rate, and keys [38]. The IETF has chartered a working group [37].

PRACTICAL APPLICATION ALREADY EXISTS

Sites behind Cloudflare can enable rules of the form "show product details only to verified bots" or "publish prices only to verified agents" [38].

CURRENT MATURITY

As of 12 August 2026 the IETF datatracker lists nine active drafts on this subject, all nine individual submissions, none formally adopted by a working group [37].

WHY THIS MATTERS IN THE CONTEXT OF THE SOURCE DOCUMENT

The industry's direction of travel on cryptographic trust runs the opposite way from the sixth layer. The sixth layer supposes that you sign your data so that it will be believed. The industry is building a mechanism in which the agent's request is signed so that you can decide whom to admit. This is not a contradiction of the idea but a different vector of the same need, and it is the one with deployments and money behind it.

WHO IT APPLIES TO

E-commerce, publishers, anyone being scraped heavily.