The audit doesn't start when you press submit. It starts the moment the crawler requests your URL with a raw HTTP GET, strips every visual layer your browser would render, and reads your site the way a language model does: as a stream of tokens, structured signals, and semantic relationships. Most auditing tools measure what your site looks like; this engine measures what it means to a machine. Six independent Audit Architects each test a layer of AI Readability, and their weighted outputs sum to a 0-to-100 score that maps to real-world citation probability.
1. The Audit Hypothesis: Websites Are Built for the Wrong Reader
When a user asks ChatGPT, Perplexity, or Gemini a question in 2026, the AI isn't clicking a link and reading your page like a human. It's either working from training data ingested months ago, or running a live retrieval step where a RAG pipeline fetches your content, chunks it, embeds it into a vector, and runs a similarity search against the query. In both cases your ability to be cited depends on one thing: whether the raw signal your HTML produces is clean enough, structured enough, and semantically distinct enough to survive that pipeline intact. The audit engine simulates exactly that process. The score is not a grade for your design, it's a diagnostic of your machine-readable infrastructure, the composite quality measured across the 1,500-site forensic study.
2. The Six Audit Architects
Each Architect tests a specific failure mode the forensic study identified as a primary cause of AI invisibility. They run in parallel and their weighted outputs sum to the final score.
Architect 1: Rendering Signal (the "Empty Shell" test)
The engine first checks whether your HTML is rendered server-side or depends on client-side JavaScript hydration. Modern frameworks (Next.js in CSR mode, Vue, Angular) ship an empty shell to the initial HTTP response; the page looks complete in a browser because the browser executes the JS and populates the DOM, but an AI crawler that doesn't execute JavaScript receives a blank document. The engine makes two requests, one standard GET and one mimicking a non-JS bot, then compares the readable token count between them; a significant delta, where core content like headings, body text, and pricing only appears post-hydration, triggers a penalty. This matters because Perplexity's retrieval agents, Common Crawl's ingestion pipeline, and most RAG tooling operate below the 200ms response threshold and do not wait for hydration, the failure dissected in the empty shell audit. The fix is not to abandon your framework, it's to ensure SSR or SSG delivers the full semantic payload in the initial response.
Architect 2: Schema Validity (the structured-data audit)
Structured data is the closest thing to a direct API contract between your site and an AI system: implement Schema.org markup correctly and you're not asking the AI to infer what your page is about, you're telling it explicitly. This Architect runs three layers. A presence check (is any JSON-LD, Microdata, or RDFa present? 70% of studied pages had none). A type-specificity check (generic Organization or Article with only a name and URL gives minimal signal; the engine looks for rich types like TechArticle, Product, FAQPage and high-signal properties like knowsAbout, sameAs, mentions). And a nesting-integrity check for "orphaned schema," where a Product entity exists but its Offers, AggregateRating, or MerchantReturnPolicy children are missing or disconnected, telling the AI a product exists but not its price, availability, or return terms, a complete disqualification for buying agents. Well-defined entities with unique sameAs associations also reduce the risk of being classified as a near-duplicate and excluded by Google's GIST algorithm.
Architect 3: Token Efficiency (the signal-to-noise ratio)
LLMs operate on token budgets, and every tag and inline CSS class your page serves costs tokens. If 80% of those tokens are structural noise (navigation, footer boilerplate, tracking remnants, decorative classes), the AI spends most of its budget before reaching your content. This Architect computes your semantic density ratio: strip <script>, <style>, <nav>, <footer>, and <header>; tokenize the remaining text with a BPE approximation; divide by the total token count of the raw HTML; and compare against the benchmark distribution. A page serving 150KB of HTML for 500 words has a ratio near 0.03, and since most RAG pipelines truncate after 512 to 2,048 tokens, value that appears after the truncation point because of HTML overhead is never retrieved, the principle quantified in the token tax work.
The first 100 tokens of readable content, what an LLM ingests before any chunking boundary, must contain your entity name, your primary service or product category, and your primary differentiator. If that information appears only at the 800th token because the first 700 were navigation and boilerplate, you fail the retrieval test even if your content is excellent.
Architect 4: Entity Clarity (the knowledge-graph signal)
This is the most misunderstood signal in AEO. Entity clarity isn't about spelling your company name correctly, it's about whether an LLM can unambiguously resolve who you are in a global knowledge graph. A "string" is text ("Apple"); an "entity" is a node (Apple Inc., the technology company in Cupertino, Wikidata Q312). Without disambiguation markup, an AI has to guess which Apple you mean, and in a RAG pipeline that ambiguity causes retrieval failures or hallucinated associations. The Architect checks for sameAs properties linking to canonical authority nodes (Wikidata, LinkedIn, Crunchbase, Wikipedia), author disambiguation via verified Person entities, and "About This Result" verifiability as a proxy for entity-resolution strength. The fix for most sites is a single well-structured Organization JSON-LD block in the site-wide head, with sameAs linking to at least three canonical references, the entity home pattern.
Architect 5: Crawl Access (the front-door audit)
Your robots.txt is the access policy for the AI economy; get it wrong and nothing else matters. This Architect parses robots.txt for three conditions. Unintentional blanket blocks: a User-agent: * / Disallow: / rule, often left from staging, blocks GPTBot, PerplexityBot, and ClaudeBot; 30% of audited sites had such blocks and none of them were a strategic decision. Selective-blocking accuracy: there's a legitimate distinction between training bots (CCBot, feeding Common Crawl and most LLM training data) and retrieval bots (GPTBot, ClaudeBot), and a site may block training while permitting real-time retrieval; the Architect validates whether your config achieves your actual intent. llms.txt presence: the llms.txt standard, a markdown map of your most valuable content, is the new sitemap.xml for the AI web, and adoption in the study was 0.2% (3 of 1,500), a significant first-mover advantage detailed in the llms.txt guide.
Architect 6: Semantic Structure (the HTML hierarchy audit)
This is the oldest signal and still among the most violated. Semantic HTML is the document outline LLMs use to understand which claims belong to which topics. The failure modes: header hierarchy abuse (developers use <h4> and <h5> for small decorative text, and a jump from <h1> to <h4> breaks the outline, so an LLM chunking by headers misassigns supporting points to the wrong parent claims); chunking boundary misalignment (fixed-size chunking is the default, and a broken hierarchy gives the chunker no structural fallback, the problem dissected in sliding window chunking); and <details> / <summary> optimization (these native elements, used for Q&A patterns, are recognized by several retrieval systems as high-confidence answer candidates because their structure explicitly signals "this is a question, here is its answer").
3. The Score: What 0 to 100 Actually Measures
The six Architects produce weighted sub-scores. Schema Validity and Token Efficiency carry the heaviest weights because they have the highest variance in the dataset and the most direct impact on retrieval quality. The final score is a composite.
| Score | Classification | Practical meaning |
|---|---|---|
| 0–40 | Invisible | AI systems cannot reliably parse or retrieve your content. Citation probability is near zero. |
| 41–70 | Readable | AI can extract basic information but lacks the structured signals to use your site as a primary source. |
| 71–100 | Optimized | Your content is structured for reliable retrieval, entity resolution, and AI citation. |
A score of 71 or above doesn't guarantee citation, the quality of your content and its semantic distinctiveness relative to competitors remain critical. What it does guarantee is that the structural barriers to citation have been removed.
4. The Validation Loop: Confirming Fixes Actually Work
The most important feature isn't the initial audit, it's the re-audit. SEO has always had a verification problem: you implement a change, wait three months, and hope traffic moves, with the causal link nearly impossible to isolate. This validation loop is deterministic. When you implement a fix (adding schema, enabling SSR, deploying llms.txt), you submit your URL for re-audit, the engine runs the same six Architects against the same signals, and the score delta is the direct measurement of whether the fix worked, with no waiting period and no confounding variables. That's the core product loop: Audit. Identify. Fix. Validate. Each credit represents one complete run of all six Architects against one URL, and re-auditing after a fix costs one additional credit, which is why the fix-validate cycle is central to how credits are consumed in active optimization.
See what the six Architects find on your site
Run a free audit. One scan reports your rendering signal, schema validity, token efficiency, entity clarity, crawl access, and semantic structure as a single 0-to-100 readability score.
Audit your site →5. What the Engine Does Not Measure
Being precise about scope matters. The engine measures structural AI readability. It does not measure content quality: whether your content is accurate, authoritative, or more useful than a competitor's is outside automated structural analysis, and a site can score 95 yet still fail to be cited because its content adds no unique information value, since GIST compliance requires content strategy, not just structural fixes. It does not measure training-data inclusion: whether you've already been ingested by GPT, Claude, or Gemini is a function of historical crawl access and timing, while the audit measures current structural readiness for future crawls. And it does not measure link authority: domain authority, backlinks, and PageRank remain relevant for traditional SERP rankings but do not directly influence whether an AI retrieval pipeline selects your content.
The contrarian point that reframes the entire score: a high number is permission to compete, not proof you've won. The engine deliberately refuses to measure the one thing every vendor pretends to quantify, content quality, because that's the part no crawler can fake and no fix can shortcut. Treat 71+ as clearing the structural toll booth: necessary, automatable, and ultimately the easy half. The hard half is being worth citing once the machine can finally read you, and that's a strategy problem, not an audit one.
6. Reference Sources
- Schema.org Technical Documentation: Organization, Product, and TechArticle type specifications. schema.org
- Common Crawl Foundation: Crawl data specifications and bot documentation. commoncrawl.org
- OpenAI GPTBot Documentation: GPTBot user-agent and robots.txt guidance. platform.openai.com/docs/gptbot
- llms.txt Proposal: Jeremy Howard's specification for the llms.txt standard. llmstxt.org
- Website AI Score Research: 1,500-site forensic audit data. View report
- GIST Algorithm Research: reverse-engineering Google's Max-Min Diversity protocol. View article
