Acme

--- url: /learn/what-ai-agents-receive-from-javascript-websites title: "What AI agents actually receive from JavaScript websites — Siere Learn" description: "Most AI crawlers fetch the initial HTML and move on. If your page is a client-rendered application, that response can be an empty shell. Here is what is in the response, how to inspect it, and how the free audit measures it." type: article language: en author: Tomislav Mladenov canonical: https://www.siere.ai/learn/what-ai-agents-receive-from-javascript-websites last_modified: 2026-08-18 aeo_generated: 2026-08-24T15:04:44.676Z --- # What AI agents actually receive from JavaScript websites Most AI crawlers fetch the initial HTML and move on. If your page is a client-rendered application, that response can be an empty shell. Here is what is in the response, how to inspect it, and how the free audit measures it. AuthorTomislav Mladenov Technical reviewerTomislav Mladenov Last verified2026-08-18 Evidence classDocumented behaviour Audit dimensionsSource completeness, Access Licence[CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) Scope · Client-rendered web applications (React, Vue, Angular, Svelte and similar); crawler behaviour as documented by vendors on the verification date. Reading is optional. Running the check is the point. [See what an AI agent receives from your URL](/tools/agent-response-check) ## The response is the page For a crawler, a page is whatever comes back from an HTTP request. There is no scroll, no click, and — unless the vendor documents otherwise — no JavaScript execution. What you see in a browser is the result of a second phase in which the browser downloads scripts, runs them, fetches data and builds the DOM. A crawler that stops after the first phase never sees that DOM. That is why "the page looks fine" and "the crawler received the page" are different statements. The first is about the browser. The second is about the response body. ## What is in the source response of a client-rendered app A typical client-rendered application (React, Vue, Angular, Svelte, or a framework built on them without server rendering) returns something close to this for every route: ` Acme

` Everything a person reads — headings, product names, prices, descriptions, links to other pages — arrives later, produced by `app.js`. In the source response the visible text is the title and nothing else. The free audit calls this an **empty or near-empty application shell**: fewer than 80 characters of normalized visible text is recorded as a failure, fewer than 300 as partial. Two consequences follow: - **The crawler cannot read what is not there.** No amount of copywriting or schema markup on the rendered page changes a response that does not contain it. - **Every route looks the same.** The shell is identical for `/`, `/pricing` and `/products/123`, so from the crawler's side the site has one page repeated many times. ## What the vendors say The claims below are about what vendors document, checked on the verification date. Vendors change behaviour without notice; check the linked source before relying on it. Vendor Documented crawlers Documents JavaScript execution? Verification method published OpenAI ([source](https://platform.openai.com/docs/bots)) OAI-SearchBot (search), GPTBot (training), ChatGPT-User (user-initiated), OAI-AdsBot Not documented Published IP address lists per crawler Anthropic ([source](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)) ClaudeBot, Claude-SearchBot, Claude-User Not documented Published IP address list Perplexity ([source](https://docs.perplexity.ai/guides/bots)) PerplexityBot (search), Perplexity-User (user-initiated) Not documented Published IP ranges Google ([source](https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics)) Googlebot and others Yes — Googlebot renders with a recent Chromium, in a separate, deferred phase Reverse/forward DNS and published IP ranges Google's own guidance, even with rendering, is that important content and links should be present in the server response, and that rendering is queued and may take longer than crawling. For crawlers that document no rendering at all, the source response is the whole story. ## How to look at what a crawler receives You do not need a browser or a tool for this. From a terminal: `# 1. The raw source response, as any HTTP client receives it curl -sL https://example.com/pricing -o source.html # 2. Bytes and a rough count of visible words wc -c source.html sed 's/]*>.*<\/script>//g; s/<[^>]*>//g' source.html | tr -s ' \n' | wc -w # 3. The same request with a documented crawler User-Agent curl -sL -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot" \ https://example.com/pricing -o agent.html diff <(sed 's/<[^>]*>//g' source.html) <(sed 's/<[^>]*>//g' agent.html) | head ` Then open the same URL in a browser, select all, and compare the amount of text. If the terminal shows a few dozen words and the browser shows several hundred, the content is produced client-side and a non-rendering crawler receives the shell. Step 3 also catches a second failure: sites that answer the crawler User-Agent differently — a bot-management block page, a captcha, a redirect to a login — while the ordinary request succeeds. That is a delivery failure of a different kind and is covered in [Recognized crawler versus verified crawler identity](/learn/recognized-vs-verified-crawler-identity). ## What the free audit records The audit runs the comparison above with documented search-crawler headers (OpenAI, Anthropic and Perplexity search crawlers), and records: - **Source completeness** — the normalized visible-text length of the ordinary response, with the thresholds described above, and, where a bounded renderer is configured, the length of the rendered DOM for comparison. It never infers rendered content from source HTML. - **Access** — the HTTP status of each request and whether the body contained a deterministic access-gate marker despite a 200. - **Fidelity** — whether the title, canonical URL, visible words, important links and redirect destination match between the ordinary and the crawler-header request. Every observation carries an evidence label. A request the audit sent with a crawler header is a *simulated request*: it reproduces the documented header and proves what your server does with it, not that the vendor actually crawled you. ## Fixing it There are three families of fix, covered in the next two guides: - **Produce complete HTML on the server or at build time** — server-side rendering, static generation or prerendering, so the source response contains the content ([React, Vue, Angular: making client-rendered content agent-readable](/learn/react-vue-angular-agent-readable)). - **Serve a complete representation at request time** — a request-time path that gives recognized agents (or any client that asks for `text/markdown`) a complete, structured version of the same content while the human page is untouched ([Markdown endpoints versus SSR, prerendering, and edge middleware](/learn/markdown-endpoints-vs-ssr-prerendering-edge-middleware)). - **Fix access** — if the crawler is blocked or redirected, no rendering strategy helps until the policy is corrected. Whichever you choose, verification is the same: run the terminal comparison, or the audit, again. The visible-text count in the source response is the number that has to change. ## What this proves - The initial HTML response of a client-rendered page can be inspected without a browser and its visible text measured. - Vendor documentation for the main AI crawlers does not commit to executing client-side JavaScript; Google documents rendering for Googlebot specifically. ## What this does not prove - That a particular vendor never renders JavaScript for a particular URL — only what the vendor documents and what the initial response contains. - That server-rendered pages are automatically well represented; completeness and structure are separate checks. ## Reproduce it yourself - curl -sL -A 'Mozilla/5.0' https://example.com/ | wc -c — record the byte count. - curl -sL -A 'Mozilla/5.0' https://example.com/ | sed 's/<[^>]*>//g' | tr -s ' \n' | wc -w — count visible words in the source response. - Open the same URL in a browser, copy the rendered text and count words. A large gap means the content is produced client-side. - Run the free audit; the Source completeness dimension records the normalized visible-text length of the source response. ## Sources - [Google Search Central — Understand the JavaScript SEO basics](https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics) — Google, accessed 2026-08-18 - [OpenAI — Overview of OpenAI crawlers](https://platform.openai.com/docs/bots) — OpenAI, accessed 2026-08-18 - [Anthropic — Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) — Anthropic, accessed 2026-08-18 - [Perplexity — Perplexity crawlers](https://docs.perplexity.ai/guides/bots) — Perplexity, accessed 2026-08-18 ## Change history - 2026-08-18First published. ## Related guides - [React, Vue, Angular: making client-rendered content agent-readable Framework-specific options for giving crawlers a complete response — server rendering, static generation, prerendering and a request-time agent path — with the trade-offs of each and how to verify the result.](/learn/react-vue-angular-agent-readable) - [Markdown endpoints versus SSR, prerendering, and edge middleware Four ways to give an agent a complete, structured response — compared on what they change, what they cost, what they can break, and how you prove each one worked.](/learn/markdown-endpoints-vs-ssr-prerendering-edge-middleware) - [AI visibility dashboards show the problem. How do you fix it? Monitoring tools tell you a model did not mention or cite you. They cannot tell you what an agent actually received from your URL, or repair it. This guide separates measurement from remediation and shows the loop that closes the gap.](/learn/from-ai-visibility-dashboard-to-fix) [← AI visibility dashboards show the problem. How do you fix it?](/learn/from-ai-visibility-dashboard-to-fix)[Is llms.txt enough? Discovery versus delivery →](/learn/is-llms-txt-enough) See what a recognized agent receives from your own URL. [See what an AI agent receives from your URL](/tools/agent-response-check) ## Structured Data ```json { "@context": "https://schema.org", "@type": "Organization", "@id": "https://www.siere.ai/#organization", "name": "Siere", "url": "https://www.siere.ai/", "logo": "https://www.siere.ai/social-square.png", "sameAs": [], "contactPoint": { "@type": "ContactPoint", "contactType": "customer support", "email": "contact@siere.ai", "url": "https://www.siere.ai/contact" } } ``` ```json { "@context": "https://schema.org", "@type": "WebSite", "@id": "https://www.siere.ai/#website", "name": "Siere", "url": "https://www.siere.ai/", "description": "Your website is invisible to AI agents. Siere fixes that. We make your site readable, structured, and optimized for the next generation of AI-powered search.", "publisher": { "@type": "Organization", "name": "Siere", "@id": "https://www.siere.ai/#organization" } } ``` ```json { "@context": "https://schema.org", "@type": "SoftwareApplication", "@id": "https://www.siere.ai/#software", "name": "Agent Experience Optimization", "url": "https://www.siere.ai/", "description": "Your website is invisible to AI agents. Siere fixes that. We make your site readable, structured, and optimized for the next generation of AI-powered search.", "applicationCategory": "BusinessApplication", "operatingSystem": "Web", "offers": [ { "@type": "Offer", "price": "0", "priceCurrency": "USD", "name": "Starter" }, { "@type": "Offer", "price": "99", "priceCurrency": "USD", "name": "Pro" } ] } ``` ```json { "@context": "https://schema.org", "@type": "FAQPage", "@id": "https://www.siere.ai/#faq", "url": "https://www.siere.ai/", "mainEntity": [ { "@type": "Question", "name": "What is Agent Experience Optimization?", "acceptedAnswer": { "@type": "Answer", "text": "AEO makes your website readable, structured, and reliably citable by AI agents like ChatGPT, Claude, Perplexity, and Gemini. Most modern websites are JavaScript-rendered and look like blank pages to AI crawlers — Siere fixes that without changing your visitor experience." } }, { "@type": "Question", "name": "How does Siere work?", "acceptedAnswer": { "@type": "Answer", "text": "Siere sits between your server and incoming requests. When an AI agent visits, Siere serves an optimized markdown version of your content. When a human visits, your normal page is served unchanged." } }, { "@type": "Question", "name": "Do I need to change my site?", "acceptedAnswer": { "@type": "Answer", "text": "No. Siere installs as middleware. There are no content changes or rewrites required." } } ] } ``` ```json { "@context": "https://schema.org", "@type": "TechArticle", "headline": "What AI agents actually receive from JavaScript websites", "description": "Most AI crawlers fetch the initial HTML and move on. If your page is a client-rendered application, that response can be an empty shell. Here is what is in the response, how to inspect it, and how the free audit measures it.", "url": "https://www.siere.ai/learn/what-ai-agents-receive-from-javascript-websites", "mainEntityOfPage": "https://www.siere.ai/learn/what-ai-agents-receive-from-javascript-websites", "dateModified": "2026-08-18", "author": { "@type": "Person", "name": "Tomislav Mladenov" }, "publisher": { "@type": "Organization", "name": "Siere", "url": "https://www.siere.ai/" }, "license": "https://creativecommons.org/licenses/by/4.0/", "isAccessibleForFree": true, "keywords": "JavaScript, SPA, client-side rendering, GPTBot, initial HTML, application shell", "citation": [ "https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics", "https://platform.openai.com/docs/bots", "https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler", "https://docs.perplexity.ai/guides/bots" ] } ```