--- url: /learn/recognized-vs-verified-crawler-identity title: "Recognized crawler versus verified crawler identity — Siere Learn" description: "A User-Agent string is a claim, not proof. This guide explains the difference between recognizing a crawler and verifying it, which vendors publish verification methods, and why the audit labels every request with an evidence level." type: article language: en author: Tomislav Mladenov canonical: https://www.siere.ai/learn/recognized-vs-verified-crawler-identity last_modified: 2026-08-18 aeo_generated: 2026-08-24T15:04:45.068Z --- # Recognized crawler versus verified crawler identity A User-Agent string is a claim, not proof. This guide explains the difference between recognizing a crawler and verifying it, which vendors publish verification methods, and why the audit labels every request with an evidence level. AuthorTomislav Mladenov Technical reviewerTomislav Mladenov Last verified2026-08-18 Evidence classDocumented behaviour Audit dimensionsAccess, Discovery Licence[CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) Scope · Crawler verification methods as published by OpenAI, Anthropic, Perplexity and Google on the verification date; evidence levels as implemented in the Siere audit and SDK. Reading is optional. Running the check is the point. [Check which crawlers your site lets in](/tools/crawler-access-check) ## A User-Agent is a claim Every HTTP request carries a `User-Agent` header, and the client chooses what to put in it. When a request arrives saying `GPTBot/1.4`, you know one thing: the sender chose to say that. It may be OpenAI's crawler. It may be a monitoring tool, a competitor's script, a security scanner, or you, testing your own site from a laptop: `curl -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot" https://example.com/ ` Nothing in that request distinguishes it from the real thing at the header level. This is why it is worth keeping two words apart: - **Recognized** — the User-Agent (or another header) matches a documented pattern for a known crawler. This is *classification*. - **Verified** — the network identity of the request was checked, independently of its headers, against something the vendor publishes. This is *authentication*. Most of the time you will only ever have the first. That is fine, as long as you say so. ## How verification works, per vendor Vendors that want site owners to be able to tell real crawls from impostors publish a way to check. The methods differ; on the verification date of this guide: Vendor Method Source OpenAI Published IP address lists, one file per crawler (search crawler, training crawler, user-initiated fetches) [OpenAI crawler documentation](https://platform.openai.com/docs/bots) Anthropic Published IP address list; Anthropic notes that blocking by IP "may not consistently guarantee an opt-out" and points to `robots.txt` for that purpose [Anthropic crawler documentation](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) Perplexity Published IP ranges [Perplexity crawler documentation](https://docs.perplexity.ai/guides/bots) Google Reverse DNS on the requesting IP must resolve to a Google hostname, and a forward lookup of that hostname must return the same IP; Google also publishes IP range files [Verifying Googlebot](https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot) Two practical notes: - **Verification is per request, using the client IP.** Behind a CDN or proxy, that means the *original* client IP, not the edge's. Get this wrong and every request looks unverified — or worse, every request looks verified. - **Published ranges change.** Cache them for a bounded time and refresh; a range file downloaded a year ago verifies nothing. Where a vendor publishes no method, the honest state of any request claiming to be that crawler is *recognized, unverified* — and it stays that way. ## Why the distinction matters in practice **For access decisions.** If you block a crawler by User-Agent, you block anyone who claims to be it and no one who lies. If you *allow* something special by User-Agent — a rate-limit exemption, a bypass of a bot-management rule — you have opened it to anyone who types the string. Verified identity is what makes an allow-list defensible. **For delivery decisions.** A delivery layer that serves recognized agents a complete representation of a page has to choose how much to trust the header. The Siere SDK expresses this as an explicit strategy: *observe* (classify and log only), *negotiate-markdown* (serve Markdown only to clients that ask for it via `Accept`), *recognized-agent-markdown* (serve on a documented User-Agent match — the default, appropriate when the representation is equivalent to the page), or *verified-agent-markdown* (serve only when network identity checks out). The right choice depends on what is being served: an equivalent representation of public content is low-risk to show to an impostor; anything more selective should require verification. **For evidence.** A report that says "GPTBot visited 412 times last week" is making a claim about identity. Unless those requests were verified, the accurate statement is "412 requests carried a GPTBot User-Agent". The number may be the same; the claim is not. ## Evidence levels Because the difference is easy to blur, the free audit and the SDK label every observation with how it was obtained: Level Meaning Simulated request Siere sent a request carrying a documented crawler header. This proves how your server responds to that header — not that the vendor crawled you. Unverified request observed A real request arrived with a recognized header; its network identity was not checked. Network-verified request observed A real request arrived and its identity was checked with the vendor's published method. External probe Siere fetched a public URL of yours (`robots.txt`, `sitemap.xml`, the page itself) from outside and recorded the response. Downstream platform observation The evidence came from a third-party platform's output, not from your site's response. The audit's crawler-header comparisons are all *simulated requests*, and every observation says so in its text. That is not a weakness; it is the only honest label for a test you ran yourself. ## `robots.txt` is a third thing `robots.txt` ([RFC 9309](https://www.rfc-editor.org/rfc/rfc9309)) is a statement of policy: which user-agent *tokens* may fetch which paths. It is not identification and not enforcement. A well-behaved crawler reads it and complies; a request that ignores it still arrives at your server. Some vendors additionally document that certain user-initiated fetches — a person asking an assistant to open a specific URL — may not be governed by `robots.txt` at all. So there are three separate questions for any request: - Does the User-Agent match a documented crawler? (recognition) - Does the network identity check out against the vendor's published method? (verification) - Does `robots.txt` allow that token on that path? (policy) The audit reports 1 and 3 for each simulated crawler and labels the evidence level of every observation. Verification (2) is something you do on real traffic in your own logs, or that a delivery layer does per request when it is configured to require it. ## What to do - Never describe recognized traffic as verified. Use the two words on purpose, in dashboards and in reports to clients. - If you make an access or delivery decision that gives a crawler *more* than an ordinary visitor gets, require verification for it. - Keep `robots.txt` as the statement of intent it is, and check that your delivery layer's actual behaviour matches it — the audit's Access and Discovery dimensions compare the two. - Test your own site as each crawler you care about, and read the result as what it is: your server's response to a header. ## What this proves - User-Agent matching classifies a request but cannot verify who sent it; anyone can send any User-Agent. - Some vendors publish IP ranges or DNS methods that allow independent verification; where they do not, a request can only ever be 'recognized, unverified'. ## What this does not prove - That unverified traffic is malicious; most recognized-but-unverified requests are ordinary tools and tests. - That verification methods are stable; vendors change ranges and hostnames, so verify against the current published source. ## Reproduce it yourself - Send a request to your own site with -A 'GPTBot/1.0' from your laptop and observe that it is indistinguishable from OpenAI's crawler by User-Agent alone. - For a real request in your logs, resolve the client IP against the vendor's published ranges (OpenAI, Anthropic, Perplexity publish JSON lists) or run reverse-then-forward DNS where the vendor documents hostnames (Googlebot). - Run the free audit and read the evidence badge on each observation: simulated request, unverified request observed, network-verified request observed, external probe. ## Sources - [OpenAI — Overview of OpenAI crawlers (published IP ranges)](https://platform.openai.com/docs/bots) — OpenAI, accessed 2026-08-18 - [Anthropic — Does Anthropic crawl data from the web?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) — Anthropic, accessed 2026-08-18 - [Perplexity — Perplexity crawlers (published IP ranges)](https://docs.perplexity.ai/guides/bots) — Perplexity, accessed 2026-08-18 - [Google Search Central — Verifying Googlebot and other Google crawlers](https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot) — Google, accessed 2026-08-18 ## Change history - 2026-08-18First published. ## Related guides - [Markdown endpoints versus SSR, prerendering, and edge middleware Four ways to give an agent a complete, structured response — compared on what they change, what they cost, what they can break, and how you prove each one worked.](/learn/markdown-endpoints-vs-ssr-prerendering-edge-middleware) - [AI visibility dashboards show the problem. How do you fix it? Monitoring tools tell you a model did not mention or cite you. They cannot tell you what an agent actually received from your URL, or repair it. This guide separates measurement from remediation and shows the loop that closes the gap.](/learn/from-ai-visibility-dashboard-to-fix) - [Is llms.txt enough? Discovery versus delivery llms.txt tells an agent where to look. It does not change what the agent receives when it gets there. This guide separates the two problems and shows how each is checked.](/learn/is-llms-txt-enough) [← Markdown endpoints versus SSR, prerendering, and edge middleware](/learn/markdown-endpoints-vs-ssr-prerendering-edge-middleware) See what a recognized agent receives from your own URL. [Check which crawlers your site lets in](/tools/crawler-access-check) ## Structured Data ```json { "@context": "https://schema.org", "@type": "Organization", "@id": "https://www.siere.ai/#organization", "name": "Siere", "url": "https://www.siere.ai/", "logo": "https://www.siere.ai/social-square.png", "sameAs": [], "contactPoint": { "@type": "ContactPoint", "contactType": "customer support", "email": "contact@siere.ai", "url": "https://www.siere.ai/contact" } } ``` ```json { "@context": "https://schema.org", "@type": "WebSite", "@id": "https://www.siere.ai/#website", "name": "Siere", "url": "https://www.siere.ai/", "description": "Your website is invisible to AI agents. Siere fixes that. We make your site readable, structured, and optimized for the next generation of AI-powered search.", "publisher": { "@type": "Organization", "name": "Siere", "@id": "https://www.siere.ai/#organization" } } ``` ```json { "@context": "https://schema.org", "@type": "SoftwareApplication", "@id": "https://www.siere.ai/#software", "name": "Agent Experience Optimization", "url": "https://www.siere.ai/", "description": "Your website is invisible to AI agents. Siere fixes that. We make your site readable, structured, and optimized for the next generation of AI-powered search.", "applicationCategory": "BusinessApplication", "operatingSystem": "Web", "offers": [ { "@type": "Offer", "price": "0", "priceCurrency": "USD", "name": "Starter" }, { "@type": "Offer", "price": "99", "priceCurrency": "USD", "name": "Pro" } ] } ``` ```json { "@context": "https://schema.org", "@type": "FAQPage", "@id": "https://www.siere.ai/#faq", "url": "https://www.siere.ai/", "mainEntity": [ { "@type": "Question", "name": "What is Agent Experience Optimization?", "acceptedAnswer": { "@type": "Answer", "text": "AEO makes your website readable, structured, and reliably citable by AI agents like ChatGPT, Claude, Perplexity, and Gemini. Most modern websites are JavaScript-rendered and look like blank pages to AI crawlers — Siere fixes that without changing your visitor experience." } }, { "@type": "Question", "name": "How does Siere work?", "acceptedAnswer": { "@type": "Answer", "text": "Siere sits between your server and incoming requests. When an AI agent visits, Siere serves an optimized markdown version of your content. When a human visits, your normal page is served unchanged." } }, { "@type": "Question", "name": "Do I need to change my site?", "acceptedAnswer": { "@type": "Answer", "text": "No. Siere installs as middleware. There are no content changes or rewrites required." } } ] } ``` ```json { "@context": "https://schema.org", "@type": "TechArticle", "headline": "Recognized crawler versus verified crawler identity", "description": "A User-Agent string is a claim, not proof. This guide explains the difference between recognizing a crawler and verifying it, which vendors publish verification methods, and why the audit labels every request with an evidence level.", "url": "https://www.siere.ai/learn/recognized-vs-verified-crawler-identity", "mainEntityOfPage": "https://www.siere.ai/learn/recognized-vs-verified-crawler-identity", "dateModified": "2026-08-18", "author": { "@type": "Person", "name": "Tomislav Mladenov" }, "publisher": { "@type": "Organization", "name": "Siere", "url": "https://www.siere.ai/" }, "license": "https://creativecommons.org/licenses/by/4.0/", "isAccessibleForFree": true, "keywords": "User-Agent, crawler verification, reverse DNS, IP ranges, GPTBot, ClaudeBot, spoofing, evidence level", "citation": [ "https://platform.openai.com/docs/bots", "https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler", "https://docs.perplexity.ai/guides/bots", "https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot" ] } ```