---
url: /tools/crawler-access-check
title: "Crawler access check — robots.txt and AI crawler policy — Siere"
description: "Free check: which AI crawlers your robots.txt allows or disallows for a URL, whether they receive an access gate, and whether your policy and your delivery agree. Evidence-labelled report you can share."
type: website
language: en
site_name: Siere
canonical: https://www.siere.ai/tools/crawler-access-check
aeo_generated: 2026-08-24T15:04:38.064Z
---

[Learn](/learn)/Tools

# Check which AI crawlers your site lets in — and what they get

Enter a URL. Siere reads your robots.txt, evaluates the effective rule for each documented AI search crawler token on that path, then requests the URL with the matching crawler headers to see whether your server's actual behaviour agrees with the stated policy.

Website URL

Email address

Request Audit

Next

Free. The report is yours to share; observations beyond the first three need a sign-in.

## What is checked

- 01

robots.txt availability

Whether robots.txt exists and returns a usable policy, or is missing (permissive default) or replaced by an access gate.

- 02

Effective rule per crawler

For each documented crawler token, the group that applies and the winning allow/disallow rule for the requested path, following RFC 9309 matching.

- 03

Actual response per crawler header

The HTTP status and any access-gate marker returned to a request carrying that crawler's documented header — so a policy that says allow and a WAF that says block are caught together.

- 04

Policy versus behaviour

The agent access matrix compares intent (allow / deny / unspecified) with what was observed (allowed / blocked / unavailable) and flags unexpected blocks and unexpected exposure.

- 05

Sitemap and discovery files

sitemap.xml validity, and whether llms.txt / llms-full.txt exist, reported as separate surfaces.

## How the evidence is labelled

A crawler-header request is a simulated request: it shows how your site treats that header, not that the vendor visited. Verifying real crawler identity in your logs uses the vendor's published IP ranges or DNS method and is described in the identity guide below.

## Read the reasoning

- [Recognized crawler versus verified crawler identity A User-Agent string is a claim, not proof. This guide explains the difference between recognizing a crawler and verifying it, which vendors publish verification methods, and why the audit labels every request with an evidence level.](/learn/recognized-vs-verified-crawler-identity)
- [Is llms.txt enough? Discovery versus delivery llms.txt tells an agent where to look. It does not change what the agent receives when it gets there. This guide separates the two problems and shows how each is checked.](/learn/is-llms-txt-enough)
- [What AI agents actually receive from JavaScript websites Most AI crawlers fetch the initial HTML and move on. If your page is a client-rendered application, that response can be an empty shell. Here is what is in the response, how to inspect it, and how the free audit measures it.](/learn/what-ai-agents-receive-from-javascript-websites)

## Structured Data

```json
{
  "@context": "https://schema.org",
  "@type": "Organization",
  "@id": "https://www.siere.ai/#organization",
  "name": "Siere",
  "url": "https://www.siere.ai/",
  "logo": "https://www.siere.ai/social-square.png",
  "sameAs": [],
  "contactPoint": {
    "@type": "ContactPoint",
    "contactType": "customer support",
    "email": "contact@siere.ai",
    "url": "https://www.siere.ai/contact"
  }
}
```

```json
{
  "@context": "https://schema.org",
  "@type": "WebSite",
  "@id": "https://www.siere.ai/#website",
  "name": "Siere",
  "url": "https://www.siere.ai/",
  "description": "Your website is invisible to AI agents. Siere fixes that. We make your site readable, structured, and optimized for the next generation of AI-powered search.",
  "publisher": {
    "@type": "Organization",
    "name": "Siere",
    "@id": "https://www.siere.ai/#organization"
  }
}
```

```json
{
  "@context": "https://schema.org",
  "@type": "SoftwareApplication",
  "@id": "https://www.siere.ai/#software",
  "name": "Agent Experience Optimization",
  "url": "https://www.siere.ai/",
  "description": "Your website is invisible to AI agents. Siere fixes that. We make your site readable, structured, and optimized for the next generation of AI-powered search.",
  "applicationCategory": "BusinessApplication",
  "operatingSystem": "Web",
  "offers": [
    {
      "@type": "Offer",
      "price": "0",
      "priceCurrency": "USD",
      "name": "Starter"
    },
    {
      "@type": "Offer",
      "price": "99",
      "priceCurrency": "USD",
      "name": "Pro"
    }
  ]
}
```

```json
{
  "@context": "https://schema.org",
  "@type": "FAQPage",
  "@id": "https://www.siere.ai/#faq",
  "url": "https://www.siere.ai/",
  "mainEntity": [
    {
      "@type": "Question",
      "name": "What is Agent Experience Optimization?",
      "acceptedAnswer": {
        "@type": "Answer",
        "text": "AEO makes your website readable, structured, and reliably citable by AI agents like ChatGPT, Claude, Perplexity, and Gemini. Most modern websites are JavaScript-rendered and look like blank pages to AI crawlers — Siere fixes that without changing your visitor experience."
      }
    },
    {
      "@type": "Question",
      "name": "How does Siere work?",
      "acceptedAnswer": {
        "@type": "Answer",
        "text": "Siere sits between your server and incoming requests. When an AI agent visits, Siere serves an optimized markdown version of your content. When a human visits, your normal page is served unchanged."
      }
    },
    {
      "@type": "Question",
      "name": "Do I need to change my site?",
      "acceptedAnswer": {
        "@type": "Answer",
        "text": "No. Siere installs as middleware. There are no content changes or rewrites required."
      }
    }
  ]
}
```