Validates robots.txt and the crawlers it blocks
Build the robots URL — takes the scheme, host and port of the audited URL and appends /robots.txt. An audited URL without a scheme or host ends the audit with 500.
Fetch — requests robots.txt as 37AuditsBot, following redirects. This is a probe of another path, not the audited page, so a 403 is a finding (400), not a blocked bot: this auditor never emits 590. HTTP 5xx fails (401), because RFC 9309 tells crawlers to treat it as a full Disallow. A network error ends the audit with 500.
Scan lines — when the answer is HTTP 200, removes a byte-order mark and # comments and reads each line as directive: value. That feeds the syntax rules (302–308), the sensitive-paths list (103) and Crawl-delay (300, 301).
Match crawlers — reads the User-agent: * group, then every crawler of the bundled catalog that a User-agent line names. A crawler counts as blocked when / is disallowed for it. When the * group blocks every crawler (402), only crawlers with a blocking group of their own are reported again, so one staging leftover does not turn into dozens of checks.
Audited page — when / is open to User-agent: * and Googlebot but the audited URL is not, fails with 404.
Aggregate — FAIL wins over WARNING. INFORMATION checks never change the status. When no rule warns or fails the audit is SUCCESS (200) and the message quotes the declared Crawl-delay.
Every row maps to one checkRule* method and one CheckCode. CheckCode suffixes use status ranges: 100–199 information, 200 success, 300–399 warning, 400–499 fail, 500–599 error.
| CheckCode | Status | When it fires | Recommendation | Why it matters |
|---|---|---|---|---|
37A-RobotsAuditor-100 | INFORMATION | Always: names RFC 9309, the specification this auditor implements. | — | — |
37A-RobotsAuditor-101 | INFORMATION | One check listing the AI training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot…) that robots.txt keeps out of the whole site. | — | Blocking them is a legitimate choice: their owners will not train models on the site. |
37A-RobotsAuditor-102 | INFORMATION | One check listing the SEO and marketing crawlers (AhrefsBot, SemrushBot, MJ12bot…) kept out of the whole site. | — | Those tools will not report the site's backlinks or rankings. |
37A-RobotsAuditor-103 | INFORMATION | Disallow paths that contain backup, .git, .svn, .env, config, secret, admin or .sql (WordPress's wp-admin is skipped). | — | robots.txt is public, so these lines tell anyone where to look; only access control protects such paths. |
37A-RobotsAuditor-104 | INFORMATION | One check listing the regional or niche search engines (Yandex, Baidu, Naver, Seznam, Petal, Applebot, DuckDuckBot…) kept out of the whole site. | — | Many sites block them on purpose; the site will not appear in those engines. |
37A-RobotsAuditor-200 | SUCCESS | No rule warned or failed and robots.txt declares a numeric Crawl-delay. The message quotes the value. | — | — |
37A-RobotsAuditor-300 | WARNING | robots.txt was fetched (HTTP 200) but contains no Crawl-delay directive. | Add a "Crawl-delay: <seconds>" line (for example "Crawl-delay: 10") to {robotsUrl} under the User-agent groups whose crawl rate you want to limit. | Without a Crawl-delay, crawlers that honor it request pages as fast as they like, which can slow the site for real visitors and inflate hosting costs. |
37A-RobotsAuditor-301 | WARNING | A Crawl-delay directive is present but its value is empty or not a non-negative number. | Replace "Crawl-delay: {crawlDelay}" in {robotsUrl} with a number of seconds, for example "Crawl-delay: 10". | Crawlers ignore a Crawl-delay they cannot read as a number of seconds, so the site gets none of the rate limiting the directive was meant to provide. |
37A-RobotsAuditor-302 | WARNING | The Content-Type is not text/plain, or the body is an HTML document (a soft 404 or a catch-all route). | Serve {robotsUrl} as a plain-text file with "Content-Type: text/plain" that contains only robots.txt directives. | A robots.txt served as an HTML page holds no directives crawlers can read, so the site gets none of the rules it meant to publish. |
37A-RobotsAuditor-303 | WARNING | The body is larger than 500 KiB. | Shrink {robotsUrl} below 500 KiB: merge rules with wildcards and remove one-off paths. | RFC 9309 only requires crawlers to read the first 500 KiB and Google ignores the rest, so rules past that point never apply. |
37A-RobotsAuditor-304 | WARNING | One check per line (at most 20) that has no colon or uses an unknown or misspelled directive. Host, Clean-param, Request-rate and Visit-time are tolerated. | Fix line {lineNumber} to use one of User-agent, Allow, Disallow, Sitemap or Crawl-delay followed by a colon, or remove it. | Crawlers silently skip such a line, so the rule its author meant never applies. |
37A-RobotsAuditor-305 | WARNING | One check per Allow or Disallow line that comes before the first User-agent line. | Move line {lineNumber} under a "User-agent: *" line (or the group of the crawler it targets). | RFC 9309 parsers ignore rules that belong to no group, so those paths stay open to every crawler. |
37A-RobotsAuditor-306 | WARNING | One check per Noindex, Nofollow or Noarchive line. | Remove the line and send a <meta name="robots"> tag or an X-Robots-Tag header on those pages instead, keeping them crawlable. | Google stopped honoring indexing directives in robots.txt in September 2019 and other crawlers never did, so those pages can still appear in search results. |
37A-RobotsAuditor-307 | WARNING | One check per Allow or Disallow value that is not empty and does not start with / or * (a bare word or a full URL). | Rewrite line {lineNumber} with a path that starts with "/" (for example "Disallow: /admin"). | Rules match the URL path, which always starts with "/", so such a pattern never matches anything. |
37A-RobotsAuditor-308 | WARNING | One check per Sitemap line whose value is not an absolute http(s) URL. Whether the sitemap answers is checked by the Sitemap auditor. | Write line {lineNumber} as a full URL, for example "Sitemap: https://example.com/sitemap.xml". | The sitemaps protocol requires a fully qualified URL in robots.txt, so crawlers may skip a relative one and miss the pages it lists. |
37A-RobotsAuditor-309 | WARNING | One check per AI search or assistant crawler (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot…) kept out of the whole site. | If the site should appear in that company's AI answers, remove "Disallow: /" for its User-agent. | AI assistants only cite and link pages their crawlers may fetch, so a blocked crawler takes the site out of their answers. |
37A-RobotsAuditor-400 | FAIL | robots.txt answered an HTTP status other than 200 and below 500 (404, 403, 401, a 3xx left after redirects…). | Publish a robots.txt file at {robotsUrl} that answers HTTP 200 and declare a Crawl-delay directive in it. | Without a reachable robots.txt, crawlers get no crawl-rate or exclusion guidance, so they may hit the site harder and index pages that should stay out of search results. |
37A-RobotsAuditor-401 | FAIL | robots.txt answered HTTP 5xx. | Make {robotsUrl} answer HTTP 200 (or 404 when there are no rules); check the server or CDN rule that serves it. | RFC 9309 tells crawlers to treat a robots.txt that answers 5xx as a full Disallow, so search engines stop crawling the whole site until it answers again. |
37A-RobotsAuditor-402 | FAIL | The User-agent: * group disallows /, unless Googlebot and Bingbot each have a group of their own that lets them in (a deliberate allow list). | Remove "Disallow: /" from the "User-agent: *" group; disallow only the paths that must stay out of search. | It keeps every compliant crawler out, so the site drops out of search engines and AI answers. It is often left over from a staging site. |
37A-RobotsAuditor-403 | FAIL | One check each when Googlebot or Bingbot is kept out of the whole site. | Remove "Disallow: /" for that User-agent. | Googlebot and Bingbot feed most search traffic (Bing also feeds DuckDuckGo, Yahoo and ChatGPT search), so blocking either removes the site from those results. |
37A-RobotsAuditor-404 | FAIL | / is open but the audited URL is disallowed for User-agent: * or Googlebot. The check's resource is the audited URL. | Remove or narrow the matching Disallow rule, or add an "Allow" rule for the page's path, if the page should appear in search. | A disallowed page is not crawled, so search engines never see its content and keep a stale or empty entry for it. |
37A-RobotsAuditor-500 | ERROR | robots.txt could not be fetched: a network error or an audited URL without a scheme or host. | Make sure {robotsUrl} is reachable over HTTP(S), then run the audit again. | — |
| Status | Description | Test logic |
|---|---|---|
| SUCCESS | robots.txt is healthy and declares a usable Crawl-delay | HTTP 200, no WARNING or FAIL rule fired, and the first Crawl-delay line has a non-negative numeric value (200). |
| WARNING | Crawlers misread part of the file, or an AI assistant is shut out | Crawl-delay missing or invalid (300, 301); wrong content type, oversize file or a line crawlers skip (302–308); an AI search crawler blocked (309). |
| FAIL | Crawlers cannot read the file, or search engines are shut out | robots.txt missing (400) or answering 5xx (401); every crawler blocked (402); Googlebot or Bingbot blocked (403); the audited page disallowed (404). |
| ERROR | The auditor could not fetch robots.txt | Network error or invalid audited URL (500). |
Every check carries robotsUrl, httpStatus, contentType, sizeBytes, crawlDelayFound, crawlDelay (the raw value, or null) and crawlDelaySeconds (the parsed number, or null) in its data. Line rules add lineNumber and line, crawler rules add bots (the product tokens), 103 adds paths, and 404 adds auditedUrl and agents. The resource is the robots.txt URL, except for 404 (the audited URL) and 100 (the RFC).
The crawlers are bundled with the auditor and identified by the product token each owner documents for robots.txt.
| Group | Crawlers | Blocked means |
|---|---|---|
| Major search engines | Googlebot, Bingbot | FAIL (403) |
| Other search engines | Applebot, DuckDuckBot, YandexBot, Baiduspider, Yeti, SeznamBot, PetalBot, Slurp, Qwantbot, MojeekBot | INFORMATION (104) |
| AI search and assistants | OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, DuckAssistBot, MistralAI-User, Meta-ExternalFetcher, YouBot | WARNING (309) |
| AI training | GPTBot, ClaudeBot, anthropic-ai, Google-Extended, Applebot-Extended, CCBot, Bytespider, Meta-ExternalAgent, Amazonbot, cohere-ai, cohere-training-data-crawler, Diffbot, ImagesiftBot, Omgilibot, AI2Bot | INFORMATION (101) |
| SEO tools | AhrefsBot, SemrushBot, MJ12bot, DotBot, rogerbot, BLEXBot, DataForSeoBot, serpstatbot, Barkrowler | INFORMATION (102) |
Scrapers, email harvesters and vulnerability scanners are not in the catalog: they ignore robots.txt, so naming them in the file does not keep them out.
A crawler is reported only when / is disallowed for it. A group that disallows some paths (Disallow: /blog/) is a normal exclusion and is not reported, except for the audited page (404).
Disallow: with an empty value allows everything, as RFC 9309 says.
Allow rules and wildcards (*, $) count: Disallow: / with Allow: /$ still blocks every page except the home page, so auditing any other page fails with 404 even though / is open.
Crawl-delay is not part of RFC 9309. Bing, Yandex and many SEO and AI crawlers honor it; Googlebot ignores it, so it is not a substitute for rate limiting at the server or CDN.
The first Crawl-delay line in the file wins, whichever User-agent group it sits in.
Only /robots.txt on the audited scheme, host and port is read. A robots.txt on another subdomain or served only over the other scheme is not seen.
robots.txt asks; it does not enforce. A crawler that ignores the file is not stopped by it, which is why the sensitive-paths list (103) is information, not protection.
This auditor replaced the Robots Crawl Delay auditor, deprecated in October 2026. Results it recorded keep their 37A-RobotsCrawlDelayAuditor-* codes, explained on its page.
An HTTP 5xx answer on robots.txt used to be an ERROR (500); it is now a FAIL (401), because RFC 9309 crawlers treat it as a full Disallow.
Every check now starts with the RFC 9309 specification pin (100).
Rules 101–104, 302–309 and 402–404 are new.
This documentation is built and hosted on Mintlify, a developer documentation platform.
Add this auditor to your monitoring setup and start identifying issues on your websites today.