Validates XML sitemap availability and freshness
3 of 3 free audits left today
See what Sitemap finds on your site. No sign-up needed.
Site root — takes scheme://host[:port] from the audited URL. A URL without a scheme or hostname ends the audit with 501.
robots.txt — fetches <root>/robots.txt (as 37AuditsBot, with the site's Basic Auth when configured) and reads only its Sitemap: lines, with crawler-commons' robots parser: relative values are resolved against robots.txt, duplicates and empty lines are dropped. Allow/Disallow rules are not evaluated. The probe never calls ensureNotBlocked: a 403 or any other answer on robots.txt is recorded in issues, not treated as a blocked audit.
Declared sitemaps — every Sitemap: URL is fetched and parsed, and its outcome is recorded in declaredSitemaps. The first one that parses is the audited sitemap.
Fallbacks — when no declared sitemap parses (or none is declared), /sitemap.xml, /sitemap_index.xml and /sitemap/sitemap.xml under the site root are tried in turn, and the first that parses is audited (selectedFromFallback).
Parse — every sitemap is fetched through the auditor's HTTP client (redirects followed, bodies up to 50 MB, 30 s per request, gzip detected by magic bytes even without Content-Encoding) and parsed by crawler-commons 1.6, which detects the format from the bytes: XML <urlset> and <sitemapindex>, RSS, Atom and plain-text URL lists all count. An HTML page, an unknown XML root or a non-2xx answer is a broken candidate.
Index children — when the audited sitemap is a <sitemapindex>, its first 50 children are fetched and parsed in parallel (8 at a time, 45 s in all); the rest are counted in childSitemapsSkipped. Only one level is read: a child that is itself an index is reported as broken. Children not read within the budget are recorded in childSitemaps but not reported.
Listed URLs — across the sitemap and its children, the auditor counts the URLs, keeps the latest <lastmod>, counts <lastmod> values more than one day in the future (and leaves them out of freshness), and counts URLs whose scheme or host differs from the audited sitemap's final URL.
Sample — sampleSize URLs (default 20) are picked by the smallest CRC32 of their URL, so the sample is spread over the sitemap and identical from run to run. Each is fetched with a GET in parallel (8 at a time, 30 s in all, 5 MB per body): an HTTP 4xx/5xx final answer, a redirect, and noindex in an X-Robots-Tag header or a robots/googlebot meta tag are reported. HTTP 403, 429, a Cloudflare challenge or a bot wall leaves the URL unchecked. Every sampled URL becomes a check of its own: 307, 308 or 309 when it has a problem, 201 when it is healthy, 101 when it could not be checked.
Freshness window — SITEMAP_MAX_AGE_DAYS (Lambda environment, default 7 days). Date-only <lastmod> values are read as the start of that day in UTC.
Aggregate — FAIL wins over WARNING; a sitemap whose latest <lastmod> is within the window emits 200, and the audit is SUCCESS when no warning fired.
Specification pin — every run emits 37A-SitemapAuditor-100 (INFORMATION), naming the Sitemaps XML format, protocol version 0.9. It does not change the overall status.
| Key | Type | Default | Description |
|---|---|---|---|
sampleSize | integer | 20 | How many URLs listed in the sitemap are fetched to check for errors, redirects and noindex. Set it per monitored URL on the Configuration tab; 0 turns sampling off, values above 100 are capped at 100. |
A missing, negative, fractional or non-numeric value falls back to 20.
Every row maps to one checkRule* method and one CheckCode, in the order the auditor runs them. CheckCode suffixes use status ranges: 100 information, 200 success, 300–399 warning, 400–499 fail, 500–599 error.
| CheckCode | Status | When it fires | Recommendation | Why it matters |
|---|---|---|---|---|
37A-SitemapAuditor-100 | INFORMATION | Always. Records the Sitemaps protocol version this auditor implements. | — (recommendation is null; nothing for the site to change) | — |
37A-SitemapAuditor-400 | FAIL | No candidate sitemap (declared in robots.txt or at the fallback paths) could be fetched and parsed. The message lists every candidate and why it failed, so 402 and 303 do not fire. | Publish an XML sitemap at {siteBase}/sitemap.xml (or any URL) that answers HTTP 200 with a <urlset> or <sitemapindex> root, and reference it from {robotsTxtUrl} with a "Sitemap:" line. Candidates tried: {candidates}. | Without a sitemap, search engines only discover pages by following links, so deep or new pages are indexed late or not at all and bring in no search traffic. |
37A-SitemapAuditor-402 | FAIL | Every Sitemap: line of robots.txt is broken, but a fallback path answered with a sitemap. One check per declared sitemap; resource is the declared URL. | Replace the line "Sitemap: {declaredUrl}" in {robotsTxtUrl} with "Sitemap: {fallbackUrl}", or make {declaredUrl} answer HTTP 200 with a sitemap. | Search engines do not guess sitemap paths: they read the Sitemap: lines of robots.txt, so when every declared sitemap is broken they see no sitemap at all, even though a working one exists at another URL. |
37A-SitemapAuditor-401 | FAIL | The audited sitemap lists no URLs: an empty <urlset>, an index without <sitemap> entries, or an index whose readable children are all empty. An index none of whose children could be read is left to 304. | Generate {sitemapUrl} from the site's published pages so it lists every URL that should be indexed. | An empty sitemap tells search engines nothing, so pages are only discovered by following links, exactly as if there were no sitemap at all. |
37A-SitemapAuditor-403 | FAIL | A sitemap file (the audited one or a child) lists more than 50,000 URLs or is larger than 50 MB uncompressed. One check per file; resource is the file. | Split {sitemapUrl} into files of at most 50000 URLs and 50 MB uncompressed, and list them in a sitemap index. | Search engines reject a sitemap file over the protocol limits, so none of the pages it lists get the sitemap signal. |
37A-SitemapAuditor-303 | WARNING | A Sitemap: line of robots.txt cannot be fetched or parsed while another declared sitemap works. One check per broken line; resource is the declared URL. | Fix {declaredUrl} so it answers HTTP 200 with a sitemap, or remove the line "Sitemap: {declaredUrl}" from {robotsTxtUrl}. | Crawlers fetch every Sitemap: line and report the broken one as an error; it wastes crawl requests and hides the pages it was meant to list. |
37A-SitemapAuditor-300 | WARNING | The latest <lastmod> across the sitemap and the children read is older than the freshness window (SITEMAP_MAX_AGE_DAYS, default 7 days). Resource is the sitemap URL. | Regenerate {sitemapUrl} whenever pages change so its <lastmod> values stay within {maxAgeDays} days; the latest is currently {latestLastmod}. | Search engines use lastmod to decide which pages to recrawl; a sitemap that looks abandoned is trusted less, so new and updated pages are found and ranked later. |
37A-SitemapAuditor-301 | WARNING | Neither the sitemap nor the children read have a parsable <lastmod> (future values excluded), so freshness cannot be assessed. Resource is the sitemap URL. | Add a <lastmod> element in W3C Datetime format (for example 2026-10-16 or 2026-10-16T09:30:00Z) to each <url> (or <sitemap> and <url>) entry of {sitemapUrl}, set to the date the page last changed. | Without lastmod dates, search engines cannot tell which pages changed and crawl them all alike, so updates to important pages are picked up later. |
37A-SitemapAuditor-302 | WARNING | robots.txt answered HTTP 200 but lists no Sitemap: line. Fires in addition to whatever the fallback paths yielded. Resource is the robots.txt URL. | Add the line "Sitemap: {sitemapUrl}" to {robotsTxtUrl}. | robots.txt is the first file every crawler reads, and a Sitemap: line there is how search engines discover the sitemap without a manual submission; without it some crawlers never find the sitemap at all. |
37A-SitemapAuditor-304 | WARNING | A child sitemap of the index cannot be fetched or parsed, or is itself an index. One check per child; resource is the child URL. | Make {childUrl} answer HTTP 200 with a <urlset> (an index cannot list another index), or remove its <sitemap> entry from {sitemapUrl}. | Search engines skip a child sitemap they cannot read, so every page it lists loses its sitemap signal and is found only through links, if at all. |
37A-SitemapAuditor-305 | WARNING | Listed URLs use another scheme or host (port included) than the audited sitemap's final URL, for example http:// URLs in an https sitemap or a staging domain. One check with the count and up to five examples. | List only {origin} URLs in the sitemap: fix the scheme and host the sitemap generator uses (http:// instead of https://, a staging or old domain), and give every other host its own sitemap. | Search engines ignore sitemap URLs on a host or scheme the sitemap does not belong to, and URLs pointing at http://, staging or old domains send crawlers to the wrong pages instead of the live ones. |
37A-SitemapAuditor-306 | WARNING | <lastmod> values are more than one day ahead of the audit's clock. One check with the count and up to five examples. | Set each <lastmod> of {sitemapUrl} to the date the page actually last changed, never to a scheduled or generated future date. | Google ignores lastmod when it is not consistently accurate; future dates make every date in the sitemap untrustworthy, so real updates stop speeding up recrawls. |
37A-SitemapAuditor-307 | WARNING | A sampled URL's final answer is HTTP 4xx or 5xx (403 and 429 are left unchecked). One check per URL. | Remove {url} from the sitemap, or restore the page so it answers HTTP 200. | Search Console reports listed URLs that fail as sitemap errors; every broken URL spends crawl budget on a dead end and lowers the trust search engines put in the sitemap. |
37A-SitemapAuditor-308 | WARNING | A sampled URL redirects to a page that answers below HTTP 400. One check per URL. | List the final URL {finalUrl} in the sitemap instead of {url}. | A sitemap should list final, canonical URLs; redirecting entries cost an extra request per crawl and search engines may index the redirect target late or not at all. |
37A-SitemapAuditor-309 | WARNING | A sampled URL answers 2xx with noindex (or none) in an X-Robots-Tag header (for every agent or googlebot) or a robots/googlebot meta tag. One check per URL. | Either remove noindex from {url} so it can be indexed, or remove it from the sitemap. | Listing a page in the sitemap asks search engines to index it while noindex forbids it; the mixed signal shows up as a Search Console error and the page stays out of results. |
37A-SitemapAuditor-200 | SUCCESS | A sitemap parsed and its latest <lastmod> is within the freshness window. Message names the sitemap URL, its type and counts, the latest <lastmod> and the window. Resource is the sitemap URL. | — | — |
37A-SitemapAuditor-201 | SUCCESS | A sampled URL answers HTTP 2xx without a redirect and has no noindex. One check per URL; does not change the overall status. | — | — |
37A-SitemapAuditor-101 | INFORMATION | A sampled URL could not be checked: HTTP 403, 429, a Cloudflare challenge or bot wall, a fetch error, or the 30 s sampling budget ran out. One check per URL; the reason is in the message and in unchecked. Does not change the overall status. | — | — |
37A-SitemapAuditor-501 | ERROR | The audited URL is missing, malformed or has no scheme or hostname, so no site root can be built. | Audit a URL with a scheme and a hostname, such as https://example.com/, instead of {url}. | — |
| Status | Description | Test logic |
|---|---|---|
| SUCCESS | A sitemap was found and is healthy | A candidate parsed, its latest <lastmod> is within SITEMAP_MAX_AGE_DAYS (200), and no warning fired. |
| WARNING | A sitemap was found but needs attention | Broken declaration while another works (303); stale (300) or missing (301) <lastmod>; robots.txt without a Sitemap: line (302); broken child (304); out-of-scope URLs (305); future <lastmod> (306); sampled URLs that fail (307), redirect (308) or are noindex (309). |
| FAIL | Crawlers get no usable sitemap | Nothing parsed (400); every declared sitemap broken (402); empty sitemap (401); a file over the protocol limits (403). |
| ERROR | The auditor could not run | The audited URL has no scheme or hostname (501). |
| INFORMATION | Specification pin, unchecked sampled URLs | Always emitted (100); one per sampled URL that could not be checked (101). Neither changes the overall status. |
The site-level checks (400, 402, 401, 403, 303, 300–306, 200) carry the same data map: siteBase, robotsTxtUrl, robotsTxtHttpStatus, robotsTxtListsSitemap, declaredSitemaps (url, ok, outcome per Sitemap: line), candidateSitemaps, selectedSitemapUrl, selectedFromFallback, sitemapType (urlset, sitemapindex, rss, atom or text), sitemapHttpStatus, contentType, entryCount (URLs, or <sitemap> entries of an index), childSitemaps (url, httpStatus, urlCount, error), childSitemapsSkipped, totalUrlCount, latestLastmod (ISO instant or null), futureLastmodCount, outOfScopeUrlCount, sampleSize, maxAgeDays and issues. The per-URL checks (101, 201, 307–309) carry only that URL: url, httpStatus, finalUrl, noindex (where it was found, or null), unchecked (why it was not checked, or null) and selectedSitemapUrl.
Search engines discover and prioritise pages from the sitemap; a site without one relies on internal links alone, and pages far from the home page may never be crawled.
<lastmod> is the signal crawlers use to recrawl changed pages first. Dates that never move, are missing or point to the future make every page look equally stale.
Listing the sitemap in robots.txt is the one discovery mechanism every crawler shares; Search Console submissions only reach Google. Crawlers do not try /sitemap.xml on their own, which is why a broken Sitemap: line is a FAIL even when the fallback works.
The window is SITEMAP_MAX_AGE_DAYS (default 7 days). A brochure site that rarely changes will warn with 300 even though nothing is wrong with its sitemap; the warning is a prompt to keep <lastmod> honest, not a defect of the site.
For an index, freshness comes from the <lastmod> of the index entries and of every URL in the children read, so an index without dates (like WordPress' wp-sitemap.xml) is judged by its posts.
robots.txt Allow/Disallow rules, including sitemap URLs that robots.txt blocks; those belong to a robots.txt auditor.
Children of an index past the first 50, and indexes nested deeper than one level.
Listed URLs outside the sample: a broken URL is only found when it is sampled. Raise sampleSize (up to 100) for a wider net.
Canonical tags, hreflang, image and video extensions, <priority> and <changefreq>.
A plain-text sitemap over 50,000 lines: crawler-commons stops reading it there, so 403 cannot fire for text files.
Sitemaps are parsed by crawler-commons: RSS, Atom and plain-text sitemaps are now evaluated instead of failing with an unknown root element.
Every Sitemap: line of robots.txt is fetched, and the fallback paths are tried whenever none of them parses (they used to be tried only when robots.txt declared nothing). A site whose declared sitemap is broken but whose /sitemap.xml works moves from 400 to 402, with the fallback sitemap now audited. A broken line next to a working one warns with 303.
The children of a sitemap index are read: freshness comes from their URLs, so an index without <lastmod> can move from 301 to 200 or 300; broken children warn with 304.
New checks: 401 empty sitemap, 403 over the protocol limits, 305 URLs on another scheme or host, 306 future <lastmod>, and one check per sampled URL: 307/308/309 for problems, 201 when healthy, 101 when it could not be checked (new sampleSize preference, default 20).
Date-only <lastmod> values are read as the start of that day in UTC (they used to be the end of the day), and unparsable values are no longer listed in issues.
An invalid audited URL (no scheme or hostname) is now an ERROR with 37A-SitemapAuditor-501; it used to be a FAIL with no check.
A sitemap without any parsable <lastmod> now gets its own warning, 37A-SitemapAuditor-301; it used to be reported as "out of date" together with the stale case.
A robots.txt that answers HTTP 200 without a Sitemap: line now warns with 37A-SitemapAuditor-302, so a site whose sitemap was only found at a fallback path is a WARNING rather than a SUCCESS.
The available field is gone from the check data; selectedSitemapUrl is null when no sitemap parsed.
This documentation is built and hosted on Mintlify, a developer documentation platform.
Add this auditor to your monitoring setup and start identifying issues on your websites today.