Validates robots.txt crawl-delay directive presence
This checker validates that websites have a robots.txt file with crawl-delay directive to control search engine crawling frequency. The checker fetches the robots.txt file from the website, scans for crawl-delay directive, and validates the directive format and presence.
This checker validates that websites have a robots.txt file with crawl-delay directive to control search engine crawling frequency. It fetches the robots.txt file from the website, scans for crawl-delay directive, and validates the directive format and presence.
| Status | Condition | Test Logic |
|---|---|---|
| SUCCESS | Crawl-delay directive found | robots.txt contains crawl-delay directive |
| WARNING | No crawl-delay directive | robots.txt exists but lacks crawl-delay directive |
| FAIL | robots.txt not found | robots.txt file inaccessible or missing |
Server Overload: Without crawl-delay, search engines may overwhelm your server with requests, causing performance issues. Increased Costs: Excessive crawling can increase bandwidth and hosting costs. Poor User Experience: Server overload can slow down your website for actual users.
Add this checker to your monitoring setup and start identifying issues on your websites today.