Mission File 08 · Crawl control
Robots.txt Checker
Review crawler instructions and identify rules that may accidentally block pages, folders, assets, or search engine access.
Check robots.txt
Enter the full website address you want reviewed.
●Enter a website URL and click Check robots.txt to begin.
Analysis startedYour crawler instructions are being analyzed.
Robots.txt Analysis Results
Ready for analysis—crawl score
Crawler Access Health
The public report shows the number and type of robots.txt concerns detected without exposing the exact blocking rules or implementation changes.
What the checker reviews
6 checksRobots.txt availabilityWhether the standard robots.txt file can be retrieved.
Sitemap declarationsHow many XML sitemap references are declared.
Blocked directoriesDirectory-style crawl restrictions found in the file.
Blocked pagesPage-specific crawl restrictions found in the file.
Syntax problemsMalformed directives or incomplete crawler groups.
Crawl warningsBroad blocking, asset restrictions, or sitemap-reference problems needing review.
Robots.txt Analysis Results
Your completed public report appears here after the scan finishes.
Your completed public report appears here after the scan finishes.
—Robots.txt found
0Sitemap entries
0Blocked paths
0Total concerns