Mission File 08 · Crawl control

Robots.txt Checker

Review crawler instructions and identify rules that may accidentally block pages, folders, assets, or search engine access.

Check robots.txt

Enter the full website address you want reviewed.

●Enter a website URL and click Check robots.txt to begin.

Analysis startedYour crawler instructions are being analyzed.

Robots.txt Analysis Results

Ready for analysis

—crawl score

Crawler Access Health

The public report shows the number and type of robots.txt concerns detected without exposing the exact blocking rules or implementation changes.

What the checker reviews

6 checks

Robots.txt availabilityWhether the standard robots.txt file can be retrieved.

Sitemap declarationsHow many XML sitemap references are declared.

Blocked directoriesDirectory-style crawl restrictions found in the file.

Blocked pagesPage-specific crawl restrictions found in the file.

Syntax problemsMalformed directives or incomplete crawler groups.

Crawl warningsBroad blocking, asset restrictions, or sitemap-reference problems needing review.

Robots.txt Analysis Results
Your completed public report appears here after the scan finishes.

—Robots.txt found

0Sitemap entries

0Blocked paths

0Total concerns

How to use the Robots Checker results

Robots directives control crawling and indexing in different ways, so they should be reviewed carefully before making changes. A robots.txt rule can limit crawler access, while a page-level noindex directive addresses whether a page should appear in search results. Use this check to identify directives that may block important content, but do not remove intentional restrictions from private, administrative, login, test, or other pages that should not be indexed.