What this robots.txt checker checks
- Whether robots.txt is reachable
- Allow and Disallow directives for the selected user agent
- Declared XML sitemap locations
- The final robots.txt URL and HTTP status
How to read the result
Robots.txt controls crawling, not guaranteed indexing. Blocking a page can prevent crawlers from seeing a noindex directive. Important public pages should normally be crawlable and linked from the site.
How to use it
- Enter the complete public HTTPS URL you want to inspect.
- Select the check button and wait for the live response.
- Review the result and correct any issue at the website or hosting provider.
- Run the checker again to confirm the public response changed.
Common questions
Does Disallow remove a page from Google?
Not reliably. A blocked URL can still appear if Google discovers it elsewhere. Use noindex on a crawlable page when removal is required.
Where should robots.txt live?
At the root of the hostname, such as https://example.com/robots.txt.
Should a robots file include the sitemap?
It is helpful and widely supported. The sitemap can also be submitted directly through search-engine webmaster tools.