Before you publish
- Place the file at your hostname’s root, for example https://example.com/robots.txt. Rules apply only to that host, protocol and port.
- Robots.txt requests crawler cooperation. It is not access control, and it does not protect private files.
- A blocked URL can still appear in search. To prevent indexing, use appropriate indexing directives on a page the crawler can access.
- Do not block resources needed to render your public pages. Check the published file and test important URLs afterward.
How these rules work
User-agent identifies a crawler group. Allow and Disallow specify case-sensitive paths. Google uses the most specific matching path; Allow wins an equally specific conflict. Other crawlers can behave differently.
Specific crawler groups do not inherit the general group’s rules. Sitemap points to a complete HTTP or HTTPS URL. This generator checks syntax; it does not fetch your sitemap or verify your site’s rules.
See Google’s robots.txt reference and Anthropic’s current crawler documentation. The legacy Claude-Web option is retained for existing configurations.