Crawler configuration
Generate a safe robots.txt starting point.
This generator allows normal crawling and adds an absolute sitemap URL. Review any private paths before deployment.
Generated file
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xml
Do not use robots.txt to guarantee de-indexing. Use appropriate access controls or noindex where applicable.