How to use it
- Type your sitemap address, for example https://www.yourshop.com/sitemap.xml.
- List folders you do not want search engines to read, one per line.
- Choose whether AI answer tools may read your site.
- Download robots.txt and upload it to the top folder of your website.
- Open yourwebsite.com/robots.txt in a browser to check it.
AI crawlers
Allowing AI crawlers lets tools such as ChatGPT, Gemini and Perplexity learn about your business and mention it in answers. If you do not want your content used by them, switch the option off and the file asks them to stay out. It is a request that well-behaved crawlers follow.
Do not rely on it for secrets
Robots.txt does not hide pages from people, and anyone can read the file. Never use it to protect private data. Use a password or keep such pages off the website.