Robots.txt Generator
Build a robots.txt file to control which crawlers can access which parts of your site. Free, no sign-up, runs in your browser.
Loading tool…
About this tool
Set the default rule, add paths to block, choose whether to block AI training crawlers and add your sitemap. Copy the result to the root of your domain.
How to use it
- Enter or paste your values in the fields above.
- The result updates as you type.
- Use the Copy button to copy the result.
Frequently asked questions
Does robots.txt hide pages from Google?
No. It stops crawling, not indexing. To keep a page out of search results use a noindex meta tag.
Where does the file go?
At the root of your domain, for example https://www.example.com/robots.txt.
Let’s talk
Need a team to build it?
We build the software behind tools like this: web apps, APIs, SaaS products and AI features.