How to use it
- 1Choose the crawler user-agent.
- 2Enter the paths you want to allow or disallow.
- 3Add your sitemap URL if available.
- 4Generate and download your robots.txt file.
SEO tool
Robots.txt Generator
Generate a robots.txt file to define crawler access rules for your website and optionally include your XML sitemap URL.
Crawler rules
Generated file
Your robots.txt content will appear here.
Important: robots.txt is a crawler instruction file, not an access control or security mechanism. Check your rules before publishing.
Features
- ✓ Custom user-agent directives
- ✓ Allow and disallow rules
- ✓ Optional sitemap directive
- ✓ Plain-text file output
- ✓ Copy or download generated rules
- ✓ No account required
Privacy
Files are processed in your browser in this Version 1 implementation. Your selected file is not sent to a Toolbox server for processing.
Frequently asked questions
What is robots.txt?
Robots.txt is a text file that communicates crawler access rules for URLs on a website.
Does robots.txt prevent a page from appearing in search results?
Not necessarily. Blocking crawling does not guarantee that a URL will be excluded from search results.
Where should I upload robots.txt?
It is normally placed at the root of your website, such as https://example.com/robots.txt.