Robots.txt Generator
Create a robots.txt file for your website in seconds. Pick a preset or add your own rules, then copy or download the file.
Configure Robots.txt
Control which search engine bots can crawl your website and what content they can access.
Start from a preset
Crawling Rules
How to create a robots.txt file
A robots.txt file is a small text file in the main folder of your site, like yoursite.com/robots.txt. It tells search engine bots which parts of your site they may visit.
1. Choose your rules
Pick a preset or add rules. Allow the pages you want in Google. Block admin pages, cart pages and your own site's search pages.
2. Add your sitemap
Put your sitemap link at the bottom so bots find every page faster.
3. Upload and test
Save the file as robots.txt in your site's main folder, open yoursite.com/robots.txt to see it, then test it with our robots.txt validator.
Careful: one line like Disallow: / blocks your whole site from Google. A robots.txt file also does not hide private pages; use a password or a noindex tag for those. Related tools: XML sitemap generator, robots.txt validator, website analyzer.
Frequently Asked Questions
Common questions about this tool and how to use it effectively
A robots.txt file is a text file placed in your website's root directory that tells search engine crawlers which pages or sections of your site they can or cannot access. It's essential for controlling crawler behavior, preventing duplicate content indexing, protecting sensitive areas, and managing your crawl budget efficiently.
The robots.txt file must be placed in the root directory of your website (e.g., https://example.com/robots.txt). It cannot be placed in a subdirectory or subfolder. Search engines will only look for it at the domain root, and it must be publicly accessible without authentication.
User-agent directives specify which crawler bot the rules apply to. Using '*' applies rules to all bots universally. You can also target specific bots like 'Googlebot' for Google or 'Bingbot' for Bing. If you want different rules for different bots, create multiple user-agent sections in your robots.txt file.
Create separate user-agent sections for each bot. For example, to block only Bingbot while allowing others, use 'User-agent: Bingbot' followed by 'Disallow: /' to block everything. Then add 'User-agent: *' with specific Allow/Disallow rules for other bots. Each user-agent section works independently.
Yes, including your sitemap URL in robots.txt helps search engines discover all your pages more efficiently. Add it at the end of your file using the format 'Sitemap: https://example.com/sitemap.xml'. You can include multiple sitemap URLs if your site has several sitemaps for different content types.
Yes, you can use wildcards like '*' (matches any sequence of characters) and '$' (matches end of URL). For example, 'Disallow: /*.pdf$' blocks all PDF files, and 'Disallow: /private*' blocks any URL starting with /private. However, wildcards are not supported by all crawlers, so use them carefully.
Use Google Search Console's robots.txt Tester tool to validate your file and test specific URLs. This tool shows you exactly how Googlebot will interpret your rules and helps identify syntax errors. Always test before deploying to avoid accidentally blocking important pages from search engines.
Yes. Make as many robots.txt files as you like, with no signup.
Yes. Pick the Blogger preset and add your blog address. Then in Blogger, open Settings, turn on Custom robots.txt under Crawlers and indexing, and paste the file.
Still have questions?
Contact our team for helpRelated Tools You Might Like
Continue optimizing your website with these complementary tools
Website Analyzer
Evidence-backed checks across search, speed, security, accessibility, and conversion basics
Canonical URL Checker
Detect canonical tag issues that hurt SEO rankings
Duplicate Content Checker
Compare text blocks to detect duplicate or similar content (lexical + AI semantic similarity)