Generate robots.txt files for search engine crawlers. Set User-agent, Allow, Disallow rules, Sitemap URL. Validates syntax. Download robots.txt instantly.
Use this free online robots.txt Generator directly in your browser. No signup required, no data leaves your device. Part of Utilier — a collection of 133+ developer utilities.
What is robots.txt Generator & Validator?
robots.txt is a file placed at the root of a website (example.com/robots.txt) that tells search engine crawlers (Googlebot, Bingbot, etc.) which pages or sections of the site to crawl or avoid. It uses directives like User-agent (which crawler), Disallow (do not crawl these paths), Allow (override Disallow), and Sitemap (where to find the XML sitemap). This tool generates robots.txt files visually with syntax validation, preventing errors that could accidentally block important pages from search engines.
User-agent directives: Specify rules for all crawlers (*) or specific bots (Googlebot, Bingbot, GPTBot). Different rules for different bots.
Disallow rules: Block crawlers from specific paths: /admin/, /private/, /api/. Prevents indexing of sensitive or duplicate content.
Allow rules: Override Disallow for specific paths. Example: Disallow /admin/, Allow /admin/public/.
Crawl-delay: Optional delay (seconds) between requests. Prevents server overload from aggressive crawlers.
Why use robots.txt generator?
Incorrect robots.txt syntax can accidentally block your entire site from search engines, causing traffic loss. This tool ensures correct syntax and prevents common errors.
Prevent indexing of sensitive pages: Block /admin/, /login/, /checkout/, or duplicate content from search results. Keeps private pages out of Google.
Avoid syntax errors: Syntax errors (wrong capitalization, missing slashes, invalid wildcards) can cause rules to fail or block the wrong pages.
Control crawl budget: For large sites, blocking low-value pages (tags, filters, search results) saves crawl budget for important pages.
Comply with regulations: Block AI crawlers (GPTBot, CCBot) from training on your content if required by policy or licensing.
Test before deploying: Validate robots.txt syntax before uploading. Avoid accidentally blocking the entire site with Disallow: /.
Learn robots.txt syntax: See how directives work. Understand User-agent, Disallow, Allow, and wildcard patterns.
When to use robots.txt generator
Use when setting up a website or controlling search engine access.
Launching a new website to guide crawlers to the sitemap and block admin pages.
Blocking duplicate or low-quality pages (tags, search results, filters) from indexing.
Preventing sensitive content (login, checkout, user profiles) from appearing in search results.
Controlling which AI crawlers can access your content (block GPTBot, allow Googlebot).
Debugging crawl issues (check if pages are accidentally blocked by robots.txt).
Optimizing crawl budget for large sites by blocking non-essential pages.
Specifying sitemap location for better search engine discovery.
Set User-agent: Choose * (all crawlers), Googlebot, Bingbot, or specific bot names. Use * for global rules, specific names for bot-specific rules.
Add Disallow rules: Specify paths to block: /admin/, /private/, /api/. Use / for root, * for wildcards, $ for end-of-path. Example: /admin/* blocks all admin subpages.
Add Allow rules (optional): Override Disallow for specific paths. Example: Disallow /files/, Allow /files/public/. Allow rules must be more specific than Disallow.
Specify Sitemap URL: Add Sitemap: https://example.com/sitemap.xml. Helps search engines find all pages. Sitemap directive is global (not per User-agent).
Set Crawl-delay (optional): Add Crawl-delay: 10 (10 seconds between requests). Used to slow down aggressive crawlers. Not supported by Googlebot.
Validate and download: The tool validates syntax (case-sensitivity, missing slashes, invalid patterns). Download robots.txt and upload to the root of your site.
Key features
User-agent selection: Set rules for all crawlers (*) or specific bots (Googlebot, Bingbot, GPTBot, etc.).
Disallow and Allow directives: Block paths with Disallow, override with Allow. Supports wildcards (* and $).
Sitemap directive: Specify sitemap URL for search engines to discover pages.
Crawl-delay: Set delay between crawler requests (seconds). Helps prevent server overload.
Syntax validation: Checks for common errors: missing User-agent, invalid wildcards, wrong case, missing slashes.
Comment support: Add comments with # for documentation. Example: # Block admin pages.
Instant download: Download robots.txt file ready to upload to the root of your site.
Common use cases
Block admin and login pages: Prevent /admin/, /login/, /wp-admin/ from appearing in search results.
Block duplicate content: Prevent tag pages, search results, or filter URLs from indexing (saves crawl budget).
Control AI crawler access: Block GPTBot, CCBot, or other AI crawlers from training on your content.
Specify sitemap location: Add Sitemap: https://example.com/sitemap.xml for search engines to discover all pages.
Slow down aggressive crawlers: Use Crawl-delay to prevent server overload from bots making hundreds of requests per second.
Debug indexing issues: Check if important pages are accidentally blocked by robots.txt.
Blocks /files/* except /files/public/*. Allow must be more specific than Disallow. Order matters (Allow after Disallow).
Technical reference
robots.txt syntax and directives:
File location
Must be at the root: https://example.com/robots.txt. Subdirectories (https://example.com/blog/robots.txt) are ignored.
User-agent directive
Specifies which crawler the rules apply to. * = all crawlers. Googlebot = Google only. Case-insensitive.
Disallow directive
Paths the crawler should not access. Disallow: /admin/ blocks /admin/ and all subpages. Disallow: / blocks entire site.
Allow directive
Overrides Disallow for specific paths. Must be more specific. Example: Disallow: /files/, Allow: /files/public/.
Wildcards (* and $)
* matches any sequence (Disallow: /*.pdf blocks all PDFs). $ matches end of path (Disallow: /page$ blocks /page but not /page/subpage).
Sitemap directive
Sitemap: https://example.com/sitemap.xml. Tells crawlers where to find the sitemap. Global (not per User-agent).
Crawl-delay
Crawl-delay: 10 (10 seconds between requests). Not supported by Googlebot. Used by Bing, Yandex, and some other crawlers.
Comments
Lines starting with # are comments. Example: # Block admin pages. Used for documentation.
Case-sensitivity
Directives (User-agent, Disallow) are case-insensitive. Paths (/Admin/ vs /admin/) are case-sensitive on case-sensitive servers (Linux).
Comparison to meta robots tag
robots.txt blocks crawling (fetching pages). <meta name='robots' content='noindex'> blocks indexing (showing in results). Use both for full control.
Common mistakes to avoid
Using Disallow: / thinking it only blocks the homepage, accidentally blocking the entire site
Why it happens: Disallow: / means 'block the root path and everything under it' (the entire site). Many beginners think it only blocks the homepage (example.com/) but it actually blocks example.com/, example.com/page, example.com/blog, etc. This is the most common robots.txt error — it delists the entire site from search engines.
How to avoid it: To block only the homepage, use Disallow: /$ ($ means end of path). To block a specific page, use the full path: Disallow: /page.html. To block a directory, use trailing slash: Disallow: /admin/. Never use Disallow: / on a live site unless you want to block everything.
Forgetting the trailing slash in Disallow: /admin, blocking unintended paths
Why it happens: Disallow: /admin (no trailing slash) blocks anything starting with /admin: /admin, /admin/, /admin-panel/, /administrator/. If you meant to block only /admin/, you accidentally block /admin-notes/ or /administrator/ too. Without the slash, it is a prefix match, not a directory match.
How to avoid it: For directories, always use a trailing slash: Disallow: /admin/. This blocks /admin/ and subpages, but not /admin-panel/. For specific files, use the full path: Disallow: /file.html. Test with Google Search Console's robots.txt tester to verify behavior.
Assuming robots.txt prevents pages from appearing in search results (it does not)
Why it happens: robots.txt blocks crawling (fetching pages), not indexing. If a page is linked from external sites, Google may index it even if robots.txt blocks crawling. The page appears in search results with 'A description for this result is not available because of this site's robots.txt'. To prevent indexing, use <meta name='robots' content='noindex'> on the page itself.
How to avoid it: For truly private pages, use both: robots.txt to block crawling AND meta robots tag (or X-Robots-Tag HTTP header) to prevent indexing. Or use authentication (login required). robots.txt is not a security measure — it is a crawl guideline.
Not specifying a Sitemap in robots.txt, making page discovery slower
Why it happens: If you do not specify a sitemap location, search engines must discover pages by crawling links. For new sites or sites with poor internal linking, this takes weeks. Adding Sitemap: https://example.com/sitemap.xml in robots.txt tells crawlers where to find all pages immediately, speeding up indexing.
How to avoid it: Always include Sitemap: https://example.com/sitemap.xml in robots.txt. Place it at the end of the file (not inside a User-agent block). You can specify multiple sitemaps if needed. Also submit the sitemap to Google Search Console and Bing Webmaster Tools for redundancy.
Using wildcards (* or $) without understanding their behavior, blocking unexpected paths
Why it happens: Disallow: /*.pdf blocks all PDFs anywhere on the site (correct). But Disallow: /*admin blocks anything with 'admin' anywhere in the path: /admin/, /user-admin/, /myadmin/ (probably too broad). Wildcards are powerful but can match more than intended if not carefully tested.
How to avoid it: Test wildcard patterns in Google Search Console's robots.txt tester. For specific directories, use exact paths without wildcards: Disallow: /admin/. For file types, use *: Disallow: /*.pdf. For end-of-path matching, use $: Disallow: /page$ (blocks /page but not /page/subpage).
Frequently asked questions
What is the difference between robots.txt and meta robots tag?
robots.txt blocks crawling (fetching pages from the server). Meta robots tag (<meta name='robots' content='noindex'>) blocks indexing (showing pages in search results). robots.txt is a file at the root. Meta robots is a tag on individual pages. Use robots.txt to save crawl budget, meta robots to prevent indexing.
Where do I upload robots.txt?
At the root of your website: https://example.com/robots.txt. Not in a subdirectory (https://example.com/blog/robots.txt is ignored). Upload via FTP, cPanel, or your hosting control panel. Verify it is accessible by visiting example.com/robots.txt in a browser.
Does robots.txt prevent pages from appearing in Google search results?
No. robots.txt blocks crawling, not indexing. If a page is linked from external sites, Google may index it without crawling (shows in results with 'A description for this result is not available'). To prevent indexing, use <meta name='robots' content='noindex'> on the page.
Can I block specific crawlers like GPTBot or Bingbot?
Yes. Use User-agent: GPTBot, Disallow: / to block OpenAI's crawler. User-agent: Bingbot, Disallow: / to block Bing. Each User-agent block applies only to that bot. User-agent: * applies to all bots not explicitly listed.
What does Disallow: / mean?
Blocks the entire site from the specified crawler. Disallow: / means 'do not crawl the root path or anything under it' (the whole site). Use this for staging sites or to block specific bots. Never use on a live production site unless you want to block all crawling.
How do I allow everything in robots.txt?
Use User-agent: *, Disallow: (empty). This means 'allow all crawlers to access everything'. This is the default if no robots.txt exists. You can also add Sitemap: https://example.com/sitemap.xml to guide crawlers.
Does Googlebot respect Crawl-delay?
No. Googlebot ignores Crawl-delay. Use Google Search Console to adjust crawl rate instead. Bing, Yandex, and some other crawlers do respect Crawl-delay. Set it to slow down aggressive crawlers (Crawl-delay: 10 = 10 seconds between requests).