robots.txt Generator
Build a clean robots.txt: User-agent groups, Allow and Disallow rules, Crawl-delay and sitemap lines — ready to drop in your site root, right in your browser.
🔒 Your data never leaves the browser — not one character is sent anywhere, and nothing is stored
How to use it
- Name the user-agentWrite * to cover all crawlers, or a specific one like Googlebot or Bingbot — each gets its own rules.
- Set allowed and blocked pathsOne path per line, each starting with / — Allow admits it, Disallow blocks it. Optionally set Crawl-delay in seconds.
- Add your sitemap (optional)Paste the full sitemap.xml URL on its own line — it becomes a Sitemap: line at the end of the file.
- Copy the fileThe text generates live — the copy button takes all of it; place it in your site’s root named robots.txt.
Enjoying the tool? Help us keep it free forever — Support OMXHub ❤️
Frequently Asked Questions
What is robots.txt?
A simple text file named robots.txt placed at the root of your site (like https://example.com/robots.txt) telling search engines which paths they may crawl and which are off-limits. It has been the core standard for over 30 years and every major engine — not just Google — reads it, though not all enforce the rules identically.
What’s the difference between Allow and Disallow?
Disallow blocks a given crawler from a specific path (like images you don’t want in search results), while Allow admits one path despite a broader rule blocking it. The tool emits both for each User-agent group based on what you write, and every rule must start with / to count as a valid path.
Why must every rule start with /?
Because Allow and Disallow define paths relative to your site’s root: /admin means the admin folder under your root. Any value that doesn’t begin with / (just a plain word) isn’t a valid path, so the tool refuses to emit it — it would be a wrong entry in your file. The only exception is the * wildcard pattern matching anything.
Why doesn’t Crawl-delay always work?
Because it’s an advisory directive that not every engine honors: Bing and Yandex take it into account, but Google ignores it entirely. It sets how many seconds should pass between a crawler’s requests to your site. If your site is fast and lightweight you probably don’t need it at all — the tool flags this difference and emits the rule only if you supply a value.
What does the Sitemap line in the file do?
A Sitemap: line tells search engines where your XML sitemap lives so they can discover and index your pages faster and more accurately. It must be a full URL starting with https:// (the same one you submit to Google Search Console, but written here as plain text). The tool emits the line only when the URL is complete.
Does robots.txt actually hide content from the internet?
No, and we don’t pretend otherwise. robots.txt asks well-behaved crawlers to stay out of blocked paths; outside sites and direct links still open. If privacy genuinely matters (private data), real protection means permissions and authentication, not a text file. And everything here runs inside your browser with zero uploads.