How It Works Under the Hood
The evaluator selects the most specific matching `User-agent` group (falling back to `*` only when no targeted agent group matches), converts wildcard `*` and end-anchor `$` directives into anchored path matchers, and picks the longest matching pattern per RFC 9309 (preferring `Allow` on length ties).
Step-by-Step Instructions
Paste your `robots.txt` contents into Section 1 and specify Crawler User-Agent (`Googlebot`, `GPTBot`, `*`) + URL Path.
Check whether the URL returns `ALLOWED` or `BLOCKED` and which exact line rule triggered.
In Section 2, paste canonical `https://` URLs (one per line) and choose `changefreq`, `priority`, and `lastmod`.
Copy or download `sitemap.xml` directly into your site's `public/` directory.
Key Features & Performance
- True RFC 9309 longest-pattern precedence (handles `Allow: /tools/` overriding `Disallow: /`)
- Supports `*` wildcards and `$` end-of-path anchors inside `Allow` and `Disallow` patterns
- Simulates targeted bot rules (`GPTBot`, `Bingbot`, `Googlebot`) cleanly with wildcard `*` fallback
- XML-escapes URL characters cleanly (`&`, `<`, `>`) when compiling `<urlset>` sitemaps
- 100% browser-local: test unreleased staging site architectures privately
Methodology, Formulas & Technical Standards
Follows RFC 9309 §2.2.2 longest-match specificity rather than naive top-to-bottom first match, resolving ties in favor of `Allow` directives and defaulting to allowed when no pattern matches.
Common Real-World Use Cases
- Testing complex `Disallow: /*?preview=` and `Allow:` overrides before deploying SEO updates
- Verifying AI crawler policies (`GPTBot`, `ClaudeBot`, `CCBot`) without touching live production files
- Generating clean `sitemap.xml` manifests for landing pages, microsites, or Cloudflare Pages deploys
- Debugging unexpected Search Console "Blocked by robots.txt" warnings
Worked Examples & Edge Cases
Specific Allow Beats Shorter Disallow
Disallow: /tools/ | Allow: /tools/env-diff-sanitizer/ | Path: /tools/env-diff-sanitizer/
ALLOWED (longer Allow rule overrides shorter Disallow)
RFC 9309 resolves by matched character length, not file line number.
Agent-Specific Block
User-agent: GPTBot -> Disallow: /
BLOCKED for GPTBot, ALLOWED for Googlebot (falls back to `*`)
Targeted agent groups do not inherit wildcard `*` group rules.
Limitations, Scope & Accuracy Notes
- • Sitemap output assumes clean canonical URLs under the 50,000-URL single-file limit; larger catalogs require splitting into a `<sitemapindex>`.
- • Non-standard proprietary extensions (`Crawl-delay`) are ignored safely during path allow/block resolution.
Local Browser Processing & Privacy
Traditional web converters upload your documents, snippets, or media files to remote servers. robots.txt RFC 9309 Tester & sitemap.xml Builder runs inside your browser tab:
Your input text and files are processed locally without uploading payloads to a processing server.
Microdeck does not log or store the files or inputs you paste into tool workspaces (normal static hosting/CDN requests still load page assets).
Working outputs stay in your browser tab and are released when you reset the tool or close the tab.
Frequently Asked Questions
Why does `Allow: /tools/` beat `Disallow: /` even if written second?
Modern crawlers following RFC 9309 pick whichever `Allow` or `Disallow` path pattern has the longest match length, breaking exact length ties in favor of `Allow`.
Does `Googlebot` inherit rules from `User-agent: *` if a dedicated `User-agent: Googlebot` block exists?
No. Once a bot matches a specific `User-agent:` group, crawlers use ONLY that block and ignore `User-agent: *`.
Are special `&` query characters escaped properly in `sitemap.xml`?
Yes. XML special characters (`&`, `<`, `>`, `"`, `'`) are escaped safely inside `<loc>` elements.
Does this fetch my live website?
No. Paste your policy text and URLs locally.
Explore Related Micro-Tools
Open Graph & Social SERP Previewer
Simulate link previews across Google, Twitter/X, Discord, Slack, Facebook & iMessage
JSON-LD Structured Data & Schema Builder
Build valid Schema.org `<script type="application/ld+json">` embeds for FAQPage, Article, Product, SoftwareApplication & BreadcrumbList
Bulk HTTP 301/302 Redirect Rules Generator
Compile bulk URL redirect lists into Cloudflare/Netlify `_redirects`, Cloudflare Bulk CSV, Nginx, Apache `.htaccess`, or `vercel.json`