Microdeck.tools 100% In-Browser

robots.txt RFC 9309 Tester & sitemap.xml Builder

New

Avoid accidentally deindexing production pages or blocking Googlebot assets. Paste any `robots.txt` file and simulate `Googlebot`, `GPTBot`, or custom crawlers against exact site paths using RFC 9309 longest-specificity rule precedence—then compile clean, escaped `<urlset>` XML `sitemap.xml` manifests ready to deploy.

100% In-Browser No Server Storage Instant Results

1. RFC 9309 `robots.txt` Longest-Match Crawler Tester

ALLOWED (*)
Matched Rule: Allow: /tools/

2. XML `sitemap.xml` Generator

Did this tool save you time? Help keep Microdeck free and maintained. Completely optional · From $3.

How It Works Under the Hood

The evaluator selects the most specific matching `User-agent` group (falling back to `*` only when no targeted agent group matches), converts wildcard `*` and end-anchor `$` directives into anchored path matchers, and picks the longest matching pattern per RFC 9309 (preferring `Allow` on length ties).

Step-by-Step Instructions

1

Paste your `robots.txt` contents into Section 1 and specify Crawler User-Agent (`Googlebot`, `GPTBot`, `*`) + URL Path.

2

Check whether the URL returns `ALLOWED` or `BLOCKED` and which exact line rule triggered.

3

In Section 2, paste canonical `https://` URLs (one per line) and choose `changefreq`, `priority`, and `lastmod`.

4

Copy or download `sitemap.xml` directly into your site's `public/` directory.

Key Features & Performance

  • True RFC 9309 longest-pattern precedence (handles `Allow: /tools/` overriding `Disallow: /`)
  • Supports `*` wildcards and `$` end-of-path anchors inside `Allow` and `Disallow` patterns
  • Simulates targeted bot rules (`GPTBot`, `Bingbot`, `Googlebot`) cleanly with wildcard `*` fallback
  • XML-escapes URL characters cleanly (`&`, `<`, `>`) when compiling `<urlset>` sitemaps
  • 100% browser-local: test unreleased staging site architectures privately

Methodology, Formulas & Technical Standards

Follows RFC 9309 §2.2.2 longest-match specificity rather than naive top-to-bottom first match, resolving ties in favor of `Allow` directives and defaulting to allowed when no pattern matches.

Common Real-World Use Cases

  • Testing complex `Disallow: /*?preview=` and `Allow:` overrides before deploying SEO updates
  • Verifying AI crawler policies (`GPTBot`, `ClaudeBot`, `CCBot`) without touching live production files
  • Generating clean `sitemap.xml` manifests for landing pages, microsites, or Cloudflare Pages deploys
  • Debugging unexpected Search Console "Blocked by robots.txt" warnings

Worked Examples & Edge Cases

Specific Allow Beats Shorter Disallow

Input
Disallow: /tools/ | Allow: /tools/env-diff-sanitizer/ | Path: /tools/env-diff-sanitizer/
Output
ALLOWED (longer Allow rule overrides shorter Disallow)

RFC 9309 resolves by matched character length, not file line number.

Agent-Specific Block

Input
User-agent: GPTBot -> Disallow: /
Output
BLOCKED for GPTBot, ALLOWED for Googlebot (falls back to `*`)

Targeted agent groups do not inherit wildcard `*` group rules.

Limitations, Scope & Accuracy Notes

  • • Sitemap output assumes clean canonical URLs under the 50,000-URL single-file limit; larger catalogs require splitting into a `<sitemapindex>`.
  • • Non-standard proprietary extensions (`Crawl-delay`) are ignored safely during path allow/block resolution.

Local Browser Processing & Privacy

Traditional web converters upload your documents, snippets, or media files to remote servers. robots.txt RFC 9309 Tester & sitemap.xml Builder runs inside your browser tab:

Processed in This Tab

Your input text and files are processed locally without uploading payloads to a processing server.

No Tool Payload Logging

Microdeck does not log or store the files or inputs you paste into tool workspaces (normal static hosting/CDN requests still load page assets).

Tab Memory Lifecycle

Working outputs stay in your browser tab and are released when you reset the tool or close the tab.

Frequently Asked Questions

Why does `Allow: /tools/` beat `Disallow: /` even if written second?

Modern crawlers following RFC 9309 pick whichever `Allow` or `Disallow` path pattern has the longest match length, breaking exact length ties in favor of `Allow`.

Does `Googlebot` inherit rules from `User-agent: *` if a dedicated `User-agent: Googlebot` block exists?

No. Once a bot matches a specific `User-agent:` group, crawlers use ONLY that block and ignore `User-agent: *`.

Are special `&` query characters escaped properly in `sitemap.xml`?

Yes. XML special characters (`&`, `<`, `>`, `"`, `'`) are escaped safely inside `<loc>` elements.

Does this fetch my live website?

No. Paste your policy text and URLs locally.

Explore Related Micro-Tools

Support Microdeck

Completely optional · From $3 · No account required

Choose an amount (USD):Selected:$5

Minimum support amount is $3 USD.

Completely optional · From $3 · Checkout opens on Gumroad.