CrawlReady

Free tool · No signup

robots.txt tool

One wrong line in your robots.txt can block ChatGPT, Perplexity or Google from your entire site — and you'd never know. Pick your site below and we'll check the file line by line, in plain English, using the same rules as a full scan — then give you a fixed file that keeps your own rules and welcomes AI crawlers.

We'll fetch the selected site's robots.txt and check it.

What we check

Five things that break AI visibility.

🚫 Blocking everything

Disallow: / under User-agent: * locks out every search engine and AI crawler. Fine for a staging site — disastrous for a live one.

🤖 Blocking AI crawlers

Rules aimed at GPTBot, ClaudeBot or PerplexityBot stop AI tools from reading and recommending your site. Sometimes deliberate — usually an accident.

⚠️ Syntax errors

Typos in directive names, rules before any User-agent line, or malformed entries that crawlers silently ignore.

🗺️ Missing sitemap

No Sitemap: line means crawlers have to discover your pages on their own — and may miss some.

🃏 Wildcard mistakes

Wildcards like User-agent: GPT* aren't understood by every crawler. Full bot names work everywhere.

Fixing it

Three steps, about five minutes.

  1. Copy or download the fixed file above. If you downloaded it, it's already named robots.txt — keep that exact name, all lowercase.
  2. Upload it to the very top level of your website — the same place your homepage lives. Use your hosting company's file manager, an FTP app, or your website builder's file upload.
  3. Check it worked: open yoursite.com/robots.txt in your browser. If you see the new text, you're done.
Don't want to do it yourself? Send this to your web person:

Hi — our robots.txt file has some issues (it's blocking AI crawlers / has errors). Could you please replace the file at the root of our website (so it loads at yoursite.com/robots.txt) with the corrected version attached? Thanks!