Beta
AI crawler checker
Check whether ChatGPT, Perplexity, and other AI search crawlers can access your website.
credit cost
0
Overview
What it does
Find blocked pages, robots.txt restrictions, and other technical issues that could prevent AI systems from retrieving your content.
The checker fetches each URL you paste and runs the technical checks that decide whether an AI search crawler can retrieve the page.
Why check AI crawler access?
AI search engines need to retrieve content before they can use it in certain search and answer-generation workflows. If your pages are blocked by robots.txt, return errors, or contain restrictions, that can limit how some AI systems access your content.
These issues are not always obvious. A page might load normally in your browser but return a different response to an automated crawler.
The AI crawler checker helps identify technical restrictions worth investigating before you spend time making other changes to improve AI search visibility.
What does the AI crawler checker check?
Check | What it tells you |
|---|---|
Status and redirects | Whether the page can be retrieved successfully, with the final HTTP status and the redirect chain up to five hops. |
robots.txt by crawler | Whether your crawler rules allow or disallow each supported crawler: OAI-SearchBot, GPTBot, PerplexityBot, Claude-SearchBot, ClaudeBot, Bingbot, and Googlebot, shown one crawler at a time. |
noindex | Whether the page carries noindex in a meta robots tag or an X-Robots-Tag header, including crawler-specific tags. A noindex addressed to any crawler is flagged. |
Canonical | Whether the canonical points to the page itself or to another URL, with the target shown. This is diagnostic context rather than a crawler block. |
Crawler response | Whether the page answers the OAI-SearchBot user agent the same way it answers a browser. This result is approximate: real crawlers are verified by IP address, so the check catches blocking based on user agent only. |
Sitemap | Whether the URL appears in a sitemap listed in robots.txt, or at /sitemap.xml if none is listed. This affects URL discovery rather than whether a crawler can reach the page. |
Text in the HTML | Whether the main text is in the raw HTML. The checker does not run JavaScript, so a page with little raw text is flagged as possibly depending on JavaScript for its content. |
The crawlers are reported separately because they do different jobs. Some fetch pages for search and answer features, others collect pages to train models, so allowing one does not mean allowing another.
For pages that pass, the checker also picks up to two distinctive passages from the main content and gives you prefilled links to ask ChatGPT, Perplexity, and Claude whether that exact text comes back in their search results. You run those links in your own accounts.
How to interpret your results
Each URL gets an overall result of Ready, Issues found, or Blocked, followed by one row per check with a pass or fail and a plain-language reason.
Ready. The page passed the checks performed by the tool. No blocking issue was identified, but this does not guarantee that an AI search engine has crawled, indexed, or cited the page.
Issues found. One or more checks identified something worth reviewing. This could include a canonical pointing elsewhere, missing sitemap inclusion, or content that depends on JavaScript. Not every issue requires a change.
Blocked. The tool identified a restriction or error that may prevent a crawler from retrieving the page. Review the specific failed checks before deciding what to change.
What should you fix first?
HTTP errors preventing access to important pages.
Unintended robots.txt restrictions for the AI crawlers you want to allow.
Accidental indexing restrictions, where relevant.
Important content missing from the initial HTML.
Canonical and sitemap inconsistencies that could affect URL discovery or interpretation.
Some restrictions are deliberate. A staging path disallowed on purpose, or a noindex on a thin tag page, should stay as it is. Check what a restriction is there for before removing it.
Does crawler access mean your page will appear in AI search?
No. Passing an AI crawler check means the tool did not identify a technical issue preventing access under the conditions tested.
It does not tell you whether ChatGPT, Perplexity, or another AI search engine has crawled your page, selected it as a source, or will include it in an answer.
AI search visibility also depends on the content, the question being asked, the sources an AI system uses, and how that system retrieves and selects information.
Method
How it works
1. Enter your URLs
Add up to 50 URLs you want to check. Start with important pages such as product pages, service pages, articles, or other content you want AI search engines to access.
2. Review the results
See whether each URL is marked Ready, Issues found, or Blocked. Expand the individual checks to understand what was detected.
3. Investigate and fix issues
Use the results to identify restrictions, server errors, or content accessibility problems. Make changes where appropriate, then run the check again.
FAQ