CheckLLMs: Which AI crawlers can reach your site?
Audit your domain against 23 AI crawlers, graded 0–100.
- In development
- 23 Crawlers Indexed
- CF Workers
Audit your domain against 23 AI scrapers — including GPTBot, ClaudeBot, and Google-Extended — and grade your exposure from 0 to 100.
CheckLLMs is a free AI crawler audit tool that checks any domain against 23 known AI training and search bots. It grades your exposure on a 0–100 scale (Grade A–F) and outputs ready-to-paste robots.txt, ai.txt, and edge config rules. No login, no account, results in under a minute.
One scan. Clear fixes.
Most site owners configured their robots.txt before LLM bots became widespread. CheckLLMs resolves your domain and fetches your robots.txt and ai.txt files live, straight from your origin server.
It reads every directive, maps it against 23 scrapers (GPTBot, ClaudeBot, Google-Extended, Bytespider, CCBot, and more), and grades how well your domain is protected. Then it outputs clean, copy-paste config scripts for Nginx, Apache, Cloudflare WAF, and Next.js edge middleware.
Weighting critical bots
CheckLLMs grades your protection A–F using a weighted scoring model. Critical bots count 2× because they handle most of the LLM data ingestion happening today:
A crawler counts as protected if it is blocked either at the path level in robots.txt or at the purpose level via the newer ai.txt standard.
- GPTBot (OpenAI) · Weighted 2×
- ClaudeBot (Anthropic) · Weighted 2×
- Google-Extended (Google) · Weighted 2×
- Applebot-Extended (Apple) · Weighted 2×
- CCBot (Common Crawl) · Weighted 2×
- Bytespider (ByteDance) · Weighted 2×
- Meta-ExternalAgent (Meta) · Weighted 2×
- PerplexityBot (Perplexity) · Weighted 2×
Audit, grade, and fix in one scan
23-crawler directory
Indexed database - Training crawlers (GPTBot, ClaudeBot, CCBot) - Inference and RAG crawlers (ChatGPT-User, Perplexity-User) - Search indexers (OAI-SearchBot, PerplexityBot)
Fix generator
Edge blocks and configs - Generates standard robots.txt files - Outputs ai.txt flags (No-Training, No-Inference) - Provides edge rules for Cloudflare WAF, Nginx, and Next.js - Outputs single-prompt configs you can hand to an AI agent
Edge native
Fast, private execution - Scans run from the Cloudflare Workers edge network - DNS and domain ranking data via the Cloudflare Radar API - Results cached per domain using Workers KV - No user data saved to disk, no cookies
How it's built
Edge Infrastructure
Cloudflare Workers, Workers KV Caching, Cloudflare Radar API
Application
Next.js 15 (App Router), TypeScript Strict Mode, Tailwind CSS
Security & Verification
SHA-256 IP Hashing, Default-deny WAF generation, DNS record verification
Build Process
Built with Claude Code, Built in the open, Deployed globally
About CheckLLMs
CheckLLMs is built by Felipe Millán, who spent the past decade working at developer platforms (SendGrid, Twilio) and on SaaS go-to-market systems. He built it to give site owners clear visibility into which AI scrapers reach their content — and how to stop them.
The scanner runs on the same crawler database that powers BoostLLMs. One database, two tools: audit your exposure here, then work on your AI visibility over at BoostLLMs.
Common questions
faq
Is this another AI website builder?
No. CheckLLMs is a security and crawling audit tool. It resolves your site's domain configuration, fetches robots.txt and ai.txt live, and checks whether AI scrapers are allowed to take your content for training or search indexing.
What does "at the edge" mean for CheckLLMs?
The audit runs on Cloudflare's edge functions (Cloudflare Workers). Requests go out from the same kind of network infrastructure AI bots use, so the result reflects what those bots would actually see — and it comes back fast.
Do I need to know how to code?
Not to audit your site. Just submit your URL. If gaps are found, CheckLLMs outputs ready-made code snippets for robots.txt and server configs (Nginx, Cloudflare WAF, and more) that you or your developer can paste in seconds.
Is my visitors' data safe?
Yes. CheckLLMs collects no personal data. Scan outputs are cached by domain in Cloudflare Workers KV, and visitor IPs are only hashed using SHA-256 for rate limiting purposes. Hashed IPs are purged automatically.
What does it cost to run?
CheckLLMs is free and requires no account. It runs on Cloudflare Workers, which falls within Cloudflare's free serverless tier. There are no paid plans.
stack
- Cloudflare Workers
- Workers KV Caching
- Cloudflare Radar API
- Next.js 15 (App Router)
- TypeScript Strict Mode
- Tailwind CSS
- SHA-256 IP Hashing
- Default-deny WAF generation
- DNS record verification
- Built with Claude Code
- Built in the open
- Deployed globally