P/02 · In development # CheckLLMs: Which AI crawlers can reach your site? Audit your domain against 23 AI crawlers, graded 0–100. - In development - 23 Crawlers Indexed - CF Workers Audit your domain against 23 AI scrapers — including GPTBot, ClaudeBot, and Google-Extended — and grade your exposure from 0 to 100. fig. — Scan log — myblog.dev, protection score 0 / 100 23 Known AI training, RAG, and search crawlers indexed 0–100 Weighted protection score (Grade A–F) 100% Runs at the edge — no cookies, no tracking CheckLLMs is a free AI crawler audit tool that checks any domain against 23 known AI training and search bots. It grades your exposure on a 0–100 scale (Grade A–F) and outputs ready-to-paste robots.txt, ai.txt, and edge config rules. No login, no account, results in under a minute. What it does ## One scan. Clear fixes. Most site owners configured their robots.txt before LLM bots became widespread. CheckLLMs resolves your domain and fetches your robots.txt and ai.txt files live, straight from your origin server. It reads every directive, maps it against 23 scrapers (GPTBot, ClaudeBot, Google-Extended, Bytespider, CCBot, and more), and grades how well your domain is protected. Then it outputs clean, copy-paste config scripts for Nginx, Apache, Cloudflare WAF, and Next.js edge middleware. Scoring framework ## Weighting critical bots CheckLLMs grades your protection A–F using a weighted scoring model. Critical bots count 2× because they handle most of the LLM data ingestion happening today: A crawler counts as protected if it is blocked either at the path level in robots.txt or at the purpose level via the newer ai.txt standard. - GPTBot (OpenAI) · Weighted 2× - ClaudeBot (Anthropic) · Weighted 2× - Google-Extended (Google) · Weighted 2× - Applebot-Extended (Apple) · Weighted 2× - CCBot (Common Crawl) · Weighted 2× - Bytespider (ByteDance) · Weighted 2× - Meta-ExternalAgent (Meta) · Weighted 2× - PerplexityBot (Perplexity) · Weighted 2× Features ## Audit, grade, and fix in one scan ### 23-crawler directory Indexed database - Training crawlers (GPTBot, ClaudeBot, CCBot) - Inference and RAG crawlers (ChatGPT-User, Perplexity-User) - Search indexers (OAI-SearchBot, PerplexityBot) ### Fix generator Edge blocks and configs - Generates standard robots.txt files - Outputs ai.txt flags (No-Training, No-Inference) - Provides edge rules for Cloudflare WAF, Nginx, and Next.js - Outputs single-prompt configs you can hand to an AI agent ### Edge native Fast, private execution - Scans run from the Cloudflare Workers edge network - DNS and domain ranking data via the Cloudflare Radar API - Results cached per domain using Workers KV - No user data saved to disk, no cookies fig. — Full audit — 23 of 23 crawlers, robots.txt vs ai.txt, DNS, TLS and headers Tech stack ## How it's built ### Edge Infrastructure Cloudflare Workers, Workers KV Caching, Cloudflare Radar API ### Application Next.js 15 (App Router), TypeScript Strict Mode, Tailwind CSS ### Security & Verification SHA-256 IP Hashing, Default-deny WAF generation, DNS record verification ### Build Process Built with Claude Code, Built in the open, Deployed globally The builder ## About CheckLLMs CheckLLMs is built by Felipe Millán, who spent the past decade working at developer platforms (SendGrid, Twilio) and on SaaS go-to-market systems. He built it to give site owners clear visibility into which AI scrapers reach their content — and how to stop them. The scanner runs on the same crawler database that powers BoostLLMs. One database, two tools: audit your exposure here, then work on your AI visibility over at BoostLLMs. FAQ ## Common questions ## faq **Is this another AI website builder?** No. CheckLLMs is a security and crawling audit tool. It resolves your site's domain configuration, fetches robots.txt and ai.txt live, and checks whether AI scrapers are allowed to take your content for training or search indexing. **What does "at the edge" mean for CheckLLMs?** The audit runs on Cloudflare's edge functions (Cloudflare Workers). Requests go out from the same kind of network infrastructure AI bots use, so the result reflects what those bots would actually see — and it comes back fast. **Do I need to know how to code?** Not to audit your site. Just submit your URL. If gaps are found, CheckLLMs outputs ready-made code snippets for robots.txt and server configs (Nginx, Cloudflare WAF, and more) that you or your developer can paste in seconds. **Is my visitors' data safe?** Yes. CheckLLMs collects no personal data. Scan outputs are cached by domain in Cloudflare Workers KV, and visitor IPs are only hashed using SHA-256 for rate limiting purposes. Hashed IPs are purged automatically. **What does it cost to run?** CheckLLMs is free and requires no account. It runs on Cloudflare Workers, which falls within Cloudflare's free serverless tier. There are no paid plans. ## stack - Cloudflare Workers - Workers KV Caching - Cloudflare Radar API - Next.js 15 (App Router) - TypeScript Strict Mode - Tailwind CSS - SHA-256 IP Hashing - Default-deny WAF generation - DNS record verification - Built with Claude Code - Built in the open - Deployed globally Related: [BoostLLMs: AI readiness scores for your website — and the fixes](https://fmillan.com/projects/boostllms/) Want to talk it through? Email [hello@fmillan.com](mailto:hello@fmillan.com) or see the [contact page](https://fmillan.com/contact/). [previous project — P/01 BoostLLMs](https://fmillan.com/projects/boostllms/) [next project — P/03 AstroEdge](https://fmillan.com/projects/astroedge/) [← back to the workbench](https://fmillan.com/projects/) [open in the 3D site →](https://fmillan.com/#p-checkllms)