Æ STUDIO · free tool
robots.txt is the first thing every crawler checks, including the AI ones. Decide who gets in, generate the file, done. Free, no signup, runs in your browser.
Every listed AI crawler can read your site. Traditional search engines are always allowed. Best choice if you want to be found and cited by ChatGPT, Perplexity, and Claude.
robots.txt decides who can crawl. It doesn't tell them what you offer — that's llms.txt. It doesn't state your business facts — that's structured data. It doesn't make your answers quotable — that's FAQ schema. Do all four, then verify with the free AI visibility checker. Want it all done and confirmed? See the agent-ready retrofit ($60-175).
Policies drift too. A plugin update or a migration can quietly reset them. Pulse re-checks robots.txt and the rest of your AI-visibility score on a schedule and emails you if it changes, from $9/mo.
Want the full 12-point checklist this file is one line of? Take the free AI Visibility Checklist. Pay what you want, including nothing.
AI crawlers — GPTBot (OpenAI), ClaudeBot and Claude-User (Anthropic), PerplexityBot, OAI-SearchBot, Google-Extended, anthropic-ai — check robots.txt before reading anything. Not deciding means someone else decided for you, usually by leaving you off a default deny-list you never wrote. Built by Æ Studio, AI-operated and human-reviewed. Contact: alexander.k.eliot@gmail.com