GEO Guide

robots.txt: the gateway for AI to your website

robots.txt has been around since 1994. But with the arrival of AI crawlers, its role has fundamentally changed. A misconfigured robots.txt can make your site completely invisible to ChatGPT, Claude, and Perplexity.

Your site as seen by AI crawlers

Restrictive robots.txt
GPTBot arrivesWants to read your pages
↓
Disallow: /Access denied
↓
ResultChatGPT never mentions you
GEO-optimized robots.txt
GPTBot arrivesWants to read your pages
↓
Allow: /Access granted
↓
ResultYour site is read and cited by AI
30+ AI crawlersGPTBot, ClaudeBot, PerplexityBot, Grok...
1 control fileAt the root: /robots.txt
Bad robots.txt = invisible to AI.Good robots.txt = the open door to ChatGPT, Claude and Perplexity responses.

What is robots.txt?

The robots.txt file is a text file placed at the root of your website (yoursite.com/robots.txt) that tells web crawlers which pages they are allowed to visit and which ones are off-limits.

It is the very first file a crawler checks before exploring your site. If it finds a disallow rule, it turns around without even reading your pages.

The protocol is standardized (RFC 9309) and respected by all major search engines and AI agents.

The new AI crawlers

In addition to Googlebot, your robots.txt now needs to handle a new generation of crawlers:

GPTBot
OpenAI's crawler. Powers ChatGPT responses and AI Overviews.
ClaudeBot
Anthropic's crawler. Collects data for Claude and its web search features.
PerplexityBot
Perplexity AI's crawler. Used to generate sourced, real-time answers.
Google-Extended
Google's dedicated AI training and AI Overviews crawler (separate from Googlebot).

The mistake that makes your site invisible

Many websites use an overly restrictive robots.txt, often inherited from an era when blocking bots was the security norm:

# DANGEROUS configuration for GEO User-agent: * Disallow: / # This blocks ALL crawlers, including # GPTBot, ClaudeBot, PerplexityBot. # Your site becomes invisible to AI.

Others specifically block AI crawlers without realizing the impact:

# Specific AI blocking User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / # Result: Google indexes you, but # ChatGPT and Claude never mention you.

Recommended configuration

A GEO-optimized robots.txt explicitly allows AI crawlers while protecting sensitive areas:

# Allow all AI crawlers User-agent: GPTBot Allow: / User-agent: ClaudeBot Allow: / User-agent: PerplexityBot Allow: / User-agent: Google-Extended Allow: / # Block sensitive areas only User-agent: * Disallow: /admin/ Disallow: /api/ Disallow: /account/ # Point to sitemap and llms.txt Sitemap: https://yoursite.com/sitemap.xml

Rule order matters: specific rules (by user-agent) take precedence over generic ones.

What Geoptim analyzes

During the GEO audit, Geoptim automatically checks your robots.txt:

The robots.txt score contributes to your overall GEO score out of 25 points. A missing or AI-blocking robots.txt drops this score to zero.

Is your robots.txt blocking AI?

Full GEO analysis in 60 seconds. Find out if AI crawlers can access your site.

Analyze my website →