GEO Unveiled: Principles, Practices, and Trade-offs of AI Visibility

GEO Tutorial for Creators: Let AI find your content proactively.

Recently, several friends tagged me saying my open-source tools were recommended by AI when they asked questions. It got indexed even though I did nothing. I thought, why not spend an hour structuring the content to make it even better? This article isn't about shortcut tactics or spamming content; it is about helping AI better understand your existing content itself.

The following diagram summarizes the 7 steps of the workflow that will be detailed below:

Diagram: The 7 steps to make your site discoverable and citeable by LLM crawlers.

1. Categorize Crawler Types using robots.txt

Many treat robots.txt as a simple binary switch. However, AI crawlers are divided into distinct categories with different purposes:

• Training Crawlers (e.g., GPTBot, ClaudeBot, CCBot): Scrape content to train models. Blocking them doesn't affect active search results.
• Search & Retrieval Crawlers (e.g., OAI-SearchBot, Claude-SearchBot, PerplexityBot): Retrieve real-time content to answer active queries. Blocking these makes you disappear from AI search.
• User-Triggered Crawlers (e.g., ChatGPT-User, Claude-User): Active only when a user drops your URL in chat.
• Opt-out Signals (e.g., Google-Extended, Applebot-Extended): Declare opt-out from training.

Table: AI scrapers categorization and recommended robots rules.

# Search & retrieval: allow
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

# User-triggered: allow
User-agent: ChatGPT-User
Allow: /

User-agent: Claude-User
Allow: /

# Training: block
User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

2. Implement llms.txt and Cross-Referencing

llms.txt is a new markdown standard placed in the root directory. Over 840,000 websites (Anthropic, Cloudflare, Stripe, etc.) have adopted it, yet adoption rate sits at 10% in broad samples.

Screenshot: llms.txt configuration file placed at root directory.

# Your Project Name

> One-line description of what this is.

## Links
- [Documentation](https://yoursite.com/docs)
- [Blog](https://yoursite.com/blog)

## About
Short paragraph explaining the project.

If you run multiple related web properties, link their llms.txt files together to form a mesh. This allows AI bots starting from any property to naturally discover the rest.

Diagram: Multi-site interlinked llms.txt mesh structure.

Once indexed, querying LLMs on your project name will show clean, accurate source attributions.

Screenshot: ChatGPT 5.5 accurately citing Kami and its sources.

Tell AI that a markdown version is available by embedding a link in your HTML <head>:

<link rel="alternate" type="text/markdown" href="/page.md" />

Comparison: Markdown reduces crawler token overhead by 80% compared to heavy HTML templates.

3. Register with Search Platforms

AI search engines run on top of search indices. ChatGPT's search queries Bing; Google's Gemini / AI Overviews queries Google's index. Verify your property in GSC and Bing Webmaster Tools, and integrate **IndexNow** to broadcast page modifications.

Diagram: IndexNow and Sitemap data distribution to downstream search agents.

4. Build a Dedicated AI Knowledge Hub

Instead of letting AI crawl randomly, provide a clean entrance: `llms.txt` (summary), `llms-full.txt` (comprehensive), and structured JSON endpoints. I built a template called Yobi to output these automatically.

Screenshot: Yobi - tw93's machine-readable directory homepage.

It also offers structured JSON endpoints for automated agent retrieval:

API Result: A clean JSON profile with GitHub statistics.

5. Use Clean, Descriptive Slugs and Apex Domains

Provide each project its own self-contained page with a descriptive slug (e.g., /projects/pake) instead of opaque parameters (e.g., /?id=47). Keep all structured data on the apex domain to ensure crawler discovery.

Screenshot: Self-contained project documentation for Pake.

Mirror structured content from subdomains to your root domain (e.g., via CI workflows or reverse proxy), making it immediately visible to bots finding your root page.

Flow: Syncing content from Yobi to tw93.fun using GitHub Actions.

6. What We Tried but Failed & JSON-LD Limits

• <meta name="ai-content-url"> is ignored by all mainstream engines.
• HTML comments to instruct AI are stripped out by content parsers.
• LLMs read JSON-LD as raw text and often fail to comprehend structured schemas unless rendered in user-visible content.

7. What the Research Data Tells Us

Studies show that citing authoritative references increases visibility by 115%. High-citation pages average around 2000 words. Furthermore, 85% of crawled pages are never cited. Structure your content so it represents a clear, semantic match to user queries.

Chart: Impact of optimization types on AI visibility (Princeton & IIT Delhi).

8. Retrieval Does Not Equal Citation

Only 15% of retrieved pages end up in active ChatGPT responses. The engine applies an extraction filter. Giving bots clean Markdown structures like `llms.txt` creates an anchor point, ensuring your properties get credited.

Result: ChatGPT accurately citing Mole and listing its install instructions cleanly.

If you have a product, blog, or document repository, give these steps a spin. Let your AI agent do the heavy lifting!

Extended Reading (Internal Archive)