July 7, 2026·7 min read
Beyond SEO: The New Web Visibility Stack
Search isn't just Google anymore. Here's everything that's changed - Answer Engine Optimization, Generative Engine Optimization, LLMs.txt, and how to get found in an AI-first web.
For the last 20 years, getting found online meant one thing: ranking on Google. You optimized for the 10 blue links. You chased Page 1.
That game still exists. But a second game has opened up alongside it - and most developers aren't playing it yet.
The old model looked like this:
USER has a question
│
▼
Opens Google
│
▼
Sees 10 links
│
▼
Clicks one
│
▼
Reads YOUR website
The new model looks like this:
USER has a question
│
▼
Asks ChatGPT / Claude / Perplexity / Google AI Overview
│
▼
Gets a direct answer
│
▼
Maybe clicks a source link. Maybe doesn't.
The middleman changed. Instead of a search engine sending you traffic, an AI is summarizing your content - and either crediting you or not.
This is the shift. Three new disciplines have emerged to deal with it.
Traditional SEO is not dead. It's just no longer enough on its own.
CLASSIC SEO SIGNALS
│
├── On-page: keywords, headings, meta tags, structured data
├── Technical: Core Web Vitals, crawlability, sitemaps
├── Authority: backlinks, domain reputation, EAT signals
└── Content: depth, freshness, topical coverage
Google still sends the most organic traffic on the internet. Mobile search is up. Local search is up. You still need this foundation.
What's changed is the top of the results page. Google now shows AI Overviews (formerly SGE) for a growing percentage of queries. That's where the new game starts.
AEO is about getting your content selected as the source for AI-generated answers. Not just ranking - being cited.
WITHOUT AEO:
User asks AI: "What is hydration in React?"
AI answers from training data
No source cited
You get zero traffic
WITH AEO:
User asks AI: "What is hydration in React?"
AI answers: "According to [your blog]..."
Source link shown
User clicks through
How to optimize for answer engines:
1. Write in question-answer format
AI models are trained on Q&A patterns. Structure content to match:
## What is hydration in React?
Hydration is the process of attaching event listeners to
server-rendered HTML. React runs on the browser, finds
the existing HTML, and "wakes it up" to be interactive.Not: a wall of prose where the definition is buried in paragraph 4.
2. Use FAQ schema
{
"@context": "https://schema.org",
"@type": "FAQPage",
"mainEntity": [
{
"@type": "Question",
"name": "What is hydration in React?",
"acceptedAnswer": {
"@type": "Answer",
"text": "Hydration is the process of attaching..."
}
}
]
}Structured data tells crawlers exactly where your answers are.
3. Be the definitive source, not a summary
AI systems prefer to cite authoritative, specific explanations - not general overviews. Depth beats breadth. One topic explained extremely well beats ten topics explained superficially.
GEO is a step beyond AEO. Where AEO is about getting cited in answers, GEO is about influencing how AI generates responses about your topic or brand.
Researchers at Princeton published a study in 2024 showing that certain content patterns increase the likelihood of being cited by generative AI by 40%+:
CONTENT PATTERNS THAT INCREASE AI CITATION RATE:
High impact:
├── Adding quotable statistics with sources
├── Including authoritative citations in your own content
├── Using fluent, clear prose (not SEO keyword stuffing)
└── Including definitions with precise language
Medium impact:
├── Adding technical terms and domain vocabulary
├── Using structured lists and tables
└── Q&A format sections
Low/negative impact:
└── Keyword density optimization (optimized for old-school SEO)
The key insight: write for humans, not crawlers. AI systems are trained to recognize and prefer natural, authoritative human writing over SEO-optimized filler.
robots.txt tells search engine crawlers which pages to index. LLMs.txt is a proposed standard that tells AI systems what content you want them to use.
The format is simple markdown, placed at yoursite.com/llms.txt:
# Shivam Gaur - Developer Portfolio
> Full-stack developer writing about React, Next.js, and web architecture.
## Blog Posts
- [React vs Next.js: The Full Picture](/blog/react-vs-nextjs):
In-depth comparison of rendering strategies, routing, and data fetching.
- [Realtime Updates Without Polling](/blog/realtime-updates-without-polling):
How to use Ably and RTK Query for event-driven cache invalidation.
## Key Facts
- Based in India
- Specializes in Next.js App Router, TypeScript, and system design
- Open to full-time roles and freelance projects
## Contact
- Email: hello@shivamgaur.dev
- GitHub: github.com/ishivamgaurWhy this matters:
WITHOUT LLMs.txt:
AI crawler visits your site
Scrapes whatever it can find
Might index your cookie banner, your 404 page,
your old draft content
Has no idea what matters
WITH LLMs.txt:
AI crawler visits yoursite.com/llms.txt
Gets a curated, human-written summary
Understands exactly who you are and what you do
Better chance of accurate representation in AI answers
The llmstxt.org spec is gaining adoption. Tools like Perplexity and some Claude integrations already parse it. Adding a llms.txt is a 20-minute investment with no downside.
Google is no longer the only answer engine sending traffic:
ANSWER ENGINES TO OPTIMIZE FOR (2026):
├── Google AI Overview
│ └── Still uses traditional crawling + Gemini
│ Optimize with: structured data, E-E-A-T, AEO
│
├── Perplexity AI
│ └── Crawls the web like a search engine
│ Cites sources prominently
│ Adding llms.txt helps
│
├── ChatGPT Search (SearchGPT)
│ └── Uses Bing index + OpenAI's crawlers
│ robots.txt controls what GPTBot indexes
│
├── Claude (Anthropic)
│ └── Primarily uses training data
│ Web browsing via third-party integrations
│
└── Apple Intelligence / Siri
└── Uses multiple sources
Local + web context
Each has its own crawlers, its own index, and its own signals. The old "rank on Google" monoculture is fragmenting.
Just like robots.txt for search engines, you can allow or block AI crawlers:
# Block OpenAI's crawler
User-agent: GPTBot
Disallow: /
# Allow Perplexity's crawler
User-agent: PerplexityBot
Allow: /
# Block Anthropic's crawler
User-agent: ClaudeBot
Disallow: /
# Block Common Crawl (used for training data)
User-agent: CCBot
Disallow: /
Whether to block or allow is a business decision:
ALLOW AI CRAWLERS → content gets used in training + answers
├── Benefit: brand visibility in AI answers
├── Benefit: cited as a source → traffic
└── Risk: content used to train competitors' models
BLOCK AI CRAWLERS → content stays yours
├── Benefit: protects original work
├── Benefit: forces users to come to your site
└── Risk: invisible in AI-generated answers
For a portfolio and technical blog, allowing crawlers generally helps more than it hurts.
Putting it all together, modern web visibility now requires multiple layers:
LAYER 1: Technical Foundation (same as always)
├── Fast page loads (Core Web Vitals)
├── Clean HTML structure
├── Crawlable by all bots
└── HTTPS, sitemaps, canonical tags
LAYER 2: Traditional SEO (still necessary)
├── Keyword research and topical authority
├── Quality backlinks
├── E-E-A-T signals (Experience, Expertise, Authoritativeness, Trust)
└── Fresh, well-structured content
LAYER 3: AEO / GEO (the new layer)
├── Q&A structured content
├── FAQ schema markup
├── Authoritative, citation-worthy writing
└── Statistics and precise definitions
LAYER 4: AI-Specific Signals
├── llms.txt file
├── robots.txt rules for AI crawlers
├── Consistent brand/author identity across the web
└── Content that holds up in AI training data
LAYER 5: Distribution (beyond Google)
├── Answer Perplexity's question format
├── Create content ChatGPT would cite
├── Publish on platforms AI systems trust (GitHub, dev.to, etc.)
└── Build a presence on platforms used for AI training
If you have a blog or portfolio and you want to stay visible as AI reshapes how people find information:
1. Add llms.txt - 20 minutes, zero cost. Create public/llms.txt in your Next.js project.
2. Add FAQ schema to your most-visited posts. One structured data block per page.
3. Rewrite your "About" page as a clear, quotable explanation of who you are and what you do. AI systems synthesize identity from consistent signals.
4. Update robots.txt - decide which AI crawlers you want and explicitly allow them.
5. Write at least one "definitive" piece - pick one topic you know deeply, and write the most complete, accurate explanation of it that exists. These pieces get cited.
The window where early movers benefit from this is open right now. The developers who figure out AI visibility in 2026 are in the same position as the developers who figured out SEO in 2008.
Traditional SEO is the foundation. But it's no longer the whole game.
AEO is about being cited by AI answers. GEO is about being represented accurately in how AI talks about your domain. LLMs.txt is about curating that representation yourself. Crawler rules are about deciding who gets to use your content.
None of this is rocket science. It's the same core principle as always: write good content, make it easy to find, and make it easy to parse. The audience now includes both humans and AI systems - so optimize for both.