AI crawlers, robots.txt and llms.txt: what to allow in 2026
GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended: which AI crawlers to allow, which to block, and whether llms.txt does anything.
INTELAI SEARCH AND GEO
Field notes on ai search and geo, newest first.
GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended: which AI crawlers to allow, which to block, and whether llms.txt does anything.
First-party ChatGPT Ads data from two live campaigns: CPM, CPC and CTR in Türkiye and Europe, account setup, and what conversion tracking changes.
AI agents now research and shortlist products for buyers. What ACP and UCP are, what an agent reads, and how a startup gets onto the shortlist.
What a pre-seed startup needs from brand identity and what it can skip: positioning line, distinctive assets, consistency, and AI-readable brand facts.
Reddit leads citations in Perplexity and Google AI answers. What the 2025 and 2026 studies show, and how a startup earns threads AI engines quote.