How to get your product named in AI answers. A pecking order with no shortcuts

· Founder of Magpie AI

Published · 7 min read

In short

AI answers name the products their web search finds and can read. Let each assistant's search crawler in, send your key sentences in the HTML rather than drawing them with scripts, say plainly what the product is, who it's for and what it costs, and get written about on other sites. Rewording pages with GEO tricks did little or harm in a 2025 benchmark.

Contents

You asked ChatGPT for the best tool in your category and it named four of your rivals. Not you. That stings, and I think it'll matter more each year. Pew Research tracked 900 US adults' searches in March 2025 and found that when Google showed an AI summary, people clicked a normal result on 8% of visits, against 15% when there wasn't one.

The assistants are a lot less mysterious than the people selling "GEO" make them sound. They run a web search, read what comes back and write their answer from it. So there are two things worth finding out about your site, and neither needs a consultant. Does their search find it? And once it has, can it read what's there?

Why an assistant leaves your product out#

Google is blunt about its side. A page in AI Overviews or AI Mode must be "indexed and eligible to be shown in Google Search with a snippet", and both features may run "multiple related searches across subtopics and data sources" to build a single answer. The others send search crawlers of their own (OAI-SearchBot for ChatGPT, PerplexityBot for Perplexity, Claude-SearchBot for Claude), and those fetch your pages to answer the searches their assistant runs.

So you can be missed in three ways. The crawler was turned away at the door. Or it got in and found an empty shell where your words should've been. Or it read you perfectly well and other pages simply ranked higher for that search, which is the slow one, and mostly ordinary search work, right down to what your Google title and description say.

The first two you can often fix in an afternoon.

Let each assistant's search crawler in#

I'd guess most sites that block AI crawlers meant to block training and caught search by accident. Each company runs separate crawlers for the two jobs, and a robots.txt rule written in a hurry (say, a list of every AI user agent copied from a forum post) shuts you out of the answers as well as the training data.

AssistantSearch crawlerWhat its owner says about blocking it, as of November 2026
ChatGPTOAI-SearchBotSites opted out "will not be shown in ChatGPT search answers, though can still appear as navigational links"
PerplexityPerplexityBotIt is "designed to surface and link websites in search results" and "not used to crawl content for AI foundation models"
ClaudeClaude-SearchBotBlocking it may reduce your "visibility and accuracy in user search results"
Google AI OverviewsGooglebotThe same crawler as Google Search, so blocking it takes you out of both

Training is a separate choice. OpenAI says disallowing GPTBot means your content "should not be used in training", and that it's an independent setting from OAI-SearchBot. You can keep your words out of training and still turn up in ChatGPT's search, and I think that's the right default for most products.

Check your CDN as well. Since 1 July 2025 Cloudflare has asked every new domain at sign-up whether to let AI crawlers in, so a site can be shut to them at the edge without a single line in robots.txt. If you clicked no in a hurry while sorting out DNS, you've locked your site in a birdcage. You can change it later in the dashboard.

Send your words in the HTML, not only by script#

In December 2024 Vercel and MERJ published a study of AI crawler traffic and reported that "none of the major AI crawlers currently render JavaScript". GPTBot, ClaudeBot and PerplexityBot fetch the script files and never run them. Googlebot does render pages, which is how a site can look fine in Google and be blank to ChatGPT (and why a shared link's card can come up empty too).

Take shiftjar.example, a made-up rota app for cafés. Open it in a browser and the first screen says "Rotas for cafés, done in ten minutes a week". Ask its server for the page, though, and you get one empty div and a script. A crawler that runs no scripts reads nothing there at all, so no answer to "rota app for cafés" can name it, however good the product is. Rendering that first screen on the server (any framework's static or server rendering will do) puts the sentence where every crawler can read it.

Then give it something to quote. Someone asking for the "best rota app for a small café" wants a line saying what the product is, who it's for and what it costs, and a slogan gives the assistant nothing to lift. That's one more reason for showing prices on a landing page, and the plain words are what a landing page's first screen should open with anyway, for people and assistants alike.

Magpie's full report checks this part. It reads your page as the server sends it and says how many of its words arrive without scripts, reads your robots.txt for each assistant's search crawler, and picks the sentence an assistant could quote for five questions (what it is, who it's for, how it works, how it's different and what it costs). It also runs buyers' searches and says whether today's answer names you.

What GEO tricks do, according to the research#

C-SEO Bench is the paper I'd send to anyone selling GEO. Accepted at NeurIPS's datasets and benchmarks track in 2025, it took eight rewriting methods from the original GEO paper (adding statistics, quotes, citations, an authoritative tone, technical terms and so on) and ran them across six kinds of search (from retail and books to news and debate) on four models, gpt-4o-mini, o3, o4-mini and Claude 3.5 Haiku. Out of 54 cases, three ranked better by a margin that held up statistically.

Three. And the Statistics method, which sounds like the safest of the lot, lowered rankings in 19 of 24 settings.

What did move a product up was sitting high in the search results the model read. On one retail test with gpt-4o-mini, putting a page first in the model's results raised its place in the answer by 2.77 positions on average, against 0.36 for the best rewriting method. The authors also found that the early adopters' gains "decrease steadily as more players adopt the same method", and called it "both a congested and a zero-sum game".

To be fair to the 2024 GEO paper that started all this, it's a real paper, accepted at KDD, and it reported visibility gains of up to 40%. It tested engines and a benchmark from 2023, though, and the newer benchmark didn't find those gains on the models it tried. Models change fast enough that I hold both results a bit loosely.

Google's own guidance ends up in much the same place. You "don't need to create new machine readable files, AI text files, or markup", there's "no special schema.org structured data" for AI features (so FAQ markup isn't a way in), and there's "no requirement to break your content into tiny pieces". I wouldn't spend a day on an llms.txt file, or on chopping pages into question-and-answer chunks.

Get written about on other sites#

A 2025 preprint by Chen, Wang, Chen and Koudas compared the sources AI search cites with Google's results. It isn't peer reviewed yet, so treat it as a lead rather than a verdict. It found "a systematic and overwhelming bias towards Earned media", meaning reviews, articles and comparisons on other sites, over a brand's own pages and social posts. My reading, and it's only a reading, is that a "best tools for" question gets answered largely from best-of lists.

Which makes the slow work getting onto those lists. Comparison articles line a category's products up like birds on a wire, and one that names your four rivals and leaves you out is a likely place the assistant learned its list in the first place. Getting into the review sites, directories and comparison pieces in your category (with your own site saying the same plain sentences they'll quote) is the part no line in robots.txt can do for you.

How to see whether answers name you#

Ask the same question several ways, on more than one assistant, on more than one day. The searches behind an answer shift with the wording and the date, so a single "it named me" (or didn't) tells you very little. And ask what a buyer would type, something like "best rota app for a small café", rather than your brand name, which any assistant finds far more easily than it finds your category.

Microsoft's side has real data now. On 10 February 2026 Bing Webmaster Tools added an AI Performance report, in public preview, that counts how often your pages are cited as sources in Copilot and Bing's AI summaries, and shows a sample of the searches the AI ran when it cited them. It counts citations, not where in an answer you appear. Still, it's free, and it's the nearest thing I know of to a search console for AI answers.

Questions people ask#

Is an llms.txt file worth adding?#

Not for Google, which says Google Search doesn't use them. Other services might read one and it won't do any harm, so if it takes you five minutes, fair enough. I just wouldn't expect it to change what anyone's assistant says about you.

Should I block GPTBot?#

That one's really about training rather than being found, and it's easy to mix the two up because the names look so alike. Blocking GPTBot keeps your words out of OpenAI's training. Whether you turn up in ChatGPT's search is down to OAI-SearchBot, which is a separate setting, so you can shut one and leave the other open. I said earlier that I'd block training and leave search alone, and on most product sites that's still what I'd do, though I can see someone deciding their marketing copy is something they'd quite like repeated everywhere and leaving both open. Either is a fair call. The one I'd argue against is blocking the search crawlers.

Sources

  1. Google Search Central, AI features and your website (opens in a new tab)
  2. Google Search Central, Optimizing your website for generative AI features on Google Search (opens in a new tab)
  3. OpenAI, Overview of OpenAI crawlers (opens in a new tab)
  4. Perplexity, Perplexity crawlers (opens in a new tab)
  5. Anthropic, Does Anthropic crawl data from the web, and how can site owners block the crawler (opens in a new tab)
  6. Zecchini, Moore, Ubl and Siddle, The rise of the AI crawler (Vercel and MERJ, 2024) (opens in a new tab)
  7. Puerto, Gubri, Green, Oh and Yun, C-SEO Bench: Does Conversational SEO Work? (NeurIPS Datasets and Benchmarks, 2025) (opens in a new tab)
  8. Aggarwal et al., GEO: Generative Engine Optimization (KDD, 2024) (opens in a new tab)
  9. Chen, Wang, Chen and Koudas, Generative Engine Optimization: How to Dominate AI Search (preprint, 2025) (opens in a new tab)
  10. Pew Research Center, Google users are less likely to click on links when an AI summary appears in the results (2025) (opens in a new tab)
  11. Bing Webmaster Blog, Introducing AI Performance in Bing Webmaster Tools (2026) (opens in a new tab)
  12. Cloudflare, Cloudflare just changed how AI crawlers scrape the internet at large (2025) (opens in a new tab)

About the author

· Founder of Magpie AI

Alex founded Magpie AI and writes about what turns a landing page's visitors into customers, drawing on what its audits find on real sites, from the first screen to the checkout

Related articles

Fix what loses you customers.Fix what loses you customers. Then bring more of them in

  • Free score, no sign‑upNo sign‑up
  • No credit card neededNo card needed