AI assistants recommend what they can read reliably and what the rest of the web talks about most. Marketplaces supply structured product data to search providers, carry millions of reviews and are mentioned on nearly every "best X" page, so for a generic question like "where can I buy a cast iron pan" they are the safe answer. An independent shop beats them on the questions a marketplace listing answers badly: local, specialist, gift, made-in-UK and expert-advice questions. This guide covers what the assistants document, where you can win, and what to publish.
Key takeaways
- OpenAI says ChatGPT's product results come from structured metadata supplied by first-party and third-party providers and other third-party content, and that merchants are ranked on availability, price, quality and whether they are the maker or primary seller.
- Amazon's robots.txt disallows OpenAI's, Perplexity's, Anthropic's and Microsoft Copilot's crawlers and Google's AI tokens. It is still recommended, because assistants also use third-party product data and what other sites say.
- Independents win on specific questions: "near me", "independent", "made in the UK", "gift for", "which one if".
- Publish buying guides that answer those questions, product pages with the facts in plain text, and a "why buy from us" page.
- Google's free Merchant Center listings, a niche marketplace and public review sites are the cheapest ways to get your product data in front of assistants.
Why do AI assistants favour marketplaces?
Because marketplaces give assistants the three things they lean on: structured product data, reviews and mentions.
Structured product data. OpenAI's help page on shopping says ChatGPT considers "structured metadata from first-party and third-party providers (e.g., price, product description) and other third-party content" when choosing products. The merchant list under a product "is generated by ChatGPT based on merchant and product metadata from third-party providers or directly from the merchants themselves", ranked on "availability, price, quality, and whether they are the maker or primary seller of that item" (OpenAI Help Center, 2026). A marketplace has clean price and stock data for millions of items; a small shop often has it in an image or a tab that loads on click.
Reviews. ChatGPT's product review summaries are "based on reviews from public websites", and "reviews and ratings are not verified by OpenAI" (OpenAI Help Center, 2026). Google's AI features "can show what's being said about products and services across the web, including in blogs, videos, and forum discussions" (Google Search Central, 2026). Thousands of public reviews give a model more to summarise than three.
Mentions. Google's AI features use query fan-out, "issuing multiple related searches across subtopics and data sources" (Google Search Central, 2025). Each sub-search returns the pages that already rank for product questions, which are mostly marketplaces, big retailers and review publications. If your shop is on none of them, there is nothing to pick you from.
None of this is payment for placement. OpenAI says product results "are selected independently by ChatGPT and are not ads, nor influenced by any OpenAI partnerships" (OpenAI Help Center, 2026), and Perplexity says its product cards "aren't sponsored" (Perplexity, 2024). The bias is towards available data, not a brand.
Does Amazon let AI crawlers read its site?
Mostly not, which is the surprising part. We checked both robots.txt files on 10 November 2026.
amazon.co.uk and amazon.com both carry a Disallow: / rule for these AI user agents, among others:
| Company | User agents Amazon disallows site-wide |
|---|---|
| OpenAI | GPTBot, OAI-SearchBot, ChatGPT-User |
| Perplexity | PerplexityBot, Perplexity-User |
| Anthropic | ClaudeBot, Claude-SearchBot, Claude-User, Claude-Web |
| Google (AI tokens) | Google-Extended, GoogleAgent-Shopping, GoogleAgent-Mariner, Gemini-Deep-Research, Google-NotebookLM, Google-CloudVertexBot |
| Microsoft | Copilot, CopilotNative, CopilotSapphire |
| Others | CCBot, Bytespider, meta-externalagent, meta-externalfetcher, DuckAssistBot, MistralAI-User, cohere-ai |
amazon.com also blocks DeepSeekBot. Neither file blocks Googlebot or bingbot site-wide, and Google-Extended "does not impact a site's inclusion in Google Search" (Google Search Central, 2026), so Amazon stays in Google Search and AI Overviews while opting out of Gemini training and grounding.
So ChatGPT, Perplexity and Claude cannot crawl Amazon's product pages. OpenAI says sites that opt out of OAI-SearchBot "will not be shown in ChatGPT search answers", though they may still appear as navigational links (OpenAI, 2026); Anthropic says blocking Claude-SearchBot "prevents our system from indexing your content for search optimization" (Anthropic, 2026). Amazon still appears because the assistants are fed product data by third-party providers and read the rest of the web, where Amazon is mentioned constantly. OpenAI does not name its providers, so we cannot say whether Amazon supplies a feed; we only know crawling is not how it gets there. (The user-initiated fetchers are looser: OpenAI says for ChatGPT-User "robots.txt rules may not apply", and Perplexity-User "generally ignores robots.txt rules" (Perplexity, 2026).)
The lesson for a small shop is the opposite of Amazon's. You have no third-party data pipeline to fall back on, so your own pages must be readable. If your robots.txt or firewall blocks OAI-SearchBot, PerplexityBot or Claude-SearchBot, you have removed your only route. Our guide to allowing ChatGPT to crawl your website has a tested robots.txt.
Which questions can an independent shop win?
The ones where "buy it on Amazon" is a poor answer. These examples are the kinds of questions shoppers put to an assistant, not search volumes.
| Question type | Example (illustrative) | Why a marketplace answers it badly |
|---|---|---|
| Local | "Where can I buy a proper cheese knife in Sheffield today?" | A marketplace has no shop door; your address, hours and stock are a direct answer. |
| Independent | "Best independent bookshop for children's books in Bristol" | The question excludes marketplaces by definition. |
| Specialist or expert | "Which fountain pen nib should a left-handed beginner buy?" | Needs an explanation; the assistant quotes whoever wrote the clearest one. |
| Gift | "Gift for a dad who has just started sourdough baking, under £40" | The assistant wants a short curated list with reasons. |
| Made in the UK or ethical | "Wooden toys made in the UK" | Provenance and values are facts you can state and a listing rarely does. |
| Fit for a case | "Walking boots for wide feet I can try on in Edinburgh" | Local plus specialist plus stock, which only a shop can confirm. |
Pick five to ten questions like these for your category. They become your buying guides, your product facts and the questions you track.
What should a small shop publish?
Three kinds of page: buying guides, product pages with the facts in plain text, and one page that says why to buy from you.
- Buying guides, one question each. Put the answer in the first two sentences, then the detail: "Which cast iron pan for two people?" opens with a size and a price band. Name the products you stock, link to them, and say who the recommendation is not for. Google's advice is to focus on "high-quality content" rather than "inauthentic mentions" (Google Search Central, 2026), and one good page serves Google, ChatGPT and Perplexity alike. See comparison pages that AI quotes.
- Product pages with the facts. OpenAI's feed specification lists nine fields "required for a useful discovery feed": item ID, title, description, URL, brand, seller name, image, availability and price (OpenAI, 2026). Its help page tells shoppers to confirm "the final price, including taxes, fees, and shipping costs", size and colour options and "return and warranty policies" before buying (OpenAI Help Center, 2026). Put all of that on the page in text. The same facts feed Google's merchant listings, where Product structured data lets shoppers "see price, availability, review ratings, shipping information, and more right in search results" (Google Search Central, 2026). Our guide to AI shopping agents and your product pages covers the markup.
- A plain "why buy from us" page. Who runs the shop and for how long, where you are, delivery costs and times by region, your full returns policy, how to reach a person, and what you specialise in. Link it from every product page. "Independent", "family-run", "made in the UK" and "we repair what we sell" belong here, stated plainly so they can be quoted.
Where should an independent shop be listed?
Wherever product data and reviews are collected, because that is what the assistants read.
- Google Merchant Center free listings. Google says "the products you sell can appear at no cost on Google Search, Google Maps, Gemini, YouTube", the Shopping tab, Images and Lens, though "your products aren't guaranteed to show across Google". You must follow its policies, "add your return policy information to your website" and set up UK delivery settings (Google Merchant Center Help, 2026). Google's AI guidance recommends "checking that your Merchant Center and Business Profile information is up-to-date" (Google Search Central, 2025). A physical shop also needs a complete Google Business Profile for the local questions.
- ChatGPT product data. "For merchants on Shopify, product data is already integrated into ChatGPT through Shopify Catalog" with no extra work (OpenAI Help Center, 2026). Direct feeds are "currently available to approved partners", who apply at chatgpt.com/merchants (OpenAI, 2026). Otherwise your route is crawlable product pages; see ChatGPT shopping for small online shops.
- Perplexity. Its shopping results are "powered by platform integrations including Shopify", and its Merchant Program was built "to make it easy for large retailers to share their product specs" (Perplexity, 2024). Small shops rely on crawlable pages and mentions.
- A second marketplace front. Etsy, Not On The High Street or a specialist marketplace for your category gives assistants a second well-structured place to find your products and reviews. Treat it as a listing that points back to your site, not your home.
- Public review sites. OpenAI's review summaries come "from public websites", so reviews on Google, Trustpilot or the platform your sector trusts count in a way that testimonials only on your own site do not. Ask after every delivery; never pay for or filter reviews. See does AI use Trustpilot reviews?.
- Mentions. Local press, "best independent shops in [town]" lists, trade bodies and suppliers' stockist pages are the sources an assistant fans out to. Our directories guide lists the UK ones worth claiming.
A 30-day plan for an independent shop
- Week 1: be readable. Check
yourshop.co.uk/robots.txtand your hosting or CDN bot settings allowOAI-SearchBot,PerplexityBot,Claude-SearchBot,Googlebotandbingbot. View the source of a product page and confirm name, price, stock, delivery cost and returns are in the text. Fix the template once. - Week 2: list the products. Set up Merchant Center free listings with a returns policy on your site and UK delivery settings. Add Product structured data if your platform does not. On Shopify, check your catalogue is published and accurate.
- Week 3: answer the questions. Write your "why buy from us" page and your first three buying guides, each answering one question from the table above, answer first. Link guides to products and products to guides.
- Week 4: earn reviews and mentions. Set up a post-delivery review request on a public platform. Email three local or trade sites that list shops like yours and ask to be included. Then ask ChatGPT and Perplexity your ten questions and note who they name.
After that, a buying guide and a mention a fortnight is sustainable for one person.
How SearchCrow helps
SearchCrow asks ChatGPT, Perplexity and the other assistants the shopping questions your customers ask and shows whether you were named, which assistants named you, and the "other businesses AI recommends" list for each question, so you can see whether you are losing a question to Amazon, to another independent or to nobody. When the website check finds product pages, shop mode reads them the way shopping agents do (price, stock, delivery, returns, product data and bot checks) and shows which of your products AI recommends. The free check takes about 30 seconds with no card; the independent shops page covers the shop-specific setup.
FAQ
Why does ChatGPT keep recommending Amazon instead of my shop? OpenAI says ChatGPT picks products from structured metadata supplied by first-party and third-party providers, plus other third-party content, and ranks merchants on availability, price, quality and whether they are the maker or primary seller. Marketplaces have the most product data, the most reviews and the most mentions on other sites, so they are the easy answer to a generic question. The fix is to answer the questions marketplaces answer badly, and to make your own product data just as complete.
Does Amazon block AI crawlers? Yes, in robots.txt. On 10 November 2026, amazon.co.uk and amazon.com both disallowed OpenAI's GPTBot, OAI-SearchBot and ChatGPT-User, Perplexity's PerplexityBot and Perplexity-User, Anthropic's ClaudeBot, Claude-SearchBot and Claude-User, Google's Google-Extended and GoogleAgent-Shopping tokens, and Microsoft's Copilot agents, among others. Googlebot and Bingbot are not blocked site-wide. Amazon still appears in answers because assistants also use third-party product data and what the rest of the web says.
Can a small shop get into ChatGPT shopping results? Partly. OpenAI says product results come from first-party and third-party structured metadata, and that merchants on Shopify already have product data integrated through Shopify Catalog. Direct product feeds are currently limited to approved partners who apply at chatgpt.com/merchants. Everyone else should make sure OAI-SearchBot can read their product pages, keep price, stock, delivery and returns in plain text, and collect reviews on public sites.
Which questions can an independent shop win against marketplaces? Questions a marketplace listing cannot answer well: where to buy something near a particular town, who makes it in the UK, which option suits a specific need, what to buy as a gift for a particular person, and which independent or specialist retailer stocks a niche category. Publish a page that answers each of those questions directly, with the facts an assistant needs to quote you.
Is it worth listing on Google Merchant Center if I'm a tiny shop? Yes. Google's free listings show your products at no cost on Google Search, Maps, the Shopping tab, Google Images, Google Lens, YouTube and Gemini, although placement is not guaranteed. You need to follow Google's policies, publish a returns policy on your site and set up delivery settings for the UK. Google's own guidance on AI features recommends keeping Merchant Center data up to date.
Sources
- Amazon.co.uk, robots.txt (checked 10 November 2026)
- Amazon.com, robots.txt (checked 10 November 2026)
- OpenAI Help Center, Shopping with ChatGPT Search
- OpenAI, Overview of OpenAI crawlers
- OpenAI, Get started with Agentic Commerce
- OpenAI, Product feed specification
- Perplexity, Perplexity crawlers
- Perplexity, Shop like a Pro
- Anthropic, Does Anthropic crawl data from the web, and how can site owners block the crawler?
- Google Search Central, Optimizing for generative AI features on Google Search
- Google Search Central, AI features and your website
- Google Search Central, Google's common crawlers (Google-Extended)
- Google Merchant Center Help, Show your products for free across Google
- Google Search Central, Product structured data