Shopify robots.txt for AI Crawlers: What to Allow

In This Guide

• Why robots.txt for AI crawlers matters on Shopify
• Know the bots: training, search and user-initiated
• How to edit robots.txt.liquid step by step
• Three policy templates you can copy
• What robots.txt cannot do
• How to measure and monitor AI crawler traffic
• Frequently asked questions

Why Robots.txt for AI Crawlers Matters on Shopify

For two decades, a robots.txt file was mostly a conversation with Googlebot and Bingbot. Today the same file is a conversation with a much larger cast: model trainers, AI search engines, and assistants that fetch a page the moment a shopper asks a question. A well-built robots.txt for AI crawlers is now part of basic ecommerce hygiene, because the rules you publish decide whether your products can be cited in an AI answer, whether your catalog can be used to train a model, and how much server capacity you hand to bots that never buy anything.

Shopify makes this more nuanced than it looks. Every store ships with a generated robots.txt that already blocks cart, checkout and admin paths, and it says nothing specific about AI user agents. That means your default posture toward GPTBot, ClaudeBot and PerplexityBot is simply "allowed unless something else stops them." If you have never made a deliberate decision, you have made an accidental one.

There is also a revenue angle. AI search and shopping assistants are becoming a discovery channel, and the bots that power them need to read your product pages to recommend them. Blocking everything with a blunt rule can quietly remove you from that channel. Allowing everything can mean paying for crawl load and handing over your catalog without any return. The goal is a policy that matches your business, written precisely enough that each bot gets the treatment you intended.

If you want the strategic side of the decision first, our guide on whether your store should allow AI shopping agents covers the commercial trade-offs. This article is the hands-on companion: which user agents exist, what to put in the file, and where the file stops working.

Know the Bots: Training, Search and User-Initiated

The single most useful mental model is to stop thinking of "AI crawlers" as one thing. Each major vendor now runs separate bots for separate jobs, and you can set a different rule for each. Official documentation from OpenAI, Anthropic, Perplexity and Google all draw the same three-way split.

Training crawlers

These collect content that may be used to train foundation models. GPTBot is OpenAI's training crawler; disallowing it signals that your content should not be used for that purpose. ClaudeBot is Anthropic's equivalent, and Anthropic states that disabling it excludes future materials on your site from training datasets. Google-Extended is a little different: it is a standalone robots.txt product token that controls whether content Google crawls can be used for future Gemini models, and it has no separate HTTP user agent of its own. Google is explicit that Google-Extended does not affect inclusion or ranking in Google Search.

Search crawlers

These index pages so an AI product can surface and link to them. OAI-SearchBot powers ChatGPT search, and OpenAI notes that opted-out sites will not appear in ChatGPT search answers, though they may still show up as navigational links. Claude-SearchBot analyzes content to improve the relevance of Claude's search responses, and Anthropic warns that blocking it may reduce your visibility in user search results. PerplexityBot surfaces and links sites in Perplexity results, and Perplexity says it is not used to crawl content for foundation models. For a store that wants AI-driven discovery, these are the bots you generally want to welcome.

User-initiated fetchers

These act on behalf of a person who just asked a question. ChatGPT-User, Claude-User and Perplexity-User visit a page in real time to answer that person. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores robots.txt because a user triggered the fetch. Anthropic describes Claude-User as accessing sites when a user's question calls for it. This category is the reason robots.txt alone is never a complete answer, which we return to below.

One more detail worth knowing: each setting is independent. OpenAI notes that if you allow both OAI-SearchBot and GPTBot, it may use a single crawl for both purposes. So a clean separation in your file is a statement of intent, and you should expect vendors to honor it for the bot named in the rule.

How to Edit robots.txt.liquid on Shopify, Step by Step

Shopify supports custom robots.txt rules through a theme template called robots.txt.liquid. According to Shopify's developer documentation, adding this template replaces the generated robots.txt output, which is why the first step matters most.

Step 1: Save what you have today

Open yourstore.com/robots.txt in a browser and copy the full output into a safe place. Shopify itself advises saving a copy of your current file and comparing it with the rendered template after you make changes. This gives you a rollback and a diff.

Step 2: Create the template

In the theme code editor, right-click the Templates folder, choose New File, and name it robots.txt.liquid. Always make this change in a duplicate of your live theme first, then publish once the rendered file looks right.

Step 3: Keep the default rules

Start the template with a loop over Shopify's default groups so you keep every built-in rule. A minimal version looks like this:

{% for group in robots.default_groups %}

{{- group.user_agent }}

{% for rule in group.rules %}{{ rule }}{% endfor %}

{%- if group.sitemap != blank %}{{ group.sitemap }}{% endif %}

{% endfor %}

Shopify recommends using the supported Liquid objects (robots, group, rule, user_agent, sitemap and request) wherever possible, because the rendered output may not exactly match Shopify's generated file otherwise. The loop above preserves the defaults, including the sitemap line.

Step 4: Add your AI bot groups below the loop

Custom rules go outside the default-rules loop as plain text. Shopify's documentation shows the pattern for blocking or allowing a named crawler: a user agent line followed by Disallow or Allow. For example, to opt out of OpenAI training:

User-agent: GPTBot

Disallow: /

Repeat the pair for each bot you want to treat differently, leaving a blank line between groups. Remember that robots.txt matching is by user agent token, so spell each name exactly as the vendor documents it.

Step 5: Test, then publish

Preview the duplicate theme, load /robots.txt, and diff it against the copy you saved. Confirm the default Disallow lines for cart, checkout and admin paths are still present and that your new groups appear once each. Then publish the theme and re-check the live file. OpenAI notes that robots.txt changes for its search bot can take about 24 hours to take effect, so do not expect instant results.

Three Policy Templates You Can Copy

There is no universal right answer, but most Shopify brands land on one of three postures. Pick the one that matches your goals, then adjust the exceptions.

Template A: Open to AI discovery, closed to training

This is the most common choice for brands that want to be recommended without contributing to model training. Allow the search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) and disallow the training bots (GPTBot, ClaudeBot and the Google-Extended token). The reasoning is simple: search bots send shoppers who can click through to your store, while training crawlers take content without a visit in return. Because Google-Extended does not affect Search inclusion or ranking, opting out of it does not touch your organic Google traffic.

Template B: Fully open

If brand awareness inside AI answers is worth more to you than controlling training use, leave everything allowed and focus your effort on crawl load and verification instead. This suits brands with strong editorial content, educational catalogs, or a goal of becoming the default cited source in their category. You still benefit from naming bots in your file so the policy is documented, even when every rule says Allow.

Template C: Locked down

Disallow every AI-related user agent, search and training alike. This fits brands with exclusive imagery, licensed content, or real concerns about price scraping. Be honest about the cost: you give up visibility in AI search products that rely on the bots you block. Anthropic and OpenAI both state plainly that blocking their search bots may reduce your visibility in those products.

What Robots.txt Cannot Do

This is where many store owners get a false sense of security. Robots.txt is a voluntary protocol: it works only on bots that choose to read and respect it. That creates three gaps you should plan for.

First, user-initiated fetchers may not obey it. As noted above, OpenAI says robots.txt rules may not apply to ChatGPT-User and Perplexity says Perplexity-User generally ignores them. If you need to control what an assistant can retrieve in the moment, you need a control that works at the request level, not a text file.

Second, anyone can lie about their user agent. A scraper can announce itself as GPTBot or as a normal Chrome browser, and a robots.txt rule will not stop it. Vendors publish IP ranges precisely so you can verify a claim: Anthropic points to a published list at claude.com/crawling/bots.json, OpenAI publishes JSON files for each of its bots, and Perplexity describes its published ranges as the source of truth for firewall rules. Checking a request's source IP against those lists is how you separate a real crawler from an impostor. Our write-up on verifying AI agents with Web Bot Auth goes deeper on cryptographic verification.

Third, do not block vendor IP ranges instead of using robots.txt. Anthropic specifically warns that blocking its IPs may not reliably opt you out, because it prevents the bots from reading your robots.txt in the first place. Use robots.txt to express policy and IP or signature checks to verify identity; the two jobs are complementary.

The practical lesson is that robots.txt is the polite layer of a larger system. Hostile scrapers, credential stuffers and inventory bots will never read it. For those, you need active detection, which is covered in our complete guide to stopping bot traffic on Shopify and in the breakdown of good bots versus bad bots.

How to Measure and Monitor AI Crawler Traffic

A policy you cannot observe is a guess. After you publish your rules, build a simple feedback loop so you can see who is actually visiting and whether they behave.

• Review server or edge logs by user agent. Group requests by the named AI bots and compare volume before and after your change. Log analysis is the most reliable source of truth, and our primer on log file analysis for large ecommerce sites shows how to structure it.
• Verify the source. For each named bot, compare request IPs against the vendor's published ranges. Requests that claim a user agent but come from elsewhere are impersonators, and they are the traffic you most want to challenge or block.
• Watch crawl behavior. A bot that hammers collection and filter URLs can compete with Googlebot for resources. Anthropic supports the non-standard Crawl-delay directive for ClaudeBot, which can ease load, though not every vendor honors it. See our notes on crawl budget and speed for how crawl load interacts with search performance.
• Check your analytics for skew. Bots that execute JavaScript can inflate sessions and depress conversion rate. Segment suspicious traffic before you trust a dashboard, especially during A/B tests.

Frequently Asked Questions

Does blocking GPTBot or ClaudeBot hurt my Google rankings?

No. GPTBot and ClaudeBot are separate crawlers from Googlebot, and Google states that Google-Extended does not affect inclusion or ranking in Google Search. Blocking training crawlers changes who can use your content for model training, not how Google ranks your pages.

Can I edit robots.txt on Shopify without a developer?

Yes, if you are comfortable in the theme code editor. You create a robots.txt.liquid template, keep the default-rules loop, and add plain-text groups for the bots you want to treat differently. Work in a duplicate theme and compare the rendered file against a saved copy of your original before publishing.

Should I allow OAI-SearchBot but block GPTBot?

Many brands do exactly this. OAI-SearchBot controls whether your pages can appear in ChatGPT search answers, while GPTBot relates to training. OpenAI says each setting is independent, so you can allow one and disallow the other, though it may use one crawl for both if you allow both.

Will robots.txt stop AI agents that shop on behalf of customers?

Not reliably. User-initiated fetchers such as ChatGPT-User and Perplexity-User may not follow robots.txt because a person triggered the request. To manage agentic browsing you need request-level controls such as verification of the agent's identity and rate limits.

How often should I update my AI crawler rules?

Review them at least quarterly. Vendors add new bots, split existing ones, and update their published IP ranges, so a rule set that was complete when you wrote it can drift out of date.

Is blocking AI crawler IP addresses a good alternative?

Generally no. Anthropic warns that blocking its IP ranges may prevent its bots from reading your robots.txt, which can defeat the opt-out. Use robots.txt to state policy and use IP verification to confirm that a request really comes from the vendor it claims.

Turn Your Crawler Policy Into Real Protection

A robots.txt file states what you want; it cannot enforce it. Nostra's edge-based bot protection verifies who is really behind each request, challenges impersonators and scrapers, and keeps real shoppers on a fast path, so your AI crawler policy holds up in practice instead of only on paper. If you want to see how much of your traffic is automated and which AI bots are really visiting, talk to the Nostra team and we will walk you through it.