How to check if your Shopify store blocks AI crawlers

The short answer: open yourstore.com/robots.txt and look for Disallow: / (a slash and nothing after it). If it sits under User-agent: *, or under the name of an AI search crawler such as OAI-SearchBot, Claude-SearchBot or PerplexityBot, that assistant can't read your store. If you don't find it, robots.txt isn't what's keeping you out of AI answers.

That takes a minute. The rest of this post covers how to read what you find, the cases where a crawler is blocked without Disallow: / appearing next to its name, and what robots.txt doesn't show you.

Which crawlers to check

AI companies run two kinds of crawlers, and they do different jobs.

Search crawlers fetch pages so an assistant can answer with live sources and link to them. These decide whether your store can appear in answers today:

Training crawlers collect text to train future models. Blocking them doesn't remove you from today's answers:

There's a third group, fetchers such as ChatGPT-User and Perplexity-User, which load a page when a person asks the assistant to look at it. Their operators say robots.txt rules may not apply to user-requested fetches, so they aren't what this check is about.

The two groups get mixed up all the time. A "block AI bots" snippet copied from a forum often lists OAI-SearchBot next to GPTBot, so a store that only meant to opt out of training drops out of ChatGPT search too.

Step 1: Read your robots.txt

Go to https://yourstore.com/robots.txt in a browser. It's plain text, split into groups. Each group starts with one or more User-agent: lines naming crawlers, followed by rules:

User-agent: *
Disallow: /admin
Disallow: /cart
Disallow: /checkout
Sitemap: https://yourstore.com/sitemap.xml

How to read it:

• Disallow: / blocks the whole site for the crawlers named in that group.
• Disallow: /something blocks one path. Shopify's default file has a long list of these (/admin, /cart, /checkout, /account and so on). They keep crawlers out of pages with nothing to recommend. They're fine and you should leave them alone.
• User-agent: * is the fallback group. It applies to every crawler that doesn't have a group of its own.

Then ask two questions:

1. Does the User-agent: * group contain Disallow: /? If so, every crawler without its own group is locked out, AI search crawlers included.
2. Does any of the search crawlers above have its own group with Disallow: /? Search the page (Cmd+F or Ctrl+F) for each name.

Shopify's default robots.txt doesn't block any of the search crawlers in the table. If yours does, the file was edited, by a person, an app or a theme.

Step 2: Watch for blocks you won't see next to the crawler's name

Two patterns catch people out.

A crawler with its own group ignores the * group completely. It follows only the most specific group that names it. That cuts both ways. A group like this:

User-agent: OAI-SearchBot
Allow: /

lets OAI-SearchBot in, but also lets it into /cart, /checkout and every other path the * group was keeping it out of. And if someone gave a crawler its own group and left out a rule you were relying on, the * group won't fill the gap.

Several crawlers can share one group. Consecutive User-agent: lines share the rules that follow them:

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: CCBot
Disallow: /

This blocks all three. If you search the page for "OAI-SearchBot" you'll find its name, but the Disallow: / is a few lines further down. Read the whole group, not just the line with the name.

Step 3: Check whether the file was customized

On Shopify, a custom robots.txt comes from a theme file called robots.txt.liquid. To see whether yours has one, go to Online Store > Themes, then ... > Edit code on your live theme, and look in the templates folder.

• No robots.txt.liquid: you're on Shopify's generated file, which doesn't block AI search crawlers. Your result from step 1 should match this.
• There is one: someone replaced the default. Read it. Make sure it still outputs Shopify's default rules (it should loop over robots.default_groups) rather than a hand-written list, and check any groups added below that loop.

If you plan to change it, save a copy of your current /robots.txt first. Shopify's own documentation notes that a customized template's output doesn't always match the file Shopify generates, so compare the two before you publish. A wrong Disallow: / takes effect as soon as the theme is saved.

Step 4: Check what robots.txt doesn't cover

A clean robots.txt is necessary, not sufficient. Three other things keep crawlers out:

• Private mode. While the store is restricted under Store access in Online Store > Preferences, crawlers get the password page, not your products. This is common on stores that recently launched or are being rebuilt.
• noindex on individual pages. Open a product page, view the source and search for noindex. A <meta name="robots" content="noindex"> tag asks search engines to leave that page out. SEO apps, theme code and Shopify's seo.hidden metafield can all add it.
• Something answering crawlers differently. You can request a page the way a crawler would from a terminal:

curl -A "OAI-SearchBot" -s -o /dev/null -w "%{http_code}\n" https://yourstore.com/products/your-product

200 means the page came back normally. A 403 or a challenge page means something in front of the store is treating bots differently from people. This test only changes the user agent: it can't reproduce rules that check where a request really comes from, so a 200 is a good sign rather than proof.

Some guides also mention an "Allow AI bots" switch in Shopify's settings. Shopify's help center doesn't document one: the Preferences page controls access to the whole store, not individual crawlers, and Shopify handles bot management itself. If you can't find the switch, that's why.

One more boundary: crawling and Shopify Catalog are separate pipes. According to Shopify, blocking AI crawlers only affects how your site is discovered on the open web. It doesn't stop Shopify Catalog from sending your product data to the agentic storefronts you've turned on, which you manage in the admin's agentic settings.

If you find a block

• Disallow: / under User-agent: *: almost always a mistake. Remove that line. If it's in robots.txt.liquid, remove it there.
• A search crawler with its own Disallow: /: remove that group, or take the search crawler's name out of a shared group. Without a group of its own it falls back to the * group and Shopify's default path rules.
• You only want to opt out of training: give the training crawlers their own groups with Disallow: / after the default rules, and leave the search crawlers alone:

User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

Whether to allow training crawlers is a business decision, and it's yours. It just shouldn't take the search crawlers down with it.

After a fix, load /robots.txt again to confirm the live file changed. Crawlers re-read it on their own schedule, so give it some days before expecting to show up in answers.

Checking this without editing theme code

Discoverly runs this check as part of a read-only scan of your store. It reads your live robots.txt group by group, the same way crawlers do, and reports which assistants can reach the store and which ones only train on it. A store fails the check only when a search crawler is actually locked out. Shopify's default file passes.

If something needs fixing, the fix arrives as a draft. You see the full diff against your live file before anything is published, and any rule that would block a crawler is highlighted so it can't slip through. The original is saved, so you can roll back in one click. Training crawlers get a switch of their own. The audit and the robots.txt fix are free.