Your Store's robots.txt and the Block AI Crawlers Setting

Search engines and other crawlers ask every website for a small text file at /robots.txt before they look around. It tells them which addresses to leave alone. PrizeX, BidKing and RaffKing stores now answer that request themselves, with a file built from what your store runs. This guide covers what it asks, why the sign-in pages are handled differently, the Block AI crawlers setting, what happens if you keep your own file, and how to check the result.

Who has it

• The store's /robots.txt and the Block AI crawlers setting are part of the core platform, in every edition: PrizeX, BidKing Starter, BidKing Plus, BidKing Pro and RaffKing Standard.
• They arrive with a software update. Nothing needs switching on for /robots.txt itself; Block AI crawlers is off until you turn it on.

1. What /robots.txt asks

The file starts with a group for every crawler, User-agent: *, followed by Disallow lines: addresses no visitor needs to find from a search engine. In plain words, the store's own list covers:

• The admin and setup: the admin panel, the installer and the cron address.
• Background addresses: the endpoints pages call behind the scenes, including plugins' own.
• Buying: the cart, checkout, payment and order pages.
• The customer's account: profile, wallet, winnings, addresses, the pages that start with /my-, account deletion and data download.
• Sign-out and language switching, which only make sense for a visitor who clicks them.
• Extra sign-in steps: the address that asks a signed-in visitor to confirm it's them again, and single sign-on returns.

Plugins add their own lines without you doing anything. A route that answers with data rather than a page, or one that needs the visitor to be signed in, is listed for every crawler. Addresses set by a standard, such as /.well-known/, are never listed, and each address appears once.

Your listings, category pages, the home page, the blog and your policy pages aren't on the list, so crawlers spend their visits on the pages you want found. You choose the title and description those pages show in search results on the same SEO Optimization screen.

Sitemap lines

A Sitemap line is added at the end when two things are true: your store's address is saved as the Site URL under Settings › General, and the store has a sitemap. That means a sitemap.xml or sitemap_index.xml file in the site's folder, or a plugin that answers /sitemap.xml. Without both, there's no Sitemap line.

2. Why the sign-in pages use noindex instead

The sign-in, sign-up, forgot password and reset password pages aren't in /robots.txt. Each one carries a noindex tag in its page header instead, which asks search engines not to list it.

That's on purpose. A crawler that follows robots.txt doesn't read a page it's told to skip, so it would never see the page's noindex, and a disallowed page that other sites link to can still turn up in results as a bare address. Leaving these pages open to crawlers is what lets the noindex be read. A search engine drops a page it had already listed once it reads the page again and finds the tag.

3. The Block AI crawlers setting

In the admin sidebar, open Pages & Policies, then SEO Optimization. On the SEO screen, the section headed Search engines and AI crawlers starts with a line saying your site's robots.txt "keeps sign-in, account, cart and admin pages out of search results", with a link that opens /robots.txt. Below it is the Block AI crawlers tick box, with this help text:

Tick it and press Save. The screen confirms "AI crawlers are now asked to stay off your site." Untick it and save to go back; the message then reads "AI crawlers may read your site again." Each change is recorded in the Admin Logs.

With the setting on, each crawler on the list gets its own group in /robots.txt that asks it to stay off the whole site. The list includes GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot, Bytespider, Amazonbot, Applebot-Extended and meta-externalagent, among others. Every other crawler keeps the shared group described above, so ordinary search engines still list your pages.

Two things to weigh before you tick it:

• It's a request. robots.txt is followed by well-behaved crawlers. It doesn't lock anything, and as the help text says, a crawler that ignores robots.txt is not stopped.
• AI search reads these crawlers too. Some of the listed crawlers feed AI search tools, such as ChatGPT search and Perplexity. While the setting is on, they're asked to stay away, so your pages may stop appearing in those tools' answers. If buyers find you that way, that's a cost to set against keeping your pages out of AI models.

4. If you keep your own robots.txt file

A robots.txt file in your site's folder is served as it is, and the store's version isn't used. The SEO screen then shows a note:

You have two choices:

• Delete the file to use the store's version. Copy anything in it you still need first: a line you wrote yourself is gone once the file is.
• Keep it on purpose if you'd rather manage robots.txt by hand. Then the store's list, plugins' lines and the Block AI crawlers setting don't apply, and keeping your file up to date is up to you.

5. When changes show

The store keeps the finished text for an hour, and the response tells browsers and crawlers they may keep it that long too. Saving the Block AI crawlers setting builds it again on the next request. Anything else, such as a plugin you've just switched on or off, shows within the hour.

6. How to check it

1. Open your store's address followed by /robots.txt in a browser, or use the link on the SEO screen.
2. Check it starts with User-agent: * and lists the admin, cart and account addresses.
3. If you turned on Block AI crawlers, look for groups such as User-agent: GPTBot followed by Disallow: /.
4. If you see something else entirely, look for a robots.txt file in your site's folder.

On nginx, if /robots.txt answers "not found", your server's shared config may already have its own location = /robots.txt block. The store's nginx.conf.example shows the line it needs:

    location = /robots.txt {
        try_files $uri /error.php$is_args$args;
    }

If your config already has a location = /robots.txt block, put that try_files line in it rather than adding a second one. Our VPS and nginx install guide covers where the server block lives.

What it doesn't do

• force any crawler to obey: robots.txt is a request;
• hide a page from people who have its address;
• let you pick which AI crawlers to ask: the setting covers the whole list;
• make a sitemap: it lists one only when your store has one.

Common questions

Does Block AI crawlers stop AI companies reading my pages?

It asks them not to. robots.txt is a request that well-behaved crawlers follow, and the setting's own help says a crawler that ignores robots.txt is not stopped. With it on, every crawler on the list, such as GPTBot, ClaudeBot, Google-Extended and PerplexityBot, is asked to stay off the whole site.

Will my store still show up in search results?

In ordinary search engines, yes: the setting doesn't ask them to stay away. AI search tools whose crawlers are on the list, such as ChatGPT search and Perplexity, are asked to stay off too, so your pages may stop appearing in their answers while the setting is on.

Why aren't the sign-in pages in robots.txt?

A crawler that follows robots.txt doesn't read a page it's told to skip, so it would never see that page's noindex tag. The sign-in, sign-up, forgot password and reset password pages are left open to crawlers and carry noindex, which asks search engines not to list them.

I already have my own robots.txt file. What happens?

Your file is served as it is, and the store's own robots.txt isn't used. The SEO screen tells you so, and the Block AI crawlers setting has no effect until you delete the file. Delete it to use the store's version, or keep it if you want to manage robots.txt yourself.