10 agents,
tracked.
Before an AI can recommend your store, its crawler has to be allowed to read it. Rank Sniper tracks the 10 agent crawlers that matter and checks, on every scan, that your store is open to them. Block the wrong one and you are not ranked low — you are absent.
GPTBot
Trains and grounds GPT models. Blocking it removes your catalog from what ChatGPT can learn about your products.
OAI-SearchBot
Powers ChatGPT Search — the live retrieval layer that surfaces and links products in an answer.
ChatGPT-User
Fetches a page in real time when a user asks ChatGPT about it directly. A blocked page cannot be read on demand.
ClaudeBot
Anthropic’s crawler for Claude. Reads your public catalog to answer and recommend within Claude.
anthropic-ai
A second Anthropic user-agent; allowing both keeps Claude’s view of your store complete.
PerplexityBot
Indexes for Perplexity’s answer engine, which cites sources — a citation is a link back to you.
Google-Extended
Controls whether Gemini and Google’s AI features may use your content, independent of classic Search indexing.
Applebot-Extended
Governs Apple Intelligence’s use of your pages, separate from Siri and Spotlight crawling.
Bytespider
ByteDance’s crawler, feeding AI products across a very large user base.
CCBot
The open web archive most models train on downstream — the widest-reaching single crawler on this list.
Why the crawler index matters
The single fastest way to disappear from AI answers is to block the crawler that builds them — usually by accident. A robots.txt copied from a template, a security app with an over-eager bot filter, or a CDN rule meant to stop scrapers can quietly slam the door on GPTBot or ClaudeBot, and nothing on your store looks broken. Your pages still load, your classic search ranking is fine, and yet the moment a shopper asks an assistant for a recommendation, your catalog is not in the room.
These agents are not one thing. A training crawler, a live-retrieval search bot and an on-demand fetcher do different jobs and are controlled by different rules — blocking the training crawler but allowing search is a real, defensible choice; blocking all of them by accident is not. Rank Sniper checks each one individually so “crawlers open” means every agent that should be able to read you actually can.
Checked on every scan, not once
Crawler access is not a set-and-forget setting. Themes update, apps change their defaults, and new AI user-agents appear every few months — a rule that was correct in spring can be silently blocking a new agent by autumn. Every Rank Sniper scan re-checks access for the full list above and flags the day it changes, so a door that closes gets noticed the next morning rather than the next quarter. This access check feeds the GEO pillar, where readability by generative engines is scored.
Open the door, then earn the answer
Access is necessary, not sufficient. Once an agent can read your store, it still has to find structured data it can trust, answer content it can quote, and transaction fields it can act on — which is what the other three pillars grade. Access gets you into the room; the rest of the loop is what gets you recommended. Run a free scan to see, in thirty seconds, whether every crawler above can read your catalog today.
