This is our crawler.
Here is how to stop it.
RankSniperBot fetches a handful of public pages from a Shopify storefront to check whether an AI shopping agent could read that catalog. It identifies itself on every request, it obeys robots.txt, and two lines in that file turn it off permanently.
RankSniperBot/1.0 (+https://ranksniperhq.com/bot)
- Who runs it
- Rank Sniper. The crawler exists for one product: a catalog legibility report a merchant asks us for.
- What it reads
- The delivered HTML and JSON of publicly accessible storefront pages —
/products.json, up to three product pages,/robots.txt,/sitemap.xmland/llms.txt. That is the whole list, and a single scan is a handful of requests, not a crawl of your store. - What it never does
- No form submission. No login, checkout or account pages. No personal data of any kind — it reads product records and page markup, and nothing behind an authentication boundary. It does not follow links to discover more of your site.
- When it runs
- When somebody asks for a report on a specific storefront. It is not a continuous crawler and it does not queue your store off the back of someone else’s scan.
- Crawl complaints
- Tell us through the contact form and name the hostname. A request to stop is honoured whether or not it comes with a robots.txt change.
Blocking it takes two lines
RankSniperBot reads /robots.txt before it fetches anything else from your store, and it obeys what it finds there. To refuse it entirely, add this to your robots.txt:
User-agent: RankSniperBot Disallow: /
Narrower rules work too, and are matched the way the standard says they should be (RFC 9309: the longest matching rule wins, and Allow beats Disallow on a tie). Disallowing only /products.json stops the catalog read; disallowing /products/ stops the product-page sample. A rule that applies to User-agent: * applies to us as well, because we fall back to the wildcard group when no group names RankSniperBot — which is what the standard requires and what most crawlers get wrong.
A refusal ends the scan, not just one request. When robots.txt disallows the catalog endpoint, nothing is fetched at all and the person who asked for the report is told the store declined it. We do not retry a refused host under a different user-agent, and we do not treat a browser-shaped request as a way around a rule written for a crawler — a firewall that permits what robots.txt forbids is one of the failures this product exists to find, and it would be absurd to be that failure.
If your robots.txt is unreachable when we ask for it, we treat that as no restriction, which is the standard’s own default. A file that times out is not a refusal, and a momentary outage should not be read as one.
Why it announces itself at all
A crawler that hides is a crawler you cannot make a decision about. This one carries its name, its version and the address of this page in every request, so anybody reading a server log can find out what it was, who sent it, and how to stop it — inside a minute, without guessing from an IP address.
There is a real caveat and it belongs here rather than in a footnote. Shopify’s edge refuses self-identified bots on most stores, so when a declared request is hard- refused we retry that single request with a standard browser profile. Which path served a scan is recorded and shown in the scan trace, never hidden. That fallback does not apply to a robots.txt refusal: a disallowed path is not fetched by any identity.
What it is actually checking
The same thing we ask merchants to care about: whether an AI shopping agent can read a catalog well enough to recommend it. That means structured data on product pages, the fields an agent needs to state a price and availability, and whether the store’s own robots.txt lets the agents in. The full method is published in the rubric, and the agents we track are listed on the crawler index.
