ShelfIndex
ShelfIndex reads public product listings to compare prices, package sizes and availability across independent online stores (food & beverage).
User-Agent
ShelfIndex/1.0 (+https://shelfindex.app)
Earlier versions of the crawler identified as ShopIndexBot, StandardGlobalResearch and StandardResearch. robots.txt rules written for any of these names are still honoured.
What it does, and how often
- Reads public pages and public product data only (store pages, sitemaps, public product feeds). Only GET and HEAD requests: no logins, no forms, nothing is ever posted.
- Cookies: nothing a store sets is stored or sent back. The one thing we send, to Shopify stores, is the storefront's own country and currency selector (
localization,cart_currency), to read prices for a market. - One pass per store per week, plus a few retries after a failed visit (never sooner than two hours later).
- Pace: at least one second between requests to the same store (two seconds when product pages are read one by one), and longer when the store's Crawl-delay or its responses ask for it. Shopify stores: one request every 15 seconds across all Shopify stores together.
- Respects
robots.txt(RFC 9309). Crawl-delay is honoured: we wait the longer of our own pace and yours. - When a store answers 429 or 503 we do not repeat the request in the same visit, and we do not come back before its
Retry-After(never sooner than two hours). The one exception: a 429 with a Retry-After of up to 30 seconds is waited out and asked once more. A 429 from any Shopify store pauses all our Shopify requests for at least 30 minutes.
Verify that a request is ours
Requests are signed with Web Bot Auth (HTTP Message Signatures, RFC 9421). The public key is published in our key directory; each request carries Signature-Agent: "https://shelfindex.app" and a Signature you can check against it.
Block or slow us down
User-agent: ShelfIndex Disallow: /
Add this to your robots.txt; it takes effect on our next visit. To ask for a slower pace instead, use Crawl-delay or write to us.