ShelfIndex

ShelfIndex reads public product listings to compare prices, package sizes and availability across independent online stores (food & beverage).

User-Agent

ShelfIndex/1.0 (+https://shelfindex.app)

Earlier versions of the crawler identified as ShopIndexBot, StandardGlobalResearch and StandardResearch. robots.txt rules written for any of these names are still honoured.

What it does, and how often

Verify that a request is ours

Requests are signed with Web Bot Auth (HTTP Message Signatures, RFC 9421). The public key is published in our key directory; each request carries Signature-Agent: "https://shelfindex.app" and a Signature you can check against it.

Block or slow us down

User-agent: ShelfIndex
Disallow: /

Add this to your robots.txt; it takes effect on our next visit. To ask for a slower pace instead, use Crawl-delay or write to us.

Contact

bot@shelfindex.app