About the Kaupr Crawler

What this is

Kaupr operates an identifiable, well-behaved crawler that indexes publicly available product data from online stores. We use this data to build a global product discovery platform that helps shoppers find products through natural language search, visual similarity, and AI assistants.

If you arrived here because you saw the Kaupr User-Agent in your server logs, this page tells you who we are, what we’re doing, and how to control it.

Identity

• User-Agent: Kaupr/1.0 product indexer (+https://kaupr.com/about/crawler)
• Contact: crawler@kaupr.com
• Source IP range: available on request from the contact above

What we crawl

We fetch publicly available product information. Specifically, on Shopify stores we read the /products.json endpoint that Shopify exposes by default. This is the same endpoint Google, Bing, and price comparison services have crawled for years. We do not attempt to access:

• Customer data of any kind
• Admin or merchant-only pages
• Pages behind authentication or a password gate
• Files explicitly disallowed by your robots.txt

How we behave

• We respect robots.txt. If you disallow our User-Agent, or disallow /products.json generally, we stop.
• We rate-limit to roughly one request per second per domain — well within polite crawl rates.
• We back off and retry later on 429 or 503 responses.
• We re-crawl each store roughly once per day to keep prices and availability current.
• We never spoof other User-Agents or rotate identities to evade detection.

How to block Kaupr

Add this to your robots.txt:

User-agent: Kaupr
Disallow: /

Our crawler will pick up the change within 24 hours and stop. No further action is required.

How to request removal of existing pages

If your products are already on kaupr.com and you want them removed, submit a request at kaupr.com/merchants/removal. We acknowledge within 48 hours and complete removal within 7 days. We also add your domain to our internal blocklist so we will not re-index your store in future crawls.

Why we exist

AI assistants like ChatGPT and Perplexity send shoppers to the product pages that best answer the shopper’s query. Independent stores often have thin product data and lose out to large retailers. Kaupr enriches independent product data so smaller merchants compete on equal terms in the AI shopping era. When a shopper buys, we earn a small affiliate commission and you keep the customer. See our affiliate disclosure for the economics.

Get in touch

If you want to be a Kaupr direct merchant, see better attribution, or just have questions — email hello@kaupr.com. We respond.