Catalog Bot
SuggestAPI Bot
SuggestAPI Bot supports catalog crawling for product data import into SuggestAPI search indexes.
Purpose
Import product catalog data
Use Case
Build and refresh search indexes
Control
Managed through `robots.txt` rules
What it supports
SuggestAPI Bot is used to crawl product catalog pages and related product data so that catalog content can be imported into SuggestAPI search indexes. This page is intended for store owners, catalog publishers, and platform operators who want to identify or manage that crawler access.
User-Agent
The crawler identifies itself with the following user-agent string:
Web Bot Auth
SuggestAPI Bot signs catalog crawl requests with HTTP Message Signatures so Cloudflare and other verifiers can confirm the traffic is ours. The crawler sends Signature, Signature-Input, and Signature-Agent headers. The public key directory is:
Signature-Agent is "https://scrape.suggestapi.com". Register that directory URL in Cloudflare’s Verified Bots form using the Request Signature verification method.
robots.txt examples
To explicitly allow or disallow catalog crawling for search index import, add rules for the crawler in your site's `robots.txt` file.
Allow
User-agent: SuggestAPI-CatalogBot/0.1 Allow: /
Disallow
User-agent: SuggestAPI-CatalogBot/0.1 Disallow: /
Contact
Questions about catalog crawling or product data import can be sent to help@suggestapi.com.