If you have been paying attention to technical SEO lately, you have probably noticed the growing anxiety around AI crawlers. Publishers and ecommerce brands are trying to control how ChatGPT, Gemini, Claude, and other AI systems represent their products, and one popular answer is the llms.txt file.
The idea is simple: publish a Markdown file at the root of your site that tells AI systems what you sell, what your policies are, and where to find your best content. It can be a useful, curated entry point for systems that choose to read it.
The problem is that llms.txt is fundamentally passive. It can describe your business, but it cannot give an active AI shopping agent a reliable way to search current inventory, apply filters, compare live offers, or hand a selected product to checkout. If you want your catalog to be discoverable and useful to shopping assistants, a text file is not enough. You need a structured capability catalog. You need Agentic Resource Discovery (ARD).
The rise of llms.txt in ecommerce
The llms.txt proposal addresses a real problem: complex ecommerce sites can be difficult for AI systems to parse. JavaScript-rendered product listings, dynamic filters, and inventory data spread across multiple services can prevent a reader from correctly understanding what a store sells. A plain-text Markdown file at the domain root offers a clean summary and can point to authoritative content.
That is useful for brand descriptions, support policies, documentation, and retrieval systems that consume the file. It may also reduce the chance that a system relies on outdated third-party summaries when official information is available.
But llms.txt has real limits for live commerce. It cannot practically contain an entire changing catalog, current prices, or real-time stock levels. It also cannot define how an agent should search by size, apply a price ceiling, or create a safe handoff to a merchant-controlled cart.
Adoption does not guarantee consumption, either. An Ahrefs study of 137,000 sites found that 97% of published llms.txt files received no requests during the study period. Google also states in its guidance for generative AI features in Search that it does not use llms.txt for visibility or ranking. The file can still serve systems that support it, but it is a passive aid rather than an active commerce engine.
Why shopping agents need more
Consider how an AI shopping assistant works. When a shopper asks, “Find me waterproof hiking boots in size 11 under $150,” the agent does not need another paragraph describing the brand. It needs to execute a live search against current catalog data.
To do that reliably, the agent must know where the search capability lives, how the request is structured, what the response contains, and how to verify that it is connecting to the intended publisher. A static text file cannot answer those operational questions.
That is the gap Agentic Resource Discovery was built to fill.
Enter Agentic Resource Discovery
Agentic Resource Discovery is an open specification for publishing, discovering, and verifying AI-facing capabilities across the web. Rather than a document meant primarily for reading, ARD uses a structured ai-catalog.json file designed for capability discovery.
A merchant or infrastructure provider can publish a catalog that declares: “I have a product search capability. Here is its technical artifact. Here are representative requests. Here is publisher and trust information.” An ARD-compatible agent or registry can inspect that catalog, identify a relevant capability, and then connect through the capability’s native API or protocol.
| Feature | llms.txt | Agentic Resource Discovery |
|---|---|---|
| Format | Markdown text | Structured JSON catalog |
| Primary role | Passive reading and retrieval context | Capability discovery by agents and registries |
| Data freshness | Static and manually maintained | Points to live APIs and technical artifacts |
| Commerce use | Brand summaries and policies | Live search, comparison, routing, and handoff discovery |
One distinction is important: ARD handles discovery, not execution. It helps an agent determine what capability exists and where to find its supporting artifact. Authentication and the actual interaction remain under the control of the selected API or protocol. That makes ARD composable with OpenAPI, MCP, A2A, HTTP QUERY, and other parts of the agentic stack.
SuggestAPI bridges the gap
At SuggestAPI, we believe your search infrastructure should bridge your catalog and the agentic web. You should not have to build a custom integration for every new AI assistant.
The SuggestAPI Knowledge Gateway publishes an ARD-compatible catalog at https://agent.suggestapi.com/.well-known/ai-catalog.json. When search is routed through SuggestAPI in front of an existing Algolia, Typesense, Elasticsearch, Meilisearch, or SuggestAPI index, the catalog gains a discoverable Agent Search Layer without replacing the underlying engine.
- Discovery: An AI agent or registry uses ARD to find the relevant SuggestAPI-powered commerce capability.
- Execution: The agent sends a structured request to search for what the shopper wants.
- Retrieval: SuggestAPI routes the request to the configured search engine and returns normalized, agent-ready product data.
- Handoff: SuggestAPI provides safe links into the merchant’s existing cart or checkout flow, leaving the transaction under merchant control.
Your search engine stays. Your storefront stays. Your checkout stays. SuggestAPI provides the discovery and search layer that makes the catalog findable and useful to the agentic web.
Do both, but rely on live capabilities for commerce
Should you publish an llms.txt file? It can be worthwhile when you have a clear audience or system that consumes it. It is inexpensive to maintain, gives supported readers an authoritative summary, and can point them toward official policies and documentation.
But a text file is not an agentic commerce strategy. If you want AI assistants to actively search your catalog, compare current products, and guide shoppers toward a transaction, expose those capabilities through structured APIs and open discovery protocols.
Use llms.txt to explain your business. Use ARD and live search APIs to let agents work with it.