SuggestAPISuggestAPI

Blog • November 16, 2026 • Elena Voss

Visa’s AI Shopping Benchmark: What Merchants Can Learn From a 42.8% Failure Rate

Visa’s new benchmark found that AI shopping models failed 42.8% of test runs. Reliable shopping starts with understanding the buyer’s requirements, checking them against product facts, and knowing when not to recommend anything.

Shopper requirements for a TV under $900 with HDR10+ video and 120 Hz gaming, beside a $799 television whose HDR10+ support is marked unknown.

Summary. Visa’s AI shopping benchmark found a 42.8% failure rate. Here’s what merchants can do to make product discovery more reliable.

A shopper asks an AI assistant for help buying a television. They know roughly what they want to spend and mention a brand a relative recommended. They don’t initially mention the video format their family uses or the gaming features they need.

A helpful assistant has to ask. Otherwise, it can return a convincing recommendation that misses the point.

That’s the kind of challenge in Visa’s new Agentic Commerce Benchmark. Visa tested 22 AI models from 12 labs across 15 shopping scenarios, with three attempts per scenario. Overall, 57.2% of runs passed and 42.8% failed. Visa announced the result on November 5, 2026.

For merchants, the result is a useful prompt to look beyond whether an assistant can produce a product recommendation. Can it discover what matters to the shopper? Can it check those requirements against the catalog? And can it say when it doesn’t have enough information?

What Visa tested—and what a failure means

Visa’s benchmark puts AI models through shopping tasks involving products such as TVs, laptops, and personal-care appliances. The shopper begins with an incomplete request. The assistant has to uncover the relevant requirements through conversation, research the options, and make an informed decision.

To pass, an assistant must identify the task-defined customer requirements and select an offer that meets them—or correctly decide not to buy if no suitable option exists. A lucky guess doesn’t count: even if a selected product happens to fit, the model can fail if it hasn’t uncovered the information needed to justify the choice. The Visa Commerce Benchmark research whitepaper describes the test in more detail.

Several results stand out:

  • 42.8% of runs failed. That does not mean 42.8% of real-world AI purchases are wrong. It is the result for the models and scenarios in Visa’s benchmark.
  • 54% of failed runs still ended in a purchase decision. In those runs, the assistant selected an offer despite a mismatch or an incomplete understanding of the customer’s needs. This is 54% of failed runs, not 54% of all runs.
  • Only 34.2% of model-and-scenario combinations passed all three attempts. A model could succeed once and struggle when given the same challenge again.
  • The five models that uncovered the most hidden buying factors averaged a 70.2% pass rate. The five that uncovered the fewest averaged 44.4%. That’s an association in Visa’s test, not proof that asking more questions alone caused the difference.

There’s an important boundary here. Visa used a controlled market and simulated shoppers, and the benchmark stopped before checkout. The “purchase” in this test was an offer-selection decision, not a completed payment or order. Visa says it plans to extend the benchmark to checkout, payment, and order placement in the future.

A good recommendation starts before the search

Imagine a shopper says: “I need a TV under $900.”

That’s enough to retrieve a lot of TVs. It isn’t necessarily enough to recommend one.

The assistant might need to learn that the shopper wants to play family videos in HDR10+ and use the TV for 120 Hz gaming. Those details change which products qualify. So do the shopper’s budget, the relevant variant, and whether an offer is actually available.

Now consider what happens if the first search finds a popular TV under $900, but the product record doesn’t clearly state whether it supports HDR10+. The assistant has a few honest options: find a trustworthy source for that specification, ask for help if appropriate, or explain that the requirement can’t be verified. It shouldn’t fill the gap with a confident guess.

Visa’s benchmark makes this distinction visible. A recommendation can sound plausible while overlooking something the customer would have considered essential. A successful interaction might also end with “none of these options meets your requirements”—not a product card.

The merchant’s opportunity: make the facts easier to check

Visa’s study primarily tests the models’ shopping judgment in a controlled environment. It does not establish that merchants’ catalog data caused the failures, or that any particular search integration would fix them.

But merchants can still act on the lesson. Once an assistant understands a requirement, it needs a dependable way to check it against the products being offered.

Start with the attributes that actually change a buying decision. For a TV, that might mean supported formats, screen size, gaming specifications, and compatible devices. For shoes, it might be size, fit, material, and whether the requested variant is in stock. For replacement parts, the exact model compatibility may matter more than the product description.

Those facts need to be accessible as consistent data, not just implied by a product name or buried in marketing copy. A useful product result should also identify the specific item or variant and provide a clear path to its product page.

Price and availability need attention too. A search result that shows an old price or an unavailable size can lead the shopper to the wrong conclusion. Google’s Merchant Center guidance similarly calls for price and availability information to match the landing page, structured data, and checkout. Where shipping, membership fees, or other costs affect a decision, make those terms available and clear as well.

Crucially, missing data is not the same as a negative answer. If the catalog doesn’t say whether a TV supports HDR10+, the assistant should treat that as unknown—not as confirmed support or confirmed incompatibility. And “I can’t verify a match” is different from “there are no matching products.”

Keep the shopper’s requirements through the conversation

A shopping request rarely arrives fully formed. Someone might start with “a TV under $900,” then add a gaming requirement, and later ask to compare two models.

If each follow-up is treated as a fresh keyword search, earlier constraints can disappear. The assistant may compare attractive options without noticing that one exceeds the budget or lacks a feature the shopper already marked as essential.

The safer approach is to carry the shopping brief forward: distinguish must-haves from preferences, update it when the shopper clarifies something, and use it to filter or evaluate results. When a product doesn’t meet every preference, the assistant can explain the trade-off instead of quietly changing the brief.

This is where discovery infrastructure matters. The model needs to conduct the conversation and make the decision. The catalog and search backend need to provide the product facts. The discovery layer connects the two, translating the request into something the search system can use and returning results the assistant can inspect.

How to test your own AI shopping experience

You don’t need to recreate Visa’s entire benchmark to start learning from it. Pick one shopping task and make the test specific enough that your team can tell whether the assistant helped.

  1. Choose realistic shopping requests. Use anonymized customer searches or support questions as inspiration. Include requests where the shopper’s first sentence leaves out an important requirement.
  2. Write down the answer key. For each scenario, define the shopper’s must-haves, preferences, and the facts a tester should reveal only when asked. Check the expected outcome against a known catalog snapshot: is there a qualifying offer, no qualifying offer, or insufficient data to tell?
  3. Review the recommendation, not just the response. Did the assistant uncover the necessary requirements? Did the specific product and variant satisfy them? Were the price and availability supported by the data? Did it acknowledge anything it couldn’t verify?
  4. Repeat each scenario. Run the same case several times and record how often it passes, along with the number of attempts. Keep the test conditions and catalog snapshot consistent when comparing changes. One convincing demo is not the same as a reliable experience.
  5. Check the handoff separately. Open the returned product link and check that the right item or variant, price, and availability are reflected on the storefront. Checkout and payment deserve their own tests; Visa’s initial benchmark did not evaluate them.

Track failures by type, not just pass rate. A missed customer requirement, a stale price, and an incorrect product comparison need different fixes. If a scenario has no verified match, check whether the assistant says so rather than recommending the nearest-looking product.

For a broader review of the catalog and handoff, see our six-point AI shopping catalog checklist.

Where SuggestAPI fits

SuggestAPI is a discovery layer for the search and catalog systems merchants already use. It is designed to extract constraints from natural-language requests, guide the existing search backend, preserve context through refinements, and return structured results that an application or agent can use. Its listed integrations include Algolia, Typesense, Elasticsearch, Meilisearch, and Shopify.

For example, an assistant could use the shopper’s confirmed budget and TV requirements to search and compare catalog matches, then send the shopper to the merchant’s storefront. SuggestAPI supports the discovery side of that flow; it doesn’t replace the assistant’s judgment, guarantee the accuracy of the underlying catalog, or take over the merchant’s cart and checkout.

Visa’s results don’t tell us that a discovery layer will make an assistant pass the benchmark. They do offer a useful standard for the work: a recommendation should be grounded in what the shopper actually needs and what the merchant can verify.

Want to make your catalog easier for assistants to search and compare? Explore SuggestAPI or read The SuggestAPI Guide to Agentic Commerce.

References

  1. Visa, “Visa Releases First-of-Its-Kind Industry Benchmark for AI Shopping; Almost Half of Test Runs Fail,” November 5, 2026
  2. Visa Research, Can an AI agent shop with you? Visa Commerce Benchmark, October 2026
  3. Google Merchant Center Help, “Product data specification”
  4. SuggestAPI, “The discovery layer”

Make the facts easier to check

SuggestAPI turns a shopper’s requirements into structured matches from the catalog and search stack you already run.