Why Your E-commerce Search Sucks

Abstract

Most e-commerce search systems are fast, stable, and irrelevant. They return results in under 100 milliseconds, scale horizontally without breaking a sweat, and consistently fail to surface what the user actually wanted. The gap between "search works" and "search works well" is where revenue leaks. Search users convert at 2 to 3 times the rate of non-search visitors (Forrester Research, via Nosto), yet only 15% of companies dedicate resources to optimizing site search (Algolia, 2024). This article examines the specific failure modes that plague keyword-based search systems, explains why manual patches fail at scale, and argues that the highest-leverage investment most teams can make is treating their own query data as a first-class product input.

The 30-Year-Old Hack Under the Hood

OpenSearch, Elasticsearch, Solr. Pick one. They're all running some variant of BM25, a term-frequency scoring algorithm that dates to the early 1990s. BM25 is excellent at what it does: given a query containing specific terms, it finds documents containing those terms and ranks them by statistical relevance. It's fast. It's battle-tested. Nobody complains about latency.

The problem surfaces when you measure retrieval quality instead of response time. Metrics like NDCG (Normalized Discounted Cumulative Gain) and Precision@K expose what raw query latency hides: the results are often wrong, and wrong results served quickly are still wrong results.

BM25 scores documents using term frequency and inverse document frequency. The core formula:

$\text{BM25}(q, d) = \sum_{t \in q} \text{IDF}(t) \cdot \frac{f(t, d) \cdot (k_1 + 1)}{f(t, d) + k_1 \cdot (1 - b + b \cdot \frac{|d|}{\text{avgdl}})}$

Where $f(t, d)$ is the frequency of term $t$ in document $d$, $|d|$ is the document length, $\text{avgdl}$ is the average document length across the index, and $k_1$ and $b$ are tuning parameters that control term frequency saturation and length normalization respectively. The formula is elegant and effective for what it does: matching query terms against document terms. It is fundamentally incapable of understanding that "sneakers" and "athletic shoes" refer to the same product category, or that "affordable laptop for students" should match a product listing that says "budget notebook computer."

The instinctive response is to bolt on a fancier model. Swap in semantic search. Add a vector database. The assumption is that the algorithm is the bottleneck. It rarely is.

The actual bottleneck is data. Specifically, the domain-specific data that organizations already own but don't use. Adapting retrievers to specific domains can yield up to 9.3 nDCG@10 improvement over generic dense models (Wang et al., 2022). Dense models trained on generic web data fail dramatically when evaluated on specialized domains like biomedical literature, scientific fact-checking, and financial question answering (Thakur et al., 2021). Transfer and adaptation techniques for domain-specific answer selection have demonstrated 8 to 12% relative MAP gains in industrial settings (Garg, Vu & Moschitti, 2020).

These gains don't come from swapping models. They come from organizations that invest in logging queries and results, labeling relevance, and building domain-specific datasets. Only 7% of companies report learning from their site search data and applying those insights elsewhere in their business (Algolia, 2024).

Three Failure Modes That Repeat Everywhere

Different stacks. Different domains. Different team sizes. The same three problems show up every time.

Failure Mode 1: Synonym Blindness

A user searches "sneakers." The catalog says "athletic shoes." BM25 returns zero results.

This is not an edge case. Baymard Institute found that 70% of the top 50 e-commerce sites cannot return relevant results for product type synonyms (Baymard Institute). The search system is matching strings, not concepts. Any mismatch between the user's vocabulary and the catalog's vocabulary produces a miss.

Failure Mode 2: Intent Misclassification

"Apple" could mean fruit or electronics. "Light jacket" could mean lightweight or light-colored. "Crane" could refer to a bird, a machine, or a yoga pose.

Keyword systems cannot disambiguate. They match tokens. Word sense ambiguity degrades information retrieval systems, and disambiguation accuracy above 90% is required before it even starts to help keyword retrieval (Sanderson, 1994). Below that threshold, disambiguation introduces more noise than it removes.

Failure Mode 3: Zero-Result Dead Ends

The user typed something reasonable. The system returned nothing. The user left.

Baymard Institute's testing found that 31% of product-finding tasks end in failure when users rely on search (Baymard Institute). Every zero-result query is a user signaling "I wanted to give you money but you wouldn't let me."

The industry average zero-result rate sits between 10 and 20%, while top performers keep it under 2 to 3% (Algolia, 2024; Baymard Institute). For a site processing 100,000 queries per day, a 10% zero-result rate means 10,000 daily dead ends. Given that search users convert at 2 to 3 times the rate of browsing users, each dead end carries disproportionate revenue cost.

Why Manual Patches Don't Scale

The standard response to all three failure modes is manual rules. Synonym lists. Query rewrites. Redirect maps. Curated results for high-traffic queries.

This works, for a while. Then the catalog changes, user behavior shifts, new jargon emerges, and the team is maintaining a brittle system that nobody fully understands. Hand-built knowledge resources are expensive, limited in coverage, and domain-dependent. Only 38.6% of queries with synonym expansion showed significant retrieval improvement (Azad & Deepak, 2019). The remaining queries either saw no benefit or degraded.

The underlying issue is that rules encode yesterday's vocabulary. They can't anticipate tomorrow's queries. Every new product category, every seasonal trend, every shift in user language creates a gap that manual rules have to chase.

The Scalable Fix: Domain-Adapted Models

Neural hybrid models that learn from domain data consistently beat traditional expansion methods across out-of-domain datasets (Chen et al., 2022). Instead of maintaining synonym lists, the system learns that "sneakers" and "athletic shoes" are the same thing because users treated them that way in click logs and purchase patterns.

The performance of dense retrievers severely degrades under a domain shift (Wang et al., 2022). This means that off-the-shelf embeddings with a generic reranker, while better than pure keyword search, still miss the bigger opportunity. Every domain has jargon, synonyms, and intent patterns that no general-purpose model understands out of the box. Medical terminology. Legal shorthand. Product attribute combinations. Manufacturing specifications.

That domain-specific data is a competitive asset. Generic models can't replicate it. Competitors can't buy it. It exists in query logs, click streams, session recordings, support tickets, and product catalogs. Organizations that extract training signal from this data and feed it into their retrieval models build a compounding advantage.

Consider the difference in concrete terms. A general-purpose embedding model encodes "running shoes" and "jogging sneakers" as similar because it learned from broad web text that these phrases co-occur in similar contexts. That is useful. But a model fine-tuned on a specific retailer's data also learns that customers who search "trail runners" and "trail running shoes" have different intent: the first group tends to browse high-end brands while the second group is more price-sensitive. This kind of domain-specific query understanding is invisible to general models and impossible to encode in manual synonym lists. It can only be learned from data that the organization already generates through normal operations.

Search Relevance Is the Product

When engineering teams treat search as a component ("search is on the roadmap for Q3"), they frame it as one feature among many. This framing misses the point: for any product where users look for things, search quality IS the user experience.

The data is unambiguous. Amazon's internal search converts at 12% compared to 2% for browsing (6x). Walmart reports 2.4x higher conversion through search. Etsy reports 3x (via Opensend; via Nosto). Most users only view the top 2 to 3 results, and attention drops sharply after the first few positions (Granka, Joachims & Gay, 2004). If those top results are wrong, nothing else on the page matters.

Yet Algolia reports only 15% of companies dedicate resources to search optimization. The typical setup: one or two engineers maintaining an Elasticsearch cluster. No relevance metrics. No evaluation pipeline. No feedback loop.

Nosto's research reinforces the stakes: 80% of shoppers exit a site because of poor search, and retailers attribute 39% of all bounced traffic to poorly performing search (Nosto, 2023).

Teams that treat search as a product own metrics (NDCG, MRR, zero-result rate), own an evaluation framework, own a data pipeline from query logs to training data, and own a feedback loop that measures, improves, and measures again. They consistently outperform teams that treat search as a component.

A quick diagnostic: if users search in your product and you cannot answer what your NDCG@10 baseline is, what your zero-result rate is, which query types underperform, and when relevance last improved (and by how much), there is a gap.

The gap is not about resources. It is about framing. Teams that frame search as "a feature to maintain" allocate a fraction of engineering time. Teams that frame search as "the product experience" allocate dedicated engineers, evaluation infrastructure, and data pipelines. The second framing leads to compounding improvement. The first leads to stagnation punctuated by periodic firefighting when a major query class breaks and customer complaints spike.

Zero-Result Queries as Free Training Data

Zero-result queries deserve special attention because they are simultaneously the most expensive failures and the most valuable signals. Every zero-result query tells you exactly where the system fails. Failed search interactions trigger reformulations that reveal system weaknesses and guide improvement (Huang & Efthimiadis, 2009).

Logging these queries, categorizing them, and analyzing patterns reveals vocabulary gaps (terms the system doesn't understand), missing categories (product types that exist in demand but not in the index), and broken entity recognition (queries the system can't parse). Wizzy.ai documented an 80% reduction in zero-result queries within 30 days by systematically analyzing failure patterns and feeding them back into the system (Wizzy.ai case study).

The upside of fixing zero-result queries is immediate and measurable. The data to fix them is already being generated by users. The only requirement is capturing it.

Conclusion

The biggest search improvements don't come from better algorithms. They come from teams that treat their own query data as a first-class product input. Fast response times are table stakes. Retrieval quality is the actual game.

Baymard's 2024 benchmark (5,000+ ratings) shows 41% of e-commerce sites fail to support 8 key search query types (Baymard Institute, 2024). The organizations that improve fastest stop patching with rules and start learning from their data. The path forward is domain adaptation, evaluation infrastructure, and treating search as the product it is, not the feature teams pretend it to be.

References

• Agichtein, E., Brill, E., & Dumais, S. (2006). "Improving Web Search Ranking by Incorporating User Behavior Information." SIGIR 2006, pp. 19-26. https://dl.acm.org/doi/10.1145/1148170.1148177
• Algolia (2024). "40+ Stats on E-Commerce Search and KPIs." https://algolia.com/blog/ecommerce/e-commerce-search-and-kpis-statistics
• Azad, H. K., & Deepak, A. (2019). "Query Expansion Techniques for Information Retrieval: A Survey." Information Processing & Management, 56(5). https://arxiv.org/abs/1708.00247
• Baymard Institute (2024). "Deconstructing E-Commerce Search UX: 8 Most Common Search Query Types." https://baymard.com/blog/ecommerce-search-query-types
• Baymard Institute. "E-Commerce Search Usability Report." https://baymard.com/research/ecommerce-search
• Chen, X., Zhang, N., Lu, K., Bendersky, M., & Najork, M. (2022). "Out-of-Domain Semantics to the Rescue! Zero-Shot Hybrid Retrieval Models." ECIR 2022, pp. 95-110. https://arxiv.org/abs/2201.10582
• Forrester Research. Site search conversion data, via Nosto. https://nosto.com/blog/new-search-research
• Garg, S., Vu, T., & Moschitti, A. (2020). "TANDA: Transfer and Adapt Pre-Trained Transformer Models for Answer Sentence Selection." AAAI 2020. https://arxiv.org/abs/1911.04118
• Granka, L. A., Joachims, T., & Gay, G. (2004). "Eye-Tracking Analysis of User Behavior in WWW Search." SIGIR 2004, pp. 478-479. https://dl.acm.org/doi/10.1145/1008992.1009079
• Huang, J., & Efthimiadis, E. N. (2009). "Analyzing and Evaluating Query Reformulation Strategies in Web Search Logs." CIKM 2009, pp. 77-86. https://dl.acm.org/doi/10.1145/1645953.1645966
• Nosto (2023). "The Future of Ecommerce Search." https://nosto.com/blog/new-search-research
• Opensend. "On-Site Search Conversion Rate Statistics." https://opensend.com/post/on-site-search-conversion-rate-statistics-ecommerce
• Sanderson, M. (1994). "Word Sense Disambiguation and Information Retrieval." SIGIR 1994, pp. 142-151.
• Thakur, N., Reimers, N., Rücklé, A., Srivastava, A., & Gurevych, I. (2021). "BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models." NeurIPS 2021. https://arxiv.org/abs/2104.08663
• Wang, K., Thakur, N., Reimers, N., & Gurevych, I. (2022). "GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrieval." NAACL 2022. https://aclanthology.org/2022.naacl-main.168/
• Wizzy.ai. Zero-result reduction case study. https://wizzy.ai