How TensorOpt evaluates retrieval systems

This page summarizes the public-facing version of the TensorOpt methodology. The full runbook stays internal; it is the engagement IP, not a free download. What I share below is enough for a buyer to evaluate whether the approach is rigorous and whether it would surface anything their team would not.

The 4-step evaluation process

Every Evaluation runs the same four steps. A Readout works from your log or your experts' questions and plans the labelling for an Evaluation.

1. Engagement. Scoping call and SOW. I get read-only access to the client retrieval pipeline (logs, vector store, search vendor, RAG service) plus a representative sample of query traffic. No production write access is requested.
2. Judgment-set construction. I build a stratified evaluation dataset sized for the pipeline under evaluation. Where an internal judgment set already exists, I use it; where it doesn't, I design one and you keep the artifact for future vendor evaluation cycles.
3. Measurement. I run the metrics that match the pipeline (NDCG@k, MRR, hit-rate@k, and citation accuracy for RAG) and wrap each one in a bootstrap 95% confidence interval. Where two pipelines or two configurations are being compared, I run a paired significance test.
4. Report. A report, the judgment set (you keep it), reproducible harness configuration, and a walkthrough.

Statistical rigor layer

This is the differentiation that matters for procurement buyers.

Evaluation-tooling vendors (Galileo, Patronus, Arize Phoenix, FutureAGI, Confident AI, Maxim AI) publish NDCG@k, MRR, Precision@k, and similar metrics as platform features. What they typically do not publish, and what I apply on top, is the classical IR statistical rigor layer:

• Bootstrap confidence intervals. Every reported metric ships with a 95% CI estimated by resampling the judgment set. An NDCG@10 of 0.62 with a CI of [0.58, 0.66] tells a different story from 0.62 with [0.50, 0.74]. Without the CI, a 2-point lift in a vendor comparison cannot be distinguished from a sampling artifact.
• Paired significance tests. When the engagement compares two pipelines (e.g., current vendor vs. challenger, or current chunking strategy vs. alternative), I run paired tests (the same query against both pipelines) rather than independent samples. This is dramatically more sensitive and reflects how A/B test reads are actually structured.

TensorOpt is also independent third-party rather than a SaaS vendor: I am not grading my own platform, and I do not own the eval surface. Many of the metrics buyers most want to see (vendor-vs-vendor, vendor-vs-replacement) are by definition not produced by the incumbent vendor's own dashboard.

What gets measured

The metric mix depends on the engagement and the pipeline. Short notes on each:

• NDCG@k. Graded relevance, position-weighted. The standard IR metric for ranked retrieval. The Evaluation reports NDCG@10.
• MRR. Reciprocal rank of the first relevant result. Useful when a question has one canonical answer and the user stops scrolling once they find it.
• Hit-rate@k. Binary "is at least one relevant item in the top k". The cleanest metric to communicate to non-IR stakeholders.
• Faithfulness and citation accuracy. For RAG generation, whether the generated answer is supported by the retrieved context. I score this on a sample of generated responses; methodology details (rubric, scoring model, sample size, human spot-check rate) live in the report.

What stays internal

The internal methodology runbook is engagement IP. It covers the operational detail (judgment-set sampling strategy, prompt templates for LLM-as-judge scoring, edge cases on chunking review, the specific bootstrap implementation, how I triage remediations into the report's recommendations). I share it inside an engagement, not on this page.

I mention this explicitly because I'd rather a team read this page and decide to book a call than read this page and decide they can replicate an Evaluation over a weekend. The interesting parts of the methodology are the parts I don't publish.

Third-party LLM API consent

Some scoring tasks (faithfulness in particular, and some LLM-as-judge configurations) require sending client data to a third-party LLM API (OpenAI, Anthropic). Where the engagement uses these APIs, explicit pre-engagement consent is recorded in the SOW and the relevant data flows are listed by name. No third-party LLM API is used without written consent. A self-hosted model option is planned but not available yet.

I publish this policy here so data requirements are transparent before an engagement begins.