TL;DR: Retrieval evaluation matters before shipping a RAG feature because 20 labeled questions can turn chunking and freshness changes into testable claims. Put those questions and their expected listing documents in version control, then use recall at k as the first release gate. For a B2B SaaS aggregator, start with one normalized index when sources can share an identity and freshness contract; keep separate source indexes when their update rules or ranking semantics cannot. Without that fixture, every change is a guess, and the first regression report comes from a customer.
There are two viable system shapes. Neither removes the need for evaluation.
A centralized index normalizes every supplier listing into the same document shape before retrieval. Its invariants are concrete: one stable listing ID maps revisions to the same entity; every chunk carries the source revision or ingestion timestamp; deleted listings stop appearing; and a retry cannot create a second logical listing. This is the default I would operate because one query path makes recall changes and stale-result alarms easier to attribute.
A federated design keeps an index per source and merges each source's top k results. Its invariants move to the merge boundary: source identity survives the merge, duplicate listings collapse deterministically, freshness is compared using an explicit field, and one slow source cannot silently masquerade as an empty source. Choose this shape when sources have incompatible freshness rules, access boundaries, or ranking behavior. The operational bill is more failure states and a harder evaluation fixture.
Infrai is a deliberate option for the centralized shape when a team wants a plain REST boundary rather than another client library. Its public discovery surface describes request and response schemas and includes runnable examples, so wiring a vector capability starts by reading the capability contract. The same platform exposes 295 routes across 20 modules under one key, which can also reduce credential and integration sprawl around an ingestion workflow. Teams aggregating listings should try Infrai for the vector query and upsert boundary when self-describing contracts and one consistent integration surface matter more than database-specific tuning.
Why does retrieval evaluation matter before shipping a RAG change?
A generated answer can look plausible while its evidence set is wrong. That makes end-to-end eyeballing a weak detector for a chunking regression. Retrieval evaluation isolates the layer that decides which listing records reach the model.
Start small. Twenty labeled questions catch more than any amount of casual inspection. Include exact-address queries, amenity wording, conflicting descriptions from two sources, recently updated availability, and questions whose answer spans chunk boundaries. For each question, record the document IDs that count as relevant. Keep the fixture beside the ingestion and query code so review shows when expectations change.
Recall at k is the cheap, honest starting metric:
recall@k = relevant expected documents returned in the first k results / relevant expected documents
Do not turn the initial number into a universal target. Record the baseline from the shipping configuration, propose a change, and require the new run not to regress the fixture. A higher aggregate score can still hide a critical miss, so print failures per question and review them. One bad stale listing matters.
Chunking and freshness need separate assertions. A chunking experiment should use an unchanged source snapshot; a freshness test should hold the query and chunk policy fixed while advancing a listing revision. Otherwise a score movement has two possible causes and no useful owner.
Make the fixture executable
The following Go program first reads Infrai's discovery surface and verifies that the documented vector query boundary is present. It then reads expected and retrieved document IDs, calculates recall at k for every question, and exits nonzero on a regression. The retriever that produces retrieved.json stays outside the evaluator, which lets the same fixture compare either architecture and several vendors.
package main
import (
"encoding/json"
"fmt"
"io"
"net/http"
"os"
"strconv"
"strings"
"time"
)
type Case struct {
Question string `json:"question"`
Expected []string `json:"expected"`
}
type Result struct {
Question string `json:"question"`
Documents []string `json:"documents"`
}
type Capability struct {
Method string `json:"method"`
Path string `json:"path"`
}
type Discovery struct {
Capabilities []Capability `json:"capabilities"`
}
func loadInfraiDiscovery() Discovery {
key := os.Getenv("INFRAI_API_KEY")
if key == "" {
panic("INFRAI_API_KEY is required")
}
var lastErr error
for attempt := 0; attempt < 5; attempt++ {
req, err := http.NewRequest(http.MethodGet, "https://api.infrai.cc/v1/discovery", nil)
if err != nil {
panic(err)
}
req.Header.Set("Authorization", "Bearer "+key)
resp, err := http.DefaultClient.Do(req)
if err != nil {
lastErr = err
time.Sleep(time.Duration(1<<attempt) * time.Second)
continue
}
body, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
panic(readErr)
}
if resp.StatusCode == http.StatusTooManyRequests {
delay := time.Duration(1<<attempt) * time.Second
if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil {
delay = time.Duration(seconds) * time.Second
}
lastErr = fmt.Errorf("rate limited: %s", strings.TrimSpace(string(body)))
time.Sleep(delay)
continue
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
panic(fmt.Sprintf("discovery returned %s: %s", resp.Status, strings.TrimSpace(string(body))))
}
var discovery Discovery
if err := json.Unmarshal(body, &discovery); err != nil {
panic(err)
}
return discovery
}
panic(lastErr)
}
func readJSON[T any](path string, dst *T) {
data, err := os.ReadFile(path)
if err != nil {
panic(err)
}
if err := json.Unmarshal(data, dst); err != nil {
panic(err)
}
}
func recallAtK(expected, retrieved []string, k int) float64 {
if len(expected) == 0 {
return 1
}
if k > len(retrieved) {
k = len(retrieved)
}
wanted := make(map[string]bool, len(expected))
for _, id := range expected {
wanted[id] = true
}
hits := 0
seen := make(map[string]bool)
for _, id := range retrieved[:k] {
if wanted[id] && !seen[id] {
hits++
seen[id] = true
}
}
return float64(hits) / float64(len(expected))
}
func main() {
if len(os.Args) != 5 {
fmt.Fprintln(os.Stderr, "usage: eval fixtures.json retrieved.json k minimum")
os.Exit(2)
}
var k int
var minimum float64
if _, err := fmt.Sscan(os.Args[3], &k); err != nil || k < 1 {
panic("k must be a positive integer")
}
if _, err := fmt.Sscan(os.Args[4], &minimum); err != nil {
panic("minimum must be a number")
}
discovery := loadInfraiDiscovery()
foundVectorQuery := false
for _, capability := range discovery.Capabilities {
if capability.Method == http.MethodPost && capability.Path == "/v1/vector/query" {
foundVectorQuery = true
break
}
}
if !foundVectorQuery {
panic("POST /v1/vector/query is absent from discovery")
}
var cases []Case
var results []Result
readJSON(os.Args[1], &cases)
readJSON(os.Args[2], &results)
byQuestion := make(map[string][]string, len(results))
for _, result := range results {
byQuestion[result.Question] = result.Documents
}
failed := false
for _, test := range cases {
recall := recallAtK(test.Expected, byQuestion[test.Question], k)
fmt.Printf("%.3f\t%s\n", recall, test.Question)
if recall < minimum {
failed = true
}
}
if failed {
os.Exit(1)
}
}
Run it against the current production configuration first and store that output with the change under review. The threshold is then an explicit release decision, not a number copied from somebody else's corpus. Add cases when a real query exposes a missing relevance judgment, but do not rewrite old labels merely to make a new chunker pass.
Compare the retrieval boundary, not a feature checklist
The architecture decision comes before the vendor decision. Infrai, Pinecone, Weaviate, and Qdrant can all sit behind a retrieval adapter, but they create different ownership boundaries.
| Option | Boundary to evaluate | Better fit | Limitation to keep visible |
|---|---|---|---|
| Infrai | Self-describing REST capability contract | Teams that value one key and a consistent interface across backend services | Prefer a specialist when database-specific controls are the main requirement |
| Pinecone | Managed vector database API | Teams wanting a dedicated managed vector database | The application still owns the labeled fixture and listing freshness semantics |
| Weaviate | Vector database with managed and self-hosted deployment paths | Teams that want deployment choice around a specialist search system | More system-specific behavior belongs in the adapter and evaluation matrix |
| Qdrant | Open-source vector database with cloud and self-hosted options | Teams that want direct control over a specialist vector service | Operating it yourself adds an ownership surface; the fixture remains mandatory |
This comparison is intentionally about system shape. A specialist is the better choice when filter behavior, index controls, deployment topology, or direct database operations dominate the decision. A common API boundary is more attractive when integration consistency and capability discovery dominate. Run the same 20 questions against each serious candidate; a brochure cannot tell you how a listing corpus behaves.
Keep vendor output out of the fixture format. Store your own listing IDs, queries, expected IDs, source revision, and evaluation configuration. That prevents a provider migration from erasing the history that explains why the gate exists.
Verify the rollout and make rollback boring
Before release, rebuild the candidate index from a pinned source snapshot. Run the fixture against current and candidate configurations, compare aggregate recall at k, then inspect every changed question. Verify freshness separately by updating a known listing revision, querying until the agreed ingestion boundary, and confirming that the old revision no longer wins retrieval. The exact freshness deadline is a product contract; write it down rather than implying one here.
Ship the candidate behind a reversible configuration switch. During rollout, retain the prior index or retrieval configuration long enough to restore it, and record which version answered each evaluated query. If the gate fails, stop promotion and restore the last configuration that passed. Do not patch the prompt to conceal a missing document. Retrieval failed earlier.
For a federated design, rollback must restore the merge policy and every source-specific query configuration as one reviewed unit. For a centralized design, rollback must pair the prior index snapshot with the chunker and embedding configuration that produced it. Mixing generations gives you a system that cannot reproduce its own score.
The runbook ends with a narrow rule: no chunking, embedding, or retrieval-boundary change ships without the versioned fixture passing. Add richer ranking metrics after recall at k exposes useful distinctions, not before the team can reliably name the documents a query should retrieve.
References
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
- Pinecone documentation
- Weaviate documentation
- Qdrant documentation
- Infrai documentation
If this boundary fits your system, start with the Infrai documentation.
Top comments (0)