RAG Systems:
Build the Retrieval Evaluation Set
Before Choosing a Model

Versioned sources and unresolved cases

Dmytro Nasyrov PhD

The retrieval evaluation coverage sheet

Field

Review job

Question class

Name the information need and failure case.

Expected supporting source

Identify the passage that would support the answer.

Source version

Bind the judgment to a declared corpus snapshot.

Answerable or absent

Record whether that snapshot supports an answer.

Judgment rule

State what counts as adequate retrieval evidence.

Review owner

Name the role responsible for resolving the label.

Synthetic corpus snapshot C1

Fictional policies and questions. No retriever or model has been run.

Source / version

State

Passage

D1 / v1

Retired

Cancel by email.

D1 / v2

Current

Cancel only in Settings.

D2 / v1

Current

Annual plans include exports.

D3 / v1

Current

Annual plans exclude exports.

C1 contains only these four passages. No precedence rule resolves D2 against D3.

Four question classes

Case

Question

Expected evidence

Q1 / lookup

How can I cancel now?

D1 / v2

Q2 / freshness

Can I cancel by email?

D1 / v2 (exclude v1)

Q3 / absent

What refund applies after 45 days?

None in C1

Q4 / conflict

Does an annual plan include exports?

D2 / v1 and D3 / v1

A short seed set exposes different review jobs. It does not establish production coverage.

A complete freshness judgment

Q2: Can I cancel by email?

Field

Coverage row

Question class

Q2 / freshness

Expected supporting source

D1

Source version

v2 in C1

Answerable or absent

Answerable from the current passage

Judgment rule

Require D1 / v2. Reject v1 as current evidence.

Review owner

Support content owner

A question with no supporting passage

Q3 asks what refund applies after 45 days. C1 contains no refund policy, so the coverage row is labeled absent for C1.

An empty retrieval result alone does not establish absence from the corpus. The label comes from reviewing the declared snapshot.

Judgment rule: report the evidence gap. Assign the refund policy owner to review it before treating the question as answerable.

A question with conflicting passages

Q4 source

Supporting text

D2 / v1

Annual plans include exports.

D3 / v1

Annual plans exclude exports.

Record both references and mark the answer unresolved. Retrieving only one passage cannot establish which policy governs.

Review owner: product policy owner. Resolve authority or revise the source before accepting a single factual answer.

The basis for retrieval judgments

Elastic ranking evaluation uses typical queries and document relevance ratings for each query. The ratings reflect the information need behind that query.

Unrated results need an explicit scoring policy. Elastic precision can ignore unlabeled documents or treat them as irrelevant, depending on its configuration.

Source: Elastic ranking evaluation documentation

The comparison handoff

Record

Evidence to preserve

Inputs

Freeze corpus C1 and the question-set revision.

Labels

Preserve source versions and reviewer decisions.

Outputs

Log returned passage IDs and ranks per question.

Assessment

Score retrieval separately from generated-answer support.

Add real workflow questions and access-control cases. Keep held-out checks separate from tuning. Record each retriever or model revision.

RAG evaluation in the delivery scope

When a RAG pilot returns plausible answers without traceable evidence, Pharos Production offers RAG development that includes retrieval evaluation and citation checks. Define reviewed question sets and separate retrieval judgments from answer-quality checks in the agreed scope.

Bring a versioned corpus sample and representative questions to scope the evaluation.

About the author

Written by Dmytro Nasyrov PhD, software architect with 24 years of production experience. Dmytro is the founder and CTO of Pharos Production. He works on production software architecture for FinTech, AI, Web3 and blockchain systems.

Dmytro Nasyrov. Photo supplied by the author.