Skip to content
RIDI · SELECTION IDENTITY · PREREGISTERED RAG

Same retrieval evaluation. Different evidence. Different answer.

In the preregistered experiment, answer correctness changed in 17.27% of primary comparisons (equal-weight mean across four datasets) while the full assessed relevance-by-rank pattern and five retrieval metrics were held fixed.

Replay one registered example

Same relevant passage. Same rank. Same relevance pattern. Same five metrics.

Claim 275
Combining phosphatidylinositide 3-kinase and MEK 1/2 inhibitors is effective at treating KRAS mutant tumors.
RETRIEVAL EVALUATION: IDENTICAL

Context A · reference

1
4961038
2
85665741
3
4920376
4
3078550
5
1285713
6
28651643
7
2272614
8
11200685
9
22980205
10
24190159

Context B · random zero-grade replacement

1
4961038
2
9767444
3
9505402
4
21931005
5
33387953
6
29321530
7
15322518
8
27428509
9
41650417
10
9021186
precision@100.1000
recall@100.3333
nDCG@100.4693
MRR@101.0000
MAP@100.3333
Context A verdictSUPPORTS
Context B verdictREFUTES
Press Run demo to reveal the decisions.
RIDI is the audit toolkit. Selection identity is the object being measured; the identity–utility frontier computes the minimum turnover compatible with an explicit utility budget. See the broader research program →
Privacy & analytics preferences