Exact retrieval-audit equality
Across 800 preregistered queries, the complete assessed relevance-grade sequence and five standard retrieval metrics were identical, yet benchmark-defined correctness changed in 17.27% of queries.
AI evaluation · decision identity · ranking updates
Equal evaluation scores do not certify equivalent AI behaviour.
RIDI asks what aggregate metrics leave unresolved: whether two system states act on the same evidence and identities, produce the same downstream decisions, and how much observed selection change was actually required.
ManuscriptSubmission-ready · not yet peer reviewed
Softwareridi-audit 1.1.1 · MIT License
Updated26 September 2026
17.27%benchmark-defined correctness changes under exact registered retrieval-audit equality
9.33 ppGPT-5.6 Sol repeat-adjusted excess in the registered frontier-model transfer
4.36 ppP5b semantic-disagreement excess above matched repeat variability
565 / 1,000priorities replaced in the EPSS v2→v3 production update
Research question
What does an evaluation score fail to identify?
Two systems can receive the same recorded evaluation while acting on different evidence, selecting different identities, or producing different downstream answers.
RIDI separates evaluation equivalence from action equivalence. It measures selected-set identity directly and, for ranking updates, computes the exact identity–utility frontier: the smallest replacement count compatible with a declared updated-score utility budget.
This is not a claim that change is harmful. It is a claim about what aggregate evaluation does—and does not—certify.
Findings
A layered evidence chain.
Positive, negative and boundary results are reported separately rather than collapsed into one headline.
Frontier-model transfer
GPT-5.6 Sol produced 10.70% equal-dataset correctness change versus a 1.37% repeat floor: 9.33 percentage points excess (95% CI 7.13–11.57).
P5 remained negative
The registered natural-prevalence inferential gate qualified 0/760 cells. No P5 answer-generation stage was entered and no threshold was relaxed.
P5b practitioner-equivalence extension
A separately locked rule qualified 103/760 cells. Held-out semantic disagreement was 8.12% versus a 3.76% repeat floor: 4.36 points excess (95% CI 3.27–5.58).
Production update
EPSS v2→v3 replaced 565/1,000 selected priorities. Adjacent same-version controls changed 0 and 7, showing that production turnover exceeded ordinary daily drift.
Exact control frontier
RIDI computes the minimum replacements compatible with a declared utility loss. Avoidable turnover differs materially across domains rather than following a universal constant.
Method
Measure, separate, control.
1. Measure identity
Record overlap, changed slots and Jaccard distance between the finite selections a system actually acts on.
2. Separate from performance
Report conventional accuracy or retrieval measures alongside selected identities rather than assuming one determines the other.
3. Control required change
Use the exact identity–utility frontier to determine the minimum turnover compatible with an explicit updated-score utility tolerance.
Open-source implementation
Use RIDI on your own selected lists or score tables.
The public package is designed to make the core audit inspectable without reproducing the manuscript experiments.
pip install ridi-audit
from ridi_audit import compare_allocations
before = ["doc-1", "doc-2", "doc-3", "doc-4"]
after = ["doc-1", "doc-2", "doc-9", "doc-4"]
print(compare_allocations(before, after))
Clean-environment verification on 26 September 2026: 21/21 tests passed; Quick Start, CLI compare/control, package build, Twine checks and a fresh PyPI install were smoke-tested successfully.
Scientific boundaries
What the project does not claim.
- Not a harm metricSelection or answer change is not automatically harm, unfairness or error.
- P5 remains negativeP5b is a distinct post-P5, pre-generation extension and does not relabel the registered P5 result.
- Finite-roster inferenceP5b is conditional on the frozen 20-configuration roster and common BM25 candidate pool; it is not a prevalence estimate over all retrievers.
- Sparse outcomesEPSS delayed outcomes are retrospective and sparse; no causal benefit is claimed.
- No CODECHECK certificateExternal execution evidence is documented, but no formal CODECHECK certificate has been issued.
Resources
Evidence, code and reproducibility.
Paper & evidence synopsisCurrent manuscript framing, registered extensions, negative results and scientific boundaries.
60-second Quick StartInstall the public package and reproduce the basic allocation-identity audit.
RAG preregistrationProspective registered design for the retrieval experiment and extension programme.
PyPI: ridi-auditPublic Python package for selected-set comparison and the identity–utility frontier.