I want evaluation to follow the decision

I value predictive accuracy, calibration and relevant fairness measures. I need them to understand what a model does well and where it fails. But I do not consider them a complete account of a system that allocates interviews, inspections or other limited opportunities. I want to follow the score into the action, because that is where the institution changes something for a person.

I use a hypothetical comparison to make the question concrete. Two models receive similar aggregate evaluations on the same population. Each is then used to select a fixed number of people. I would not assume that the two lists contain the same individuals. Before calling the systems operationally equivalent, I want to examine whether their apparent similarity survives the decision rule that turns a ranking into a selection.

What I mean by allocation identity

I use allocation identity to focus attention on who receives the outcome. A change near a selection boundary may alter a person’s access even when the overall evaluation changes little. I want to know which individuals enter or leave the selected set, what caused that change, and whether it is justified. I do not treat a changed selection as an error by definition; a correction can properly change who is selected.

I also want to keep individual and group questions visible together. Similar proportions for groups need not mean the same people were selected within each group. I would examine that distinction where individual access has meaningful consequences. I would not infer fairness or unfairness from identity overlap alone. The applicable goals, evidence, policy and opportunities for review still have to carry that judgment.

The audit I would ask an institution to defend

I would ask for an account that connects model performance, relevant group measures, the allocation rule and the available capacity. I would then ask what changes in the selected set when the model or rule changes, including how uncertainty and ties are handled. I want the explanation to identify which choices belong to the institution and which behavior comes from the model, so responsibility does not disappear between them.

I would apply that expectation proportionately. I do not need a complex identity audit for every low-consequence recommendation. I would be more demanding where opportunities are scarce, repeated decisions accumulate consequences, or a person has little chance to re-enter. I also want identity analysis performed with appropriate access controls. Making the decision inspectable for authorized reviewers does not require exposing identifiable records to the public.

The question behind my research

I connect this argument to my research on RIDI and a broader question: why does equivalence on declared performance measures leave downstream action unresolved? I want to examine what an audit identifies, what remains free to change and which of those changes matters in context. Selection identity is one part of that inquiry; the evidence a system uses and the action it produces can also require examination. When someone tells me two systems perform similarly, I ask which measures support that statement and whether the systems act equivalently where it matters.