We are used to evaluating AI systems by asking how well they score. Accuracy, AUROC, calibration, ranking quality, fairness metrics and error rates all matter. They tell us something important about the behavior of a model.
But many real systems do not stop at a score. A score becomes a queue. A rank becomes an interview list. A risk estimate becomes an inspection priority. A probability becomes a treatment decision. A recommendation becomes access to a scarce opportunity.
At that point, the evaluation problem changes.
When a model participates in allocation, the question is no longer only “How good are the scores?” It is also “Who receives what because of those scores?”
The missing layer between prediction and consequence
Imagine two systems evaluated on the same population. Their aggregate performance is nearly identical. Their calibration looks comparable. Their top-line fairness metrics are within the same range.
Yet when each system is used to fill a fixed number of places, select a limited set of cases, or prioritize scarce resources, the identities of the people selected can differ substantially.
Nothing in the headline metric necessarily tells you that.
This is the distinction that motivates my work on allocation identity. In a capacity-constrained setting, system behavior has at least two layers: the quality of the scoring function, and the identity of the downstream allocation that follows from it.
Why capacity changes the evaluation problem
Capacity forces a boundary. If only 100 people can be selected, then a model is not merely describing risk or merit. It is participating in a decision over the boundary between selected and not selected.
Small changes near that boundary can therefore matter much more than small changes elsewhere in the ranking. Two models can preserve almost the same overall ordering and still swap a meaningful number of individuals around the cutoff.
That is not necessarily evidence that either model is wrong. It is evidence that aggregate evaluation and allocation evaluation answer different questions.
Fairness can also look different after the cutoff
Group-level fairness measures remain essential, but they do not always reveal allocation identity either. Two systems may produce similar group proportions while selecting different individuals within those groups.
For some applications, that distinction may not matter. For others, it can be central. If the system is used repeatedly, if individuals have limited opportunities to re-enter, or if allocations trigger downstream interventions, identity can become part of the audit trail.
This suggests a useful principle: evaluate the model at the level at which the institution actually acts.
What should an audit include?
For consequential systems that convert scores into limited allocations, I think evaluation should increasingly report four things together:
- Predictive or ranking performance.
- Group-level fairness and calibration where appropriate.
- The allocation rule and capacity constraint.
- Allocation identity: who changes when the model, threshold, ranking rule or policy changes.
The point is not to replace existing metrics. It is to make the decision layer explicit.
Why this matters beyond AI research
Institutions often treat a model as if it ends where the score ends. Operationally, that is rarely true. Scores enter workflows, policies, budgets, queues and human judgment. The model becomes one component in a larger decision system.
That is why I increasingly think of AI evaluation as a systems problem rather than only a modeling problem.
The score is one representation. The allocation is an action. Good governance needs to understand both.
The question I want people to ask
When two systems receive similar evaluation scores, I do not want the discussion to end with “they perform the same.” I want one more question asked:
Do they make the same decisions about the same people?
Sometimes the answer will be yes. Sometimes the answer will be no. Either way, it is information worth having before a model enters a consequential workflow.