Why good models fail validation (and what to do about it)
Most models that stall in validation are not badly built. They are badly evidenced. Here is where the gaps actually come from.
Here is a pattern anyone who has worked in model risk will recognize. A data science team spends four months building a model. It performs well. The business is excited. Then it goes to validation — and sits there for six months, sometimes longer, while validators and developers trade documents that neither side finds satisfying.
The instinct is to blame the validators for being slow or the developers for being sloppy. Usually neither is true. The model is fine. The problem is that the evidence a validator needs was never produced, because nobody treated evidence as a deliverable.
The three gaps that stall reviews
First: data lineage. A validator asks a simple question — where did this training data come from, and what happened to it along the way? If the answer involves someone re-running notebooks to remember, the review has already stalled. Lineage has to be captured while the work happens. Reconstructing it later is expensive, and reconstructed lineage is inherently less credible than recorded lineage.
Second: decision history. Validators are not only assessing the model; they are assessing the judgment behind it. Why this feature set? Why was that segment excluded? Why this threshold? When these decisions are documented at the moment they are made — even in two sentences — the review moves. When the answers are "we tried a lot of things and this worked," it does not.
Third: fairness methodology. Teams often run disparate impact analysis after the model is built, find results they can live with, and write the methodology to match. Validators can tell. A fairness testing plan written before results exist, with pass criteria committed in advance, is worth more than any number of favorable post-hoc analyses.
Evidence as a by-product, not a project
The fix is not more documentation effort at the end — it is restructuring development so evidence accumulates automatically. Versioned data and seeded training runs make reproducibility a property of the pipeline rather than a promise. Lightweight decision logs, maintained as the work happens, cost minutes per week. Pre-registered validation plans remove entire categories of dispute before they start.
Teams sometimes object that this slows development. In our experience it is the opposite, in aggregate: a few percent slower to first candidate model, dramatically faster to production. Because the slowest part of shipping a model in a regulated environment was never the training. It was always the proving.