ROC AUC, worst-fold performance, and reporting honestly

A classifier trained to predict whether tomorrow’s return will be positive reports 0.58 ROC AUCROC AUC The area under the receiver operating characteristic curve; the probability a classifier ranks a random positive case above a random negative one.. Is that good? The honest answer depends on three things that the single number does not itself contain: what the class balance was, how that 0.58 compares to a model that guesses nothing at all, and whether it held up across every out-of-sample fold or was carried by one favourable stretch. This article works through all three.

What ROC AUC actually measures

ROC AUC is the probability that a classifier ranks a randomly chosen positive example above a randomly chosen negative one. Formally, for a classifier producing a score $s(x)$:

$$\text{AUC} = P\big(s(x^+) > s(x^-)\big)$$

where $x^+$ is drawn from the positive class and $x^-$ from the negative class. A value of 0.5 means the classifier’s ranking is no better than a coin flip; 1.0 means perfect separation. Critically, AUC is a ranking metric — it says nothing about calibration (whether a predicted probability of 0.7 actually corresponds to a 70% chance of the positive outcome) and nothing about the actual class balance in the data. Two classifiers can report identical AUC on a 50/50-balanced target and on a 95/5-imbalanced target, even though the practical difficulty of the second problem is far greater.

Why financial classification targets are usually imbalanced, and what that does to AUC

A target like “will this triple-barrier exit hit profit-take before stop-loss” is rarely close to 50/50 once realistic barrier widths are applied — it might be closer to 30/70. AUC remains a valid ranking measure under imbalance, which is part of why it is preferred over accuracy for these targets: an accuracy score of 70% on a 70/30-imbalanced target is achieved by a classifier that always predicts the majority class and does nothing else. AUC does not reward that degenerate strategy, because a classifier that outputs the same score for every row cannot rank a positive above a negative any better than chance, and will report close to 0.5.

This is precisely why the workspace’s walk-forward reporting includes both accuracy and ROC AUC per fold rather than accuracy alone — a model that looks strong on accuracy under imbalance can be entirely uninformative on AUC, and the discrepancy between the two is itself diagnostic.

The number ROC AUC still cannot answer alone

An AUC of 0.58 sounds close to the useless value of 0.5. Whether it represents a real, exploitable edge depends on what a reasonable non-model baseline achieves on the same data, and on how tightly the same number holds up across independent out-of-sample periods. Both questions are answered elsewhere: the comparison against a baseline that predicts nothing is the subject of beating the naive baseline, and the question of consistency across folds is the subject of this section.

Worst fold, worked

Take the same five-fold ROC AUC sequence used elsewhere in this validation series: 0.61, 0.58, 0.72, 0.55, 0.64. The mean is 0.62. The worst foldWorst fold The single lowest-performing fold in a walk-forward or cross-validated evaluation, the number that should headline a validated result, not the average. — the minimum across the sequence — is 0.55.

Statistic	Value
Mean ROC AUC	0.62
Std dev	0.066
Worst fold (min)	0.55
Best fold (max)	0.72

A report built around the mean alone reads as “this model achieves 0.62 ROC AUC out of sample” — an unambiguously positive framing. A report that includes the worst fold reads as “this model achieves 0.62 ROC AUC on average, and as low as 0.55 in its weakest out-of-sample period” — a framing that gives a reader the information needed to judge whether the strategy’s worst-case behaviour is tolerable. Both sentences describe the same experiment. Only the second one lets a reader make an informed decision about risk.

This is why the workspace’s out-of-sample validation section, one of the eight fixed sections in every generated evidence report, names fold count, mean metric, standard deviation and worst fold explicitly, rather than reporting a single blended score. It is also why experiment comparison tables in the workspace are sorted by deflated SharpeDeflated Sharpe ratio The Sharpe ratio adjusted for the number of configurations tried and for non-normal returns, so a result picked from many looks less impressive. rather than raw performance — a related discipline of refusing to let the most flattering number stand in for the full picture, covered in overfitting: detection, not prevention.

The class-balance question, made concrete

Return to the accuracy-versus-AUC discrepancy. On a target with a 70/30 class split, a classifier that always predicts the majority class scores 70% accuracy and approximately 0.50 ROC AUC — the accuracy number looks respectable, the AUC number correctly identifies that nothing informative is happening. A second classifier that predicts correctly 65% of the time overall but ranks positives above negatives with real skill might score a lower 65% accuracy and a genuinely useful 0.63 ROC AUC. Judged by accuracy alone, the second classifier looks worse. Judged by AUC, the ordering reverses, and correctly so — the second classifier is the one actually extracting signal from an imbalanced target, while the first is exploiting the imbalance itself.

When the worst fold is the real story

Consider two models with identical mean ROC AUC of 0.63 across five folds:

• Model A: 0.60, 0.62, 0.65, 0.64, 0.64 — worst fold 0.60, standard deviation roughly 0.019.
• Model B: 0.80, 0.78, 0.55, 0.52, 0.50 — worst fold 0.50, standard deviation roughly 0.135.

Model A is the more deployable candidate, despite having the identical mean to Model B and a lower peak fold score. Its worst fold barely dips below its mean, and its stability statistic — mean divided by standard deviation, as defined in walk-forward validation, fold by fold — is roughly 33, against Model B’s roughly 4.7. Model B’s headline mean is carried entirely by two unusually favourable folds; its remaining three folds are at or below the level of a coin flip. A researcher who reports only the mean would present these as equivalent. They are not.

Balanced accuracy as a second check

The workspace reports balanced accuracy alongside ROC AUC and F1 for every fold, precisely because ROC AUC’s independence from the operating threshold — it evaluates ranking across every possible threshold at once — can obscure how the model behaves at the specific threshold a strategy would actually use. Balanced accuracy is computed as the average of sensitivity (true positive rate) and specificity (true negative rate) at a chosen threshold:

$$\text{Balanced accuracy} = \frac{1}{2}\left(\frac{TP}{TP+FN} + \frac{TN}{TN+FP}\right)$$

A model can report a respectable ROC AUC of 0.62 while its balanced accuracy at the threshold actually used in production is close to 0.50, if the chosen threshold sits in a region of the ROC curve where the model’s ranking advantage happens to be thin. The two metrics answer different questions — ROC AUC asks whether the ranking is informative anywhere along the curve, balanced accuracy asks whether the specific operating point chosen is informative — and a report that includes only one of them can miss a mismatch between where a model is strong and where it is actually being deployed.

Reporting honestly

None of this argues against ROC AUC as a metric — it remains the right ranking metric for an imbalanced classification target, and the primary metric the workspace uses to compute walk-forward stability for classification tasks. The argument is about what accompanies it in a report. A single mean ROC AUC, reported without its fold-level spread, its comparison against a naive baseline, and its worst-case value, is a number that has been groomed rather than measured. All three pieces are needed before a reader — a supervisor, an allocator, a peer reviewer — can tell whether 0.62 means “consistently, modestly informative” or “got lucky twice out of five tries.”