Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Classification Metrics and Decision Thresholds › Implementation in NumPy and scikit-learn

Michael BrenndoerferOctober 24, 20269 min read

Machine Learning from ScratchAll chapters

Part 4 of 15Model Evaluation and Selection

Chapter 32Classification Metrics and Decision Thresholds › Implementation in NumPy and scikit-learn

Reading section 7 of 10 · Implementation in NumPy and scikit-learn · 9 minSection 7 of 10 · 9 min

Section 7 of 10 · 9 min read

Classification Metrics and Decision Thresholds › Implementation in NumPy and scikit-learnSection view · Sections in this chapter7 / 10 ↓

Sections in this chapter10

Reading tools Definitions, links & notes

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Notes & comments

Notes & commentsLeave notes and join the discussionSelection toolbarShow comment actions when selecting text. Turn off for the browser menu only.Comment indicatorsShow small counts beside annotated passagesPassage highlightsGently shade annotated text selections

We now run the full protocol on the simulated alerting data, using the toolkit from the visualization section. The pipeline has three roles and we keep them strictly separate: a training split would fit the scorer, the validation split selects the threshold, and the test split reports. Since the scores are given, only the last two appear in the code.

The comparison we want is between rules, not models. We evaluate three decision rules: the default 0.50 cut, the closed-form threshold that would be correct if the scores were calibrated, and the threshold selected by minimizing empirical cost on validation. Each fixed rule is applied to both splits. Their realized costs can be compared, but one validation/test difference does not identify the expected selection bias.

PythonIn [21]
CollapseExpandrule_rows = []
for rule_name, t in [
    ("default 0.50", 0.50),
    ("closed form", t_analytic),
    ("selected on validation", t_star),
]:
    for split_name, y_split, s_split in [
        ("validation", y_val, score_val),
        ("test", y_test, score_test),
    ]:
        m = evaluate(y_split, s_split, t)
        rule_rows.append(
            {
                "rule": rule_name,
                "split": split_name,
                "threshold": t,
                "TP": int(m["TP"]),
                "FP": int(m["FP"]),
                "FN": int(m["FN"]),
                "precision": m["precision"],
                "recall": m["recall"],
                "F1": m["f1"],
                "cost per account": m["cost"],
            }
        )
rule_table = pd.DataFrame(rule_rows)

ConsoleOut [22]
CollapseExpand                  rule      split  threshold  TP   FP  FN  precision  recall     F1  cost per account
          default 0.50 validation     0.5000  86   38 258     0.6935  0.2500 0.3675            9.7225
          default 0.50       test     0.5000  89   48 263     0.6496  0.2528 0.3640            9.9225
           closed form validation     0.0323 321 1507  23     0.1756  0.9331 0.2956            2.7462
           closed form       test     0.0323 330 1498  22     0.1805  0.9375 0.3028            2.6975
selected on validation validation     0.0470 314 1229  30     0.2035  0.9128 0.3328            2.6612
selected on validation       test     0.0470 319 1212  33     0.2084  0.9062 0.3388            2.7525

The table reports each rule on both splits. The default 0.50 rule has the smallest recall and largest cost per account on both. The validation-selected rule has the smallest validation cost, while the closed-form rule has the smaller test cost of the two alternatives. Those test costs evaluate the three fixed rules on the same held-out observations; the test split is not used to choose among them.

Three things are worth reading off this table.

The default rule is clearly worse than the alternatives. Recall is about a quarter of the positives, and the cost per account is more than three times that of the alternatives.
The closed-form threshold, despite resting on an assumption the empirical rule does not need, does well, because in this simulation the assumption holds; it has slightly higher validation cost than the selected threshold and slightly lower test cost in this realization. Empirical selection minimizes the validation objective; it does not guarantee the ordering on test observations.
The selected threshold performs almost identically on test and validation. The realized validation and test costs differ in this run; the difference can reflect sampling as well as threshold selection and does not by itself estimate the expected selection bias.

A flat observed cost curve shows that nearby thresholds can have similar observed costs. It does not establish the size of selection optimism or sampling stability. Assessing those properties requires the sampling design and an evaluation of the full selection procedure.

The implementation so far is our own, so it needs a check against a reference. scikit-learn provides all the same quantities, and agreement to machine precision confirms that your reading of the definitions matches the community's.

PythonIn [23]
CollapseExpandfrom sklearn.metrics import (
    average_precision_score,
    confusion_matrix,
    precision_recall_curve,
    roc_auc_score,
    roc_curve,
)

sk_auc = roc_auc_score(y_val, score_val)
sk_ap = average_precision_score(y_val, score_val)
sk_cm = confusion_matrix(y_val, score_val >= t_star, labels=[0, 1])
sk_precision, sk_recall, _ = precision_recall_curve(y_val, score_val)
sk_fpr, sk_tpr, _ = roc_curve(y_val, score_val)

agreement = pd.DataFrame(
    {
        "metric": ["ROC AUC", "average precision"],
        "from scratch": [auc_val, ap_val],
        "scikit-learn": [sk_auc, sk_ap],
    }
)
agreement["abs difference"] = (
    agreement["from scratch"] - agreement["scikit-learn"]
).abs()

ConsoleOut [24]
CollapseExpand           metric  from scratch  scikit-learn  abs difference
          ROC AUC      0.889709      0.889709             0.0
average precision      0.518929      0.518929             0.0

The two summary numbers agree to machine precision, which confirms that our from-scratch ROC AUC and average precision match the reference implementations of the same definitions. This agreement checks the implementation of those definitions; it does not bear on whether either metric is appropriate for a given decision.

ConsoleOut [25]
CollapseExpandconfusion matrices identical: True
largest recall discrepancy   : 0.00e+00
largest precision discrepancy: 0.00e+00
largest TPR discrepancy      : 0.00e+00

The confusion matrix is identical and the curves agree to floating-point noise. The one structural difference is ordering: scikit-learn returns recall in decreasing order for precision_recall_curve, because it walks the thresholds from the lowest to the highest, while our implementation walks from the highest downward. Sorting before comparing removes the disagreement, which is a useful reminder that the direction of a curve array is a convention and not a property of the data. That comparison checks array contents, not the shape of the interpolated curve; the ROC discrepancy above uses drop_intermediate=False so that scikit-learn returns all interior points.

The last piece of the protocol concerns the threshold itself. A single number from a single validation split hides how much the choice would move under resampling, and that variability is exactly what you should tell a stakeholder about.

PythonIn [26]
CollapseExpandboot_rng = np.random.default_rng(11)
boot_thresholds = np.empty(300)
for b in range(boot_thresholds.size):
    idx = boot_rng.integers(0, y_val.size, y_val.size)
    boot_thresholds[b], _ = select_threshold(y_val[idx], score_val[idx])

Out[27]:

	quantity	threshold	validation cost at that threshold
0	5th percentile threshold	0.0224	2.7850
1	median threshold	0.0408	2.6900
2	95th percentile threshold	0.0646	2.8775

ConsoleOut [28]
CollapseExpand                 quantity  threshold  validation cost at that threshold
 5th percentile threshold     0.0224                             2.7850
         median threshold     0.0408                             2.6900
95th percentile threshold     0.0646                             2.8775

ConsoleOut [29]
CollapseExpandselected threshold on the full validation split: 0.0470
validation cost at the selected threshold      : 2.6612
validation cost at the analytic threshold      : 2.7462
share of bootstrap thresholds within 2x of selected: 0.94

The 5th and 95th percentile bootstrap thresholds are about 0.0224 and 0.0646, separated by a factor of about 2.89. Evaluating those two rules on the original validation sample gives costs of 2.7850 and 2.8775 per account, about 4.7 and 8.1 percent above its selected minimum of 2.66125. The selected threshold on the full validation split is about 0.0470. These are observations under this resampling scheme, not proof of stable future cost or a guarantee that every threshold between the quantiles has similar cost. These three costs measure sensitivity at threshold quantiles on the reused validation sample; they are not a bootstrap distribution of cost. Estimating cost uncertainty requires a separate evaluation design, which this illustration does not implement. Reporting the threshold variation and observed costs separately communicates more than the threshold alone.

The 300 bootstrap thresholds were generated on the validation split with seed 11 and the midpoint plateau rule. Their spread is easier to judge as a distribution than as three summary numbers.

PythonIn [30]
CollapseExpandboot_hist_counts, boot_hist_edges = np.histogram(boot_thresholds, bins=30)
boot_hist_centers = 0.5 * (boot_hist_edges[:-1] + boot_hist_edges[1:])
boot_hist_width = np.diff(boot_hist_edges)

VisualizationOut [31]
CollapseExpand

Histogram of 300 bootstrap thresholds over a narrow threshold range, with a solid line at the full-validation selected threshold and a dotted line at the closed-form cutoff. The bulk of the draws sit above the closed-form cutoff. — The 300 validation-bootstrap thresholds using seed 11 and the midpoint rule. The solid line marks the full-validation choice near 0.047 and the dotted line the analytic cutoff near 0.0323. This is a resampling distribution, not a guarantee of future cost stability.

Key ParametersLink Copied

The threshold itself is the decision variable. It is selected on data that plays no part in fitting the scorer and is fixed before any test evaluation.
c_fp and c_fn are the two costs, and they must be in the same units. Their ratio, not their absolute size, determines the threshold that a calibrated score would use, so the closed form is $c_{FP}/(c_{FP}+c_{FN})$ and the corresponding odds are $c_{FP}/c_{FN}$ .
The class prevalence affects precision and average precision. Transporting them to a different prevalence requires assumptions about the class-conditional score distributions. The closed-form probability cutoff instead depends on the error-cost ratio; the probabilities to which it is applied must represent the deployment population.
The calibration status of the score decides whether the closed form is admissible. A rank-only score supports empirical cost minimization but not the analytic rule.
Uncertainty depends on the metric denominator and the sampling and selection design. Recall uses actual positives as its denominator; precision uses predicted positives, or alerts.

Key MethodsLink Copied

confusion_counts and binary_metrics give the four counts and the derived metrics at a single threshold. They are the primitives; everything else is a loop or a vectorization over them.
threshold_sweep computes the whole family of confusion matrices in one pass by sorting the scores once and taking cumulative sums. Its cost is $O(n \log n)$ , dominated by the sort.
curve_points, roc_auc, average_precision and pr_trapezoid turn the sweep into curves and summary numbers. Keeping average_precision and pr_trapezoid as separate functions is deliberate, because they are separate quantities.
cost_curve and select_threshold implement the decision rule. select_threshold uses a midpoint for an interior optimal plateau when it is representable, falling back to the inclusive upper score if rounding reaches the excluded lower score. It reports infinity for the no-alert state and a valid representative between zero and the minimum score for the all-alert state. These reproducible conventions do not guarantee greater sampling stability than an endpoint.
The scikit-learn counterparts are roc_auc_score, average_precision_score, precision_recall_curve, roc_curve and confusion_matrix. Recent versions also provide TunedThresholdClassifierCV, which wraps the same select-on-validation logic in a cross-validated estimator; the primitive version here is easier to audit and to adapt to asymmetric costs.

Comments

Add to the conversation

A question, a correction, or an example from your own work.

No comments yet. Be the first to share your thoughts!

Key ParametersLink Copied

Key MethodsLink Copied

Comments

Add to the conversation

Stay up to date