Evaluation & ML in Practice

The fundamental tension in supervised learning is between fitting and generalizing. A model that perfectly memorizes training data is useless — you want a model that extracts the underlying pattern and applies it to new examples it has never seen.

Formally, we assume data is drawn i.i.d. from some unknown distribution $P(X, Y)$. Training gives us a finite sample $S_{train}$ drawn from $P$. The true goal is to minimize expected risk: $$R(f) = \mathbb{E}{(x,y) \sim P}[\ell(f(x), y)]$$ but we can only compute empirical risk on our sample: $$\hat{R}(f) = \frac{1}{n}\sum{i=1}^n \ell(f(x_i), y_i)$$

The gap $R(f) - \hat{R}(f)$ is the generalization gap. Statistical learning theory (VC theory, PAC learning, Rademacher complexity) provides bounds on this gap. The key insight: if you optimize $f$ on $S_{train}$, the empirical risk on $S_{train}$ is a biased estimate of true risk — it will be optimistically low. You need a held-out set to get an unbiased estimate.

Analogy: imagine studying for an exam using practice problems. If you memorize the answers to those exact problems, you score 100% on them but learn nothing. The real exam tests generalization. The practice problems are your training set; the real exam is your test set.

This is why data splitting is not a bureaucratic nicety — it is a statistical necessity. Every example used during training, hyperparameter selection, or any model decision is 'spent' and cannot serve as an unbiased estimate of true performance.

import numpy as np
from sklearn.model_selection import train_test_split

# Canonical three-way split
X, y = load_my_dataset()

# First, carve out the test set — touch it LAST
X_temp, X_test, y_temp, y_test = train_test_split(
    X, y, test_size=0.15, random_state=42, stratify=y
)

# Then split remainder into train / validation
X_train, X_val, y_train, y_val = train_test_split(
    X_temp, y_temp, test_size=0.176,  # 0.176 * 0.85 ≈ 0.15 of total
    random_state=42, stratify=y_temp
)

# Rough rule of thumb:
# Large data (>100k): 70/15/15 or 80/10/10
# Small data (<5k):  use cross-validation instead
print(f"Train: {len(X_train)}, Val: {len(X_val)}, Test: {len(X_test)}")

A two-way split (train/test) is insufficient once you start making model selection decisions — choosing hyperparameters, selecting architectures, deciding when to stop training. Every such decision, if made by peeking at the test set, corrupts its statistical validity.

The three roles are distinct:

1. Training set — the optimizer touches this. Gradient descent, tree splits, support vector margins, all computed here.
2. Validation set (dev set) — the developer touches this. You tune hyperparameters, compare model variants, decide on features. Because you make decisions based on validation performance, validation performance is also a biased (optimistic) estimate — just less so than training performance.
3. Test set — touches this once, at the very end, to report the final number in your paper or deployment decision. It is the 'sealed envelope'. Opening it early permanently invalidates it.

Why validation performance is still biased: If you try 1000 hyperparameter combinations and pick the best on validation, you have effectively overfit to the validation set through the optimization process. The more choices you make using validation, the more optimistic it becomes. This is called validation set overfitting or researcher degrees of freedom.

Proportions matter. With 1M examples you can afford 80/10/10. With 1000 examples, a 15% test set (150 examples) gives very high-variance accuracy estimates (±4% at 95% CI). This motivates cross-validation.

Stratification: For classification, always use stratified splits so class proportions are preserved in each subset. A dataset with 95% negatives could randomly give you a validation set with 100% positives — meaningless.

# Illustrating how repeated val-set peeking inflates reported performance
import numpy as np

rng = np.random.default_rng(0)

# Simulate: true model has 60% val accuracy
# But we try 200 random hyperparameters
n_configs = 200
val_size = 200  # small val set -> high variance

# Each config's true accuracy ~60%, but measured with noise
true_acc = 0.60
measured = rng.binomial(val_size, true_acc, size=n_configs) / val_size

best_measured = measured.max()
print(f"Best measured val acc (over {n_configs} tries): {best_measured:.3f}")
print(f"True accuracy of 'best' model:       ~{true_acc:.3f}")
# Output: Best measured val acc: ~0.720  <-- looks great!
# True accuracy:                  ~0.600  <-- the honest number

When data is scarce, a single train/val split is wasteful and unreliable. k-fold cross-validation is the standard solution.

The algorithm: Partition the dataset into $k$ equal folds. For each $i \in {1, \ldots, k}$: train on all folds except fold $i$, evaluate on fold $i$. Average the $k$ evaluation scores.

$$CV_k = \frac{1}{k}\sum_{i=1}^k \text{Score}_i$$

The standard deviation across folds gives you a confidence estimate.

Why it works: Every example is used for validation exactly once, so the CV estimate uses all the data. It is a nearly unbiased estimate of generalization performance (bias decreases as $k$ increases). The variance is also measurable.

Choosing k:

• $k=5$ or $k=10$ is the empirical sweet spot. Kohavi (1995) showed $k=10$ balances bias and variance well.
• Leave-one-out (LOO): $k=n$. Minimum bias, maximum variance, expensive.
• Stratified k-fold: preserves class proportions in each fold. Use for classification.
• GroupKFold: when examples are grouped (e.g., multiple images per patient), prevents data from the same group appearing in both train and val — critical for avoiding leakage.

The test set survives: Cross-validation replaces the validation set, not the test set. The workflow is: use CV on the training portion to select your model and hyperparameters, then evaluate once on the held-out test set.

Nested cross-validation: For unbiased hyperparameter selection, use an outer CV loop for evaluation and an inner CV loop for hyperparameter tuning. Expensive but gold-standard for small datasets.

from sklearn.model_selection import StratifiedKFold, cross_val_score
from sklearn.ensemble import RandomForestClassifier
import numpy as np

X, y = load_my_dataset()

# Keep test set completely aside first
X_trainval, X_test, y_trainval, y_test = train_test_split(
    X, y, test_size=0.15, stratify=y, random_state=42
)

# CV on train+val portion to estimate performance & tune
skf = StratifiedKFold(n_splits=10, shuffle=True, random_state=42)

model = RandomForestClassifier(n_estimators=100, random_state=42)
scores = cross_val_score(
    model, X_trainval, y_trainval,
    cv=skf, scoring='roc_auc'
)

print(f"CV AUC: {scores.mean():.4f} ± {scores.std():.4f}")

# After CV confirms this config is good, train on ALL trainval
# then evaluate once on test
model.fit(X_trainval, y_trainval)
final_auc = roc_auc_score(y_test, model.predict_proba(X_test)[:, 1])
print(f"Test AUC: {final_auc:.4f}")

Accuracy — the fraction of correct predictions — is seductively simple and often deeply misleading. The confusion matrix is the antidote: it shows which mistakes your model is making.

For binary classification with classes Positive (P) and Negative (N), the confusion matrix has four cells:

Interpreting the cells:

• TP: Model says 'cancer', patient has cancer.
• TN: Model says 'no cancer', patient is fine.
• FP (Type I error): Model says 'cancer', patient is fine. Unnecessary surgery, anxiety.
• FN (Type II error): Model says 'no cancer', patient has cancer. Missed diagnosis — potentially fatal.

Notice: these two error types have completely different costs. Accuracy treats them identically. In medicine, law, fraud detection, and safety systems, they are not identical.

The accuracy paradox: Suppose 99% of credit card transactions are legitimate. A model that always predicts 'legitimate' achieves 99% accuracy and a completely useless confusion matrix with zero TPs.

Multiclass: The confusion matrix extends to $k \times k$ for $k$ classes. The diagonal is correct predictions; off-diagonal entries reveal which classes are confused with each other.

from sklearn.metrics import confusion_matrix, ConfusionMatrixDisplay
import matplotlib.pyplot as plt

# Example: email spam classifier
y_true = [1,1,1,1,1,0,0,0,0,0]  # 1=spam
y_pred = [1,1,1,0,0,0,0,0,1,0]  # model predictions

cm = confusion_matrix(y_true, y_pred)
print(cm)
# [[3 1]   <- TN=3, FP=1  (legitimate: 1 mis-flagged as spam)
#  [2 4]]  <- FN=2, TP=4  (spam: 2 slipped through)

# TP=4, TN=3, FP=1, FN=2
TP, FN, FP, TN = cm[1,1], cm[1,0], cm[0,1], cm[0,0]
accuracy = (TP + TN) / (TP + TN + FP + FN)  # 0.70
print(f"Accuracy: {accuracy:.2f}")

From the confusion matrix we derive two fundamental metrics that capture different aspects of a classifier's behavior.

Precision (Positive Predictive Value): Of all examples predicted positive, what fraction actually are? $$\text{Precision} = \frac{TP}{TP + FP}$$ Precision answers: when my model fires an alarm, how often is it right? High precision = few false alarms.

Recall (Sensitivity, True Positive Rate): Of all actual positives, what fraction did the model catch? $$\text{Recall} = \frac{TP}{TP + FN}$$ Recall answers: how many of the real positives did I find? High recall = few missed detections.

The precision-recall trade-off: These are in fundamental tension. You can achieve perfect recall by predicting everything positive (but precision plummets). You can achieve near-perfect precision by only predicting positive when extremely confident (but recall drops). This trade-off is controlled by the decision threshold applied to the model's output probability.

F1 score: The harmonic mean of precision and recall: $$F_1 = 2 \cdot \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}} = \frac{2TP}{2TP + FP + FN}$$ The harmonic mean penalizes extreme imbalances: a model with Precision=1.0, Recall=0.01 gets $F_1 = 0.020$, correctly revealing it is useless despite the high precision.

$F_\beta$ score: When one error type is more costly: $$F_\beta = (1+\beta^2) \cdot \frac{\text{Precision} \times \text{Recall}}{\beta^2 \cdot \text{Precision} + \text{Recall}}$$

• $\beta > 1$: recall matters more (e.g., $\beta=2$ for disease detection — missing a case is worse than a false alarm)
• $\beta < 1$: precision matters more (e.g., $\beta=0.5$ for spam — being overly aggressive is annoying)

Macro vs. micro vs. weighted averaging (multiclass):

• Macro: average of per-class metrics. Treats all classes equally. Sensitive to rare-class performance.
• Micro: pool all TPs/FPs/FNs across classes, then compute. Dominated by frequent classes.
• Weighted: average weighted by class support. Good default for imbalanced problems.

from sklearn.metrics import classification_report, f1_score

y_true = [1,1,1,1,1,0,0,0,0,0]
y_pred = [1,1,1,0,0,0,0,0,1,0]

print(classification_report(y_true, y_pred,
      target_names=['legit','spam']))
# Output:
#               precision    recall  f1-score   support
#        legit       0.75      0.75      0.75         4  <- TN/(TN+FN), TN/(TN+FP)
#         spam       0.80      0.60      0.69         5  <- TP/(TP+FP), TP/(TP+FN)
#     accuracy                           0.70        10
#    macro avg       0.78      0.68      0.72        10
# weighted avg       0.78      0.70      0.72        10

# F-beta: emphasize recall (beta=2) for safety-critical detection
from sklearn.metrics import fbeta_score
fb = fbeta_score(y_true, y_pred, beta=2)
print(f"F2 score: {fb:.3f}")  # penalizes FN more heavily

Both precision/recall and the confusion matrix depend on a decision threshold — the probability cutoff above which the model predicts 'positive'. The choice of threshold is a deployment decision, not a model property. How do we evaluate the model independently of this choice?

The ROC curve (Receiver Operating Characteristic — the name comes from radar engineering in WWII) plots:

• True Positive Rate (TPR = Recall): $\frac{TP}{TP + FN}$ (y-axis)
• False Positive Rate (FPR): $\frac{FP}{FP + TN}$ (x-axis)

...as the decision threshold varies from 0 to 1. Every point on the curve corresponds to a threshold. You trace the curve by sweeping the threshold.

Interpreting the ROC curve:

• The diagonal $(FPR = TPR)$ is a random classifier — it achieves TPR by randomly guessing, so every true positive comes with an equal false positive. No skill.
• A perfect classifier reaches the top-left corner $(FPR=0, TPR=1)$ — all positives caught, zero false alarms.
• A curve that bows toward the top-left is a good classifier; one that bows toward the bottom-right is worse than random (just invert its predictions).

AUC (Area Under the ROC Curve): A single scalar summary of the ROC curve, ranging from 0 to 1.

• Random classifier: AUC = 0.5
• Perfect classifier: AUC = 1.0
• AUC has a beautiful probabilistic interpretation: AUC = P(score for a random positive > score for a random negative). This is the probability that the model ranks a positive example above a negative one — a pure ranking quality measure.

When to use ROC-AUC vs. Precision-Recall AUC: For highly imbalanced datasets (e.g., 1 positive per 1000 negatives), ROC curves can look deceptively optimistic. The FPR denominator includes all true negatives, so even many false positives can look like a small FPR. Precision-Recall AUC (Average Precision, AP) is more informative in this regime — precision directly reflects how many of your positives are real.

from sklearn.metrics import roc_auc_score, roc_curve, average_precision_score
import matplotlib.pyplot as plt

# Assume y_scores are model output probabilities for positive class
y_true   = [0,0,0,0,0,1,1,1,1,1]
y_scores = [0.1, 0.2, 0.3, 0.4, 0.6,  # negatives
            0.5, 0.7, 0.8, 0.9, 0.95]  # positives

fpr, tpr, thresholds = roc_curve(y_true, y_scores)
auc = roc_auc_score(y_true, y_scores)
ap  = average_precision_score(y_true, y_scores)

print(f"ROC-AUC: {auc:.3f}")  # ~0.88
print(f"Avg Precision (PR-AUC): {ap:.3f}")

# Probabilistic interpretation:
# P(score_pos > score_neg) for random pair
from itertools import product
pairs = [(sp, sn) for sp in y_scores[5:]
                  for sn in y_scores[:5]]
prob = sum(1 for sp, sn in pairs if sp > sn) / len(pairs)
print(f"Manual AUC estimate: {prob:.3f}")  # matches roc_auc_score

Data leakage occurs when information from outside the training distribution — specifically, information that would not be available at prediction time — contaminates the training or evaluation process. It is the single most common cause of models that appear to work but fail in production.

Leakage is insidious because it makes your model look better. You will not notice it during development. You will notice it when you deploy.

Forms of leakage:

1. Target leakage (most dangerous): A feature directly encodes or strongly correlates with the label, but would not be available at prediction time. Example: predicting whether a loan will default using 'has_collection_agency_account' — which is only created after a default. The model uses future information.

2. Train-test contamination: Test (or validation) data influences the training pipeline. The most common form: fitting preprocessing on the full dataset before splitting.

   # WRONG: scaler fit on all data — test statistics leak into training
   scaler.fit(X_all)
   X_scaled = scaler.transform(X_all)
   X_train, X_test = train_test_split(X_scaled, ...)
   
   # CORRECT: fit only on training data
   X_train, X_test = train_test_split(X, ...)
   scaler.fit(X_train)  # only training stats
   X_train_s = scaler.transform(X_train)
   X_test_s  = scaler.transform(X_test)  # apply training stats
   

3. Group leakage: Multiple examples from the same entity (patient, user, document) appear in both train and test. The model memorizes entity-specific patterns rather than learning general ones. Example: medical imaging where 10 images from the same patient are in both train and test. Use GroupKFold.

4. Temporal leakage: Using future data to predict the past. Example: training a stock price predictor using features computed from data at time $t+1$ to predict labels at time $t$. For time series, always split by time, never randomly.

5. Preprocessing leakage: Feature selection, PCA, normalization, or any transformation fitted on the full dataset before splitting. Even imputing missing values with the global mean leaks test-set statistics.

Prevention recipe:

• Always split first, preprocess second
• Use sklearn Pipeline objects, which enforce fit-on-train-only semantics
• For time series, use TimeSeriesSplit
• For grouped data, use GroupKFold
• Audit every feature: ask 'would this be available at prediction time?'
• Be suspicious of surprisingly high accuracy — it usually means leakage

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score

# The Pipeline enforces the correct order:
# fit_transform on train, transform-only on test/val
# This is the canonical way to prevent preprocessing leakage.

pipe = Pipeline([
    ('scaler', StandardScaler()),   # fit on train fold only
    ('pca',    PCA(n_components=10)),  # fit on train fold only
    ('clf',    LogisticRegression(max_iter=1000))
])

# cross_val_score respects the pipeline:
# each fold fits scaler+PCA ONLY on the training portion
scores = cross_val_score(
    pipe, X, y, cv=10, scoring='roc_auc'
)
print(f"Leak-free CV AUC: {scores.mean():.4f} ± {scores.std():.4f}")

# Temporal data: always split by time
from sklearn.model_selection import TimeSeriesSplit
tscv = TimeSeriesSplit(n_splits=5)
for train_idx, val_idx in tscv.split(X_timeseries):
    # train_idx always precedes val_idx in time
    pass

Class imbalance is the norm in real problems, not the exception. Fraud rates are 0.1%. Disease prevalence is 0.01%. Defect rates are 1%. When classes are imbalanced, standard metrics and default algorithms both fail silently.

The accuracy trap revisited: With 99:1 class imbalance, a model that always predicts the majority class achieves 99% accuracy. This is why you must use precision, recall, F1, or PR-AUC as your primary metrics for imbalanced problems.

Strategies for imbalanced data:

1. Resampling:
   • Oversampling: duplicate or synthesize minority examples. SMOTE (Synthetic Minority Over-sampling Technique) generates synthetic examples by interpolating between k-nearest-neighbor minority samples.
   • Undersampling: remove majority examples. Loses information but fast.
   • Combined: SMOTE + Tomek links.
2. Class weighting: Most sklearn classifiers accept class_weight='balanced' which adjusts the loss function to weight minority errors proportionally more.
3. Threshold adjustment: The default 0.5 threshold is rarely optimal for imbalanced problems. Use the validation set to choose the threshold that maximizes your metric of interest (F1, F-beta, a cost matrix).
4. Algorithmic choices: Tree ensembles (Random Forests, Gradient Boosting) naturally handle some imbalance via bagging; linear models struggle more.

Calibration: A model's output probability should mean what it says. If the model says 'P(cancer) = 0.8' for a group of patients, 80% of them should actually have cancer. Most ML models are not well-calibrated by default:

• SVMs produce poor probabilities
• Random Forests tend to compress probabilities toward 0.5
• Neural networks are often overconfident

Measuring calibration: A calibration curve (reliability diagram) plots mean predicted probability vs. observed fraction of positives in each probability bin. A well-calibrated model follows the diagonal.

Fixing calibration: sklearn.calibration.CalibratedClassifierCV with method='isotonic' or method='sigmoid' (Platt scaling).

from sklearn.utils.class_weight import compute_class_weight
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.calibration import CalibratedClassifierCV, calibration_curve
import numpy as np

# 1. Class weighting
classes = np.unique(y_train)
weights = compute_class_weight('balanced', classes=classes, y=y_train)
cw = dict(zip(classes, weights))

model = GradientBoostingClassifier()
# GBM doesn't take class_weight, but we can use sample_weight:
sample_weights = np.array([cw[yi] for yi in y_train])
model.fit(X_train, y_train, sample_weight=sample_weights)

# 2. Calibration
calibrated = CalibratedClassifierCV(model, method='isotonic', cv=5)
calibrated.fit(X_train, y_train)

# 3. Reliability diagram
prob_true, prob_pred = calibration_curve(
    y_val, calibrated.predict_proba(X_val)[:, 1],
    n_bins=10
)
# Perfect calibration: prob_true == prob_pred (diagonal line)

Different ML tasks need different metrics. Using the wrong metric for a task is as bad as using no metric.

Multi-label classification (each example can have multiple labels): Standard F1 macro/micro averaging applies per-label and then aggregates.

Regression metrics:

• MAE (Mean Absolute Error): $\frac{1}{n}\sum |y_i - \hat{y}_i|$. Robust to outliers. Interpretable in original units.
• RMSE (Root Mean Squared Error): $\sqrt{\frac{1}{n}\sum (y_i - \hat{y}_i)^2}$. Penalizes large errors more. Dominated by outliers. Same units as target.
• MAPE (Mean Absolute Percentage Error): $\frac{100}{n}\sum |\frac{y_i - \hat{y}_i}{y_i}|$. Scale-free but undefined when $y_i=0$ and distorted near zero.
• $R^2$ (coefficient of determination): $1 - \frac{\sum(y_i-\hat{y}_i)^2}{\sum(y_i-\bar{y})^2}$. Fraction of variance explained. Ranges from $-\infty$ to 1; $R^2=0$ means 'predict the mean', $R^2=1$ means perfect.

Ranking metrics (search, recommendations):

• NDCG (Normalized Discounted Cumulative Gain): rewards relevant items ranked higher; heavily used in search evaluation.
• MAP (Mean Average Precision): mean of per-query average precisions; evaluates ranked retrieval.

Language generation metrics:

• BLEU: n-gram precision vs. reference translations. Notoriously imperfect but widely used.
• ROUGE: n-gram recall; used for summarization.
• Perplexity: $2^{-\frac{1}{N}\sum \log_2 P(w_i)}$; measures how surprised the language model is by the test corpus.

The meta-lesson: Every metric captures one aspect of performance. Before choosing a metric, ask: what behavior do I actually want to reward? What would a user or stakeholder directly care about? A metric is a proxy for real-world value — choose it carefully.

from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
import numpy as np

y_true = np.array([3.0, -0.5, 2.0, 7.0])
y_pred = np.array([2.5,  0.0, 2.0, 8.0])

mae  = mean_absolute_error(y_true, y_pred)   # 0.5
rmse = mean_squared_error(y_true, y_pred, squared=False)  # 0.612
r2   = r2_score(y_true, y_pred)              # 0.948

print(f"MAE:  {mae:.3f}")
print(f"RMSE: {rmse:.3f}")
print(f"R2:   {r2:.3f}")

# RMSE penalizes the large error (7->8) more than MAE does
# MAE is median-optimal; RMSE is mean-optimal
# Choose based on whether outlier errors matter more

A model achieves 85.2% accuracy vs. a baseline's 84.8% — is this a real improvement? With a test set of 500 examples, probably not. This is the domain of significance testing in ML evaluation.

Why it matters: Random variation in test set sampling can easily produce differences of 1-2% accuracy. If you report 85.2% without acknowledging uncertainty, you are overclaiming.

The McNemar test: The standard test for comparing two classifiers on the same test set. It examines only the discordant predictions — where the classifiers disagree — and tests whether one classifier makes significantly more correct predictions.

Bootstrap confidence intervals: Resample the test set with replacement $B$ times (e.g., $B=10000$), compute your metric on each bootstrap sample, and report the 2.5th and 97.5th percentiles as a 95% CI. This is model-free and works for any metric.

The multiple comparisons problem (Bonferroni): If you compare 20 models to a baseline, you expect one 'significant' result at $\alpha=0.05$ purely by chance. Apply Bonferroni correction: require $p < \alpha / m$ for $m$ comparisons. Or use Holm-Bonferroni for more power.

Honest reporting checklist:

• Report metric ± standard deviation (from CV or bootstrap)
• State the test set size and class balance
• Report your baseline clearly (random, majority class, previous SOTA)
• State whether you applied any significance test
• State how many models you tried before selecting the reported one
• Never report test set performance more than once per model

Ablation studies: To understand why your model works, systematically remove each component and measure performance degradation. This separates genuine contributions from coincidental correlations.

import numpy as np
from scipy.stats import chi2

def mcnemar_test(y_true, pred_a, pred_b):
    """
    McNemar's test: are classifier A and B significantly different?
    Returns test statistic and p-value.
    """
    # Discordant counts: A right/B wrong (n10) and A wrong/B right (n01)
    n10 = np.sum((pred_a == y_true) & (pred_b != y_true))
    n01 = np.sum((pred_a != y_true) & (pred_b == y_true))
    # With continuity correction (Edwards):
    stat = (abs(n10 - n01) - 1)**2 / (n10 + n01)
    p_value = 1 - chi2.cdf(stat, df=1)
    return stat, p_value

# Bootstrap confidence interval for any metric
def bootstrap_ci(y_true, y_score, metric_fn, B=10000, alpha=0.05):
    n = len(y_true)
    bootstrap_scores = []
    rng = np.random.default_rng(42)
    for _ in range(B):
        idx = rng.integers(0, n, size=n)
        bootstrap_scores.append(
            metric_fn(y_true[idx], y_score[idx])
        )
    lo = np.percentile(bootstrap_scores, 100 * alpha / 2)
    hi = np.percentile(bootstrap_scores, 100 * (1 - alpha / 2))
    return lo, hi