Stop Wasting Trials: A Progressive Hyperparameter Search Strategy with Optuna

Most hyperparameter tuning tutorials show you how to define a search space and run 100 trials. What they don't show is what to do with the results, how to pick up where you left off after your laptop turns off, or how to systematically narrow the search as you learn.

This is the strategy I developed while tuning CatBoost on 14 million flight records — where each trial took 40 minutes on the full dataset.

The Problem with One-Shot Search

Whether you use RandomizedSearchCV or a basic Optuna study, the typical approach is: define wide bounds, run N trials, take the best result. This has two problems.

You throw away information. After 30 trials, you know a lot about which regions of parameter space are promising. The next 30 trials should exploit that knowledge, not keep exploring the same wide bounds.

Everything is lost if you stop. If your computer turns off mid-run (it happened to me after 47 trials), a standard in-memory study loses everything.

The Solution: Progressive Studies with SQLite

The core idea is to run multiple sequential studies, each informed by the last:

Each study stores results in SQLite, so every completed trial survives a crash or reboot:

load_if_exists=True is the key line. Re-run the cell and it picks up exactly where it left off.

Warm-Starting Between Studies

When you move from V1 to V2, don't start V2 blind. Seed it with the best parameters from V1, clamped to the new narrower bounds:

def _clamp(val, lo, hi):
    return max(lo, min(hi, val))

V2_BOUNDS = {
    "iterations":    (900,  1200),
    "learning_rate": (0.09, 0.15),
    "l2_leaf_reg":   (6.5,  8.0),
}

if len(study_v2.trials) == 0:
    seed_params = {k: _clamp(study_v1.best_params[k], *V2_BOUNDS[k]

enqueue_trial forces the first V2 trial to use these exact parameters. Optuna's TPE sampler then uses that result as an anchor, so subsequent trials cluster around what you already know is good instead of exploring blindly.

Deriving Bounds Empirically

After V2, I queried the SQLite database directly to see what the top 10 trials had in common:

import sqlite3, pandas as pd

con = sqlite3.connect("models/optuna_catboost.db")
df_trials = pd.read_sql("""
    SELECT t.number, tp.param_name, tp.param_value, tv.value as auc
    FROM trials t
    JOIN trial_params tp ON t.trial_id = tp.trial_id
    JOIN trial_values tv  ON t.trial_id = tv.trial_id
    WHERE t.study_id = 2 AND t.state = 'COMPLETE'
    ORDER BY auc DESC
""", con)

top10_nums = df_trials.groupby("number")["auc"].first().nlargest(10).index
print(df_trials[df_trials["number"].isin(top10_nums)]
      .groupby("param_name")["param_value"]
      .agg(["min", "max", "mean"]

The result was striking: depth=12 appeared in 100% of the top 10 trials. That made V3 simple — fix depth at 12, tighten everything else to the ranges where the top trials actually lived.

Speed: Fixed Stratified Sampling

Each trial on 14 million rows with 3-fold cross-validation took about 40 minutes. The fix: create one fixed stratified sample before the study and reuse it across every trial objective:

OPT_N  = 500_000
_rng   = np.random.RandomState(42)
_pos   = np.where(y.values == 1)[0]
_neg   = np.where(y.values == 0)[0]
_n_pos = int(OPT_N * len(_pos) / len(y))
_n_neg = OPT_N - _n_pos
_idx   = np.concatenate([
    _rng.choice(_pos, _n_pos, replace=False),
    _rng.choice(_neg, _n_neg, replace=False),
])
X_opt = X.iloc[_idx].reset_index(drop=True)
y_opt = y.iloc[_idx]

This brought trial time from 40 minutes to about 5 minutes — an 8x speedup. The absolute AUC scores are slightly lower on 500k rows than the full dataset, but the relative ranking of parameter configurations is preserved. Optuna finds the same best region either way.

Recovering from a Crash

When my computer turned off mid-trial, one trial was stuck in RUNNING state. Optuna won't launch a new trial until it clears. Fix it directly in SQLite:

Restart the study cell and it resumes from where it left off.

Results

After ~130 total trials across V1–V3, the best CatBoost configuration achieved AUC = 0.8527 with depth=12, 1161 iterations, learning_rate=0.133, l2_leaf_reg=7.26. The progression was visible at each stage: V1 established depth 12 as dominant. V2 confirmed and refined. V3 converged in a small, well-understood region of parameter space.

The result was better than any single-shot random search would have found in the same compute budget — because each phase used information from the last.