LightGBM Overfitting: How One Parameter Caused a 0.114 AUC Gap

After running a randomized hyperparameter search on LightGBM for flight delay classification, I got this result:

A gap of 0.114 is significant. The model was memorizing training data rather than learning generalizable patterns. Here's how I diagnosed the root cause in five minutes — and what I changed to fix it.

Understanding LightGBM's Growth Strategy

Most gradient boosting implementations grow trees depth-first: you specify max_depth=6 and each tree has at most 64 leaves. LightGBM grows trees leaf-first: it always splits whichever leaf reduces the loss the most, regardless of depth.

This is more efficient — leaf-wise growth finds better splits faster. But it means num_leaves is now the primary complexity control, not max_depth. Without a num_leaves constraint, LightGBM can create extremely deep, narrow trees that perfectly fit the training data.

The Culprit: num_leaves=334

My randomized search returned num_leaves=334 as optimal. That sounds reasonable until you think about what it means.

With 334 leaves and a training set of ~755,000 rows, the average samples per leaf is roughly 2,260. But that average is misleading. Leaf-wise growth creates highly uneven splits — many leaves will have tens of thousands of samples, but the "interesting" leaves that capture rare high-delay patterns can have just dozens. LightGBM was learning those rare patterns in fine-grained detail, including noise.

My search range for num_leaves was 300–1000. The algorithm found 334 — at the very bottom of that range. The optimum was below 300, but I had made it impossible to find.

Reading the Other Warning Signs

The full best-params output revealed more problems compounding each other:

Three problems at once: too many leaves, too few samples required per leaf, and maximum histogram resolution memorizing fine-grained training patterns. And n_estimators=800 hitting the floor of the 800–1500 range was telling me something — fewer trees were better at these other settings. A model doing less better is a sign of overfitting.

The General Principle

When your best hyperparameters sit at the boundary of your search range, that boundary is wrong. Your search is telling you where to look next:

• num_leaves=334 (bottom of 300–1000) → search below 300

• n_estimators=800 (bottom of 800–1500) → search below 800

• min_child_samples=63 (lower half of 20–100) → push the floor higher

The Fix

# V1 — too wide, promotes overfitting
param_dist_v1 = {
    'num_leaves':        randint(300, 1000),  # wrong region
    'min_child_samples': randint(20, 100),    # floor too low
    'max_bin':           [150, 200, 255],     # too fine
}

# V2 — regularization-focused
param_dist_v2 = {
    'num_leaves':        randint(30, 150),    # primary fix
    'min_child_samples': randint(100, 500),   # higher floor
    'max_bin':           [63, 100, 150]

The Broader Lesson

LightGBM's default num_leaves=31 is conservative for a reason. When you let a search algorithm choose from a range of 300–1000, you're giving it permission to build a very complex model and hoping regularization saves you. It usually won't.

Start conservative — 31–127 is a good default search range for most problems. Expand upward only if validation performance is still improving. The gap between your train and test AUC tells you exactly which direction to move.

In this case, one parameter searched in the wrong region was responsible for nearly all of the 0.114 overfitting gap. The fix was a tighter search range — nothing more.