Why I Chose CatBoost (And What It Taught Me About Generalization)

Model selection isn't about picking the highest number. It's about understanding why a model performs the way it does.

CatBoost finished with a test AUC of 0.845. XGBoost was 0.839. LightGBM was 0.844. On raw performance, these three models are essentially tied. So why recommend CatBoost?

The answer is the train-test gap.

Overfitting is the central challenge in boosted models

Random Forest with default settings reached a training AUC of 1.000 — perfect memorization — with a test AUC of 0.825. That 0.175 gap is not a model that's learned anything generalizable. LightGBM with default settings had nearly identical behavior. These aren't bad algorithms. They're algorithms whose defaults are optimized for fit, not for generalization, and on a large dataset with 73 features and complex interactions, they will overfit without explicit intervention.

Getting CatBoost to its final performance required a multi-phase Optuna study spanning 400+ trials. The search found that a depth of 5–7 and moderate L2 leaf regularization produced the best balance — high test AUC with the tightest generalization gap in the comparison at 0.031. That number matters more than the marginal AUC advantage over XGBoost.

Why CatBoost handles this data well

Flight data is categorical-heavy. Carrier codes, airport identifiers, aircraft manufacturer groups — these features have high cardinality and meaningful signal. Most gradient boosting implementations encode these as numeric integers or use one-hot encoding, which can generate sparse, noisy feature representations at scale. CatBoost's native ordered target encoding handles high-cardinality categoricals without leaking target information across folds, which matters when the categorical feature interacts strongly with the label.

This isn't a theoretical advantage — it showed up empirically. On both datasets, CatBoost's generalization gap was tighter than comparable XGBoost runs at equivalent hyperparameter effort. That's the mechanism explaining the performance difference, not just the algorithm name.

The practical lesson about model selection

Leaderboard performance — the highest single test AUC — is a useful starting point, not an ending point. Before recommending a model for production, the question worth asking is: how does this model's training performance compare to its test performance, and what does that gap tell me about how it will behave on data it hasn't seen?

A model with a 0.003 AUC advantage and a 0.03 generalization gap beats a model with a 0.003 AUC advantage and a 0.10 generalization gap every time in production. The first model knows what it knows. The second model thinks it knows more than it does.