The Feature Engineering Trick That Improved My Model More Than Any Algorithm

When I set out to predict flight delays using 14 million rows of US Department of Transportation data, I made a common mistake: I spent weeks comparing gradient boosting libraries before I spent a single day thinking hard about my features. That was backwards.

The most important insight in this project had nothing to do with CatBoost vs. XGBoost vs. LightGBM. It was a simple observation about how delays actually work.

Delays Are Contagious

A flight doesn't exist in isolation. It's the product of an aircraft that flew somewhere yesterday, a crew that worked last night, a gate at an airport that had three late arrivals this morning, and a carrier whose operational practices compound or dampen all of that.

This means delay is serially correlated. If a route from Chicago O'Hare to JFK was delayed three days in a row, it probably isn't having a great week. If an aircraft's last five flights averaged 45 minutes late, the next one probably won't be on time either. If a carrier is running behind at a particular hub this afternoon, every departure from that hub is at elevated risk.

This is the core insight: the best predictor of a future delay is the recent history of delays on the same route, aircraft, and carrier.

Building Rolling Lag Features

To capture this, I engineered rolling average delay features across four dimensions:

def engineer_rolling_features(df):
    groups = {
        "origin":  ["origin"],
        "dest":    ["dest"],
        "carrier": ["op_unique_carrier"],
        "route":   ["origin", "dest"],
    }
    windows = [3, 7]

The shift(1) is critical. Without it, the rolling average would include the current flight's own delay — data that doesn't exist at prediction time. This is a form of data leakage that would make your model look great in training and fail completely in production.

I also built hourly congestion features — 1-hour and 3-hour rolling counts of delayed departures at the origin airport — which capture real-time airport stress rather than just historical trend.

Tail Number History: The Aircraft-Level Signal

One of the most effective features came from joining FAA aircraft registration data to track individual aircraft performance:

# For each tail number, compute stats over the last 5 flights
df["tail_prev5_avg_delay"] = (
    df.sort_values("fl_date")
      .groupby("tail_num")["dep_delay"]
      .transform(lambda x: x.shift(1).rolling(5, min_periods=1).mean())
)

df["tail_prev5_delay_count"] = (
    df.sort_values("fl_date")
      .groupby("tail_num")["dep_delay"]

An aircraft that was delayed on 4 of its last 5 flights is carrying operational debt — maintenance issues, crew fatigue, or positioning problems that haven't resolved. This signal is independent of the route or carrier average and adds real predictive power.

The Payoff

When I look at feature importance from the trained models, rolling lag features and prev_flight_delay_1 — the single prior flight delay — consistently dominate. The single most predictive feature is simply: was this flight's most recent departure late?

To put it in perspective: the gap between a logistic regression baseline and a tuned CatBoost model is about 2–3 AUC points. The gap between having no lag features and having them is likely 5–10 points. Feature engineering beat model selection by a factor of 2–3x.

The Lesson

Before you tune a single hyperparameter or try a new algorithm, ask: what do domain experts know about how this target variable behaves over time?

For flight delays, any airline operations professional would tell you immediately that delays cascade through the system. That domain knowledge, translated into rolling window features, was worth more than any model I tested.

The algorithm is the last 20%. The features are everything else.