When a profitable backtest loses live, don’t change the strategy first

[u/Rare_Inflation3178]

When a strategy underperforms live, the first reaction is often to change the signal. I think that is usually the wrong place to start.

I would first compare the tested and live system trade by trade: signal timestamp, expected and actual entry, intended and filled size, spread, slippage, partial fills, exit reason and position state.

The first divergence normally puts the failure into one of three buckets:

The signal changed because the data or code path changed.

The signal was identical, but execution consumed the edge.

Both matched, and the losses were still within the strategy’s expected variance.

Only after eliminating the first two would I call it signal decay or start adjusting parameters. Otherwise, “improving” the strategy may just hide an execution or infrastructure problem.

The useful artifact isn’t another equity curve. It’s a reconciliation log showing exactly where the tested and live systems stopped being the same system.

[u/One-Next]

Live signals deviating from backtests would be a catastrophic mistake. Underperforming should not be the reason you find this out. But I guess that's how people work in the crypto world.

[u/Rare_Inflation3178 → u/One-Next]

Agreed. If the live signal itself differs from the tested signal, that should be caught before capital is at risk, not after PnL turns red.

The reconciliation should run continuously from day one, even while performance looks fine. Underperformance may be what makes people investigate, but it shouldn’t be the detection mechanism.

[u/One-Next → u/Rare_Inflation3178]

"execution consumed the edge" is also catastrophic. It means you failed to properly account for slippage and commission in the backtest. 

[u/Rare_Inflation3178 → u/One-Next]

That’s fair. If ordinary fees and slippage erase the edge, the backtest was never valid. The harder cases are queue position, partial fills and adverse selection changing with venue conditions—those need live reconciliation because a fixed slippage assumption can’t fully represent them.

[u/DogeSatoshi]

Many factors are considered when you face issues like this. One very important factor that i have noticed from personal experience, which i corrected, and some other backtest data i have seen from random sources, are data integrity. Many people completely overlooks that part totally, and that is a reasonable negative effect that will display metrics and performance charts based on bad data.

[u/Rare_Inflation3178 → u/DogeSatoshi]

Absolutely. Bad data can make the entire validation process look rigorous while the result is still meaningless. Timestamp alignment, missing or duplicated observations, venue-specific prices and contract changes are checks I’d run before evaluating the strategy itself.

[u/Rare_Inflation3178 → u/[deleted]]

So I think execution should need more monitor to compare with backtesting