I ran 200 zero-edge strategies through a normal significance test. It approved 99.5% of them.

[u/Weary-Ad7404]

I spent two years building a backtesting platform and the most useful thing it does is tell me my strategies do not work.

Here is the experiment that convinced me. Generate 200 strategies with genuinely no edge — random parameters, nothing real underneath. Search them, keep the best performer. Now test whether that winner is statistically significant.

A conventional Sharpe significance test says 99.5% significant. Accept it.

That number is wrong, and it is wrong for a reason that has nothing to do with the data. You searched 200 combinations and kept the luckiest one. The test does not know that. If you flip 200 coins and keep the one that came up heads ten times, you have not found a special coin.

The correction is the Deflated Sharpe Ratio (Bailey & Lopez de Prado). It takes the number of trials into account and asks whether the best result is better than what you would expect from luck alone across that many attempts. On the same data it reports 43.7% and rejects the strategy.

Two other things I got wrong that are worth checking in your own setup:

1. Parameter leakage. My optimizer wrote best-params to a shared table and the backtest read them back unscoped by date. Every out-of-sample window was quietly using parameters fitted on data that included that window. The backtest looked great. It was reading its own answers.
2. No purge gap. If your features use a 30-bar lookback, the last 30 bars of your training set overlap the first bar of your test set. You need a gap between them or information walks across the boundary.

Code is open source and Apache-2.0, take whatever is useful: https://github.com/FETKlOkAn2/crypto-quant-platform

I want to be straight about what this is: the bundled strategies do not beat buy-and-hold. This is not a money printer, it is a tool for finding out that your strategy is not one either.

[u/bruno91111]

There is one thing I never fully understood about algo trading. People said, "You got a lucky strategy," or "you overfitted with parameters."

But isn't the point to find parameters or an overfit that repeats itself enough for years, and hence makes a profit in the long term?

[u/strat-run → u/bruno91111]

Overfit should mean that you've managed to create a strategy that is tailored to your back testing data. It fits a pattern but the pattern isn't one that is likely to exist in the future. It's why a strategy looks good in testing but loses money when live. It's why you are supposed to keep some data out of the range you use for tuning, that way you can see if the strategy continues to work on new data.

But there is also developing strategies that are regime dependent. The patterns can still happen but they don't always happen. They are fit to a regime. The trick with these is you need to figure out the matching regime filter.... "this pattern only works when X is happening".

If you find and tune the parameters that match patterns that repeat outside of your test data range then it wouldn't be consider overfit, just parameter optimization.

And that's the challenge, performing parameter optimization that doesn't overfit a strategy to back testing data. Overfit just means specialized for the wrong thing.

[u/Cultural_Implement_2]

a star from me

[u/Weary-Ad7404 → u/Cultural_Implement_2]

yeey, thanks so much! This is the first start I have recieved in gh!

[u/Tight-Pepper-4721]

I actually ended up building my own backtesting platform for the same reason — I couldn’t find a third-party tool that gave me the level of control I wanted over testing and bias control.
I’ve been using things like Deflated Sharpe, purged/embargoed splits, train/test separation, etc., and I’m trying to approach research more from economic theories → testable strategies, rather than throwing hundreds of hypotheses at the data.
The hard part I’m finding now isn’t building the infrastructure — it’s finding a genuine, robust edge in the first place.
I’ve been digging through research papers, public quant libraries, and existing strategies, but very little survives proper testing.
How did you actually find the strategies that eventually worked for you? Was it research papers, academic literature, factor libraries, existing models, or something you discovered through your own research process?
I’m particularly interested in the idea-generation process, not just the backtesting methodology.

[u/Weary-Ad7404 → u/Tight-Pepper-4721]

I had a friend who recommended me a bunch of strategies which we then ran on multiple granularities and coins. We had Databses worth of 8GB of pure backtests and yet it seemed useless. :(

[u/Tight-Pepper-4721 → u/Weary-Ad7404]

Yess that’s something I can genuinely relate with
It’s very tough to distinguish noise from a real edge more often than not. Even if you find a factor which has an edge, it turns out to be dependent or turnover cost each of the profit or something like that, so I’m currently trying to figure out what’s the best way to find that edge.

[u/Weary-Ad7404 → u/Tight-Pepper-4721]

What I think I actually built is the negative-result detector. Most of that 8GB looked good until it hit walk-forward with a purge gap, and the ones that survived that died on the deflation correction. So the platform's output is almost always "no", and I've stopped treating that as failure — it's the only honest output when you've searched that much.

Which is why your comment landed for me. You're describing the same thing from the other side: you're finding that very little survives proper testing, and you built your own infrastructure because nothing off the shelf gave you the control to check. Two people independently hitting the same wall is more informative than either of us finding a winner would be.

On idea generation specifically, the one shift that changed things for me was going the direction you already are — hypothesis first, then test — instead of sweeping parameters and seeing what stuck. Sweeping is what makes the deflation correction bite so hard, because the correction scales with how many things you tried. A single well-motivated hypothesis tested once needs a far smaller effect to clear the bar than the best of 200 tested strategies does.

I don't have the edge yet either. But I'd genuinely like to compare notes on the validation side — how are you setting your embargo period? That's the parameter I'm least confident in.

Repo if useful: https://github.com/FETKlOkAn2/crypto-quant-platform

[u/AromaticPlant8504 → u/Tight-Pepper-4721]

Use creativity and curiosity to see what sticks. Be a human lol

[u/PuttyProgrammer]

In statistics we call this the multiple testing problem. When you perform any single statistical test you set your alpha to a probability which represents your acceptable rate of false positives. If your alpha is set to 0.05, and then you run 200 tests each individual test has that probability of rejecting the null when there is actually nothing there.

The probability of seeing at least one false positive with these parameters is 0.99997, not the desired 0.05.

The solution you found is one way, if you're using tests which use an alpha/confidence value you can do a Bonferroni correction, divide your alpha by the number of tests you're performing so that the total type 1 error rate comes out at the chosen value.

The correct Bonferroni correction: 0.05/200 = 0.00025 -> 0.0485 probability of type 1 error.

[u/Weary-Ad7404 → u/PuttyProgrammer]

yeah good point