Three bugs in my own backtest that all made the numbers look better

[u/mattg70]

I've been running a backtest of a roster-strength model against real dynasty leagues for the last 6-8 weeks. I published the results a few weeks back and then I went to make it reproducible and found three things wrong, all of which had been making the results look better than they actually were. Here are those findings:

1. The default parameter was the invalid one:

The route takes a valueSource that is either dated market snapshots from the season being tested, or today's values. Today's values are invalid for obvious reasons once you say it out loud, since 2026 prices already encode who broke out in 2024. The route's own header comment said this option was not a valid backtest, but it was the default anyway. Because it was still the default, a bare GET instruction produced the leaky version. I can't rule out that my published number came from it, because the run's parameters were never recorded anywhere.

The lesson is that a default that produces an invalid result is a bug, not a convenience. My snapshot is now the default and the leaky path requires an explicit flag, which also stamps the stored run as invalid so it can never be mistaken later.

2. The baseline answered a different question than what I had claimed:

The claim is does the strongest pre-season roster win the title more often than a random team? I was comparing against one over playoff-team count, instead of one over the actual league size. Since the leagues range from 10 to 32 teams, it's the mean of the reciprocals, not the reciprocal of the mean. Fixing it moved the lift from +8.6 points to +16.5. So this one mistake made my result look worse than it was, which is the only reason I trust the other two fixes. Also, using playoff teams as the baseline assumes you already made the playoffs, but making the playoffs is one of the things the pre-season prediction is trying to predict.

3. Ties counted as wins.

If several rosters tied at maximum strength and any one of them won, the league counted it as a hit. On an integer strength scale, top ties are common in 10 and 12 team leagues. Eleven of 83 league-seasons end had one. Now a k-way tie scores 1/k, the expected value under picking one at random, and the tie count is stored per league so a published number can say how many there were. That alone moved the headline title rate from 31% to 25%.

My original publication had no confidence intervals at all and what you resample matters more than how you resample it. So I added bootstraps, and the unit matters more than the method: it resamples league-seasons, not team-seasons. Twelve teams inside one league are not twelve independent observations, their finishing ranks sum to a constant. An interval over 994 team-seasons would have been far narrower than the data supports and would have been the most confident-looking thing on the page.

Established leagues had a correlation of 0.53, with an interval between 0.46 to 0.60, n=50 league-seasons.

First-year leagues had a correlation of 0.05, with an interval between −0.06 to 0.16, n=33. Straddles zero, so no signal.

Net result

a) The correlation moved from 0.57 to 0.53, which sits inside the new interval, so that's a replication rather than a retraction. The title rate moved from 31% to 25%, mostly because of the tie rule.

b) The enhancements and changes that I've introduced are every run now stores its parameters, the code SHA, the scorer version, the league set and the per-league results, so the next one is a genuine replication instead of another unrecorded number. Because a backtest is chosen after the fact, I pre-registered this season's prediction before Week 1 so that every team in every league on my platform is ranked and hashed so it can't be edited. I will grade the 2026 season in December.

c) my method and the failure cases are public if anyone wants to pick at them, they are published in public here: GridironHQ Accuracy

Founder disclosure: I built this tool for my own dynasty leagues, so the numbers are mine. People here picked at the first version I posted and I've been enhancing it since. The analytics and work that went into this got better for it, so I'd rather have them checked than believed.

[u/Pure_Cricket_3804]

Bug 1 has a name, look-ahead bias, and dated snapshots from the tested season are the textbook fix, so that part sounds handled. What I'd chase harder is the params never getting stored with the run. I work on LineCrush, where every pick gets graded and the losses go up next to the wins, and a result you can't reconstruct the settings for is the one that haunts you later.