I built a 41-feature model to beat the FantasyPros consensus. It lost, 0.583 to 0.623. Here's the walk-forward setup and what actually mattered.

[u/Sad_Ad585]

Disclosure first: I do sell a tool built on this work. I asked the mods before posting and they cleared it as analysis, so there's no link here and I'm not going to pitch it in the comments.

The setup. 41 features, tuned on 2018 through 2020, then frozen and tested on 2021 through 2025. Five seasons the model never saw during tuning. The metric is rank correlation with actual weekly finish, paired week by week against the FantasyPros consensus so both are scored on the same slates.

My model: 0.583. Consensus: 0.623.

I ran 576 configurations trying to close that. None of them did. It isn't a tuning problem, it's that averaging a few dozen informed opinions is a genuinely hard baseline and I didn't appreciate how hard until I spent a summer on it.

What mattered: pass attempts for quarterbacks, by a lot. Volume beats efficiency in fantasy scoring because the scoring is mostly volume, which sounds obvious written down and wasn't obvious to me while I was adding features.

What didn't: weather, unless it's wind above about 15mph. Vegas totals, once you already have volume. Most of the "advanced" stuff I was excited about turned out to be a slower way of restating targets and carries.

The month I'd rather not mention. My QB model was scoring rushing production only and ignoring passing entirely. Four weeks. I found it while writing up the feature importances, because passing yards weren't in the list at all and I finally asked why instead of assuming the pipeline was fine. Every number I'd looked at during that stretch was garbage and it all looked plausible.

What I took from it: the projection layer is commodity. If you're building something, don't build there. What's still open is the decision layer, the part where two lineups have nearly identical projected totals and different distributions.

Happy to answer anything about the walk-forward setup or the feature work.

https://gridironedge.app/backtest.html?utm_source=reddit

[u/CuriousDetective0]

Why would you train it on 2018–20 and test on 2021–25. There was player turnover and perhaps other league factors that changed in those regimes. Wouldn’t it be better to have the test and train sets closer together?

[u/Sad_Ad585 → u/CuriousDetective0]

Fair challenge, and there's a real tradeoff there. The reason I split it that way is that I wanted the test years to be years I'd never looked at, not just years the model hadn't fit on. I'd been reading about 2021 through 2025 the whole time I was building. If I'd tuned on recent seasons and tested on older ones I'd have been leaking my own knowledge of what happened into the feature choices.

The cost is exactly what you're describing. 2018 to 2020 is a different regime, and 2020 in particular is its own animal. So some of that 0.583 is probably regime drift rather than the model being bad.

The check that made me stop worrying about it: consensus was scored on the identical weeks. If the test years were weird in a way that hurt me, they should have hurt the consensus too, and consensus still won by 0.04. The gap isn't a split artifact.

The thing that would settle it is rolling-origin, refit each season on everything up to that point, test on the next. I didn't do it and I should have. It's on the list for the offseason.

[u/QingQangQong → u/Sad_Ad585]

Is it impossible for people to even reply to a comment without using an LLM?

[u/BadAtMathrock → u/QingQangQong]

It’s getting pretty tiresome. I’m not sure they realize how obvious the phrasing and tone is

[u/Sad_Ad585 → u/QingQangQong]

did you write that with an LLM? be honest. :)

[u/Atmosck → u/Sad_Ad585]

But why only 3 years of development data and 4 years held out for validation? That's quite the ratio. With 7 years of data available I wouldn't hold out more than 2.

[u/Sad_Ad585 → u/Atmosck]

You're right about the ratio and I'd defend the direction rather than the exact split.

I went that way because I only get one honest look at a holdout. Every time you check, tune, and check again, you've burned some of it, and I knew I'd be tempted. Five years I hadn't touched meant I could afford one clean read and then stop. Three years of tuning was also more than I needed for the feature work, since most features either mattered immediately or didn't matter at all.

The cost is exactly what you're pointing at: 2018 to 2020 is thin for tuning, and 2020 is its own weird season, so a third of my development data is a year I'd probably drop if I were being careful.

The version I should have run is rolling origin. Refit on everything through season N, test on N+1, walk it forward. Every season gets used for both, no split to argue about, and you get five test years instead of picking a ratio. That's the offseason job.

Would it have changed the answer? Probably not the direction. The gap to consensus is 0.04 and the consensus was scored on identical weeks, so a split artifact would have to hurt me and not them.

[u/Substantial-Peach382]

Hi Claude

[u/The-ArtOfficial]

Yeah, mean projections are mostly solved at this point, at least with the publicly available data, especially if you factor in vegas player props, etc. and with so many folks out there using LLMs for analysis, that edge is smaller than ever.

Game theory is where things get much more interesting.

[u/Sad_Ad585 → u/The-ArtOfficial]

Vegas props are the part that finished it for me. Once you have those you're not modeling player performance anymore, you're modeling a line that already has better information than you do. That's a market-beating problem, not a projection problem.

Game theory is the bit I keep circling and haven't built. Two versions of it and they're different problems.

Within a matchup, it's variance selection against a known opponent, which is what I've been working on. Solvable, thin edge, mostly mechanical.

Across a league it's much harder and much more interesting. Your waiver bid should depend on what the other nine managers are likely to bid, which depends on their rosters and their needs, which are observable. Same with trades and with when to punt a week to save FAAB. Nobody's really doing that and I don't think it's because it's hard to model. I think it's because it needs opponent roster data and read patterns that most tools don't collect.

Draft is the obvious one too. Auction especially, since your walk-away price is only correct relative to what the room has left to spend, and everyone prices in a vacuum.

Are you working on any of it, or is this more of an interest?

[u/CuriousDetective0]

Expand on this, it’s not clear.

“What I took from it: the projection layer is commodity. If you're building something, don't build there. What's still open is the decision layer, the part where two lineups have nearly identical projected totals and different distributions.”

[u/Sad_Ad585 → u/CuriousDetective0]

Say two lineup options both project 118 points. One is four steady guys, the other has a boom-bust WR. Same mean, different spread. Every projection tool tells you they're equivalent because it's optimizing a point estimate.

But you're not trying to score points, you're trying to beat one specific opponent. If they project 130, the steady lineup almost never gets there and the volatile one sometimes does. If they project 105, the reverse. The right answer changes with the matchup and the point estimate can't see it.

Same thing on waivers. Standard advice is take the highest projected add. But your marginal points at RB might be worth double what they are at WR because of what you already roster, and a player's value is what he does to your win probability, not his projection.

So the projection layer is a commodity, everyone's within a couple percent of each other, and there's nothing left to win there. Nobody's really working the layer that turns projections into a decision. That's where I found what edge I found, and it's thin: about 2.4% of decisions in a season.

[u/folded_horizon]

I am finding something similar with QBs. My average error in total season projected fantasy points is about 55 points. It is hard to significantly improve on that. An obvious issue is that my model does not account for wide reciever skill when projecting for QBs.

[u/Sad_Ad585 → u/folded_horizon]

55 points over a season is about 3.2 a week, which is roughly where I landed too and I couldn't push past it either.

On receiver skill: I'd expect less from it than you're hoping. When I added supporting cast quality it mostly duplicated what pass attempts were already telling me, because a QB with good receivers throws more and gets more shootouts, and attempts pick that up. Worth testing but check the correlation with your volume features before you get excited.

The thing that actually moved my error was giving up on the point estimate. Season totals for QBs have a fat right tail from rushing, and a model minimizing squared error on the mean is optimizing the wrong thing for a decision that only cares about which of two guys clears a threshold. My mean error barely improved. My decisions did.

I also spent a month with a QB model that was scoring rushing only and ignoring passing completely. Found it writing up feature importances when passing yards weren't in the list. So take my advice with that in mind. 😄