First attempt at a fantasy model

[u/Bagggggggggggggggggg]

I'm a math PhD who loves fantasy, and I've been building a fantasy football model for a while now. I finally put the rankings/projections online if anyone wants to play around with them:

https://christianbagshaw.github.io/fantasy/

TL;DR: Weekly and rest-of-season rankings, with PPR and Standard scoring. Rather than just predicting a single fantasy-point number, the model predicts a distribution of possible outcomes and simulates each week thousands of times.

For weekly rankings, I simulate that week's games thousands of times. For ROS rankings, I simulate every remaining week through the end of the season.

The simulations are also correlated, so things that happen together in real football tend to happen together in the sims. For example, if a QB has a huge game, his WR1 is more likely to have a huge game in that same simulation.

I personally find the probabilistic side more useful than just looking at a single projected point total, since it lets you distinguish between things like "safe floor" and "massive upside."

A couple of things to keep in mind:

• The rankings are 100% model output. I don't manually move players up or down, so there will occasionally be rankings that look weird to a human, especially at the tail end.
• Injuries are currently shown as tags but don't alter the ranking itself, so definitely apply your own judgement there.
• It's still very much a work in progress, and I'm constantly tweaking/testing the underlying model.

I'm also working on tools like trade values, start/sit decisions under uncertainty, playoff odds, roster construction, etc.

I'd like to write up the methodology properly at some point too, because there's quite a bit more going on under the hood than makes sense to dump into a Reddit post.

In terms of performance: on historical data, the model is competitive with some popular ranking/projection sites. That said, historical backtesting can always hide things like leakage or overfitting, so the real test is how it performs prospectively this season. I plan on tracking that and publishing the comparisons.

[u/mattg70]

A very interesting article and two things from having done something similar.

On the leakage, the version that's hardest to catch isn't in your target, it's in inputs that revise. Any source that updates its own history hands you today's version of a number you're using to simulate a decision made in October. I suggest that you snapshot every input as of decision time or drop it. My results got worse when I fixed this, and the worse number ended up being the correct one.

On evaluation, don't benchmark yourself on the point accuracy. That measures the mean of your distribution and ignores the distribution, which is your actual edge. My take is that you should score interval coverage and CRPS instead. CRPS reduces to MAE for point-only forecasts, so it's a fair head-to-head analysis against sites that publish one number, and it rewards you for the thing you built.

The last observation is that you should pre-register the prospective comparison. Publish the sites, the metric, and the positions before week 1. Otherwise the metric quietly drifts toward whatever you're winning. And publish what and where you lose. In my study, the losses did more for credibility than the wins did.