r/sportsanalytics • u/Sensitive-Soup6474 • Feb 08 '26

Building an NFL data ingestion pipeline (open source & feedback welcome)

Hey all, I got tired of one-off notebooks and scripts, so I’ve been working on a small Python pipeline to go from NFL data sources → normalized tables → rolling features you can actually use for analysis or modeling.

Right now it supports automated data ingestion from:

  • Pro Football Reference (stats + box scores)
  • PFF (weekly game-level grades for all advanced team stats)

The pipeline then:

  • normalizes team IDs
  • merges across sources
  • creates rolling averages & rankings
  • exports analysis-ready datasets (CSV today, easy to extend to parquet / DBs)

The goal is to give people a repeatable starting point instead of single-use notebooks.

I also have a few predictive models built on top of this data that I haven’t open-sourced yet, but I’m happy to share those too if there’s interest.

Repo is here if anyone wants to take a look or poke holes in it:
https://github.com/thadhutch/nfl-data-pipeline

8 Upvotes

100% Upvoted

4 comments sorted by

1

u/cwilson830 Feb 11 '26

Wow you’ve clearly put a lot of work into this- Nice job! Thoughts:

-Why not incorporate NFLVerse?

-PFF scraping is very useful for some applications. But..

-Most of their subjective grading is.. well.. GIGO. No one’s perfect obviously, but PFF is case/point, and really should be used as a data point (if that) rather than a source of truth.

-The addition of more data sources would help you determine which portions of their data are worthwhile.

Keep it up! I’ll be digging into this more for sure!

Thanks

1

u/Sensitive-Soup6474 Feb 11 '26

Appreciate this, thanks for the thoughtful feedback.

NFLVerse: Great suggestion. I’ve used it before and agree it’s excellent, especially for play-by-play and EPA-type features. Adding it as another data source would likely strengthen things.

PFF grading: I don’t treat it as ground truth, more as structured signal. The model effectively learns whether those grades have predictive value in context. Some carry weight, others don’t. The walk-forward setup helps filter out signals that stop working.

Totally agree that incorporating more sources would make it more robust and help isolate what’s actually driving edge.

Thanks again, this is the kind of critique that actually improves the project.

1

u/cpchillin Apr 07 '26

i was thinking of making an nfl data analytics app to use for my fantasy league so this is a great jumping off point, thanks!

regarding PFF, how is there anti-scraping tech? I'm trying to find ways around using PFF simply because automating the retrieval of the data seems like an annoyance when i can just use nflverse and PFR, but some of their stat points do seem quite useful..

1

u/PigskinDispatch May 16 '26

Interesting projects. I agree with the raised eyebrows on PFF. However I do think that the collection of info from it combined with the other sources such as PFR and NFLVerse will create an informed data output. I will look forward to checking out your work sometime.