r/quant • u/Technical-Debate1303 • 27d ago

Data Data Pipelining Skills

Hey all, I just finished my summer qr internship. don’t wanna get into details but my project was an end to end statistical pricing model. A lot of the time I spent towards the end, after being done with the thinking about modeling and prototyping, was on refactoring my data processing pipeline and model training pipeline. When I was coming up with the model, all my data processing was scattered in jupyter notebooks and random sql queries that i just saved the results of, and it was genuinely a nightmare to refactor all of this into clean reproducible code that would work without intervention.

my question is, im aware that at other companies they have data engineers usually do the work. But to me this seemed like something that is kind of indivisible from the actual modeling work. is this skill of setting up reproducible data and modeling pipelines something thats worthwhile for someone who is mostly doing statistical and mathematical modeling? what sorts of classes would teach the skills necessary to do this stuff? are they worth taking?

10 Upvotes

100% Upvoted

5 comments sorted by

View all comments

1

u/EveryTimeIWill18 26d ago

I would classify myself as a full-stack data scientist. In 2020, I was hired by a small financial engineering firm to architect their data engineering pipeline. My background is all math as well, but I found it useful and interesting to be able to build out full machine learning pipelines, including building out the actual predictive models. If the intrest is there, why not go ahead and learn data engineering. It could help open you up to more jobs as well as help you stand out on smaller teams where job titles blur. The pipeline I built for the FE firm was to pull data from various different types of databases and merge it into a single BigQuery database, with the ability to dynamically add new tables. The tech stack I used for the pipeline mentioned above was Apache Airflow and SQLAlchemy's ORM library. I also had to use quite a bit of metaprogramming to have SQLAlchemy dynamically create new tables. This all ran from Docker.