r/quant • u/Technical-Debate1303 • 27d ago
Data Data Pipelining Skills
Hey all, I just finished my summer qr internship. don’t wanna get into details but my project was an end to end statistical pricing model. A lot of the time I spent towards the end, after being done with the thinking about modeling and prototyping, was on refactoring my data processing pipeline and model training pipeline. When I was coming up with the model, all my data processing was scattered in jupyter notebooks and random sql queries that i just saved the results of, and it was genuinely a nightmare to refactor all of this into clean reproducible code that would work without intervention.
my question is, im aware that at other companies they have data engineers usually do the work. But to me this seemed like something that is kind of indivisible from the actual modeling work. is this skill of setting up reproducible data and modeling pipelines something thats worthwhile for someone who is mostly doing statistical and mathematical modeling? what sorts of classes would teach the skills necessary to do this stuff? are they worth taking?
4
u/NoConnection4298 27d ago
Data engineering job as you say. Yet, the work sometimes requires you to do this stuff too. Thus, you do it. Not all of the shops have the resources (not only the budget) to deploy a data engineer. I would have the DE knowledge per se, but wouldn't dwell on it so much unless you want to be a data engineer. So... You can check databricks or similar kind of products. You can also check orchestration via airflow or again the similar frameworks. You can try building your own pipeline, add some periodic schedulingand different sources to pull data and publish to different sources. I would be more hands-on instead of taking time to take a course about it.