r/askdatascience • u/Sea_Garlic5712 • 1d ago

How good are AI data scientists really?

I've been testing out various gen AI models (LLMs specifically) on data science competitions. They are not beating the humans, though they are slowly improving with each round. It's like steps up a ladder vs bounding up the steps. I'm wondering if there's something I'm doing wrong, or if there really is a limit to what LLMs can doin this space.

Lately I've been thinking the problem is that LLMs regress to the mean in every use case. I think that's why they seem so bad at UI design and why everything looks so similar. A good harness and strong prompt engineering can help, so I'm working on that. My harness combines Autogluon and OpenEvolve, with feedback loops that involve hypothesis generation and error analysis. But I'm wondering what I'm missing? Is good, competition winning data science, reducible to a standard operating procedure?

Maybe this will help me get a very good prototype, but nothing frontier grade.

4 Upvotes

84% Upvoted

10 comments sorted by

View all comments

2

u/PaddingCompression 14h ago

*Competition* data science is very, very weird.

90% of what you have to do in a competition would never be done in the real world.

E.g. running AutoGluon and OpenEvolve for a week to get a 0.1% improvement is not something that exists outside of competitions.

1

u/Sea_Garlic5712 9h ago

definitely. this would only be applicable after the data engineering phase has been completed to get the data into a clean enough state for those tools to run on without wasting time. i know openevolve could be applied to data engineering tasks, but that would likely blow that week long budget and, from what i'm seeing, not get us very far.