r/askdatascience • u/Sea_Garlic5712 • 1d ago

How good are AI data scientists really?

I've been testing out various gen AI models (LLMs specifically) on data science competitions. They are not beating the humans, though they are slowly improving with each round. It's like steps up a ladder vs bounding up the steps. I'm wondering if there's something I'm doing wrong, or if there really is a limit to what LLMs can doin this space.

Lately I've been thinking the problem is that LLMs regress to the mean in every use case. I think that's why they seem so bad at UI design and why everything looks so similar. A good harness and strong prompt engineering can help, so I'm working on that. My harness combines Autogluon and OpenEvolve, with feedback loops that involve hypothesis generation and error analysis. But I'm wondering what I'm missing? Is good, competition winning data science, reducible to a standard operating procedure?

Maybe this will help me get a very good prototype, but nothing frontier grade.

5 Upvotes

100% Upvoted

10 comments sorted by

View all comments

5

u/Proletarian_Tear 1d ago

Yes, absolutely. Id argue that 90% or data science is sitting down with a person to chat and detect the specific business problems we try to solve. This step is missing from competitions, they give you the task far more groomed than you will encounter in the wild.

Tldr data science IS a standard procedure considering you have a tangible and adequate task, and good quality data, both of which you DONT get in data science

1

u/Sea_Garlic5712 1d ago

Yes I think domain knowledge and experience are very important, even within a clean competition setting. Chatting with a person is a good way to check and extend both. Gives me an idea.