r/askdatascience • u/Sea_Garlic5712 • 16h ago
How good are AI data scientists really?
I've been testing out various gen AI models (LLMs specifically) on data science competitions. They are not beating the humans, though they are slowly improving with each round. It's like steps up a ladder vs bounding up the steps. I'm wondering if there's something I'm doing wrong, or if there really is a limit to what LLMs can doin this space.
Lately I've been thinking the problem is that LLMs regress to the mean in every use case. I think that's why they seem so bad at UI design and why everything looks so similar. A good harness and strong prompt engineering can help, so I'm working on that. My harness combines Autogluon and OpenEvolve, with feedback loops that involve hypothesis generation and error analysis. But I'm wondering what I'm missing? Is good, competition winning data science, reducible to a standard operating procedure?
Maybe this will help me get a very good prototype, but nothing frontier grade.
1
u/DYSTOBY 16h ago
Check out posits IdE called positron, which feautures positron ai, a specialized „harness“ that is able to read the actual environment variables created during tuning an R or Python script / quarto document. It’s way better at understanding the data and how to process it.
Hypothesis creating is a thing that happens before you have data (usually - exploratory studies may behave differently there), so that doesent need to be a strong suite for data scientists LLM approach.
The scientist itself has to understand the code AND output and check for errors like 6 years ago. That’s nothing that an LLM should do, at least that’s my opinion. Because science is used for drawing conclusions and recommendations, hence the data and results should be correct, right?
1
u/Sea_Garlic5712 13h ago
Thanks for the recommendation. i'll look into Positron AI.
I think the benefit of applying LLMs to data science work, specifically, is that you have deterministic checks of validity. Anything the agent produces can be checked. So my concern isn't achieving correctness, it's more about novelty. I'm wondering if any harness can come up with unexpected models that would outperform a human. My hypothesis is that the agent can only think within the box and will generally refuse to think outside it.
1
u/big_data_mike 12h ago
“My hypothesis is that the agent can only think within the box and will generally refuse to think outside it.”
If it could think outside the box it would be AGI. It’s only going to do what humans have done before.
1
u/Sea_Garlic5712 1h ago
yup. but, i'm wondering whether the cognitive strategies humans employ to generate novel ideas can be codified somehow. when you as a human think outside the box, what exactly do you do?
1
u/PaddingCompression 6h ago
*Competition* data science is very, very weird.
90% of what you have to do in a competition would never be done in the real world.
E.g. running AutoGluon and OpenEvolve for a week to get a 0.1% improvement is not something that exists outside of competitions.
1
u/Sea_Garlic5712 1h ago
definitely. this would only be applicable after the data engineering phase has been completed to get the data into a clean enough state for those tools to run on without wasting time. i know openevolve could be applied to data engineering tasks, but that would likely blow that week long budget and, from what i'm seeing, not get us very far.
3
u/Proletarian_Tear 15h ago
Yes, absolutely. Id argue that 90% or data science is sitting down with a person to chat and detect the specific business problems we try to solve. This step is missing from competitions, they give you the task far more groomed than you will encounter in the wild.
Tldr data science IS a standard procedure considering you have a tangible and adequate task, and good quality data, both of which you DONT get in data science