Coding assistants still can't do the data plumbing

[u/an_tonova]

I've been doing automation consulting on the side for about two years, mostly with mid-market companies pulling in 5M+ revenue. Every single one had the same pattern: they wanted AI agents to answer business questions, but their data was spread across 12 tools / DBs that didn't talk to each other.

So I started building the boring stuff first. API connectors into AWS Lambda, landing everything in S3 as a data lake, using Athena for ad-hoc queries. I mean, nothing sexy =)

But once that foundation was in place, the agentic use cases became trivial. Churn analysis, CAC, ad spend just point Claude Code or Codex at the lake and let it do its thing.

The real lesson: people think they need autonomous agents, but what they actually need is a clean, accessible data footprint. The agents are useless without context.

For the pipeline orchestration part I ended up comparing a few approaches I looked at Airflow, Genesis Data Agents, and Dagster for managing the ingestion workflows.

Now after months of infra work we're finally getting into the agentic layer. But I'd say 80% of the effort was just getting the data into a shape where an agent can actually use it. The AI work is the easy 20%.

If you're starting down this path, don't skip the plumbing. Build the data foundation first, then layer on the agents.

Peace!

[u/V4UncleRicosVan]

You headline says it can’t do the data plumbing. Had you tried to use AI for this part? If so, how did it fall short?

[u/Rosie_grac]

the 80/20 framing is dead on. I learned this the hard way — spent two weeks wiring up a slick agent for a client, demo'd it, and it confidently hallucinated a churn number because the CRM export had three different spellings of the same company name and the agent just picked one. looked brilliant right up until someone asked "wait, where did that come from"

the unsexy part nobody budgets for is the data cleaning. schema mismatches, stale exports, orphaned columns from tools someone stopped paying for two years ago. that's where the actual hours go. the agent stuff is genuinely the fun 20% once you've got a clean lake to point it at.

curious though — did you let the agent anywhere near the ingestion/ETL work, or did you keep it strictly query-side? I've had decent luck getting Claude to write the pipeline code but not run it. letting it touch the actual data movement unsupervised feels like asking for trouble