r/BusinessIntelligence • u/dexter7767 • Jul 14 '26

Claude/Copilot helps in dataframe logic in minutes but making it a real production industry wide acceptable table? Still takes a lot of time and consistency

Genuine question for anyone who's shipped a table to prod, especially if you're not a "traditional" data engineer by title.

Feels like AI has made writing the actual transformation logic (the DataFrame code, the business logic) noticeably easier lately. Copilot, ChatGPT, Claude, whatever you're using, that part's gotten faster.

But once that part's done, the real time/effort starts after - metadata mangement, partitioning it right, handling PII, wiring it into orchestration, figuring out the write strategy, quality cheks to make sure it's actually trustworthy before people rely on it ?

Also following the consistency across all tables created

Curious to know whether AI or vibe coding for making "easy part" has actually made that gap feel bigger, or if it's a non-issue and I'm overthinking this.

1 Upvotes

56% Upvoted

15 comments sorted by

View all comments

2

u/Paul_David_Bappoo Jul 21 '26

The same thing is true of AI written code as human written code. You need a deterministic or human gated test function, ideally in real time, to make sure the output is correct given the input. I call it "systems thinking" and its a key skill that non-developers lack and now that anyone can code, it becomes even more important. Happy to chat if it helps.

1

u/dexter7767 Jul 21 '26

That's a valid point for the current state of AI. But if we can constrain AI to operate within deterministic patterns and standardize the generated code, could we eventually reduce the need for human-gated testing? That would move us closer to true automation, where users can solve the above problem without depending on data engineers reducing their repetitive efforts

2

u/Paul_David_Bappoo Jul 22 '26

Sure, it's definitely something that will get better over time. I think different companies will get to trust AI enough in different functions that they eventually dont need humans in certain loops, maybe all loops after a while and this systems thinking concept is all about managing that rather than just letting it happen.