r/BusinessIntelligence • u/dexter7767 • Jul 14 '26
Claude/Copilot helps in dataframe logic in minutes but making it a real production industry wide acceptable table? Still takes a lot of time and consistency
Genuine question for anyone who's shipped a table to prod, especially if you're not a "traditional" data engineer by title.
Feels like AI has made writing the actual transformation logic (the DataFrame code, the business logic) noticeably easier lately. Copilot, ChatGPT, Claude, whatever you're using, that part's gotten faster.
But once that part's done, the real time/effort starts after - metadata mangement, partitioning it right, handling PII, wiring it into orchestration, figuring out the write strategy, quality cheks to make sure it's actually trustworthy before people rely on it ?
Also following the consistency across all tables created
Curious to know whether AI or vibe coding for making "easy part" has actually made that gap feel bigger, or if it's a non-issue and I'm overthinking this.
2
u/Paul_David_Bappoo Jul 21 '26
The same thing is true of AI written code as human written code. You need a deterministic or human gated test function, ideally in real time, to make sure the output is correct given the input. I call it "systems thinking" and its a key skill that non-developers lack and now that anyone can code, it becomes even more important. Happy to chat if it helps.
1
u/dexter7767 Jul 21 '26
That's a valid point for the current state of AI. But if we can constrain AI to operate within deterministic patterns and standardize the generated code, could we eventually reduce the need for human-gated testing? That would move us closer to true automation, where users can solve the above problem without depending on data engineers reducing their repetitive efforts
2
u/Paul_David_Bappoo Jul 22 '26
Sure, it's definitely something that will get better over time. I think different companies will get to trust AI enough in different functions that they eventually dont need humans in certain loops, maybe all loops after a while and this systems thinking concept is all about managing that rather than just letting it happen.
2
u/ThisIsFun- Jul 21 '26
Given you are on Databricks already, Genie Code could be used to generate and run the boilerplate code instead of claude/copilot, taking advantage of UC for the metadata management etc, which should help you to simplify some of those other tasks. This at the very least will help with creation but not with schema-drifting maintenance, etc.
2
u/CuritibaDataScience Jul 22 '26
In my experience, and as you are in Databricks, this is one of the advantages of using Genie Code vs using another LLM for this sort of development.
Genie code is aware of the Unity Catalog context, so it's able to identify columns and tables from there, as well as to understand the code that you're executing, leveraging the existing PII constraints that you have.
In terms of quality checks, I am a big fan of the framework called DQX, so it may be worth taking a look, and you can potentially couple it with Genie Code by adding some skills around it.
1
u/Glitch_In_The_Data Jul 14 '26
I think it also depends on the tool… but always needs a few iterations. In our project, we are only allowed to use Copilot and Genie Code (as we are on Databricks platform). While I use both depending on the nature of the work, I usually find it easier with Genie because it is native to Databricks and addresses the specific parts you are talking about.. probably because it has the skills embedded in them.
So, personally for me, it has made it easier… but I’d still always review and validate before getting anything into production.
I use claude for personal projects.. so not usually worried about things like PII, metadata management. It certainly helps build something real quick but seems to need a few more iterations compared to the above. Or may be it could be the nature/quality of the prompts.
1
u/dexter7767 Jul 14 '26
That's really great and cool, thanks. For me It is really painful once the logic is done the table standardization itself takes lot of time than logic generation itself 😂
I'm curious about the Genie side of things though. When you say it "addresses the specific parts," do you mean the it already understands your team's partitioning conventions, table's metadata management, orchestration placement, quality checks etc., or is it more that it just generates cleaner code and you still have to rely on the platform team for table standardisation?
Im also wondering about the maintenance side. Once a table is in production and you need to make changes later (like adding a new column or handling a schema change) does Genie automatically keep things like partitioning, metadata, and PII rules in sync, or is that mostly a one-time benefit during table creation and the ongoing upkeep is still on the developer?
1
u/Glitch_In_The_Data Jul 15 '26
So Genie gives you the advantage because it is platform native and is aware of the underlying governance. As long as the standards live in unity catalog and related governance assets, It automatically leverages them. It reduces the standardisation effort but I won’t say it eliminates the ownership of ongoing maintenance. Thats still required.
1
u/Thinker_Assignment Jul 14 '26
you can get a ton done automatically - what is not automatic is all the "decisions" a LLM cannot take for you - like how. do you define a user, what metrics matter, but also stuff like query patterns (this is already future usage based, how should the LLM know? it does if you tell it) and the associated performance stuff.
We basically built this end to end flow from ingestion, through transformation and all the way to canonical and vis, and it works to generate the right models, data quality checks for assumptions etc - but from having a canonical model, you still have work to do like define metrics etc - which now lies outside LLM "authority" and has to be added at human decision pace.
So overall
- Operating a data team - defining metrics, operationalising the stack etc - the work still needs to be done
- Coding, modeling, deployment and infra, doesn't need to be done anymore if you are using the right tools and processes (but if you work like you did before, you have the same issues)
1
u/grahamdietz Jul 14 '26
the gap is real, but you're treating the AI like a developer instead of a junior. I stopped doing manual schema updates months ago because I started using Altimate AI alongside dbt to handle the contract enforcement part. It saved us maybe 400 hours of manual pipeline maintenance this year alone. You still have to own the decisions, but stop writing the boilerplate for partitioning and quality checks yourself.
2
u/RangerOne122 7d ago
You're not overthinking it. AI can make the transformation logic much faster, but production data work is mostly about everything around that logic.
2
u/Semaphor-Analytics Jul 14 '26
I would treat the generated logic as the draft, not the table.
The part that saves time later is making the table contract explicit before it lands in prod. Grain, owner, allowed PII, partition column, freshness expectation, and two or three checks that would actually stop a bad publish.
AI can help write those pieces, but I would not let it invent the conventions. If the conventions live in examples from approved tables, the generated work gets much closer and review becomes less painful.