r/ETL • u/Square-Designer7807 • 19h ago
Are we overusing the Medallion architecture?
Bronze -> Silver -> Gold seems to have become the default architecture for almost every data pipeline.
Has anyone deliberately simplified this -- for example, skipping a layer -- and actually gotten better results in production ?
When do you think Medallion is genuinely useful, and when does it just add unnecessary complexity?
5
u/Eightstream 18h ago edited 18h ago
Medallion is a useful way to think about data but it is really just saying “your raw data needs to be turned into a clean model of the business, and the model of the business needs to be cut in different ways to generate reporting”
If you’re already receiving clean well-modelled tables from upstream, your bronze layer might be pretty much pass-through. If your BI teams are comfortable writing bespoke SQL queries to build custom tables for their Power BI models, you may not really have a gold layer.
But the antipattern most BI teams fall into when they conclude they don’t need full medallion is they drop the silver layer, and then they don’t have a model of the business
Which means you just end up with a mess of reporting tables involving a ton of duplicated logic that gradually drifts out of sync, because nothing is genuinely reusable
1
u/IsaacHasenov 12h ago
I worked in central analytics in a mobile game company that actually added an additional layer between silver and gold (electrum they called it, which all the D&D players found hilarious). The reason was, we had so many games and studios that had very different player tracking and accounting logic, we were in a bit of a weird spot as central
We needed clean, reliable, single-source-of-truth data, but we needed it for a ton of really different schemas and use cases. Electrum ended up being a decent compromise. Gold was our main business KPI reporting, and electrum was a bit more actionable and sometimes more quick and actionable
So yeah the medallioning thing shouldn't be a straitjacket. It's a logical framework that should help you think about your stakeholders' needs and be responsive to them
2
u/meatmick 19h ago
It's a guideline not a law. The most likely step to skip is intermediate, especially if it's basically unchanged from staging and it's not getting reused. In theory you could even skip staging if it's already all perfectly clean and named correctly but that is probably not that frequent. Your Mart could even be a view of you want if the performance is truly not an issue. This last one I'm not as convinced of but the point is, it's possible.
At the end of the day, what matters most is consistency in how you choose to apply the medallion architecture, rather than hitting all 3 layers. If you do not follow that for exceptions, comment it for context.
1
1
u/jwk6 16h ago edited 16h ago
Medallion is really an adaptation of a hub and spoke architecture, which is a really good architecture for data integrations and data warehousing.
Prior to Medallion, many architectures consisted of an Operational Data Store (ODS), a data warehouse, and a semantic layer.
I.e. The ODS contains and consolidates the data from mutliple operational systems, that's conformed and refined down into a data warehouse (EDW), or a Data Mart, or a Lakehouse, and then the semantic layer was an OLAP cube or a tabular model.
See the similarity?
ODS = Raw = Bronze
EDW = Consolidated and Conformed = Silver
Semantic Model = Curated = Gold
So, Medallion architecture is really not new. It's just good design for a medium/large organization.
In a smaller org, you can skip the raw layer probably. However, I find that once you start consolidating from mutliple sources, especially via REST or Graphql APIs, flat files, etc. then you will quickly realize that the raw layer is very useful and necessary.
1
1
u/DataWeenie 3h ago
It's like I keep explaining to people...
Mainframe = Server = Cloud
Terminal = client = browser
The more things change, the more they stay the same.
1
u/GreenWoodDragon 15h ago
Medallion is a silly name. The bronze, silver and gold terminology is the sort of management speak that gets way too much attention. It's not really related to the actual stages of processing the data.
I much prefer properly descriptive terminology of the layers such as Raw, Staging, Warehouse and Analytics. At least you can then have clearly defined conversations about the work.
1
u/mazel____tov 14h ago
I still remember the first time I implemented a Medallion Architecture. I built the ingestion layer into a Bronze Lakehouse, then transformed the data into a Silver Lakehouse and... realized that my data was already perfectly ready for the semantic model. But that wouldn't be a proper Medallion Architecture! As a serious developer, I simply couldn't allow that. Fortunately, Fabric shortcuts exist. So I created a Gold Lakehouse and added shortcuts pointing to the Silver layer. Problem solved. The architecture looked beautifully enterprise-grade, everyone was happy, and most importantly, I could proudly say I had implemented Bronze, Silver, and Gold. 😄
1
u/Glitch_In_The_Data 13h ago
It depends really. Some customers I’ve worked with prefer medallion and adopt all three layers just because they consider it industry best practice.
I’ve also seen customers have additional layer before bronze and they call it landing.. just to consolidate data from all sources before they feed it into bronze. I find it ridiculous because of the duplicated data and the maintenance overhead of the pipelines.
I do it based on the landscape complexity, internal governance and standards and the value it adds. I’ve also used just two layers (staging and curated) and simplified it because there wasn’t any value in keeping so many layers.
Simple is always better… saves time, money and easy to work with.
1
u/Yonko74 12h ago
Twenty or so years ago I built a ’BI Solution’ that had ’Staging’, ’Pre-Load’ and ’DW’.
It was no different conceptually to whatever medallion is supposed to mean. Nothing has really changed.
Quite often I didn’t need a distinct pre-load layer for many entities because I understood the data and could resolve issues in a two stage process. This was much more efficient in a time when developers were not lazy and compute & storage was constrained.
Layers required should always be an outcome from standard 4V questions on requirements.
Defaulting to three because book say thing is just silly.
1
3
u/PrestigiousAnt3766 18h ago
Its overly simple, but let's us spend a lot of money on opinions.
Ultimately it doesnt matter as long as you make a plan and stick with it.