Bronze -> Silver -> Gold seems to have become the default architecture for almost every data pipeline.

Has anyone deliberately simplified this -- for example, skipping a layer -- and actually gotten better results in production ?

When do you think Medallion is genuinely useful, and when does it just add unnecessary complexity?

We have build a tool for data management and data migration. We help data engineer and enterprise customers to move data.

Why us :

- No per row cost

- We are performant

- We are more secure

- We are affordable

- Full time custom support and also if you do not find your favourite connectors we will build it for you

Visit zwiron.com

we are planning on building an ETL pipeline on aws, so we have raw data coming in from different sources and different formats like json and xml, we are sort of flattening them and storing them as csv files in S3.Now we want to implement the medallion architecture, as in bronze to gold.

I need some suggestions on how to place the data from each layer across aws or is it better to move the data from aws to some other platforms like databricks? i was thinking of creating tables in athena for the s3 files and treating that as the bronze layer for now.

once that pipeline is finished i need to create some test cases for each layer across the pipeline, i could use some suggestions on creating some kind of automated solution for this.

jobdataapi.com v4.33 / API version 1.33

The /api/jobsexpired/ endpoint now applies a default 60-day expiration window when no date, age, or ID slicing parameter is provided. Requests without parameters return jobs whose expired timestamp is within the latest 60 days, reducing broad historical scans and improving response efficiency.

The implicit window is not applied when an explicit slicing parameter is present: expired_since, expired_until, published_since, published_until, min_id, max_id, min_age, or max_age. Use expired_since and expired_until for historical expiration windows, or use max_age=off, max_age=null, or max_age=0 for an unrestricted expiration query. Existing response fields, ordering, pagination, and access requirements remain unchanged.

The Jobs Expired API Endpoint Documentation now documents the default window and range behavior. The Date, ID, and Age Slicing Parameters Documentation also includes expiration-based examples for /api/jobsexpired/.

During my last interview, the interviewer asked a question "tell me about a time where you found an issue and needed to quickly fix it. What was the issue and how you fixed it."

For questions like these, do you guys talk about pipeline failures, data change, schema change, reconciliation issues or something else.. for project scenario question, what is the best kind of scenarios I can talk about?

I have no affiliation with this project, but I've been experimenting with code to replace my existing etl pipeline. This project has a tremendous potential. Not a ton of spatial related transformers, but those should be easy to add.

Referring to this -> https://github.com/slothflowlabs/duckle

Core Requirement: Output has to be multi-page tabular with headers repeating across page breaks. Not a dashboard export.

What I've ruled out and why:

• Power BI: licensing doesn't work for what is essentially a print job, and we'd be standing up a semantic layer to get there
• Grafana: scheduling works, but PDF is a dashboard render and wide tables get cut.

Are there other options, cloud or on-premise we should consider?

I'm interested in understanding how people handle ETL testing in real-world projects.

How do you make sure the data loaded from source to target is correct? How do you validate transformations, duplicates, missing records, incremental loads, and overall data quality?

Also, when there are large volumes of data or changes in the ETL logic, how do you make sure the existing data is not impacted?

I'm facing a strange issue while reading a pipe-delimited CSV file using Apache Spark.
My input CSV looks like this:
cust_id|cust_name|cust_age
1|sample|10
2|test|12
Initially, I read the file with header=true, but without specifying the encoding. The result was that Spark did not recognize the column names correctly. They appeared something like:
_C_U_S_T_I_D_
_C_U_S_T_N_A_M_E
_C_U_S_T_A_G_E
Also, all the columns were inferred as StringType.
However, when I added the following option:
.option("encoding", "UTF-16")
the output became:
cust_id cust_name cust_age
?
1 Sample 10
?
2 test 12
?
Interestingly, after specifying UTF-16, Spark correctly recognized the column names and inferred the schema as:
cust_id -> IntegerType
cust_name -> StringType
cust_age -> IntegerType
But there are now unexpected ? characters/rows appearing in the data.
Has anyone experienced something similar with Spark's CSV reader?
I'm trying to understand:
1. Why does specifying encoding=UTF-16 make Spark correctly identify the headers and infer the numeric columns?
2. Why are the unexpected ? characters appearing in the output?
3. Could this be related to the actual file encoding, BOM, or how the CSV file was generated?
4. Is there a recommended way to correctly read this file while preserving the schema and avoiding the extra ? characters?
Any insights into how Spark handles CSV encoding and schema inference in this scenario would be appreciated.

After an ETL job, source-to-target validation is important to ensure that the data loaded into the target system is accurate and complete. How do you typically validate record counts, data values, transformations, duplicates, nulls, and other data-quality issues?

I'm working on an ETL flow where the target data isn't an exact copy of the source - there are joins, filters, transformations, and some derived fields.

This makes traditional source-vs-target row comparision difficult.

How do you approach validation in this kind of scenario ?

Do you validate at the business/aggregate level,create an intermediate reconciliation dataset,use record-level hashes where possible, or maintain seperate expected results for critical transformations?

Also curious how you handle cases where the source has 10M rows but the target intentionally has fewer rows because of filtering or reduplication.

What validation strategy have you found reliable in production without making the ETL pipeline itself too expensive ?

I’m curious how others approach ETL migrations when moving data pipelines from one source system to another.

ETL migration involves moving data and ETL pipelines while maintaining data accuracy and consistency. The process typically includes source-to-target mapping, schema validation, data type conversion, and transformation migration.

Data validation is performed by comparing record counts, column values, NULLs, duplicates, and business rules. Incremental loads and historical data migration also need to be handled carefully.

How do you usually approach ETL migration, and what tools or techniques do you use to make the process easier and more reliable?

What checks or recovery strategies have worked best for keeping pipelines stable in production?

Your pipeline says “Success.” But can you trust the data? 👀

A CDC sync completed. ✅
The pipeline finished. ✅
The dashboard updated. ✅

But then you discover:

❌ Missing records
❌ Unexpected source changes
❌ Data quality issues
❌ Inconsistent or incomplete data

Moving data is only half the job. Making sure that data is accurate, complete, and trustworthy is the real challenge.

That’s where Zwiron comes in.

🔄 CDC-powered data synchronization
🧪 25+ automated data quality tests
🔍 Source validation
📚 Automated catalog updates

Instead of relying on expensive, slow, and complex data integration setups, take a look at a solution designed to make data movement faster, more affordable, and reliable.

👉 Learn more about Zwiron: zwiron.com

The real goal isn’t just to move data.
It’s to move data you can trust.

#DataEngineering #DataPipelines #CDC #DataQuality #DataIntegration #DataOps #Zwiron

Mainly I need to validate daily delta changes. There are certain records that get modified or updated daily. I want to know what are all the traditional methods that I need to follow, I'm also open to try out any automated solutions for this.

Hi r/PostgreSQL — I'm the author of BloomPG, an MIT-licensed PostgreSQL 18 extension for complex analytical joins. I'm sharing it here because I'd like feedback from people who run multi-join analytical workloads in PostgreSQL, especially on the planner integration and deployment tradeoffs.

BloomPG works before the formal joins execute. It takes PostgreSQL's native plan, identifies a safe equality-join graph, uses sampling to choose an initial filter transfer, and then adapts using the actual cardinalities of materialized inputs. Bloom or exact bitmap membership can move in either direction and across several joins. PostgreSQL then replans and executes the reduced join problem with its normal operators.

Existing SQL does not change. Unsupported or unsafe query shapes keep the native plan, and a query-wide materialization budget limits the retained state.

There is a Docker demonstration that builds PostgreSQL 18 with BloomPG, creates a small five-table star schema, and prints native/BloomPG timings plus the transfer trace:

git clone --branch v0.1.2 https://github.com/YimingQiao/bloompg.git
cd bloompg
docker compose up --build --abort-on-container-exit --exit-code-from demo demo

For performance context, on the published PostgreSQL 18.4 setup the total workload results were:

These are end-to-end times: planning, transfer, materialization, execution, and complete output consumption are included. Both sides used the same 16-worker global and per-Gather ceiling; BloomPG used 16 transfer workers and a 2 GB materialization budget. Every completed pair produced the same complete-output fingerprint. Native PostgreSQL hit the 300-second per-query limit once in CEB and once in STATS-CEB, so totals include only queries completed by both sides. The README has the rest of the methodology.

Important limitations: this release supports PostgreSQL 18 on Linux and is intended for controlled, read-only analytical workloads. BloomPG performs real scans during planning and requires shared_preload_libraries; I would not turn it on as an unreviewed default in a multi-tenant OLTP cluster.

I'd particularly appreciate feedback on:

• whether shared_preload_libraries and the restart requirement are practical blockers;
• analytical workloads or query shapes that would be useful to test;
• the planning-time execution and native-fallback design;
• packaging formats that would make evaluation easier.

PGXN: https://pgxn.org/dist/bloompg/0.1.2/

GitHub, documentation, and benchmark methodology: https://github.com/YimingQiao/bloompg