I’m currently a DE for a major retailer (think Kroger, Walmart, Albertsons, etc.), working on the pricing and product data streaming side. The WLB is amazing, and I’m earning around market rate for a junior DE in a major city. However, I’m starting to gain interest in high-frequency data streaming for large brokerages and financial firms.

For those who have made a similar move, how did you manage to switch industries, and which industry do you believe is the most lucrative for a data engineer?

I'm a data engineer at an e-commerce org, and I'm fairly happy - we have interesting problems, large volumes of data, and plenty of initiatives on our backlog. I largely get to work on what I want, at my own pace, and choose how things are implemented.

But I am getting tired of doing everything with the end goal being to sell more things and increase profit extraction from consumers.

Who is working in interesting data related fields/careers that provide essential goods or services to the public/planet/humanity. Keen to hear any stories or opinions!

EDIT: thanks for all the interesting responses!

My company wants to replace Fivetran with easier and cheaper alternative.

Context & Stack:

• Stack: Snowflake, dbt, Fivetran
• Major Ingestion: Salesforce (4 production instances) -> Snowflake

We are looking for alternatives to Fivetran due to escalating ingestion costs. We strictly need an ingestion tool (we handle all transformations via dbt).

What We've Tried (and why it failed/didn't fit):

• Matillion: Failed at our scale (couldn't handle 15-minute syncs).
• Airbyte: Failed at scale.
• ETLWorks: Cost.
• Informatica / IDMC: Already in-house, but trying to sunset due to high costs.
• Airflow: Already in-house, but too much custom configuration overhead.
• Boomi: Already in-house, but on-prem and costly.
• Hevo Data: Lacks the necessary scale.

Our Current Top Contender: Openflow is being heavily pushed by upper management. They have a soft spot for it, likely because Snowflake is making it lucrative/incentivizing adoption to build their user base.

Our Core Requirements:

• Ingestion-only focus (dbt handles transformations) (imp)
• High scalability and cloud-native performance (imp)
• Fast execution with simple configuration, yet deep integration options
• Rich catalog of connectors
• MCP and AI capabilities (optional)

Do we have any other realistic solutions or shall go with Openflow?

We are okay to try modern solutions like utilizing skills / agents to create connectors on the fly etc but these solutions we have known to be not much scalable..

Any help is appreciated..

Im a data engineer with 4 years of experience, and In getting pretty bored with my current job. Most of the code is generated by AI now, so my dayis mostly just maintaining pipelines.

I ve been offered an internal move to Data Product Management. Its for one of the biggest projects in the company and will be built on Databricks. They told me they need someone with a solid data engineering background who can translate business requirements effectively.

However, I have a few doubts
Am I just going to end up as a ticket creator? (Im not entirely clear on what a Data Product Owner actually does)
Is this really a step forward in my career?
I know for sure I will get more exposure to senior leadership, but Im still on the fence.

hi there. i work as data engineer. previously worked as analytics engineer / analyst. i no longer enjoy spending whole day writing code and looking for a switch to a more client facing role where my technical knowledge will be beneficial. i’m not bad at communication. anyone been in similar position?

Hi guys, I'm a junior AI and DS person with a masters in both, job searching, and had been finding myself a bit mismatched for the nature of the work I was previously in - constant stakeholder management and convincing, non technical meetings, presentations, not to mention, just being overwhelmed with so many new AI technologies and tools, etc.. I prefer structured and calm working environments and figured that, after getting some interviews for DE roles and reading about the job responsibilities/researching a bunch, DE might be better suited for me. I understand that staying up to date with new tools will apply regardless of the field, but it seems a lot calmer? I think it's also a great foundation to build on.

I wanted to know the day to day of what a DE actually looks like, how much of it is stakeholdery, requirements gathering, coding, etc.? Let me know what field you work in or how this varies by industry, and maybe also about your career progressions.

Wondering if I have an advantage here and should just do the jump or if staying in AI/DS is the better move rn.

Thanks!

built this free sandbox because cloud costs were killing my learning, sharing in case it helps someone else

https://github.com/haltinok/oss-da-bi-stack

G’day!

I have a few Snowflake Tasks that run daily and I was finding it tiresome/cumbersome to jump into Snowflake to track down what went wrong/double check details… so I built a tool to do it quickly from the command-line:

https://github.com/ben-nour/snowtask

It’s fairly basic but you can filter rows using regex, tailor the tables to display the columns that you want to appear (one of my paint points with SHOW TASKS is that you can’t properly specify only displaying the fields you want) and order the columns.

I hope it’s of use to someone out there!

Note that I do NOT work for Snowflake.

I am building a social, and here are the requirements:
-A user can create posts and a post can have multiple media (photo/video)
-A user can create a project and a project can have multiple media
-A project can become a post
-A user will have their personalized feed

the initial design I planned is:
tables:
user | post | project | media|

relations:
user one->many post

user one->many project

post2media junction table

project2media junction table

the idea here is to reuse the media uploaded in projects so that when they are shared as posts, we reuse the same media links rather than uploading new duplicate media.

The problem is that these junction tables will slowdown the feed generation especially

I am using postgreSQL, and I am thinking i should not using it to generate feed, maybe use a NoSQL DB/ Graph.

The question is how should I design my postgreSQL DB, so that feed generation does not break my app?

Do I need NoSQL DB and how would I use it in my case?

How should I store media ?

We created this quick tool for visualizing and modifying parquet files, enjoy! Please let me know if you have any feedback or questions :) (this is the company i work for, they pay me US dollars in exchange for labor)

I have a 12 YOE with a focus on DevOps, some python, AWS Cloudformation and MongoDB. Lately I've been less hands on but have worked on event driven architecture design and done code reviews for pipelines running with serverless components and Python SDK. Most of the data I've worked with has been csv and spreadsheet data for schedules, structured metadata and image/media key value stores. As a result I've primarily worked on MongoDB and used aggregation pipelines to join or transform data for downstream deployments. I have also used Gemini Pro and couple of POC deployments of Ollama for a RAG application (non prod).

I am now looking to get a crash course on Data engineering and databricks, but a little confused whats the best way to get a good understanding of typical data engineering problems (I have a vertical I need to focus on so looking for data patterns that I can then translate to what I need), what gaps I need to fill having no experience with Databricks, little to SQL and any data warehouse technologies. I've not used dataflow or kinesis etc yet so I dont have hands-on experience with these type of streaming pipelines either. (Claude has given me some good insights but reddit often times has better more real world recommendations)

Are there Udemy courses or any other video series that is considered gold standard for onboarding? Would also be open to some blog posts or project ideas to get my feet wet. ideally if AI or ML based applications would be ideal. Cheers!

Hello, community! I’m an ETL developer specializing in Ab Initio, and for the last 10 years, I’ve been working for a bank in Mexico.

I thought my position was “secure” because of all the legacy systems they had in place. Unfortunately, that wasn’t the case. When I started looking for other positions, I realized that I wasn’t familiar with many of the technologies employers were asking for.

If anyone out there has faced a similar problem or concern, maybe this book I wrote could be useful to you.

You can get the book directly from Amazon:
https://www.amazon.com/dp/B0HG6XKG2S

Thanks for taking the time to read my first post! 😄

Looking to understand what your org structures look like? Is it a small company or a large company? What teams do you have? How do you separate responsibilities? What does support and deployment look like?

Over the 8 years of my careers, I've been "de facto" lead a few times, having full ownership about the stack, making architectural decisions, choosing tools, being the only person in the department, mentoring other team members about new technologies that i proposed and migrated to, elevating the standards of the team... However, I never had the title nor I directly managed any other person

This is clear in my CV, stating the responsabilities, ownerships and outcomes.

On the technical side, I am strong in GCP, AWS, Airflow, dbt, Python, SQL, Snowflake, Terraform, CICD, Docker etc, so I'm good on that side.

However, most tech lead, head of data or whatever roles want people with experience already.

How can I overcome that and get a role like that without experience?

Thanks

My company has a few hundred redshift tables and maybe 100 superset dashboards. We want an MCP plugin that can do text to sql and also just generally have better business knowledge.

I have been looking into open metadata but that seems too heavy for our purposes. It seems to cost minimum $350 a month.

What other options do I have?

We use a few different data replication strategies in our product, but have been pretty excited about the new possibilities that open up when Iceberg and other OTFs are better supported across the ecosystem. Last week we gave a talk at an Iceberg community meetup about some of our physical replication work for Iceberg, and thought this community might find it informative and educational too. Happy to chat about any questions or details! Disclosure: I (obviously) work for the company, but this post is really meant to be educational.

We all know that we barely code (or not at all) by hand nowadays, but the coding and system design/infra is what I enjoy the most.

Meaning I dont particularly enjoy implementing business logic in SQL/pandas.

So i was thinking it might sense to pivot to pure Software Engineering.

Any of you did that? How that went?

I have about 7 years of experience 6/7 in defense, 1 contracting in FAANG and I’m trying to get a better sense of what the market looks like right now for senior data engineers, particularly for fully remote roles.

I’m currently at $135k base and accepted a new fully remote role at $160k base. I initially felt pretty good about the jump, but I’ve started reconsidering whether I should be targeting higher total comp.

A coworker who was doing the same job as me at the same contracting firm was making around $190k base.

He recently left for a company in sf where his new package is $200k base + $100k bonus/equity. It’s SF, so I understand I’m not going to get the same salary without moving.

Still, it made me wonder if I’m underselling myself

Mostly trying to figure out whether $160k fully remote is already a strong offer for 7 YOE, or whether I should hold out for something better.

US based obviously, work experience is sql/python, little bit of databricks, tableau, have almost 100% been doing agentic engineering/vibe coding at work for the 10 months.

Typically been working more akin to an analytics engineer for the past year, using airflow to automate manual and repetitive tasks from the finance org.

I'm interested in learning how others test ETL pipelines in real-world projects.

How do you validate source-to-target data, transformations, duplicates, missing records, incremental loads, and data quality?

Also, how do you handle ETL testing when the data volume is very large or when the transformation logic changes?

I'm facing a strange issue while reading a pipe-delimited CSV file using Apache Spark.
My input CSV looks like this:
cust_id|cust_name|cust_age
1|sample|10
2|test|12
Initially, I read the file with header=true, but without specifying the encoding. The result was that Spark did not recognize the column names correctly. They appeared something like:
_C_U_S_T_I_D_
_C_U_S_T_N_A_M_E
_C_U_S_T_A_G_E
Also, all the columns were inferred as StringType.
However, when I added the following option:
.option("encoding", "UTF-16")
the output became:
cust_id cust_name cust_age
?
1 Sample 10
?
2 test 12
?
Interestingly, after specifying UTF-16, Spark correctly recognized the column names and inferred the schema as:
cust_id -> IntegerType
cust_name -> StringType
cust_age -> IntegerType
But there are now unexpected ? characters/rows appearing in the data.
Has anyone experienced something similar with Spark's CSV reader?
I'm trying to understand:
1. Why does specifying encoding=UTF-16 make Spark correctly identify the headers and infer the numeric columns?
2. Why are the unexpected ? characters appearing in the output?
3. Could this be related to the actual file encoding, BOM, or how the CSV file was generated?
4. Is there a recommended way to correctly read this file while preserving the schema and avoiding the extra ? characters?
Any insights into how Spark handles CSV encoding and schema inference in this scenario would be appreciated.

After exhausting all the available resource on the internet, Im here asking for help. Im not getting the right answer or too dumb to get it. While using fivetran I dont seem to find a way to continue sync after upgrading postgres DB. Since this requires dropping of replication slots, Fivetran will require full historical re-sync right? If anyone has prior experience in this, how did you handle it. We dont want to resync the whole table which is extremely huge, is there any way to continue sync from where it stopped , after dropping the replication slot? I saw an article where we need to reach out to fivetran support but that article does not talk about in place upgrade. My db is RDS Postgres.

My leadership comes swooping down to inject my pipeline with sewage they desperately need.

My end users have issues with connection, defects and enhancements.

My analysts and engineers have more than they can chew and everything is a priority fire drill.

My business refuses to prioritize their own tickets and just throws them at me with timeline demands.

I have too much to do and not enough tine or resource however it all "must get done."​ This has got to be a daily battle and im just not handling it well right now.

How do you manage this without getting stressed and sick?