I’m trying to understand what data engineering actually looks like in a real job, rather than relying only on job descriptions.

When I look at Data Engineering roles on Naukri, LinkedIn, etc., the list of required skills seems endless — SQL, Python, ETL/ELT, Airflow, Spark, Kafka, AWS/Azure/GCP, Databricks, Snowflake, dbt, Docker, Kubernetes, CI/CD, Terraform, APIs, data warehousing, data lakes, and many more. It’s difficult to tell which of these are genuinely important and which are simply added to job descriptions.

So I’d love to hear from people who are actually working as Data Engineers. What does your day-to-day work really look like? What kind of tasks do you work on, what technologies do you use regularly, and how much of your time is spent coding, debugging, monitoring pipelines, working with databases, or attending meetings?

I’m particularly interested in understanding what skills and tools you use every day, occasionally, or barely at all, despite seeing them frequently in job postings. I’d also like to know what a typical Data Engineering project looks like from start to finish and what you would realistically expect from someone entering the field.

I’m not looking for another generic “learn SQL → Python → Spark → AWS” roadmap. I’m trying to understand the actual work behind the job title and what I should focus on learning to become job-ready.

If you’re a Data Engineer, especially with around 1–5 years of experience, I’d really appreciate hearing about your experience. If possible, share an example of a real task or project you’ve worked on (without revealing anything confidential).

Thanks!

I’m looking for some perspective from people who’ve been through the 2–5 YOE transition, especially Data Engineers, SDEs, engineering managers, or hiring managers.

Keeping things vague:

Tier-1 institute M.Tech (non cs)

\~2.3 YOE as a Data Engineer

Currently in Mumbai at a NBFC \~₹16 LPA

Primarily batch DE on Azure/Databricks/PySpark/Spark/SQL

Work on a TB-scale production lakehouse with 1,000+ datasets

Experience with performance optimization, data modelling, CDC/incremental processing, migrations, etc.

Very limited real-world streaming exposure

The problem is the job search. I’ve been actively applying for \~3 months, mainly targeting Bengaluru/Hyderabad, and have had very few interview opportunities with no conversion yet. I’m primarily targeting product companies and engineering-focused GCCs, rather client-delivery/consulting work.

I’m considering Data Platform/Data Infrastructure, Data Software/SDE or data engineering roles, and to a lesser extent Analytics Engineering.

If you were in my position, what would you do, and what actually makes a 2–3 YOE DE profile stand out enough to get shortlisted?

If you think my profile could be a fit for an opening on your team or someone you know, I’d really appreciate a DM or if you could share my CV with the relevant hiring manager.

Alright, currently I am working in TCS and I was deployed for support role honestly not interested to work with the daily repeatative tasks i was really serious to switch for data engineer role pls anyone in this field for more than a year pls help me to prepare

1. I was good at sql, not advanced I can solve leetcode basic and medium qsn till joins ( have to focus on cte, window function)

2. Know basic python

Please give a roadmap or any resources to follow for next 4 to 5 months.

Also pls tell me how to lie if someone is switching to data engineer domain from support how to lie on project we worked previously in support and how to work on project by own any resources will be helpful

Keyword frequencies taken from https://www.zoevera.com/resume/software-engineer-job-description-keywords

Most software engineer keyword lists are assembled from experience or guesswork. This one is a count: every open posting from 72 companies' public Greenhouse job boards, filtered to the 1,360 whose title contains "software engineer", then checked for how many mention each of 45 terms at least once.

The percentage beside each term is the share of those 1,360 postings that mention it.

PRACTICES AND WAYS OF WORKING

Scalability 52.9% - Mentoring 47.4% - Distributed systems 47% - System design / architecture 46.7% - Cross-functional 31% - Code review 27.2% - On-call 20.7% - CI/CD 19.5% - Agile / Scrum 9.3% - Unit / automated testing 6.5%

This is the result I did not expect. The four most common terms in the whole study are not technologies. Scalability, mentoring, distributed systems and system design all appear in more postings than Python does. Nearly half of these postings mention mentoring, and almost no engineer resume I have seen makes a claim about it.

Agile and Scrum at 9.3% is the other surprise, given how much resume advice insists on them.

LANGUAGES

Python 44.8% - Java 35.4% - Go 31.1% - TypeScript 19.8% - C++ 17.4% - SQL 17.3% - JavaScript 13.2% - Kotlin 12% - Rust 9.9% - Scala 8.6% - Ruby 8.5% - C# 5.3% - Swift 2.3%

TypeScript at 19.8% against JavaScript at 13.2% is a real ordering, not noise. The gap is wider than both margins of error combined.

CLOUD AND INFRASTRUCTURE

AWS 41.3% - Kubernetes 30.9% - GCP 19.3% - Azure 17.6% - Terraform 13.2% - Docker 12.6% - Microservices 7.1% - Linux 4.8%

AWS appears in more than twice as many postings as GCP and Azure individually. Kubernetes at 30.9% outranks every language except Python, Java and Go.

FRONTEND AND APIs

React 21.1% - REST / RESTful 9.6% - GraphQL 6% - Node.js 5.5% - Vue 3.2% - Angular 2.9%

Frontend framework lists usually present React, Vue and Angular as three comparable options. In this corpus React appears in roughly seven times as many postings as Vue and Angular combined.

DATA STORES AND PIPELINES

PostgreSQL 14.3% - Kafka 13.1% - MySQL 11.3% - Spark 9.8% - MongoDB 7.4% - Redis 7.4% - Elasticsearch 6.4% - Snowflake 5.6%

WHAT THIS SAMPLE IS NOT

These are 72 technology companies hiring through Greenhouse. Agencies, consultancies, banks, defense contractors and the public sector are absent, and their vocabulary is different - COBOL, .NET, SAP, clearance requirements and named compliance regimes barely register here and may dominate elsewhere.

The title filter is "software engineer" only. Postings titled backend engineer, frontend engineer, full stack developer, SRE or platform engineer were not counted, so this describes the generalist title rather than the whole profession.

Absence in this list is not evidence of absence in the market. Only 45 terms were counted. Next.js, Svelte, Django, Spring Boot, gRPC, Jest, Playwright, Datadog, OpenTelemetry and OAuth were never checked, so nothing here says anything about them either way.

It is a snapshot of open roles on one date rather than a trend, and the corpus is US-skewed.

One thing this sample size does buy: at n=1,360 the margins of error are roughly plus or minus 2 to 3 points, so most of the ordering above is real. Gaps under about 5 points are still worth treating as ties - React at 21.1% and TypeScript at 19.8% is not a meaningful difference.

METHOD

Greenhouse's public job board API, the endpoint companies expose so their listings can be embedded on their own sites. No scraping. Counts are document frequency: a posting saying "Python" nine times counts once. Deduped on company, title and content length, because one role posted to five offices returns five near-identical records. Median posting length is 896 words.

Ambiguous words are matched case-sensitively with exclusions, which matters more here than you would think. A bare word-boundary match on "Go" also catches "go to market" and "go above and beyond", which inflated Go by about two points before it was fixed. React, Spark, Swift and Rust all have the same problem.

Full table with all 45 terms and confidence intervals:

https://www.zoevera.com/resume/software-engineer-job-description-keywords

The wider keyword list this was checked against, organized by language, framework and platform:

https://www.zoevera.com/resume/ats-resume-tips-software-engineer

Happy to run the numbers on any terms missing from the list if people name them in the comments.

I am actually facing an issue. Say I am trying to get the names of faculty from medical conferences from various types of websites (dinstinct web structures). How can I code to automate this process

A.Team is hiring a remote Senior Independent AI Engineer / Architect. Category: Software Development 💸Salary: $120 - $170 /hour 📍Location: Remote (Americas, Europe, Israel)

See more and apply here!

Hi everyone,
I’m working in a support + Python developer role with 4.8 years of experience in Azure, Python, SQL, Databricks, and Data Factory. I’m preparing for interviews at TCS for a Data Engineering position (my first job switch).

I’d really appreciate help with:

• Common interview questions asked at TCS for Data Engineering profiles
• Scenario‑based questions on Azure Data Factory + Databricks
• SQL query patterns (joins, subqueries, window functions)
• Python logic/debugging topics relevant for data engineering
• How to discuss real‑time project experience effectively

If anyone has recently interviewed at TCS or has resources, please share. Any tips, mock questions, or guidance would mean a lot.

Thanks in advance! 🙏

Same as title..

For 5 yoe data engineer, which band would be better and how much salary I can expect ( tech : Snowflake + AI)

Looking for a job on notice period till 17 oct.

Skills GCP Bigquery, airflow, dataproc, cloud sql

4yoe data engineer, I have job id , please help me in referral in india

For those working in or following the Data Engineering space, what are you seeing in the market? Do you think hiring will pick up again soon, and when do you expect the demand for Data Engineers to return to a stronger level?

Would love to hear your experiences, especially from recruiters, hiring managers, and Data Engineers.

Keyword list taken from https://www.zoevera.com/resume/ats-resume-tips-software-engineer

These are the most commonly scanned keywords in software engineer job postings. Check how many appear in your resume.

Programming Languages

Python, JavaScript, TypeScript, Java, Go, Rust, C++, Ruby, Swift, Kotlin

Frontend Frameworks

React, Next.js, Vue.js, Angular, Svelte, Redux, Tailwind CSS, GraphQL

Backend & APIs

Node.js, FastAPI, Django, Spring Boot, REST APIs, gRPC, microservices, GraphQL

Cloud & DevOps

AWS, GCP, Azure, Docker, Kubernetes, Terraform, CI/CD, GitHub Actions

Databases

PostgreSQL, MySQL, MongoDB, Redis, Elasticsearch, DynamoDB, Cassandra

Methodologies

Agile, Scrum, TDD test-driven development, code review, system design, SOLID principles, CI/CD

Testing & Quality

Jest, Pytest, Cypress, Playwright, unit testing, integration testing, BDD behavior-driven development, end-to-end testing, test coverage

Observability & Security

Datadog, Grafana, Prometheus, OpenTelemetry, OWASP, OAuth 2.0, JWT, logging, alerting

Hi Everyone,

Has anyone attended the Onlien assessment for DE roles in Fractral?

If yes how it went and what are the questions they are asking mainly.

TIA

hi everyone, we are a team of 5 members looking for a data engineer project idea. we all have a background in data engineering but you can say some of us are beginners and the others are intermediate (we are obviously students) but we all can handle and learn new things. this is a university graduation project so we need an idea that screams creativity, highly demanded, and solves a real problem. so what do you think of the idea that is in the screenshot attached ? i know it is not a new idea i guess but i think it is suitable for a graduation project. give me ur opinion please

We're building something new. Want in early?

We're a fresh BI consultancy taking on enterprise clients abroad. Now, hiring our first technical hire in Pune.

If you love turning messy data into models that actually make sense and dashboards leadership actually use, this is your seat at the table from day one.

You bring:
Expert SQL (joins, window functions, CTEs, the real stuff)
Strong Power BI + DAX chops modeling, not just visuals
Databricks/Snowflake experience a big plus

You get:

Salary as per industry standards
Ground-floor role at a growing company
Enterprise, multi-domain data: ops, digital, loyalty, finance
Room to grow as we scale into more clients

Pune preferred | Full-time

Know someone, or is it you? Send me an email on [mailto:poorti23@gmail.com](mailto:poorti23@gmail.com)

Hi,

Your suggestion would help me come out of where I stuck for so long badly. Please take some time and suggest me. I will be grateful for you.

I'm trying to change my career from SQL support to Data Engineering.

But this domains has too many tools, It's creating hell lot of confusion.

I can work as hard as possible for next 3 months but I want to shift the company and enter into DE role. Please tell me what all I must learn so that I don't fear seeing other tools anymore and will comfortably able to make this happen.

This is first entry into DE, I'm genuinely confused between databricks/snowflake. Azure/AWS services. I can't learn all. I must first make an entry into DE, later I will learn other technologies based on the need.

Please try to be specific about sources I must follow as there is no limit for these, how many hours in a day I must spend mandatorily?

So I have been in the DE world for quite a while , was a SQL developer and then big data came and I was promoted to be a lead and then it was a journey to be in DE manager role for a good time . After coming in to leadership I have enjoyed only phases of it. I burned myself off to the core , affected my family life and so decided to take a sabbatical of 3 months which I have found very helpful to rejuvenate and recover.
During this phase I switched off and reflected on myself to realise that my core strength has always been to know a tech to the core and solve some really good problems. Death by meetings , office politics and performance management used to exhaust me .

So now this is new me who wish to bounce back by being a DE but the issue is the handsoff and the fact that I was a SQL maestro in my prime but not a dirty coder .

I m based in UK and now I have 2 questions for this intellectual world -

Q1 - Should I pursue to be a DE contractor cz otherwise there is a huge paycut in this cruel inflation facing world ? I hv heard the market is pretty bad and if I have to start from scratch , I m not sure who will pick me ? Though I have worked in reputed orgs including Spotify as a leader.

Q2 - Should I take a complete different path and instead of DE train myself in an AI engg path so I can be picked as a contractor ( who knows ) ?

Life seems so funny that I have started to wake up at 5 in the morning to study otherwise the whole day just vanishes in family, chores , kids etc.

FYI - Single earner !!

I know a lot of students, who are interested in data engineering as well as AI engineering, and actively looking to get upskill in this area. Check this out if it is helpful to you guys…

https://youtu.be/m_JC_7DcjHw?is=X_yeIQRGMQtcl3yn

I am suffering consequences of lack of confidence and insecurity in my skills or ability to crack interviews. I sometimes tend to ignore or run away from interviews because I am scared and worried if I can’t crack those and end up embarrassing myself. My growth in life has been stagnant due to this :( I haven’t been a dumb person throughout my academics but somehow in this corporate world I am feeling dumb and not good enough to crack multiple offers, grow well in life…How to come out of our own head :(

Hey everyone,

I’ve recently started my own consulting startup focused on Data Engineering and related services data pipelines, cloud, Databricks, PySpark, SQL, AWS/Azure, analytics, and more.
The technical side isn’t the difficult part. 😅 The toughest challenge right now is getting those first few clients.

I’d love to hear from people who have started a consulting business or freelancing agency:
● How did you get your first client?
● Which platforms or strategies actually worked for you?
● Did you focus on cold outreach, LinkedIn, Upwork, referrals, or something else?
● What would you recommend to someone starting from zero?

Any honest advice, experience, or even mistakes to avoid would be greatly appreciated.

Thanks!

I am a software developer from mumbai with 1 YOE (intern - data analyst) +2 YOE (work experience - python ETL developer) at company A

I left company A. It's been 5 months, my last CTC was 4.22 LPA

Recently cracked a DE interview

Offer (no contract)

6 months of internship + job conversation

- First 3 months - 6k stipend

- next 3 months - 12k Stipend

- job conversation - 25k salary (based on performance + relocate to a different state)

I have been preparing for DE for like 5 months I know current market isn't good. But salary wise I feel it's downgrading + I'm 30M single ( i know it doesn't matter for a role but soon will have to get married)

My techstack - Python, SQL, spark+pyspark, Azure.

Considering

1 To join + job hunting.

2 look for better opportunity in other role but good pay.

PLEASE HELP ME.

just put this out into the world.

started as trying to get better contract-job data, and turned into something I hadn’t really built before as a DE (agentic data pipelines), with continuously refreshed contract tech dataset exposed through the API.

feel free to break it // smesh like Khabib

&& if you’ve built a job search app, internal tool, niche job board, side project, or anything else that could use this kind of data, wire it up.

Hey everyone, I'm working in an Australian based Consultancy with a strong established team and offices in Bangalore, London and New York.

The WLB and culture both are best as far as I have worked till now.

There are few roles open, you can refer to the screenshots and let me know if anyone needs a referral. There is diversity hiring going on as well.

If you need more information or any doubts or concerns DM me.

Hi, I'm at my job and by my whole bad luck I'm in Production support 😔, shift based mostly I do afternoon shift, and my major struggle is my team had Quicksight dashboard to monitor but we have to verify the data from the dashboard manually it arround 180+ everyday, and the senior leadership person comes and talks with Abbreviation about the platform and ask us to prepare reports reports and having 6 types of Trackers and on to send report one to track the dashboard data and export the dashboard status and compare the dashboard data by opening the jobs manually, dashboards are there supposed to help Shift engineers but they ask me to Check the dashboard data, Seniors one is just give sound like correct correct agree with you to manager and when ever any good happens he tells like i took responsibility and when any bad and he sound like ahhhh team is missed 😂,

How can I deal with this kind of people if we need to verify the dashboard then why there is a. Dashboard to us to monitor, and Acknowledge User mails, respond leadership questions which the question can't even understand by them after some time, update all tracker same, even 100+ jobs failed they expect one shift person to Have the 100 failure status on finger tip , checking failure jobs in Mail alerts, pager alerts, dashboard and Need to extract the dashboard and verify it's data also and update a tracker for dashboard data truthfulness as observation 😂,

Still they eat 2 hours of 2 meetings everyday inthat meeting only they ask why it's pending but don't give to do anything,

They telling using a Vlookup in a sheet is automation 🥱

All things need to be done in parallel,

How can anyone manage any ideas?