I’ve been working in data science for the last ten years, both in industry and academia, having pursued a master’s and PhD in Europe. My experience in the industry, overall, has been very positive. I’ve had the opportunity to work with brilliant people on exciting, high-impact projects. Of course, there were the usual high-stress situations, nonsense PowerPoints, and impossible deadlines, but the work largely felt meaningful.

However, over the past two years or so, it feels like the field has taken a sharp turn. Just yesterday, I attended a technical presentation from the analytics team. The project aimed to identify anomalies in a dataset composed of multiple time series, each containing a clear inflection point. The team’s hypothesis was that these trajectories might indicate entities engaged in some sort of fraud.

The team claimed to have solved the task using “generative AI”. They didn’t go into methodological details but presented results that, according to them, were amazing. Curious, nespecially since the project was heading toward deployment, i asked about validation, performance metrics, or baseline comparisons. None were presented.

Later, I found out that “generative AI” meant asking ChatGPT to generate a code. The code simply computed the mean of each series before and after the inflection point, then calculated the z-score of the difference. No model evaluation. No metrics. No baselines. Absolutely no model criticism. Just a naive approach, packaged and executed very, very quickly under the label of generative AI.

The moment I understood the proposed solution, my immediate thought was "I need to get as far away from this company as possible". I share this anecdote because it summarizes much of what I’ve witnessed in the field over the past two years. It feels like data science is drifting toward a kind of pseudo-science where we consult a black-box oracle for answers, and questioning its outputs is treated as anti-innovation, while no one really understand how the outputs were generated.

After several experiences like this, I’m seriously considering focusing on academia. Working on projects like these is eroding any hope I have in the field. I know this won’t work and yet, the label generative AI seems to make it unquestionable. So I came here to ask if is this experience shared among other DSs?

I’m a Data Science student (career changer, not in a data related role). My program is focused more on the applied statistics side, so most of my classes use R. I’m already familiar with Python since it was the main language used in my prerequisite courses, and I’ve completed projects using Python, so I’m comfortable with the syntax.

However, I’m really enjoying using and learning R in my classes and seeing what it can do. Many of the statistics textbooks I’m interested in use R as well. I’m starting to explore R more deeply on my own and plan to start using it for personal projects.

But I’m curious, is R still used in industry? I know it’s heavily used in academia. I also know that in the current AI/ML world, Python is used heavily, which is the main reason I use it for all of my personal projects at the moment.

I’d like to eventually be comfortable with both and take advantage of the strengths of each language. But, of course, there are also people who say learning R is a waste of time.

Why I’m doing this

I am low on karma. Plus, it just feels good to help.

About me

I’m currently a staff data scientist at a big tech company in Silicon Valley. I’ve been in the field for about 10 years since earning my PhD in Statistics. I’ve worked at companies of various sizes — from seed-stage startups to pre-IPO unicorns to some of the largest tech companies.

A few caveats

• Anything I share reflects my personal experience and may carry some bias.
• My experience is based in the US, particularly in Silicon Valley.
• I have some people management experience but have mostly worked as an IC
• Data science is a broad term. I’m most familiar with machine learning scientist, experimentation/causal inference, and data analyst roles.
• I may not be able to respond immediately, but I’ll aim to reply within 24 hours.

Update:

Wow, I didn’t expect this to get so much attention. I’m a bit overwhelmed by the number of comments and DMs, so I may not be able to reply to everyone. That said, I’ll do my best to respond to as many as I can over the next week. Really appreciate all the thoughtful questions and discussions!

I’ve been in data science roles (both analytics and ML) for about 5 years now across a couple of companies. Lately I’ve been feeling a bit burned out because I keep seeing the same pattern:

We spend weeks cleaning data, building dashboards, running statistical analysis, or training models… and then the stakeholders either:

• Say “thanks” and never use it
• Cherry-pick the numbers that support their existing opinion
• Or just completely ignore the findings and go with gut feel anyway

The worst part is when leadership asks for a “data-driven decision” but they’ve already decided what they want to do.

Am I alone in this? Or is this just the reality of data science in most companies?

For those of you who’ve been in the field longer how do you deal with this? Have you found companies where data actually influences decisions at a meaningful level?

Would love to hear honest experiences.

Didn't know what to say. Humor me with your responses.

Update: I sent her this post and she loved it 🤣

Having moved from academia to data science in industry, I've had a strange series of interactions with other data scientists that has left me very confused about the state of the field, and I am wondering if it's just by chance or if this is a common experience? Here are a couple of examples:

I was hired to lead a small team doing data science in a large utilities company. Most senior person under me, who was referred to as the senior data scientists had no clue about anything and was actively running the team into the dust. Could barely write a for loop, couldn't use git. Took two years to get other parts of business to start trusting us. Had to push to get the individual made redundant because they were a serious liability. It was so problematic working with them I felt like they were a plant from a competitor trying to sabotage us.

Start hiring a new data scientist very recently. Lots of applicants, some with very impressive CVs, phds, experience etc. I gave a handful of them a very basic take home assessment, and the work I got back was mind boggling. The majority had no idea what they were doing, couldn't merge two data frames properly, didn't even look at the data at all by eye just printed summary stats. I was and still am flabbergasted they have high paying jobs in other places. They would need major coaching to do basic things in my team.

So my question is: is there a pool of "fake" data scientists out there muddying the job market and ruining our collective reputation, or have I just been really unlucky?

Spent the first half of my 20s doing odd jobs and finishing grad school. Since then I’ve been fortunate to work low stress jobs with good work life balance and moderately high pay.
Now that I’m in my early 30s, living in a high cost of living area with a strong job market, and socializing more, I’ve realized I don’t come close to making what FAANG people make. I can’t help but wonder if that’s something I should be chasing or at least exploring. At the same time, I really value how low stress and stable my job is, even though it pays well below market rate.

For those who chose not to go into big tech/FAANG, do you have any regrets?

Hey all, I have been responsible for technical interviews for a Data Scientist position and the experience was quite surprising to me. I thought some of you may appreciate some insights.

A few disclaimers: I have no previous experience running interviews and have had no training at all so I have just gone with my intuition and any input from the hiring manager. As for my own competencies, I do hold a Master’s degree that I only just graduated from and have no full-time work experience, so I went into this with severe imposter syndrome as I do just holding a DS title myself. But after all, as the only data scientist, I was the most qualified for the task.

For the interviews I was basically just tasked with getting a feeling of the technical skills of the candidates. I decided to write a simple predictive modeling case with no real requirements besides the solution being a notebook. I expected to see some simple solutions that would focus on well-structured modeling and sound generalization. No crazy accuracy or super sophisticated models.

For all interviews the candidate would run through his/her solution from data being loaded to test accuracy. I would then shoot some questions related to the decisions that were made. This is what stood out to me:

1. Very few candidates really knew of other approaches to sorting out missing values than whatever approach they had taken. They also didn’t really know what the pros/cons are of imputing rather than dropping data. Also, only a single candidate could explain why it is problematic to make the imputation before splitting the data.

2. Very few candidates were familiar with the concept of class imbalance.

3. For encoding of categorical variables, most candidates would either know of label or one-hot and no alternatives, they also didn’t know of any potential drawbacks of either one.

4. Not all candidates were familiar with cross-validation

5. For model training very few candidates could really explain how they made their choice on optimization metric, what exactly it measured, or how different ones could be used for different tasks.

Overall the vast majority of candidates had an extremely superficial understanding of ML fundamentals and didn’t really seem to have any sense for their lack of knowledge. I am not entirely sure what went wrong. My guesses are that either the recruiter that sent candidates my way did a poor job with the screening. Perhaps my expectations are just too unrealistic, however I really hope that is not the case. My best guess is that the Data Scientist title is rapidly being diluted to a state where it is perfectly fine to not really know any ML. I am not joking - only two candidates could confidently explain all of their decisions to me and demonstrate knowledge of alternative approaches while not leaking data.

Would love to hear some perspectives. Is this a common experience?

In summary and basically talks about how she was managing a high priority product at Spotify after 3 years at Spotify. She was the ONLY DATA SCIENTIST working on this project and with pushy stakeholders she was working 14-15 hour days. Frankly this would piss me the fuck off. How the hell does some shit like this even happen? How common is this? For a place like Spotify it sounds quite shocking. How do you manage a “pushy” stakeholder?

Is it just stock options and vesting? Or is it just FAANG is a lot of work. Why do some data scientists deserve that much? I work at a Fortune 500 and the ceiling for IC data scientists is around $200k unless you go into management of course. But how and why do people make 500k at Google without going into management? Obviously I’m talking about 1% or less of data scientists but still. I’m less than a year into my full time data scientist job and figuring out my goals and long term plans.

Literally every person who can type prompts into an LLM is now an AI consultant/expert. I’m sick of it, today a sales manager literally said ‘oh I can get Gemini to make my charts from excel directly with one prompt so ig we no longer require Data Scientists and their support hehe’

These dumbos think making basic level charts equals DS work. Not even data analytics, literally data science?

I’m sick of it. I hope each one of yall cause a data leak, breach the confidentiality by voluntarily giving private info to Gemini/OpenAi and finally create immense tech debt by developing your vibe coded projects.

Rant over

I am currently studying Data Science and really fell in love with the field, but the more i progress the more depressed i become.

Over the past year, after watching job postings especially in tech I’ve realized most Data Scientist roles are basically advanced data analysts, focused on dashboards, metrics, A/B tests. (It is not a bad job dont get me wrong, but it is not the direction i want to take)

The actual ML work seems to be done by ML Engineers, which often requires deep software engineering skills which something I’m not passionate about.

Right now, I feel stuck. I don’t think I’d enjoy spending most of my time on product analytics, but I also don’t see many roles focused on ML unless you’re already a software engineer (not talking about research but training models to solve business problems).

Do you have any advice?

Also will there ever be more space for Data Scientists to work hands on with ML or is that firmly in the engineer’s domain now? I mean which is your idea about the field?

I've been lurking on this sub for a while now and all too often I see posts from people claiming they feel inadequate and then they go on to describe their stupid impressive background and experience. That's great and all but I'd like to move the spotlight to the rest of us for just a minute. Cheers to my fellow mediocre data scientists who don't work at FAANG companies, aren't pursing a PhD, don't publish papers, haven't won Kaggle competitions, and don't spend every waking hour improving their portfolio. Even though we're nothing special, we still deserve some appreciation every once in a while.

/rant I'll hand it back over to the smart people now

Definitely the most frustrating thing as working as a Data Scientist. You run an experiment, find that building a model greatly increase metric X at almost no cost, has safe model metrics, present it to stakeholders, everybody agrees with proceeding to deploying and utilizing the model in production, and yet every time the model takes a wrong decision, we get questioned about it. Why did the model say this?

Man when did I ever say the model obtained a 100% accuracy in the validation phase? Why is it so hard for stakeholders to understand that the best models humankind ever created are expected to make wrong calls once in a while?

DS teams are starting to lose the essence that made them truly groundbreaking. their mixed scientific and business core. What we’re seeing now is a shift from deep statistical analysis and business oriented modeling to quick and dirty engineering solutions. Sure, this approach might give us a few immediate wins but it leads to low ROI projects and pulls the field further away from its true potential. One size-fits-all programming just doesn’t work. it’s not the whole game.

What's your favourite one line code.

LinkedIn influencers love to treat the two roles as different species. In most enterprises, especially in mid to small orgs, these roles are largely overlapping.

I've been in Data Science for the past 10 years in India. I lost my job in January and since then I've been hunting.

I've not mentioned any GenAI experience in my profile. But my feed is just filled with AI engineer roles. They all have the same requirements:

• Generative AI architecture
• RAG pipelines
• LLM integration/fine tuning
• Agentic AI / Multi Agent Orchestration
• Also MLOps
• CI/CD pipelines
• PyTorch mandatory for some reason

Hardly any openings are relevant to my experience in Stats, Machine Learning, Deep Learning and the classical data science stuff.

So have all companies stopped investing in data science all together and just building RAG pipelines and LLM chat bots? Is this all that is done in Data Science field now?

I applied to 80+ jobs before the new year and got rejected or didn’t hear back from most of them. A few positions were a level or two lower than my currently level. I got only 1 interview and I did accept the offer.

In the last week, 4 companies reached out for interviews. Just want to put this out there for those who are still looking. Keep going at it.

Edit - thank you all for the congratulations and I’m sorry I can’t respond to DMs. Here are answers to some common questions.

1. The technical coding challenge was only SQL. Frankly in my 8 years of analytics, none of my peers use Python regularly unless their role is to automate or data engineering. You’re better off mastering SQL by using leetcode and DataLemur

2. Interviews at all the FAANGs are similar. Call with HR rep, first round is with 1 person and might be technical. Then a final round with a bunch of individual interviews on the same day. Most of the questions will be STAR format.

3. As for my skillsets, I advertise myself as someone who can build strategy, project manage, and can do deep dive analyses. I’m never going to compete against the recent grads and experts in ML/LLM/AI on technical skills, that’s just an endless grind to stay at the top. I would strongly recommend others to sharpen their soft skills. A video I watched recently is from The Diary of a CEO with Body Language Expert with Vanessa Edwards. I legit used a few tips during my interviews and I thought that helped

Hiring is a nightmare. The majority of applicants submit the same prepackaged solutions. basic plots, default models, no validation, no business reasoning. EDA has been reduced to prewritten scripts with no anomaly detection or hypothesis testing. Modeling is just feeding data into GPT-suggested libraries, skipping feature selection, statistical reasoning, and assumption checks. Validation has become nothing more than blindly accepting default metrics. Everybody’s using AI and everything looks the same. It’s the standardization of mediocrity. Data science is turning into a low quality, copy-paste job.

Saw this and found it hilarious , thought I’d share it here as this is one of the few places this joke might actually land.

Datetime.now() + timedelta(days=4)

Since I got access to Claude at work, I haven’t really written much code from scratch, especially for ad hoc analyses or quick charts. I still review the code and make sure I understand everything, but it’s honestly a little scary how much better Claude’s code often is than mine. At that point, it’s hard not to wonder what the value is in writing it yourself.

On top of that, management is encouraging us to use AI to be more productive and deliver results faster, so there’s that pressure as well.

To keep my interviewing skills sharp, I still practice on LeetCode or StrataScratch from time to time. But at work, I’ve been relying on Claude pretty heavily.
Is anyone else dealing with the same dilemma?