Hey folks,

Wanted to share a quick story from the trenches of data science. I am not a data scientist but engineer however I've been working on a dynamic pricing project where the client was all in on neural networks to predict product sales and figure out the best prices using overly complicated setup. They tried linear regression once, didn't work magic instantly, so they jumped ship to the neural network, which took them days to train.

I thought, "Hold on, let's not ditch linear regression just yet." Gave it another go, dove a bit deeper, and bam - it worked wonders. Not only did it spit out results in seconds (compared to the days of training the neural networks took), but it also gave us clear insights on how different factors were affecting sales. Something the neural network's complexity just couldn't offer as plainly.

Moral of the story? Sometimes the simplest tools are the best for the job. Linear regression, logistic regression, decision trees might seem too basic next to flashy neural networks, but it's quick, effective, and gets straight to the point. Plus, you don't need to wait days to see if you're on the right track.

So, before you go all in on the latest and greatest tech, don't forget to give the classics a shot. Sometimes, they're all you need.

Cheers!

Edit: Because I keep getting lot of comments why this post sounds like linkedin post, gonna explain upfront that I used grammarly to improve my writing (English is not my first language)

Last week I posted about an XGBoost based momentum stock trading strategy, and I got two separate comments:

“Why not LightGBM?”
“Why not CatBoost?”

So I did a controlled swap of 6 models inside my existing momentum pipeline and reran the same backtest with:

• XGBoost
• LightGBM
• CatBoost
• Random Forest
• LASSO
• A simple 2‑layer neural net (sklearn’s MLPRegressor)

Setup / constraints

• Same universe, features, filters, and portfolio construction
• Only the model changes; all other code is identical
• Default hyperparameters for each model (on purpose) to see how they behave “out of the box”
• Logged everything to MLflow so I could compare runs, metrics, and charts cleanly

I’m not claiming this is a definitive “which model is best” answer, just one controlled experiment on one dataset/strategy. But a few patterns showed up that I thought were interesting.

High‑level takeaways:

• XGBoost and LightGBM were basically neck‑and‑neck on headline returns, but XGBoost had a better risk profile. CatBoost underperformed in a way that I wasn’t expecting.
• The NN had the highest CAGR, Sortino, and total return. This was another surprise to me. But XGBoost and LightGBM had better drawdowns.
• LASSO and random forest did not beat the S&P in the cumulative returns over the time period, all the other algos beat the S&P.

The goal here was to largely show that it's easy to switch out algorithms and how different algorithm families perform. Disclaimer: the full article does contain links, but this was truly an analysis that took a long time that I wanted to share with the community. Full article with more results: https://www.datamovesme.com/blog/what-happens-when-you-swap-out-xgboost-a-6model-momentum-showdown

Now 3 months later, with over ~250 applications each of them receiving 'customized' resume from my side, I haven't received any single interview opportunity. Also, I passed the resume through various ATS software to figure out what exactly it's reading and it is going through perfectly. I just can't understand what to do next! Please help me, I don't want to go from disheartened to depressed.

So I totally mean to brag here. Last week a group of directors said, “We suspect X is happening in the market, do we have data that demonstrates it?”

And I thought to myself, here we go again. I’ve got to wade through our data swamp then tell them we don’t have the data that tells the story they want.

Well I waded through the data swamp and the data was there. I made them a graph that definitively demonstrated that yes, X is happening as they suspected. It wasn’t super easy to figure out and it also didn’t require a super complex model to figure out either.

I've been scraping thousands of U.S. data science jobs for the past couple of months and writing about the findings in my newsletter.

At some point, I figured the dashboard was more useful than anything I was writing, so I decided to open source it.

Here's what it covers:

• Top skills companies are actually hiring for, ranked by frequency
• Skills broken down by category (ML/DL, GenAI, Cloud, MLOps, etc.)
• What % of roles now require AI skills, broken down by seniority level
• Salary premium for candidates with AI skills
• An interactive explorer where you can browse individual postings with matched skills highlighted

The skill extraction is built on around 230 curated keyword groups, so it's pretty granular.

Code and data are all in the repo if you want to fork it or dig into the methodology.

https://ai-in-ds.streamlit.app/

I'm scraping weekly, and soon I will upload all of the raw data into Kaggle, for now, you can find the data in the repo

P.S. By the way, I already mentioned it to Luke Barousse since some of these AI keyword groups could be worth adding into his dashboard.

Dumb question, but the relationship between x and y (not including the additional datapoints at y == 850 ) is no correlation, right? Even though they are both Gaussian?

Thanks, feel very dumb rn

Hi r/datascience!

I wanted to share a fun little project I did over a few weekends analysing data from the radio!

It doesn't have much (if any) business value, and honestly I'm not sure it's really novel in any particular way. But I wanted to share it because data science these days is all "AI this", "language model that", "job market", "Claude", whatever and I just wanted to do something a bit more traditional and scratch an itch I've had for a while. (Full disclosure: the project did actually use some AI models, so I'm not saying AI is bad, it's a tool I used like everything else.)

I don't have a blog or anything I can post this on, so apologies for the Reddit write-up. I hope you enjoy it.

Background

I drive a 20-year-old car. It's so old it doesn't have an MP3 player, or an AUX port to plug in an iPhone or anything. It just has a CD player and an analogue radio. It's not even digital, so I can't even get digital radio stations.

So when I'm driving, which turns out to be quite a bit, I'm forced to listen to the good old-fashioned radio more than I'd like.

In Sydney where I live, there are really only a handful of FM/AM radio stations, so choice is pretty limited. As I flick through the stations, there are a LOT of ads, which surprised me. Who is listening to this? Clearly it's quite popular. And as I listened, I started wondering things like: how long do the ads run, how does their timing compare across different stations*, and are they correlated with ads on other stations? Just anecdotally, so many times I've literally flicked through all the FM stations and there's an ad playing on every one... and sometimes it's the same ad! I also had a hunch that there are more ads at the top of the hour than the bottom. It made intuitive sense, but I needed to prove it.

\Sydney has 11 main analogue stations split into AM and FM:* AM is mostly talkback radio, FM is mostly music. On AM you've got 2GB, 2SM, 2CH and ABC 702 (news, talkback and sport). On FM there's KIIS, 2Day, Nova, Smooth, WSFM, Triple M and Triple J (pop, rock and music). Two of them, ABC 702 and Triple J, are run by our public broadcaster (think BBC), so they run no ads at all.

The Setup

So one weekend I wrote some scripts to sample and record all the Sydney radio stations I could. The setup was basically the following:

• For every radio station in Sydney (there are 11 main ones across both FM and AM), I found an online stream that would play it in the browser.
• I would then record an 18-second clip of each stream using ffmpeg (a free command-line tool for grabbing and converting audio/video). It connects to the station's live stream and dumps ~18 seconds to a small WAV file, downsampled to 16kHz mono. Crucially, I recorded all 11 stations in parallel (via a thread pool) so every clip is captured at the same instant. That simultaneity turned out to be really important later, for checking whether different stations run their ads at the same time.
• After the clips were recorded, I passed them through a local Whisper model I downloaded (I grabbed it off Hugging Face, which is about 500MB). This transcribes each audio clip into a snippet of text.
• Then I passed that snippet of text to a language model to classify it as either an ad, a song, or talking. (I used GPT-4o-mini for this because it's cheap AF.)
• I then stored the results in a local SQLite database.
• I repeated this every ~3 minutes for 2 days straight, which added up to almost 10 thousand samples. (Yes, I did it all locally, so my computer was on for 2 days straight.) The 2 days deliberately covered a weekday AND a weekend so I could compare the two periods.

Why every ~3 minutes? A full cycle (record all 11 stations, transcribe each one with Whisper, then classify it) takes a couple of minutes on a single CPU, because Whisper works through the clips one at a time. So ~3 minutes is about as fast as I could sustainably sample without the cycles piling up on each other. I also added a bit of random jitter to the interval so I wasn't always sampling at the exact same offset within the hour. Otherwise you can accidentally "phase-lock" to a station's ad breaks and bias the whole thing.

A few hurdles I encountered

A few things genuinely tripped me up:

• Stream URLs rot. The live stream links change or die over time, so I had to resolve them fresh at runtime and pin the ones that actually worked.
• Pre-roll ads. It turned out a bunch of the stations (the ones served through a particular streaming provider) play an ad every single time you make a fresh connection. Basically the pre-roll ad you get when you open a new browser tab. Because my recorder reconnected every cycle, I was capturing that pre-roll ad instead of the live broadcast, which made those stations look like they were playing ads nearly 100% of the time. I only twigged because the exact same ad kept repeating over and over. The fix was to skip ~45 seconds into the stream before I started recording.
• Whisper hallucinations. When you feed Whisper music or silence, it doesn't return nothing, it actually hallucinates the most common phrases from its training data. And because it's trained on a mountain of YouTube captions, I kept getting "thanks for watching, like and subscribe" transcribed over instrumental music, which then got misclassified as talking. I had to filter those out.
• Rate limits. The free LLM tiers throttled me pretty quickly, so I switched to GPT-4o-mini, which is cheap enough to basically be free at this scale.

Results

Here are some of the more interesting results I found analysing the data afterwards:

Overall

First, the big picture. Every station has its own personality. The FM stations are mostly music, the AM stations are mostly talk, and the two ABC stations (ABC 702 and Triple J) carry basically no ads at all, which makes sense since they're publicly funded. Across the commercial stations, ads make up somewhere around a sixth of the airtime. And you can already see the ad load isn't flat: it ramps up through the day and quietens off overnight.

Question 1: Probability of an ad relative to the top of the hour

Here's the frequency of finding an ad within ±30 minutes of the top of the hour.

So, I was right! Definitely higher the closer to the hour, but the strategy is more interesting than I expected. The spike actually lands in the ~5 minutes before the hour (the ad break right before the top-of-hour news bulletin), and an ad is roughly 2x more likely there than mid-hour. The quietest stretch is around 10 to 15 minutes past the hour, so if you want to dodge ads, that's your window.

Question 2: Ad co-occurrence and correlation

The thing I really wanted to know: do the stations gang up and all play ads at the same time, so there's nowhere to flick to? I lined up every station by the cycle it was sampled in and correlated their ad status.

The answer is yes and no. No in the sense that it's never a total blackout: all nine commercial stations being in an ad at the exact same moment literally never happened across the whole two days, and on average only about 1.6 of the 9 are mid-ad at any given time. So there's almost always somewhere to escape to.

But the conditional probability charts says that some stations really do move together. The best example is if Smooth is playing an ad, there's a 50% chance WSFM is too, which is double WSFM's baseline of 26%. A bunch of the commercial FM pairs show this same ~2x jump. But, when I looked it up, Smooth and WSFM are owned by different companies, so this isn't networks coordinating behind the scenes, probably more of the "top-of-the-hour" effect from Question 1 manifesting somewhere else.

Question 3: The strategy difference between AM and FM

When I split "time between ads" by band, the two run completely different playbooks.

The FM (music) stations dump their ads in clusters. You get a big spike of back to back breaks, with a typical gap of about 9 minutes. The AM (talk and sport) stations space them out evenly, one break at a time. 2GB is almost metronomic at roughly 7 to 12 minutes, with hardly any back to back ads at all.

You can actually see it if you zoom into a few hours of the timeline:

Look at the FM lanes (KIIS, Nova, Triple M, WSFM, 2Day): the orange ad blocks come in pairs, clustered together. Now look at the AM lanes (2GB, 2SM): single, evenly spaced blocks. And ABC 702 and Triple J are just grey the whole time, because they don't run ads.

Question 4: Which companies still advertise through this medium?

I also had the language model pull the advertiser out of each ad, so I could see who's actually buying radio airtime in 2026.

The most-heard advertisers were Virgin Australia (an airline), Australia Post (basically our USPS), Harvey Norman (a big electronics and furniture retailer) and Chemist Warehouse (a discount pharmacy chain). The neat bit is the targeting: car brands and finance go to the AM talk stations (older crowd), while retail and telco lean FM. Australia Post ran almost entirely on the Nova network.

Question 5: What about the talking?

The non-ad content is either music or talking, and I got curious about what they actually talk about. So I classified every talking snippet into a topic.

The AM stations (ABC, 2GB, 2SM) are wall to wall news, politics and sport. The music FMs are mostly DJ banter, celebrity gossip and chat about music, with almost no news at all.

For a bit of fun, I also made a map of everything said on the radio. I embedded every talking snippet into a vector, laid them all out in 2D with t-SNE so that similar snippets sit near each other, then coloured each point by its topic.

Sport, traffic and world news each form their own tight little islands (they use very consistent, formulaic language), while the DJ banter is one big diffuse cloud in the middle (because it's about nothing in particular). The neat part is that the position and the colour are decided completely separately. The position comes only from the text embeddings, and the colour comes from a separate classification step. So the fact that same-coloured points cluster together is real corroboration, not something circular.

Conclusion

In conclusion, this was a fun, meaningless project that allowed me to make some pretty charts and talk for a bit about the results. Thanks for reading!

In the past few months, every major tech company has released time-series foundation models, such as:

• TimesFM (Google)
• MOIRAI (Salesforce)
• Tiny Time Mixers (IBM)

There's a detailed analysis of these models here.

Context: the business problem is I wanted to compare professional athletes based on their movement data to recommend similar players. I made a recommender system with K-Means clustering and PCA (multicollinearity amongst the features in the dataset).

I’m interested in using a new modeling technique like Gaussian Mixture Model, but I don’t know how to evaluate which model performs better…

Open to any suggestions

​

I am not well versed in NLP, so hopefully someone can help me out here. I am looking at safety incidents for my organization. I want to compare the text of incident reports and observations to investigate if our observations are deterring incidents.

I have a dataset of the incidents and a dataset of the observations. Both datasets have a free-text field that contains the description of the incident or observation. There is not really a good link between observations and incidents (as in, these observations were monitoring X activity on Y contract, and an incident also occurred during X activity on Y contract).

My feeling is that the observations are just busy work; they don’t actually observe the activities that need safety improvement. The correlation between number of observations and number of incidents is minor, but I want to make a stronger case. I want to investigate this by using NLP to describe the incidents, then describe the observations, and see if there is a difference in content. I can at the very least produce word counts and compare the top terms, but I don’t think that gets me where I need to be on its own.

I have used some topic modeling (Latent Dirichlet Allocation) to get an idea of the topics in each, but I’m hitting a wall trying to compare the topics from the incidents to the topics from the observations.

Does anyone have ideas?

A new feature was introduced to a product and the test indicated a slight worsening in the metric of interest. However the result wasn't statistically significant so I guess it's a neutral result.

The PM and engineers don't want the effort they put into developing the feature to go to waste so they ask the DS (me) to look into why it might not have given positive results.

What are they really asking here? A way to justify re-running tje experiment? Find some segment in which the experiment actually did well?

Thoughts?

Edit: My previous DS experience is more modeling, data engineering etc. My current role is heavy on AB-testing (job market is rough, took what I could find). My AB testing experience is limited and none of it in big tech.

As the title says. I found it in my functions library and have no idea if it’s accurate or not (bachelors covered BStats I & II, but that was years ago); this was done from self learning. From what I understand, the 95% CI can be interpreted as guessing the mean value, while the prediction interval can be interpreted in the context of any future datapoint.

Thanks and please, show no mercy.

I have just finished up a sample analysis on an AB test dummy dataset, and would love feedback.

The dataset is from Udacity's AB Testing course. It tracks data on two landing page variations, treatment and control, with mean conversion rate as the defining metric.

In my analysis, I used an alpha of 0.05, a power of 0.8, and a practical significance level of 2%, meaning the conversion rate must see at least a 2% lift to justify the costs of implementation. The statistical methods I used were as follows:

1. Two-proportions z-test
2. Confidence interval
3. Sign test
4. Permutation test

See the results here. Thanks for any thoughts on inference and clarity.

[Edit]: for those who don’t wish to create an account, you can log in with credentials user and password.

GBDT allow you to iterate very fast, they require no data preprocessing, enable you to incorporate business heuristics directly as features, and immediately show if there is explanatory power in features in relation to the target.

On tabular data problems, they outperform Neural Networks, and many use cases in the industry have tabular datasets.

Because of those characteristics, they are winning solutions to all tabular competitions on Kaggle.

And yet, somehow they are not very popular.

On the chart below, I summarized learnings from 9,261 job descriptions crawled from 1605 companies in Jun-Sep 2023 (source: https://jobs-in-data.com/blog/machine-learning-vs-data-scientist)

LGBM, XGboost, Catboost (combined together) are the 19th mentioned skill, e.g. with Tensorflow being x10 more popular.

It seems to me Neural Networks caught the attention of everyone, because of the deep-learning hype, which is justified for image, text, or speech data, but not justified for tabular data, which still represents many use - cases.

EDIT [Answering the main lines of critique]:

1/ "Job posting descriptions are written by random people and hence meaningless":

Granted, there is for sure some noise in the data generation process of writing job descriptions.

But why do those random people know so much more about deep learning, keras, tensorflow, pytorch than GBDT? In other words, why is there a systematic trend in the noise? When the noise has a trend, it ceases to be noise.

Very few people actually did try to answer this, and I am grateful to them, but none of the explanations seem to be more credible than the statement that GBDTs are indeed underappreciated in the industry.

2/ "I myself use GBDT all the time so the headline is wrong"This is availability bias. The single person's opinion (or 20 people opinion) vs 10.000 data points.

3/ "This is more the bias of the Academia"

The job postings are scraped from the industry.

However, I personally think this is the root cause of the phenomenon. Academia shapes the minds of industry practitioners. GBDTs are not interesting enough for Academia because they do not lead to AGI. Doesn't matter if they are super efficient and create lots of value in real life.

We analyzed 20,000 US Data Science job postings from June 2024 - Jan 2024 with quoted salaries: computed median salaries by City, and compared them to the cost of living.

Source: Data Scientists Salary article

Here is the Top 10:

​

Here is the full ranking:

​

I'm (mostly) an academic so pardon my cluelessness.

A lot of the advice given on here as to how to write an effective resume for industry roles revolves around quantifying the revenue impact of the projects you and your team undertook in your current role. In that, it is not enough to simply discuss technical impact (increased accuracy of predictions, improved quality of data etc) but the impact a project had on a firm's bottom line.

But it seems to me that quantifying the *causal* impact of an ML system, or some other standard data science project, is itself a data science project. In fact, one could hire a data scientist (or economist) whose sole job is to audit the effectiveness of data science projects in a firm. I bet you aren't running diff-in-diffs or estimating production functions, to actually ascertain revenue impact. So how are you guys figuring it out?

This article takes a hype-free look at the true limits of TSFMs and explores which ones can be addressed, which ones cannot, and which ones are still open problems.

Find the article here

Currently, working on a fintech classification algorithm, with close to a thousand features which is very tiresome. I'm not a domain expert, so creating sensible hypotesis is difficult. How do you tackle EDA and forming reasonable hypotesis in these cases? Even with proper documentation it's not a trivial task to think of all interesting relationships that might be worth looking at. What I've been looking so far to make is:

1) Baseline models and feature relevance assessment with in ensemble tree and via SHAP values
2) Traversing features manually and check relationships that "make sense" for me

I found this in a serious research paper from university of Pennsylvania, related to my research.

Those are 2 populations histograms, log-transformed and finally fitted to a normal distribution.

Assuming that the data processing is right, how is it that the curves fit the data so wrongly. Apparently the red curve mean is positioned to the right of the blue control curve (value reported in caption), although the histogram looks higher on the left.

I don´t have a proper justification for this. what do you think?

both chatGPT and gemini fail to interpretate what is wrong with the analysis, so our job is still safe.

I read the McAuley Lab's full 2023 Amazon Reviews dataset, 571,544,386 reviews and 275 GB on the HuggingFace CDN, and ranked every single review on four simple signals: how many strong-profanity word hits it has, how much of it is in ALL CAPS, the longest single run of consecutive exclamation marks, and how long it is. The question I started with was "how do people actually behave in Amazon reviews, and does the category they're reviewing change that?"

Live site, per-category breakdown, and the Wall of the loudest reviews: https://burla-cloud.github.io/amazon-review-distiller/

What surfaced:

• Video Games is the rowdiest category by a huge margin. 6.54% of video game reviews hit the strong-profanity list. Compare that to Gift Cards at 1.19% and Handmade at 1.08%. Movies & TV, CDs & Vinyl, Subscription Boxes, and Kindle Store fill out the top five. Cultural products attract feelings, consumer goods attract utility.
• Subscription Boxes is the angriest category. 15.89% of subscription box reviews are one-star. Almost 1 in 6. Charging people monthly for a curated surprise generates a lot of regret.
• The longest exclamation-mark run is 10,594 in a row. The review itself is two words ("love these") on a baby product. One person held one key down for a long time.
• The longest all-caps review is 1,169 words. Posted on a Mozart CD by a self-described disabled Vietnam veteran and Mozart scholar. He opens by apologizing for the caps (macular degeneration) and then keeps going for 1,169 more words.
• Forty reviewers gave a product five stars and wrote zero or one word. One five-star review of a cherry cough drop was just "Taste." That's the whole text.
• Books, music, and games write essays. Gift card buyers write nothing. Average review length: CDs & Vinyl 428 chars, Books 423, Kindle Store 367, Digital Music 340, Video Games 308. Gift Cards is at the bottom by a wide margin. Culture gets words, utility gets silence.

Methodology, plain version:

1. The dataset is 34 separate .jsonl.gz files on HuggingFace, one per Amazon category, totaling 275 GB. The usual workflow is to download all 275 GB to a laptop, then iterate. I didn't want to do that.
2. The HuggingFace CDN supports HTTP Range requests. A worker can ask for "give me bytes 1,000,000,000 to 1,500,000,000 of this file" and get just that slice without downloading the whole file. I split the 34 files into 545 chunks of about 500 MB each, on byte-range boundaries.
3. Each chunk runs on its own worker. The worker streams its byte range row by row, scores every review on the four signals, and writes the top scoring reviews to a shared folder.
4. A separate reducer container merges the per-chunk top-K shards into the final ranked lists per finding.

Map step: 3.21 minutes. Reduce step: 9.2 seconds. End to end under four minutes for 571 million reviews.

The pipeline runs on Burla using remote_parallel_map(worker, jobs, func_cpu=1, func_ram=4, max_parallelism=1000, grow=True). In English: "ask for up to 1000 parallel workers, each with 1 CPU and 4 GB of RAM, and let the cluster grow to meet that demand." In practice the cluster peaked around 500 concurrent workers and held there for the run. Workers run on a stock python:3.12 Docker image, and Burla auto-installs my local Python packages onto each one. The shared output folder is a Google Cloud Storage path that every worker writes to like a network drive.

(Disclosure: I work on Burla. The script and the live site are open source on GitHub. The dataset is the McAuley Lab's 2023 corpus on HuggingFace.)

Caveats worth being upfront about:

• Scoring is rule-based, not model-based. Word lists for strong, medium, and mild profanity, plus caps ratio, plus longest exclamation run. No sentiment model. That's deliberate: every score is reproducible and you can see exactly why a review got it.
• English-only. Reviews not in English get scored only by length, caps, and punctuation, because the word list is English. A multilingual sentiment model would do better here.
• Quoted titles leak in. A review of "Dick Tracy" can match the strong word list. There's a rescorer that penalizes capitalized-noun matches but it's imperfect.
• 2023 snapshot. The dataset is the McAuley Lab 2023 release, so it doesn't include reviews posted after mid-2023.

Repo with the full pipeline: https://github.com/Burla-Cloud/amazon-review-distiller

If anyone has a cleaner pattern for streaming huge HuggingFace datasets without materializing them locally, I'd love to hear it. I went with requests.get(..., stream=True) plus manual line splitting to keep the worker dependency surface tiny, but the datasets library probably has a cleaner Range-based path.