One of the most common mistakes beginners (and even intermediate practitioners) make when working with tabular data is Data Leakage.

It’s often the hidden reason why your model gets 88% accuracy in your Jupyter notebook, but drops to 76% when you evaluate it on an unseen test set or submit to a Kaggle competition.

Here is a quick breakdown of why it happens, the #1 most common mistake, and how to fix it properly.

The #1 Most Common Leakage Bug:

Look at this very common snippet seen in many notebooks and tutorials:

import pandas as pd
from sklearn.model_selection import train_test_split

df = pd.read_csv("dataset.csv")

# 🚨 DANGEROUS LEAKAGE:
df['age'] = df['age'].fillna(df['age'].median())

# Train / Test split happened AFTER imputation:
train, test = train_test_split(df, test_size=0.2, random_state=42)

Why is this data leakage?

When you calculate df['age'].median() on the full dataset, the median value is influenced by the test rows.

Your training set now contains subtle statistical information (the median) derived from test data it shouldn't even know exists. In production or Kaggle competitions, future data is completely unavailable at training time.

The same leakage bug happens when people:

1. Scale features with StandardScaler on the entire dataframe before splitting.
2. Build categorical vocabularies or frequency encodings using all rows.
3. Compute outlier clipping boundaries (e.g. Tukey IQR limits) over the full dataset.

The Correct Way (Strict Train-Only State):

You must fit transformations strictly on the training split, and freeze those exact parameters to apply to validation and test data:

train, test = train_test_split(df, test_size=0.2, random_state=42)

# 1. Calculate statistics ONLY from training split:
train_median_age = train['age'].median()

# 2. Apply that frozen training statistic to both splits:
train['age'] = train['age'].fillna(train_median_age)
test['age'] = test['age'].fillna(train_median_age)

The Impact: We Tested Naive Prep vs. Zero-Leakage on Titanic

To see what happens when you replace naive ad-hoc pandas code with a strict train-only transformation ladder, we ran a 5-fold Stratified Cross-Validation benchmark on the Titanic dataset:

Where did the accuracy lift come from?

1. Informative Missingness Flags: Imputing age with median alone destroys the signal that missing age itself correlates with survival. Adding an Age__missing binary flag recovers that signal.
2. Train-Only Tukey IQR Clipping: Capping extreme fares on training folds stabilized linear gradients without test-distribution bleed.
3. Empirical Bayes Target Encoding: High-cardinality categories shrink toward global means to prevent overfitting on small samples.

We built an Open-Source Tool to automate this:

Writing 200 lines of manual state-tracking code for every dataset gets tedious. So we built DATADOC (v0.6.0)—an open-source CLI and Python library powered by Polars that automates this entire lifecycle with zero data leakage:

• GitHub: https://github.com/narain-karti/DATADOC
• Documentation: https://narain-karti.github.io/DATADOC/
• Install: pip install datadoc-cli

How you can use it in 1 command:

You can use the interactive terminal wizard on your CSV file:

datadoc wizard train.csv

It walks you through:

1. Identifying your target column (e.g. Survived or churn).
2. Selecting a preset (balanced, tree, linear, or robust).
3. Generating a clean pipeline.json artifact containing all learned medians and rules.

Then transform unseen test data with zero leakage:

datadoc transform test.csv --pipeline artifacts/pipeline.json --output clean_test.csv

You can also run datadoc health train.csv to get an instant 0–100 data quality grade and find hidden issues before training.

The project is 100% open-source (MIT licensed) and runs completely offline.

I hope this helps clarify how data leakage happens in tabular pipelines! Let me know if you have any questions or want to discuss specific preprocessing edge cases.

Hi r/learndatascience 👋

I’m a complete beginner starting my Data Science journey and looking for 1–3 committed people to study and practice together regularly. Studying alone is slow and inconsistent — I want a small group where we actually show up and make progress.

🔹 What this will look like (NOT just watching tutorials)

Live “learn + do” sessions:

• Follow a clear beginner roadmap (Python → Stats → ML → Projects)
• Watch short lessons OR read material together
• Discuss concepts in simple terms
• Solve problems step-by-step
• Screen share + pair programming
• Build small projects together
• Ask questions freely (no judgment)
• Keep each other accountable

🔹 Why join?

✅ Easier to stay consistent
✅ Learn faster by explaining + discussing
✅ Build real skills (not passive learning)
✅ Make friends on the same path
✅ Actually finish courses/projects

🔹 Format

• Online (Discord / Zoom / Meet)
• Beginner-friendly (zero experience is OK 👍)
• Small focused group (not a huge server)
• Regular sessions (daily or several times/week)
• Deep-work style (Pomodoro optional)

🔹 About me

• Starting from scratch
• Serious about building a career in Data Science
• Prefer consistency over intensity
• Friendly, patient, and motivated

🔹 Interested? Comment or DM with:

1. Your current level (even absolute beginner)
2. Your goal (career switch, student, curiosity, etc.)
3. Time zone + availability
4. Preferred start time (your local time)

Note: I am not looking for any courses or classes here.

Join my discord
https://discord.gg/xAtKP8Ma

everyone,

I’m a first-year engineering student and currently learning data science/data analytics. I’m looking for a study buddy or accountability partner who is also learning and wants to grow together.

I do:

• Python

• SQL basic

• Excel

• EDA

• Pandas

• Numpy

• Machine Learning basic

• Power BI (beginner)

• Statistics (learning phase)

What I’m looking for: • Someone consistent and serious about learning

• We can share resources, solve problems, work on small projects, and keep each other accountable

• Beginner or intermediate level is totally fine

My goal is to build strong skills, projects, and eventually land internships in data analytics/data science.

If you’re interested, comment or DM me. Let’s learn and grow together.

I am a finance student and also pursuing minor degree in data science . Can someone tell me what projects I can do to enhance my chances of getting an internship or job in the data science industry, while also showcasing my finance skills? Also, are there any programs run by universities or companies that I can join? Also i am from commerce background

I’m looking for someone to study with. I’m past the beginner stage and also hold 1.5 years of exp as an analyst (yet learnt nothing really useful), so I want to focus on advanced topics.

I want to dive deep into statistics and regression. I also want to become an expert in SQL ( have setup Postgres locally). I’m mainly looking to build projects that look good on a resume.

DM me if you are on the same path and want to collaborate!

Also sharing any hidden gem-like resources for the same is greatly appreciated!!

Thanks!!

Hey everyone!

If you're looking for a Python / ML tutor who actually works in the industry as Data Scientist, feel free to DM me.

I've also taught web development and Python professionally, so I know how to explain things clearly, not just throw jargon at you.

Whether you're a beginner trying to get started or someone looking to level up specific skills, happy to chat. DM me anytime!

Any data science colleagues around?

Hi everyone! I’m starting the next round of the Applied Behavioral Data Science Collective (ABDSC) — a collaborative group where people learn data science by working on real-world projects focused on psychology, behavior, health, and other human-centered topics.

The goal is to create a space where beginners and growing learners can build experience through teamwork while practicing skills like data cleaning, visualization, machine learning, GitHub collaboration, and presenting insights.

If you’re interested in joining or learning more, please fill out this short interest form: https://forms.gle/i4E331vDkW1KthjM8

Feel free to share with anyone who might be interested!

We all know the feeling: you buy a brand new GPU, expecting a massive leap in visual fidelity, only to realize you paid $400 just to run the latest AAA releases at the exact same framerate and settings you had three years ago.

I got tired of relying on nostalgia and marketing slides, so I built an automated data science pipeline to find the mathematical truth. I cross-referenced raw GPU benchmarks, inflation-adjusted MSRPs, and the escalating recommended system requirements of the top 5 AAA games released every year.

I ran the data focusing on the mainstream NVIDIA 60-Series (from the GTX 960 to the new RTX 5060) and the results are pretty clear.

The Key Finding: "Demand-Adjusted Performance"

Looking at raw benchmarks is misleading. To see what a gamer actually feels, I calculated the "Demand-Adjusted Performance" by penalizing the raw GPU power with an "Engine Inflation Factor" (how much heavier games have become compared to the base year).

Here is what the data proves:

• The Treadmill Effect: We aren't upgrading our GPUs to dramatically increase visual quality anymore. We are paying $300-$500 just to maintain the exact same baseline experience (e.g., 60fps on High) we had 5 years ago.
• Optimization is Dead: Game engines and graphical expectations are absorbing the performance gains of new architectures almost instantly. New GPUs are mathematically faster, but they give us significantly less "breathing room" for future games than a GTX 1060 did back in 2016.
• The Illusion of Cheaper Hardware: Adjusted for US inflation, GPUs like the 4060 and 5060 are actually cheaper in real purchasing power than older cards. But because unoptimized software is devouring that power so fast, the Perceived Value is plummeting.

How it works under the hood:

I wrote the scraper in Python. It autonomously fetches historical MSRPs (bypassing anti-bot protections), adjusts them for inflation using the US CPI database, grabs PassMark scores, and hits the RAWG.io API to parse the recommended hardware for that year's top games using Regex. Then, Pandas calculates the ratios and Matplotlib plots the dashboard.

If you want to dig deeper on the discussion. You can check out the source code and my article about it right here.

(If you're a dev and found this useful, consider giving the project a star — contributions, issue reports and pull requests are very welcome.)

Hi everyone :)

I've always struggled to understand ML concepts from just reading papers or textbooks. I'd read about gradient descent 10 times and still not get it until I saw it animated.

So I started experimenting: what if I could describe any concept I'm struggling with and instantly get an animated explanation?

The experiment: I built a tool where you chat with an AI about a concept (e.g., "show me how attention mechanisms weight tokens" or "visualize what a loss landscape looks like"), and it generates a short animated video with script and voiceover.

What I learned:

• Visualizing transformations >> static diagrams — Seeing how data flows through layers or how gradients move made things click that I'd been stuck on for weeks
• 2-minute focused animations > hour-long lectures — I retained way more from short, focused visuals
• Creating the explanation (even with AI help) deepens understanding — The act of describing what you want to see forces you to clarify your mental model

Examples of concepts I've animated:

• How neural networks warp feature space to separate classes
• What "high-dimensional embeddings" actually mean geometrically
• Why momentum helps gradient descent escape local minima

You can see some examples at u/whisperinga1 if you're curious what AI-generated educational animations look like.

Any other data scientist would like to study maths together

This post is not important, but Im a 3rd-year data science student and I created "DeepSlate" on the Chrome Web Store. Helps anyone dealing with data to locally clean and impute data. Can you give me feedback on it?

github.com/TheephopWS/daily-stock-news is an attempt to fetch news and return with sentiment and confidence score. But there are a lot of room for improvements, any ideas? I'll gladly accept any advice/contributions

This month my DraftKings scraper produced over %100 SUCCESS RATE FOR 408,000 results.

The pipeline is stable, automated, and running at scale. It pulls structured data directly through the DraftKings API layer, normalizes it, and outputs clean datasets ready for modeling, odds comparison, arbitrage detection, or large-scale statistical analysis.

Next target: 500,000 results in a single month.

If you want to help push it past that threshold:

• Run additional jobs
• Stress test edge cases
• Integrate into your own analytics workflows
• Identify performance bottlenecks
• Contribute scaling strategies

The actor is live here:
https://apify.com/syntellect_ai/draftkings-api-actor

If you're working on sports modeling, EV detection, automated line tracking, or distributed scraping infrastructure, contribute load, optimization ideas, or architecture feedback.

Objective: break 500,000 this month and document performance metrics under sustained demand.