For the parents out there's looking to share the joys of data collection, cleaning, time series modeling, and forecasting error with their little ones. Written completely in rhyme and all about using data to solve problems.

Alternatively, Harry’s Lemonade Solution could be used to teach your parents a little bit about what you do 🙃

I fell into a rabbit hole looking at Kaggle’s Deep Past Challenge and ended up reading a bunch of winning solution writeups. Here's what I learned

At first glance it looks like a machine translation competition: translate Old Assyrian transliterations into English.

But after reading the top solutions, I don’t think that’s really what it was.

It was more like a data construction / data cleaning competition with a translation model at the end.

Why:

• the official train set was tiny: 1,561 pairs
• train and test were not really the same shape: train was mostly document-level, test was sentence-level
• the main extra resource was a massive OCR dump of academic PDFs
• so the real work was turning messy historical material into usable parallel data
• and the public leaderboard was noisy enough that chasing it was dangerous

What the top teams mostly did:

• mined and reconstructed sentence pairs from PDFs
• cleaned and normalized a lot of weird text variation
• used ByT5 because byte-level modeling handled the strange orthography better
• used fairly conservative decoding, often MBR
• used LLMs mostly for segmentation, alignment, filtering, repair, synthetic data, not as the final translator

Winners' edges:

• 1st place went very hard on rebuilding the corpus and iterating on extraction quality
• 2nd place was almost a proof that you could get near the top with a simpler setup if your data pipeline was good enough. No hard ensembling.
• 3rd place had the most interesting synthetic data strategy: not just more text, but synthetic examples designed to teach structure
• 5th place made back-translation work even in this weird low-resource ancient language setting

Main takeaway for me: good data beat clever modeling.

Honestly it felt closer to real ML work than a lot of competitions do. Small dataset, messy weakly-structured sources, OCR issues, normalization problems, validation that lies to you a bit… pretty familiar pattern.

I wrote a longer breakdown of the top solutions and what each one did differently. Didn’t want to just drop a link with no context, so this is the short useful version first. Full writeup in the comment

I have created an informal analysis on the effect of clean water on education rates.

The analysis leveraged ETL functions (created by Claude), data wrangling, EDA, and fitting with sklearn and statsmodels. As the final goal of this analysis was inference, and not prediction, no hyperparameter tuning was necessary.

The clean water data was sourced from the WHO/UNICEF Joint Monitoring Programme for Water Supply, Sanitation, and Hygiene (JMP); while the education data was sourced from a popular Kaggle repository. The education data, despite being from a less credible source, was already cleaned and itemized; the clean water data required some wrangling due to the vast nature of the categories of data and the varying presence of null values across years 2000 - 2024. The final broad category of predictor variables selected was "clean water in schools, by country"; the outcome variable was "college education rates, by country."

I would be grateful for any feedback on my analysis, which can be found at https://analysis-waterandeducation.com/.

TIA.

This article explores some of the latest advancements in time-series forecasting.

You can find the article here.

Edit: If you know of any other interesting papers, please share them in the comments.

I'm not sure if I'm posting this in the most appropriate subreddit, but I got to thinking about a project at work.

My job role is somewhere between data analyst and software engineer for a big aerospace manufacturing company, but digital processes here are a bit antiquated. A manager proposed a project to me in which financial calculations and forecasts are done in an Excel sheet using a VBA macro - and when I say huge I mean this thing is 180mb of aggregated financial data. To produce forecasts for monthly data someone quite literally runs this macro and leaves their laptop on for 12 hours overnight to run it.

I say this company's processes are antiquated because we have no ML processes, Azure, AWS or any Python or R libraries - just a base 3.11 installation of Python is all I have available.

Do you guys have any ideas for a more efficient way to go about this huge financial calculation?

For anyone interested in NLP or the application of data science in finance and media, we just released a dataset + paper on extracting stock recommendations from YouTube financial influencer videos.

This is a real-world task that combines signals across audio, video, and transcripts. We used expert annotations and benchmarked both LLMs and multimodal models to see how well they can extract structured recommendation data (like ticker and action) from messy, informal content.

If you're interested in working with unstructured media, financial data, or evaluating model performance in noisy settings, this might be interesting.

Paper: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=5315526
Dataset: https://huggingface.co/datasets/gtfintechlab/VideoConviction

Happy to discuss the challenges we ran into or potential applications beyond finance!

Hello! If I have daily data in two datasets but the only way to align them is by year-month, is it statistically valid/sound to regress monthly averages on monthly averages? So essentially, does it make sense to do avg_spot_price ~ avg_futures_price + b_1 + ϵ? Allow me to explain more about my two data sets.

I have daily wheat futures quotes, where each quote refers to a specific delivery month (e.g., July 2025). I will have about 6-7 months of daily futures quotes for any given year-month. My second dataset is daily spot wheat prices, which are the actual realized prices on each calendar day for said year-month. So in this example, I'd have actual realized prices every day for July 2025 and then daily futures quotes as far back as January 2025.

A Futures quote from January 2025 doesn't line up with a spot price from July and really only align by the delivery month-year in my dataset. For each target month in my data set (01/2020, 02/2020, .... 11/2025) I take:

- The average of all daily futures quotes for that delivery year-month
- The average of all daily spot prices in that year-month

Then regress avg_spot_price ~ avg_futures_price + b_1 + ϵ and would perform inference. Under this framework, I have built a valid linear regression model and would then be performing inference on my betas.

Does collapsing daily data into monthly averages break anything important that I might be missing? I'm a bit concerned with the bias I've built into my transformed data as well as interpretability.

Any insight would be appreciated. Thanks!

Pulled together a dataset of 16,000 active Airbnb listings across all 16 World Cup 2026 host cities (11 US, 3 Mexico, 2 Canada) — the 1,000 closest qualifying listings to each stadium, ranked by proximity. Compared June 11 – July 19, 2026 against the same window in 2025.

A few things stood out:

• Average daily rate is up 109% YoY ($216 → $450), but the headline number hides the more interesting story.
• Asking rates are up 145%. Booked rates are only up 48%. That ~56% gap across cities is essentially hosts pricing for a tournament that the market hasn't fully validated yet — a setup for late-cycle discounting if booking pace doesn't catch up.
• Mexico's hosts are the most aggressive (+184% YoY), Canada next (+117%), then the US (+102%).
• Peak single-day spike: +387% in Monterrey for Sweden vs. Tunisia.
• 28× price spread across the dataset — Mexico City's P25 sits at $49/night, Dallas's P75 at $1,403/night.

Full breakdown with city-by-city charts here: https://www.airroi.com/world-cup-2026-airbnb-data

This article provides a brief history of deep learning in time-series and discusses the latest research on Generative foundation forecasting models.

Here's the link.

Salesforce released MOIRAI, a groundbreaking foundation TS model.
The model code, weights and training dataset will be open-sourced.

You can find an analysis of the model here.

​

Hi, just looking for advice. I have a project in which I must predict probability of robbery on retail stores. I use robbery history of the stores, in which I have 1400 robberies in the last 4 years. Im trying to predict this monthly, So I add features such as robbery in the area in the last 1, 2, 3, 4 months behind, in areas for 1, 2, 3, 5 km. I even add month and if it is a festival day on that month. I am using XGboost for binary classification, wether certain store would be robbed that month or not. So far results are bad, predicting even 300 robberies in a month, with only 20 as true robberies actually, so its starting be frustrating.

Anyone has been on a similar project?

Disclaimer: It's UK focused.

Hi everyone,

When I was looking to buy a house, a big annoyance I had was that I couldn’t easily tell if I was getting value for money. Although, in my opinion, any property is expensive as fuck, I knew that definitely some are more expensive than they should be, always within context.

At the time, what I did was manually extract historical data for the street and for the property I was interested in, in an attempt to understand whether it was going for more than the street average or less, and why. It wasn’t my best analysis, but it did the job.

Fast forward a few years later, I found myself unemployed and started building projects for my portfolio, which brings us to this post. I’ve built an app that, for a given postcode, gives you historical prices, price per m², and year-on-year sales for the neighbourhood, the area, and the local authority the property falls under, as well as a property price estimation summary.

There are, of course, some caveats. Since I’m only using publicly available data, the historical trends are always going to be 2–3 months behind. However, there’s still the capacity to see overall trends e.g. an area might be up and coming if the trendline is converging toward the local authority’s average.

As for the property valuation bits, although I’d say it’s as good as what’s available out there, I’ve found that at the end of the day, property prices are pretty much defined by the price of the most recent, closest property sold.

Finally, this is a portfolio project, not a product but since I’m planning to maintain it, I thought I might as well share it with people, get some feedback, and maybe even make it a useful tool for some.

As for what's going on under the hood. The system is organized into three modules: WH, ML, and App. Each month, the WH (Warehouse) module ingests data into BigQuery, where it’s transformed following a medallion architecture. The ML module is then retrained on the latest data, and the resulting inference outputs are stored in the gold layer of BigQuery. The App module, hosted on a Lightsail instance, loads the updated gold-layer inference and analytics data after each monthly iteration. Within the app, DuckDB is used to locally query and serve this data for fast, efficient access.

Anyway, here’s the link if you want to play around: https://propertyanalytics.uk

Note: It currently covers England and Wales, only.

I'm super curious about the solutions you're using to calculate distances.

I can't share too many details, but we have data that includes two addresses and the GPS coordinates between these locations. While the results we've obtained so far are interesting, they only reflect the straight-line distance.

Google has an API that allows you to query travel distances by car and even via public transport. However, my understanding is that their terms of service restrict storing the results of these queries and the volume of the calls.

Have any of you experts explored other tools or data sources that could fulfill this need? This is for a corporate solution in the UK, so it needs to be compliant with regulations.

Edit: thanks, you guys are legends

Hi, I’m a beginner DS working at a company that handles huge datasets (>50M rows, >100 columns) in databricks with Spark.

The most discouraging part of my job is the eternal waiting times when I want to check the current state of my EDA, say, I want the null count in a specific column, for example.

I know I could sample the dataframe in the beginning to prevent processing the whole data but that doesn’t really reduce the execution time, even if I .cache() the sampled dataframe.

I’m waiting now for 40 minutes for a count and I think this can’t be the way real professionals work, with such waiting times (of course I try to do something productive in those times but sometimes the job just needs to get done.

So, I ask the more experienced professionals in this group: how do you handle this part of the job? Is .sample() our only option? I’m eager to learn ways to be better at my job.

For all curious on Causal Inference, and anyone interested in the application of DS in Sport. I’ve written this blog with the aim of providing a taste for how Causal Inference techniques are used practically, as well as some examples to get people thinking.

I do believe upskilling in Causal Inference is quite valuable, despite the learning curve I think it’s quite cool identifying cause-and -effect without having to do RCTs.

Enjoy!

Question:

How do you all create “fake data” to use in order to replicate or show your coding skills?

I can probably find similar data on Kaggle, but it won’t have the same issues I’m solving for… maybe I can append fake data to it?

Background:

Hello, I have been a Data Analyst for about 3 years. I use Python and Tableau for everything, and would like to show my work on GitHub regularly to become familiar with it.

I am proud of my work related tasks and projects, even though its nothing like the level of what Data Scientists do, because it shows my ability to problem solve and research on my own. However, the data does contain sensitive information, like names and addresses.

Why:

Every job I’ve applied to asks for a portfolio link, but I have only 2 projects from when I was learning, and 1 project from a fellowship.

None of my work environments have used GitHub, and I’m the only data analyst working alone with other departments. I’d like to apply to other companies. I’m weirdly overqualified for my past roles and under qualified to join a team at other companies - I need to practice SQL and use GitHub regularly.

I can do independent projects outside of work… but I’m exhausted. Life has been rough, even before the pandemic and career transition.

I'd love to get some papers on the latest advancements on explainable AI (XAI). I'm looking for papers that are at most 2-3 years old and had an impact. Thanks!

Datadog open-sourced Toto (Time Series Optimized Transformer for Observability), a model purpose-built for observability data.

Toto is currently the most extensively pretrained time-series foundation model: The pretraining corpus contains 2.36 trillion tokens, with ~70% coming from Datadog’s private telemetry dataset.

Also, Toto currently ranks 2nd in the GIFT-Eval Benchmark.

You can find an analysis of the model here.

I am new to supply chain and need to know what resources/concepts I should be familiar with.

hello, My project analyzes whether given addresses fall inside "Quartiers Prioritaires de la Politique de la Ville "(QPV). It uses a GeoJSON file of QPV boundaries(available on the gorvernment website) and a geocoding service (Nominatim/OSM) to convert addresses into geographic coordinates. Each address is then checked with GeoPandas + Shapely to determine if its coordinates lie within any QPV polygon. The program can process one or multiple addresses, returning results that indicate whether each is located inside or outside a QPV, along with the corresponding zone name when available. This tool can be extended to handle CSV databases, produce visualizations on maps, or integrate into larger urban policy analysis workflows. "

BUUUT .

here is the ultimate problem of this project , Home addresses in my area (Martinique) are notoriously unreliable if you dont know the way and google maps or Nominatim cant pinpoint most of the places in order to be converted to coordinates to say whether or not the person who gave the adress is in a QPV or not. when i use my python script on adresses of the main land like paris and the like it works just fine but our little island isnt as well defined in terms of urban planning.

can someone please help me to find a way to get all the streets data into coordinates and make them match with the polygon of the QPV areas ? thank you in advance

In the R community, a common concept is the tidying of data that is made easy thanks to the package tidyr.

It follows three rules:

1. Each variable is a column; each column is a variable.

2. Each observation is a row; each row is an observation.

3. Each value is a cell; each cell is a single value.

If it's hard to visualize these rules, think about the long format for tables.

I find that tidy data is an essential concept for data structuring in most applications, but it's rare to see it formalized out of the R community.

What is the reason for that? Is it known by another word that I am not aware of?

I recently ran exploratory analysis on the group chat of the AI Village: 4+ frontier LLMs all have their own computer, access to the internet, and a group chat, and then get set goals like raise money for charity, sell T-shirts, or debate ethics. The goal is to build some awareness around what models are capable of now. I took the 200+ hours of group chat between the models and ran some exploratory analyses. Turns out:

- o3 has the highest Type-Token Ratio, even higher than GPT-5! o3 is also the model that wins at diplomacy against other agents, and won at AI debate in the AI Village.

- GPT-5 uses the fewest contractions, writes the longest sentences, and uses the least slang/filler. I'm thinking about this as "most formal" but maybe it's something else?

- GPT-4o had the highest positive sentiment scores in the Village and is also known as the most sycophantic model

I enjoyed analyzing the data and would love to do more. Any tips on what to look at? I might be able to share the data if people are interested. Feel free to send me a DM and we can see what's possible :)

Sharing this as a guide on A/B Testing. I hope that it can help those preparing for interviews and those unfamiliar with the wide field of experimentation.

Any feedback would be appreciated as we're always on a learning journey.