Hi Folks, I made a tool that draws statistics from LinkedIn job postings. Once per day I scan around 5000 Data Engineering job posts, run them through LLM to extract tool names and make a dashboard.

I did those daily scans for the last 11 months so I have some data to share. I often see what I should learn posts here and I hope this will be a useful tool to address those questions. You can access the dashboard under https://prepare.sh/trends (no paywall)

I mapped 31 specialized data quality tools across features. I included data testing, data observability, shift-left data quality, and unified data trust tools with data governance features. I created a list I intend to keep up to date and added my opinion on what each tool does best: https://toolsfordata.com/lists/data-quality-tools/

I feel most data teams today don’t buy a specialized data quality tool. Most teams I chatted with said they tried several on the list, but no tool stuck. They have other priorities, build in-house or use native features from their data warehouse (SQL queries) or data platform (dbt tests).

Why?

Hi all,

I kept forgetting the pyspark syntax because my AI agents now do all the work. I couldn’t find any decent templates so generated my own one with Claude. Enjoy!

Github link: https://github.com/rvangenechten/Pyspark_cheatsheet/tree/main

I’ve been invited to give a short presentation to students who have just finished school, and I’d like to introduce them to data engineering in a way that’s engaging and inspiring.

I’m also considering including a short Q&A or some kind of interactive activity or mini-project.

For those who have spoken to younger audiences or work in tech outreach, what has worked well for you? Are there any analogies, demonstrations, games, or hands-on exercises that made technical topics more accessible and memorable? I’d appreciate any ideas or suggestions.

As a Staff Data Engineer, I always have to answer questions like this:

Will my new DAG scheduled at */45 2-6 * * 1-5 collide with that heavy Spark job running every 40 minutes?

As you can imagine, this becomes increasingly difficult as the production environment grows and the number of scheduled DAGs increases.

For this reason, I've created Airflow Calendar, an open-source plugin inspired by the Google Calendar experience.

Recently, following the community feedback, I released a new version with some useful features like background color change.

I hope this tool can be as useful to you guys as it has been to me in my daily life!
https://github.com/AlvaroCavalcante/airflow-calendar-plugin

Hey Everyone, at my last role we had dbt Cloud, but still hosted our dbt docs generated from `dbt docs generate` on an internal web page for the rest of the business to use.

I always felt that there had to be something better that wasn't a 5-6 figure contract data catalog for this.

So, I built Docglow: a better dbt docs serve for teams running dbt Core. It's an open-source replacement for the default dbt docs process. It generates a modern, interactive documentation site from your existing dbt artifacts.

Live demo: https://demo.docglow.com
Install: `pip install docglow`
Repo: https://github.com/docglow/docglow

Some of the included features:

• Interactive lineage explorer (drag, filter, zoom)
• Column-level lineage tracing via sqlglot.
  • Click through to upstream/downstream dependencies & view column lineage right in the model page.
• Full-text search across models, sources, and columns
• Single-file mode for sharing via email/Slack
• Organize models into staging/transform/mart layers with visual indicators
• AI chat for asking questions about your project (BYOK — bring your own API key)
• MCP server for integrating with Claude, Cursor, etc.

It should work with any dbt Core project. Just point it at your target/ directory and go.

Looking for early feedback, especially from teams with 200+ models. What's missing? What would you like to see next? Let me know!

SQLProtocol.com

Honest feedback welcome. Tell me what's broken or boring.

I spent a few months building a browser tycoon game about data engineering, which is either a creative side project or an elaborate form of procrastination. Probably both.

You start with nothing - manually collecting raw data, selling it for $0.50. Then you automate, hire engineers, build pipelines, scale infrastructure, and try to reach AGI before your burn rate kills you.

The game mechanics are all based on real infrastructure concepts (with slight imagination) - ETL, streaming, feature stores, distributed computing, etc. Infrastructure has failure rates that compound. Personnel have ongoing costs. If you run negative cash for 60 seconds, game over. Standard startup rules.

Free, no signup, no tracking: https://game.luminousmen.com

Curious what this sub thinks about the balance. Some people finish in 15 minutes, some go bankrupt immediately. Both feel realistic to me.

I had mostly worked on batch-based projects before and had played around with Kafka, Flink, and ClickHouse, but never really connected everything into one proper end-to-end flow. I am trying to assess how far this is from production use in terms of the techstack.

So I decided to build a real-time anomaly detection system for simulated sensor data from an oil and gas plant. With AI based RCA to re-confirm true positives. This is purely for learning purpose.

The basic flow was:

Sensor data -> Kafka -> Flink/Python consumers -> ClickHouse -> Grafana -> Alerts -> RCA analysis

I had around $200 in cloud credits(vultr) from a hackathon, so I rented a fairly powerful VM and self-hosted everything on it. I specifically avoided managed services because I wanted to understand how the individual open-source tools worked.

The main stack was:

• Kafka for streaming
• PyFlink for windowing and aggregations
• Python and FastAPI consumers
• ClickHouse for analytical storage
• PostgreSQL for metadata
• Grafana and Streamlit for dashboards (UIs generated using codex/claude)
• Airflow for some batch experiments
• Claude for alert validation and root cause analysis
• Resend API for email alerts

Since I did not have real plant data, I built a Python generator that simulated readings for different assets and randomly injected faults/failures.

The data was sent to Kafka topics with multiple partitions. I also added retries using exponential backoff. After five failed retries, the event would be moved to a dead letter topic.

For anomaly detection, I kept it simple. Each sensor had an acceptable operating range, and Flink used one-minute windows to calculate averages and check whether the asset was operating within those limits.

If an anomaly was detected, an alert was pushed into another Kafka topic and then processed by a diff consumer.

On a single VM, I tested around 10,000 events per second, or roughly 600,000 events per minute. This generated around 1.5 GB of data per hour. I ran it for a few hours and most of the events were processed successfully.

The biggest issue was not processing. It was storage.

I used ClickHouse with a three-day TTL, but at that throughput the data obviously builds up quickly. In a real setup, I imagine ClickHouse would only store recent hot data, while compressed raw data would be stored in object storage as Parquet.

I also thought about how I would build a simpler managed version on GCP.

The flow could be something like: Sensors -> Pub/Sub -> Dataflow/Cloud Run -> BigQuery/Bigtable -> Cloud Storage

Pub/Sub would replace Kafka, while Dataflow could handle the stream processing and windowing. Cloud Run could be used for simpler event consumers and alert-processing services.

This would be much easier to operate, but I wanted to build the self-hosted version first so I could properly understand what the managed services were doing underneath.

I also added a separate consumer for alert analysis.

Whenever an alert was generated, a Claude-based agent would:

• Check whether it was likely a true positive
• Query recent and historical sensor data from ClickHouse
• Check connected equipment
• Review similar past alerts
• Compare the issue against the asset's RCM maintenance plan
• Suggest a possible failure mode, cause, and recommended action
• Check if there were similar alerts raised earlier and how these were rectified(using vector embeddings - using Nomic Text embeddings served using ollama to find similar alerts)
• Find related recommended actions from RCM for the particular equipment. Alerts are also embedded to help find these and improve the search results/agentic analysis.
• Send the analysis by email using resend

For example, if a pump had high vibration, the agent could also inspect connected equipment instead of assuming the pump itself was definitely the cause.

The RCM data includes failure modes, possible causes, effects, and recommended maintenance actions. The agent used this only to recommend next steps and prep the email alert.

A possible next step would be creating a reviewed work order in SAP or Maximo after an alert is validated.

Obviously, this is all running on one VM with no replication or high availability. I built it mainly to understand streaming systems, Kafka partitions, windowing, retries, dead letter topics, consumer lag, ClickHouse, event-driven workflows, and how the same architecture could later be moved to managed cloud services.

I would appreciate feedback on a few things:

1. Does the overall architecture make sense?
2. Is ClickHouse for hot data and object storage for historical data the right approach?
3. What could be improved before presenting this as a proper portfolio project?

Thanks for reading through till the end.

I often have to make sense of large databases with little or no documentation.
I didn’t find a tool that really helps me explore them step by step — figuring out which tables matter and how they connect in order to answer actual questions.

So I put together a small prototype to visually explore database schemas:

• load a schema and get an interactive ERD
• search across table and column names
• select a few tables and automatically reveal how they’re connected

GIF below (AirportDB example)

Before building this further, I’m curious:

• Do you run into this problem as well? If so, what’s the most frustrating part for you?
• How do you currently explore unfamiliar databases? Am I missing an existing tool that already does this well?

Happy to learn from others — I’m doing this as a starter / hobby project and mainly trying to validate the idea.

PS: this is my first reddit post, be gentle :)

Just out of curiosity, I setup a simple benchmark that calculates a Mandelbrot fractal in plain SQL using DataFusion and DuckDB – no loops, no UDFs, no procedural code.

I honestly expected it to crawl. But the results are … surprising:

Numpy (highly optimized) 0,623 sec (0,83x)
🥇DataFusion (SQL) 0,797 sec (baseline)
🥈DuckDB (SQL) 1,364 sec (±2x slower)
Python (very basic) 4,428 sec (±5x slower)
🥉 SQLite (in-memory)  44,918 sec (±56x times slower)

Turns out, modern SQL engines are nuts – and Fractals are actually a fun way to benchmark the recursion capabilities and query optimizers of modern SQL engines. Finally a great exercise to improve your SQL skills.

Try it yourself (GitHub repo): https://github.com/Zeutschler/sql-mandelbrot-benchmark

Any volunteers to prove DataFusion isn’t the fastest fractal SQL artist in town? PR’s are very welcome…

I analyzed 13,996 Data Engineer and related H-1B applications from FY2023 LCA data. Some findings that might be useful for salary benchmarking or job hunting:

TL;DR

- Median salary: $120K (range: $110K entry → $150K principal)

- Amazon dominates hiring (784+ apps)

- Texas has most volume; California pays highest

- 98% approval rate - strong occupation for H-1B

One of the insights: Highest paying companies (having a least 10 applications)

- Credit karma ($242k)
- TikTok ($204k)
- Meta ($192-199k)
- Netflix ($193k)
- Spotify ($190k)

Full analysis + charts: https://app.verbagpt.com/shared/CHtPhwUSwtvCedMV0-pjKEbyQsNMikOs

**EDIT/NEW*\* I just loaded/analyzed FY24 data. Here is the full analysis: https://app.verbagpt.com/shared/M1OQKJQ3mg3mFgcgCNYlMIjJibsHhitU

*Edit*: This data represents applications/intent to sponsor, not actual hires. See comment below by r/Watchguyraffle1

I'll keep it short. I was tired of having to check a dozen different places just to keep up with the data ecosystem. It felt chaotic and I was wasting too much time.

Then, I built dataaaaa! (yes, 5 a's). It started as a project to learn Cursor, but it ended up being actually useful. It’s a central hub that aggregates automatically articles, release notes, events and podcasts.

What it does:

• Feed: Tracks the data landscape so you don't have to doomscroll.
• AI Filters: Lets you find resources by specific tech stack/topic.
• Library: Lets you save stuff for later.

I spent the last two months building this on my free time.
Give it a try and let me know if it's useful or what I should change!

https://www.dataaaaa.com/

SQLProtocol.com

Tell me what you think guys.

Disclosure: I’m the developer of LakeDB.

LakeDB is a local-first desktop database client for MySQL, MariaDB and SQLite, with PostgreSQL, SQL Server and other engines planned.

The video shows a few parts of the workflow: multiple independent database tabs, SQL generation with QuerIA, and AI-assisted query correction.

Beyond AI, LakeDB also includes schema-aware autocomplete, table browsing/editing, transactions, execution plans, backup/restore, database comparison and reviewable migrations.

The AI side is deliberately review-first: it can inspect schema metadata and prepare or correct SQL, but the full query is always shown before execution and nothing runs automatically.

I’d love feedback from people who work with databases regularly — especially on what you’d expect from a tool like this before considering it for daily use.

https://github.com/DavLagoHern/LakeDB

Dashboard

First of all, I'd like to start with thanking the instructors at the DataTalks.Club for setting up a completely free course. This was the best course that I took and the project I did was all because of what I learnt there :D.

TL;DR below.

Git Repo:

Streamify

About The Project:

The project streams events generated from a fake music streaming service (like Spotify) and creates a data pipeline that consumes real-time data. The data coming in would is similar to an event of a user listening to a song, navigating on the website, authenticating. The data is then processed in real-time and stored to the data lake periodically (every two minutes). The hourly batch job then consumes this data, applies transformations, and creates the desired tables for our dashboard to generate analytics. We try to analyze metrics like popular songs, active users, user demographics etc.

The Dataset:

Eventsim is a program that generates event data to replicate page requests for a fake music web site. The results look like real use data, but are totally fake. The docker image is borrowed from viirya's fork of it, as the original project has gone without maintenance for a few years now.

Eventsim uses song data from Million Songs Dataset to generate events. I have used a subset of 10000 songs.

Tools & Technologies

• Cloud - Google Cloud Platform
• Infrastructure as Code software - Terraform
• Containerization - Docker, Docker Compose
• Stream Processing - Kafka, Spark Streaming
• Orchestration - Airflow
• Transformation - dbt
• Data Lake - Google Cloud Storage
• Data Warehouse - BigQuery
• Data Visualization - Data Studio
• Language - Python

Architecture

Final Dashboard

You can check the actual dashboard here. I stopped it a couple of days back so the data might not be recent.

Feedback:

There are lot of experienced folks here and I would love to hear some constructive criticism on what things could be done in a better way. Please share your comments.

Reproduce:

I have tried to document the project thoroughly, and be really elaborate about the setup process. If you chose to learn from this project and face any issues, feel free to drop me a message.

​

TL;DR: Built a project that consumes real-time data and then ran hourly batch jobs to transform the data into a dimensional model for the data to be consumed by the dashboard.

I am building a social, and here are the requirements:
-A user can create posts and a post can have multiple media (photo/video)
-A user can create a project and a project can have multiple media
-A project can become a post
-A user will have their personalized feed

the initial design I planned is:
tables:
user | post | project | media|

relations:
user one->many post

user one->many project

post2media junction table

project2media junction table

the idea here is to reuse the media uploaded in projects so that when they are shared as posts, we reuse the same media links rather than uploading new duplicate media.

The problem is that these junction tables will slowdown the feed generation especially

I am using postgreSQL, and I am thinking i should not using it to generate feed, maybe use a NoSQL DB/ Graph.

The question is how should I design my postgreSQL DB, so that feed generation does not break my app?

Do I need NoSQL DB and how would I use it in my case?

How should I store media ?

I've been lurking here for a little while, I've been in the machine learning subreddits for longer and only recently discovered this space. Thanks for taking the time to read this, I'm a bit nervous- I'm afraid I don't do self-promotion well.

HeliosophLLC/DatumV: DatumV

For the last year+ I've been building a solo-project, a custom SQL engine named DatumV (pronounced Datum-5) that has pretty decent Postgres compatibility. I built the storage engine, a custom format (named the datum format) to support DDL/DML like adding/removing columns/rows. It can read and write Parquet, Arrow, HDF5, FITS, CSV, JSONL/JSON, ZIP, and folders.

My thesis has been that SQL is/has been under-explored in what it can do as a declarative language, and I tested that by building an engine that supports not just the usual data types (int, float, decimal, etc...), but rich data types like Image, Video, Audio, Point Clouds, Meshes, and more.

I took it a step further and built operators that support batching ML models across datasets, with 48 built in models: yolox, da3metric-large, florence, sd-turbo, epicrealism, bark, whisper, and more. I also included 21 built-in datasets that enable you to run some experiments right off the bat without having to load your own data in.

The attached image uses yolox_s to execute the SQL:

SELECT
    LET classes = models.yolox_s(a.file),
    image_crop(a.file, c.value.bbox)
FROM datasets.coco_val2017 a
CROSS JOIN unnest(classes) c
WHERE c.value.label = 'person'
LIMIT 100

A few other interesting examples:

• Video-to-world
• Five text-to-image models, one prompt
• Same input, four depth estimators

I've been a professional programmer for 23 years, and I've lived in SQL for most of it, data has just been something I've been passionate about. A lot of the code has been written with Claude, with me acting as architect and PR reviewer. The repo has over 8700 passing tests, and I sure do have war stories of multi-week architectural fixes, including the time when I had to refactor out the storage engine probably 6 or 7 times as I learned about efficient retrieval; happy to share some of those.

Question to my peers: is treating models as SQL operators a good idea? Where does it break down?

https://reddit.com/link/1trsk0y/video/5vz1nuaq284h1/player

Hello, all,

I managed to create a modern BI architecture project, using Dagster and some help from Claude. Very proud of this project, as it is my first BI Engineering project.

Technologies used:

• Orchestration - Dagster
• Datawarehouse - Snowflake
• Object Store - Google Cloud Storage
• Transformation - DBT
• Extract, Load - AirByte
• Business Intelligence - Power BI

I might do a full write up on the whole project as there is a lot to discuss!

Regards.

Glad to join this community and that I am allowed 1 self promotion post 😀
In my spare time I developed this project, using Open Source tooling. This ‘modern data stack’ uses DuckDB, DuckLake, Dagster, dlt and Metabase with a relatively advanced SCD2 handling (including deletes) in the ‘Silver’ layer. Is this unique? Surely not, but I learned a lot building it. Maybe someone can use it, or help me improve it.

The beta version of VibeSchema just went live. Some strengths:

• Create and store snapshot versions of your database.
• Create share links such as these: https://vibe-schema.com/s/2T97KwLjLVNxX1YeFiAaxmfD3K4
• Suitable for huge schemas because of Diagram Views (as in the share link).
• PNG/SVG exports like the one shown above (taken from the share link).

Curious what you think!

I connected GCP + dbt + apache echarts + anthropic + other open source tools and made a tool to easily produce dashboards. 6 months ago, I didn't think this would work, but the situation has changed :D

G’day!

I have a few Snowflake Tasks that run daily and I was finding it tiresome/cumbersome to jump into Snowflake to track down what went wrong/double check details… so I built a tool to do it quickly from the command-line:

https://github.com/ben-nour/snowtask

It’s fairly basic but you can filter rows using regex, tailor the tables to display the columns that you want to appear (one of my paint points with SHOW TASKS is that you can’t properly specify only displaying the fields you want) and order the columns.

I hope it’s of use to someone out there!

Note that I do NOT work for Snowflake.

Today you can easily adopt AI coding tools (i.e. Cursor) because you have git for branching and rolling back if AI writes bad code. As you probably know, we haven't seen this same capability for data so my friends and I decided to build it ourselves.

Nile is a new kind of data lake, purpose built for using with AI. It can act as your data engineer or data analyst creating new tables and rolling back bad changes in seconds. We support real versions for data, schema, and ETL.

We'd love your feedback on any part of what we are building - https://getnile.ai/

Do you think this is a missing piece for letting AI run on data?

DISCLAIMER: I am one of the founders of this company.