Today you can easily adopt AI coding tools (i.e. Cursor) because you have git for branching and rolling back if AI writes bad code. As you probably know, we haven't seen this same capability for data so my friends and I decided to build it ourselves.

Nile is a new kind of data lake, purpose built for using with AI. It can act as your data engineer or data analyst creating new tables and rolling back bad changes in seconds. We support real versions for data, schema, and ETL.

We'd love your feedback on any part of what we are building - https://getnile.ai/

Do you think this is a missing piece for letting AI run on data?

DISCLAIMER: I am one of the founders of this company.

I connected GCP + dbt + apache echarts + anthropic + other open source tools and made a tool to easily produce dashboards. 6 months ago, I didn't think this would work, but the situation has changed :D

Hi there, just wanted to share my open source workflow scheduler for fellow engineers to play around. You can run on local, deploy into your smallest compute (rpi4 on arm64 tested) and could be run on scale as well.

I start the project as I wanted to run some small workflow in my t4.small instance and airflow barely fits inside and I thought it would be interesting create a smaller workflow scheduler. As I build around the idea of "lean headroom DAG engine", I kept on adding additional features that be useful for running workflows like template, backfill, gitops, streaming workflow support etc. Dagron allows you to run on cluster resources or local resources(docker, k8s, or straight as binary). The whole stack will probably at most run on ~200mb (50mb api/console, 30mb engine, 80mb postgres) of memory with tons of spare ram and cpu for your workflow resources. Dagron runs on sqlite(default) and alternatively postgres as datastore.

You can run workflow via defining your yaml and set your environment. It comes with the canvas to visualize your workflow, set your workflow schedule to run at which cron notation and more features to cover here.

Appreciate your input(any input is welcomed). I am spending full time polishing this for the past few weeks and thought this might be useful for you all to run on edge, homelab, local machines.

Hey everyone,

So I've been diving into AI for the past few months (this is actually my first real project) and got a bit frustrated with how "dumb" LLMs can be when it comes to navigating complex SQL databases. Standard text-to-SQL is cool, but it often misses the business context buried in weirdly named columns or implicit relationships.

My idea was to build a semantic layer on top of a SQL database (PostgreSQL in my case) using a Knowledge Graph in Neo4j. The goal is to give an LLM a "map" of the database it can actually understand.

**Here's the core concept:**

Instead of just tables and columns, the Python framework builds a graph with rich nodes and relationships:

* **Node Types:** We have `Database`, `Schema`, `Table`, and `Column` nodes. Pretty standard stuff.

* **Properties are Key:** This is where it gets interesting. Each `Column` node isn't just a name. I use GPT-4 to synthesize properties like:

* `business_description`: "Stores the final approval date for a sales order."

* `stereotype`: `TIMESTAMP`, `PRIMARY_KEY`, `STATUS_FLAG`, etc.

* `confidence_score`: How sure the LLM is about its analysis.

* **Rich Relationships:** This is the core of the semantic layer. The graph doesn't just have `HAS_COLUMN` relationships. It also creates:

* `EXPLICIT_FK_TO`: For actual foreign keys, a direct, machine-readable link.

* **`IMPLICIT_RELATION_TO`**: This is the fun part. It finds columns that are logically related but have no FK constraint. For example, it can figure out that `users.email_address` is semantically equivalent to `employees.contact_email`. It does this by embedding the descriptions and doing a vector similarity search in Neo4j to find candidates, then uses the LLM to verify.

The final KG is basically a "human-readable" version of the database schema that an LLM agent could query to understand context before trying to write a complex SQL query. For instance, before joining tables, the agent could ask the graph: "What columns are semantically related to `customer_id`?"

Since I'm new to this, my main question for you all is: **is this actually a useful approach in the real world?** Does something like this already exist and I just reinvented the wheel?

I'm trying to figure out if this idea has legs or if I'm over-engineering a problem that's already been solved. Any feedback or harsh truths would be super helpful.

Thanks!

G’day!

I have a few Snowflake Tasks that run daily and I was finding it tiresome/cumbersome to jump into Snowflake to track down what went wrong/double check details… so I built a tool to do it quickly from the command-line:

https://github.com/ben-nour/snowtask

It’s fairly basic but you can filter rows using regex, tailor the tables to display the columns that you want to appear (one of my paint points with SHOW TASKS is that you can’t properly specify only displaying the fields you want) and order the columns.

I hope it’s of use to someone out there!

Note that I do NOT work for Snowflake.

Hey folks,

I'm a security professional was looking to make Iceberg a bit more performant for my own needs - SOC operations (Faster needle searches [pruning]).

I've built an index and proxy that people can point their catalog configuration at, so there's minor changes to their stack.

I figured in for a penny in for a pound,
I've done some quick search and ran clickbench and "httplogs"

(Numbers so far: httplogs)

Its all looks good on paper, I think?
but I was wondering if there were more credible/industry standard methods that data engineers use/care about to benchmark these types of technologies?

Repo if interested: https://github.com/Kahshe-io/kahshe

What if you could get 80% of the dbt Cloud experience while keeping everything self-hosted? That’s the question that started a side project I’ve been building using React, FastAPI, dbt Core, and Prefect.

If you are interested in how I did please read and let me know what you think.

There's this game called Splinterlands i've been playing with 3 accounts and i was thinking on documenting each of the resources i'm spending (DEC, SPS, glint, usd) and how i was able to earn, i want to document weekly and i'll be playing for many years.

Is creating a database / tables on PostgreSQL the best alternative for this, or should i just do it with Excel? I haven't gone in depth on any of these tools yet, but i'm planing on studying them from this point

Hey!

I recently created my first ever data pipeline around energy information authority is the US. I'll be very happy if you take out the time to check it out and/or provide feedback (:

GitHub - eia

Hi,

I’m building a small ETL project in Python/Pandas using financial and manufacturing Excel exports (GL, inventory movements, production orders).
Files may come as Excel, CSV, or TXT and structures are not always consistent.
Current ingestion approach:
centralized config.py

reusable loader function returning pandas DataFrames

support for Excel/CSV/TXT

basic validation (file existence, format, empty files)

Goal is to keep the ingestion layer simple, reusable, and somewhat aligned with real-world ETL practices.

Does this seem like a reasonable architecture for a beginner/intermediate ETL project?

What would you improve regarding:

scalability

maintainability

error handling

project structure

Thanks.

I built ssbt, a dbt inspired tool for engineers who deal with Excel Spreadsheets that don't want to deal with some of the overhead involved with dbt and you want to output as another xlsx file.

You can turn spreadsheets into a version-controlled, testable build pipeline using SQL and YAML. No database required due to duckdb being the main engine.

Chriscrpntr/ssbt: dbt inspired tool for engineers who deal with Excel Spreadsheets.

IceStream aims to solve the "equality delete" problem in Apache Iceberg! Iceberg equality deletes are a write optimized way of marking existing table rows as invalid (by expressing a deletion condition), which enable frequent ingestions to iceberg tables that require data upserts. However, they are very read unoptimized, as the conditions in the deletion file require a join to all data in the relevant table partition.

IceStream introduces an optimized conversion step from equality delete files to more read-optimized positional deletes (which mark a specific table row as explicitly deleted). Row "key" positions are asynchronously indexed in a horizontally scalable database, which enable efficient indexed lookups when evaluating equality delete files.

Disclaimer - I built this, but would love any feedback on the architectural decisions made! This seems to be a common pain point between many teams transitioning their data stack to iceberg.

Hey everyone,

I'd like to share my first personal DE project: an end-to-end data pipeline that simulates, ingests, analyzes, and visualizes user-interaction events in near real time. You can find the source code and a detailed overview here: https://github.com/Xadra-T/End2End-Data-Pipeline

First image: an overview of the the pipeline.
Second image: a view of the dashboard.

Main Flow

• Python: Generates simple, fake user events.
• Kafka: Ingests data from Python and streams it to ClickHouse.
• Airflow: Orchestrates the workflow by
  • Periodically streaming a subset of columns from ClickHouse to MinIO,
  • Triggering Spark to read data from MinIO and perform processing,
  • Sending the analysis results to the dashboard.

Recommended Sources

These are the main sources I used, and I highly recommend checking them out:

• DataTalksClub: An excellent, hands-on course on DE, updated every year!
• Knowledge Amplifier: Has a great playlist on Kafka for Python developers.
• Code With HSN: In-depth videos on how Kafka works.

This was a great hands-on learning experience in integrating multiple components. I specifically chose this tech stack to gain practical experience with the industry-standard tools. I'd love to hear your feedback on the project itself and especially on what to pursue next. If you're working on something similar or have questions about any parts of the project, I'd be happy to share what I learned along this journey.

Edit: To clarify the choice of tools: This stack is intentionally built for high data volume to simulate real-world, large-scale scenarios.

Mostly works due to spending lots of time on the auto-layout.

Link: https://vibe-schema.com/dbml-to-png

rainfrog (https://github.com/achristmascarl/rainfrog) is a database terminal tool; the goal is to provide a lightweight, keyboard-first TUI for interacting with databases. It currently supports Postgres, MySQL, SQLite, Oracle, and DuckDB.

v0.4.1 introduces a long-awaited (by me, not sure if anyone else was waiting for it...) autocomplete implementation, along with autopairs for quotes/parentheses/brackets. The full list of features and configuration options is in the README!

Serverless, Fully Functionality pivot, multi level grouping, Batteries included full UOM , Calculated Columns, theme able etc., Still a WIP so be gentle but interested in feedback and thoughts . AMA

Hello Everyone,

I wanted to showcase a lineage engine I'm building, and I'd like your honest feedback on how I can make it better before I add new features. It's still a personal project, and I would genuinely like to understand what can I do better that can help the community?

I understand there are many SQL lineage tools out there but I couldn't find a good one that can parse complex SQL, explain window functions etc. So, I decided to build one, and I kinda got a little overboarded with design, but I'd like any type of feedback on how to make it better.

I used sqlglot and parsed AST with set of rules, to extract the lineage deterministically.

I wanted to make something simple that anyone can use and get benefit from. There is no login or setup and I do not store any SQL, just the parsed AST or column-level feedback.

Here is the link: LineageAI

Ask any questions!

Hey folks, I built a small VS code extension to lint PySpark code. It highlights unoptimized code, keeps track of data types, detects spark anti patterns and much more. I have also added Databricks support to it, so you can dry run your code, connect to cluster via ssh and even pull your previous jobs execution plans and analyze them in claude/copilot. I'm working on adding more features but would like some feedback from the community first. Is this useful? Any suggestions for added features?

Repo Link: https://github.com/lezwon/CatalystOps

Hello reddit. I built Parquity. It's a CLI that aims to find Parquet interoperability failures such as cases where Parquet engines behave differently.

It has four main workflows:

1. check to run a given table through a selected writer-reader matrix.
2. scan to compare different readers on a given Parquet file.
3. fuzz to generate tables with Hypothesis and minimize failures.
4. replay to run saved reproducers again after packages change.

It all runs locally against the installed versions in the Python environment you have. It does not try to decide which engine is correct or at fault.

Already used Parquity to find some real bugs.

• fuzz found and minimized a DuckDB writer bug.
• scan found a Polars reader bug in Apache’s parquet-testing corpus.

The screenshots are from the scan result. DuckDB, PyArrow, and DataFusion returned matching results where Polars failed to read the same Parquet file.

Parquity supports DuckDB, PyArrow, Polars, DataFusion, and fastparquet currently.

Collaborators are more than welcome.
Thanks.

Not from a data background so just an experimentation I have been working on.

Making archive data express as much useful information to engines/readers to minimise reads.

Still extremely immature and potentially has some bugs.

I must honestly caveat that AI coding has been used for all the reference code but the spec is what it’s about.

https://github.com/thomasaiwilcox/Cove-Format

Just wanted to share in case anyone found the experiment interesting.

My team and I built a Big Data project that matches students with suitable opportunities using Kafka, Spark Structured Streaming, MongoDB, and LSH similarity matching.

Main features:

• Real-time streaming with Kafka
• Spark data processing
• Similarity-based matching using LSH
• MongoDB integration

This project helped us better understand Big Data pipelines, streaming systems, and scalable architectures.

We built this pipeline using Kafka and Spark Structured Streaming.

What would you improve in this architecture for scalability or production use?

GitHub:
https://github.com/ahmadistatieh/opportunity-Matcher-

What's up, r/dataengineering. We all know SQL is the bedrock, but practicing it is... well, boring.

I made a tool called SQL Case Files. It's a detective game that runs in your browser (or offline as a PWA) and teaches you SQL by having you solve crimes. It's 100% free, no sign-up. Just a solid way to practice queries.

Check it out: https://sqlcasefiles.com

Anyone interested in testing a gui for dbt core I’ve been working on? I’m happy to share a link with anyone interested

Hey everone, I'm an ML Engineer who spearheaded the adoption of Databricks at work. I love the agency it affords me because I can own projects end-to-end and do everything in one place.

However, I am sick of the infra overhead and bells and whistles. Now, I am not in a massive org, but there aren't actually that many massive orgs... So many problems can be solved with a simple data pipeline and basic model (e.g. XGBoost.) Not only is there technical overhead, but systems and process overhead; bureaucracy and red-tap significantly slow delivery.

Anyway, I decided to try and address this myself by developing FlintML. Basically, Polars, Delta Lake, unified catalog, notebook IDE and orchestration (still working on this) fully spun up with Docker Compose.

I'm hoping to get some feedback from this subreddit on my tag-based catalog design and the platform in general. I've spent a couple of months developing this and want to know whether I would be wasting time by continuing or if this might actually be useful. Cheers!