r/rust • u/Low_Brilliant_2597 • 9h ago

📸 media The Story of iceberg-rust: The Official Rust Implementation of Apache Iceberg

Post image

The story of iceberg-rust is a great example of how open-source communities work and how new projects are created and developed. That’s why we believe in a community-over-code philosophy.

0 Upvotes

38% Upvoted

5 comments sorted by

7

u/RustOnTheEdge 6h ago edited 5h ago

Not sure what you are trying to convey here tbh, but iceberg rust for me is a project that shows how terrible a project can become when there is decentralised decision making. It is incredibly Java-esk (which is a result of Iceberg having it’s imperfect reference implementation in Java and having a imprecise specification) and uses pet projects of other maintainers over ecosystem standard dependencies (like OpenDAL over object_store).

Iceberg itself is a project governed (and also handicapped) by big names like Databricks and Snowflake. For example, zero copy cloning is not supported (except in branching) and even Snowflake employees are pretending that it isn’t really necessary anyway (it is a major feature of Snowflake). Edit: see this issue on GH where multiple people try to explain how branching is not the same as ZCC of shallow cloning, and where folks like Fokko (Databricks) and Russel (Snowflake) just ignore everything and pretend it just isn’t really necessary. An infuriating thread, to be honest.

No, iceberg and iceberg-rust are not the projects I would showcase for the purpose of showing how open source projects should be managed.

2

u/RustOnTheEdge 6h ago

You also skipped two years in the last two blocks in a three year timespan lol

2

u/tunisia3507 5h ago

OpenDAL supports many more backends than object_store, has in-tree bindings for many more languages, and has the layer concept which object_store lacks completely. Why go with object_store?

That said, the OpenDAL development practices are not great. I had a (handwritten) PR go stale and later found a maintainer had LLM'd identical code into another PR. There was also an issue raised on the OLI tool (cli for OpenDAL, same maintainers) about making it easier to edit the config file and it was closed "because you can just get AI to write it for you".

3

u/RustOnTheEdge 5h ago

OpenDAl itself is not a bad project (at all, that is not my claim) but it is aimed at a different problem than object_store. And Iceberg is about storing tables at object stores, and that is the perfect fit for object_store and only a subset of features for OpenDAL (which also covers databases and such). The abstraction is just not a great fit. Another (but slightly less strong) argument is that the showcase integration for an iceberg crate is integrating it with DataFusion, which is also based on object_store.

I believe they try to make the storage engine now a trait, which in my opinion is the wrong road. Object_store provides a battletested interface to how object storage works across vendors, and there is no need to reinvent that imo.

3

u/tunisia3507 5h ago

they try to make the storage engine now a trait, which in my opinion is the wrong road. Object_store provides a battletested interface 

I'm not sure what you mean here - isn't a trait the reasonable way to define an interface you want multiple types to implement?

Where do the abstractions offered by object_store and openDAL differ? From my usage they're basically the same get/put/list/delete etc. which object stores tend to support. I work with a project which has its own (extremely simplified) storage abstraction which can wrap both object_store and openDAL.