Apache Iceberg on Rust Without Heavy Java

If you work with big data, you've likely encountered Apache Iceberg. Essentially, it's the standard for building Lakehouse architecture on top of Parquet or ORC files. For a long time, Java remained Iceberg's main home. Nearly all official libraries and integrations with Spark or Trino were written for the JVM.

However, the data processing toolstack is now shifting toward Rust. Projects like Apache DataFusion, Polars, and others need access to Iceberg tables without spinning up a heavy Java runtime or dealing with JNI bindings. This is how an official repository apache/iceberg-rust came to exist within the Apache Foundation.

Why Port to Rust

A native implementation delivers performance and low memory overhead. When you're writing a microservice for stream processing, a utility, or a custom Foreign Data Wrapper for PostgreSQL, dragging along the JVM is painful. The JVM requires hundreds of megabytes of memory at startup and periodically freezes due to garbage collection.

The Rust library can read Iceberg metadata and data directly from lightweight binary files. The project develops within the Apache Foundation itself, so this is not a private fork but an official infrastructure component.

Repository Structure

The developers didn't bundle everything into a massive monolith. The project is split into a set of independent modules. You take only the dependencies that are actually needed in your service.

The library core lives in module iceberg. It handles metadata parsing, manifest manipulations, and table schema operations according to Iceberg v1 and v2 specifications.

Separate crates have been created for working with metadata catalogs:

• iceberg-catalog-rest for REST API operations
• iceberg-catalog-glue for AWS Glue Data Catalog integration
• iceberg-catalog-hms for connecting to Hive Metastore
• iceberg-catalog-sql for storing metadata in PostgreSQL or SQLite
• iceberg-catalog-s3tables for working with Amazon S3 Tables

Module iceberg-storage-opendal handles physical storage like S3, GCS, or local disk. It uses the Apache OpenDAL library, which immediately opens access to all popular object stores.

If you need a query execution engine, module iceberg-datafusion comes in handy. This module binds Iceberg to the Apache DataFusion SQL engine, enabling you to execute SQL queries against tables directly from Rust.

Where It's Already Working

Development is active, but serious products are already using the library in production:

• Databend uses it in their cloud data warehouse.
• RisingWave leverages the library for working with Iceberg from their real-time database.
• Supabase uses Rust Iceberg components in extensions for Postgres and in streaming ETL services.
• Moonlink uses the library to solve the task of replicating CDC data from Postgres to Iceberg format in fractions of a second.
• Apache DataFusion Comet uses it to accelerate Spark performance.

Support and Versioning

The project team follows an MSRV (minimum supported Rust version) policy. Code is built and tested against stable Rust releases. The minimum supported compiler version is fixed with a buffer of approximately three months relative to recent language releases.

Development is fully open. All discussions take place on the [email protected] mailing list and in channel #rust of the official Apache Iceberg Slack community.

Is It Worth Trying

If you're building services around data warehouses, writing connectors in Rust, or developing your own data processing tools, iceberg-rust looks like the most logical choice.

The library is still being refined in terms of covering the entire Iceberg specification, but basic reading, writing, and support for key catalogs work stably. For DataFusion-based projects, it's already a ready-to-use building block.