Delta Lake Without JVM and PySpark in Rust and Python Projects

Anyone who has tried to adopt the Delta Lake format in small services or scripts quickly runs into a harsh reality. To simply read a couple of gigabytes of transaction log files, you have to drag along Apache Spark, and with it the heavyweight JVM infrastructure. In large data platforms this is justified, but for a local application, a background microservice, or an AWS Lambda serverless function, such a solution feels far too cumbersome.

Developers from the Delta Lake community decided to close this gap. They created the delta-rs project — a native library for working with Delta Lake in Rust, complete with official bindings for Python.

Why Another Data Library

The Delta Lake format is great because it solves the main pain points of open file storage. It adds an ACID transaction log on top of Parquet files, change history with the ability to roll back to previous versions (time travel), and strict schema enforcement.

The classic stack around Delta Lake has historically been tied to Scala and Java. If your primary stack is Rust or lightweight Python without heavy frameworks like PySpark, before delta-rs came along, your options were limited.

The library provides low-level APIs for Rust and a high-level interface for Python. Now you can write data with ACID guarantees to an S3 bucket or local disk in just a couple of lines of code without spinning up JVM processes.

Quick Start in Python and Rust

The Python package interface deltalake is designed so that developers don't have to relearn anything. It integrates easily with familiar analytics libraries — Pandas, Arrow, or Polars.

Writing and reading a table in Python looks very familiar:

import pandas as pd
from deltalake import DeltaTable, write_deltalake

# Создаем тестовый датафрейм и сохраняем его в формате Delta
df = pd.DataFrame({"id": [1, 2], "value": ["foo", "boo"]})
write_deltalake("./data/delta", df)

# Считываем данные обратно
dt = DeltaTable("./data/delta")
df_read = dt.to_pandas()

assert df.equals(df_read)

An interesting detail: you can instantly open the same table from a Rust application. No manual metadata assembly is required. The library will automatically parse the JSON commit log in the _delta_log directory.

Example of reading table metadata in Rust:

use deltalake::{open_table, DeltaTableError};
use url::Url;

#[tokio::main]
async fn main() -> Result<(), DeltaTableError> {
    let delta_path = Url::from_directory_path("/abs/data/delta").unwrap();
    let table = open_table(delta_path).await?;

    let files: Vec<_> = table.get_file_uris()?.collect();
    println!("{files:?}");

    Ok(())
}

Under the Hood and Integrations

High speed and low memory consumption are not accidental. The project is built on top of the Apache Arrow engine and the DataFusion vector engine. Rust is used for safe memory management and parallel I/O when working with cloud storage.

The library can work directly with AWS S3, Google Cloud Storage, Azure Blob Storage, and the local file system.

Thanks to the shared foundation on Rust and Arrow, delta-rs has quickly become part of the modern data processing ecosystem. Popular tools work with it out of the box:

• Polars uses delta-rs for direct reading and writing of Delta tables.
• DuckDB can execute analytical SQL queries against Delta logs.
• Da, Dask, and Ray use this module for distributed data processing in Python.
• AWS SDK for Pandas uses it as the native engine for the Delta format.

Practical Use Cases

In which situations does delta-rs win over the classic approach?

The first case is microservice architecture. For example, you have a Rust or Python service that collects events from a message queue and needs to append batches to a data store every five minutes. Spinning up a Spark cluster for this is expensive and complex to maintain. With delta-rs, the service simply links the library and writes data directly to S3.

The second case is lightweight ETL pipelines. If your data volumes are measured in tens or hundreds of gigabytes, Polars combined with delta-rs will process them on a single instance faster than a PySpark cluster can be provisioned.

The third case is Serverless architecture. In AWS Lambda functions with strict limits on image size and startup time, fitting a Java environment is problematic. A compiled Rust binary with delta-rs starts in milliseconds.

Does the project have limitations? Yes, the Delta Lake specification is quite extensive and is constantly updated by Databricks. Some rare or new format features are implemented in delta-rs with a slight delay. Before using complex operations like MERGE with specific conditions, you should check the supported features table in the project documentation.

Conclusion

The delta-io team has done an excellent job. The native Rust library has brought lightness back to working with the Delta Lake format.

If you need transactionality, data versioning, and reliable storage on top of Parquet, but you don't want to deal with Java infrastructure, definitely give delta-rs a try. You can install the package in Python with the command pip install deltalake, and add it to a Rust project via cargo add deltalake.