Many writers, one bucket¶
moraine is a Rust library for many processes writing the same tables on S3 or local disk. There is no server and no lock: a commit is one conditional PUT of the next Delta Lake log entry.
Start with the tutorial Read the benchmarks
moraine is pre-1.0; the API may still change.
let db = Database::builder("s3://bucket/db")
.open()
.await?;
let opts = TableOptions::default();
let table = db
.create_table("readings", schema, opts)
.await?;
// One PUT for the data file, one
// conditional PUT for the log entry.
table.append(vec![batch]).await?;
let boiler = table
.scan()
.filter(
col("device").eq(lit("boiler")),
)
.collect()
.await?;
How a commit lands¶
00000000000000000043.jsonWriter B, after reading 42: the same entry, one version up00000000000000000042.jsonWriter A: the put-if-absent landed00000000000000000041.jsonAn earlier commit00000000000000000040.json00000000000000000039.json
_delta_log/, newest entry on top. Each entry is one object, written once and never changed.- Queue. Writes from any number of tasks in one process join the table handle's queue. The first to find no commit in flight leads and takes everything compatible with it.
- Upload. The leader writes the rows as Parquet files sorted by cluster key and primary key, one PUT each. Nothing refers to them yet, so no reader can see them.
- Commit. One PUT of the next entry,
_delta_log/<version>.json, withIf-None-Match: *. When it lands, every caller in the group gets that version. - Lose a race. When another writer took the version first, the leader reads that entry with one GET, checks it for conflicts and puts its entry at the next version. No listing, no lock.
- Lose the response. When a put's outcome is unknown, the handle looks for its own transaction id in the log before it tries again, so a commit never lands twice.
What a table gives you¶
- Standard Delta Lake tables
- Parquet data files and a Delta log at reader version 1 and writer version 2, with no table features. The interoperability tests read moraine's tables with delta-rs and open tables delta-rs wrote. The Delta log
- Exactly-once writes
- Tag a write with an application id and version; a retry, a crashed process or a second replica replaying it gets "already applied". Exactly-once writes
- Reads that touch few bytes
- Files are sorted by cluster key and primary key; reads skip files by key range, bloom filter and key-index sidecar, and pages by the Parquet page index. Footers and small tables stay in memory per version. Clustering and pruning
- Indexes beside the data
- One immutable sidecar per data file: a key index, vector search with usearch or TurboQuant, and full-text search with tantivy, ranked with the whole table's BM25 statistics. A file without its sidecar is answered exactly, so results never depend on the index being built; writers build a new file's sidecars before its commit. Sidecar indexes
- Blobs
- Binary values of any size in a column: inline when small, objects of their own beside the table when large, read whole, by range or as a stream, and collected by vacuum when no row references them. Blobs
- Tested by simulation
- A fault-injecting store, a simulated clock, a seeded scheduler, a reference model and an invariant checker. Upserts, deletes and updates passed 100,000 seeds of each of their four scenarios. Testing and simulation
Measured against Lance and Delta Lake¶
How many times faster moraine is than the better of Lance and Delta Lake on each measure, median against median, with moraine in its default configuration. Solid bars are wins; outlined bars, left of 1×, are losses.
Lance and Delta Lake run through kv0, a tracker of mine with a write-ahead log, so the logging ratios include integration work moraine's adapter does not do. The per-workload charts, the method and every caveat are on the benchmarks page.