Changelog¶
All notable changes to this project are documented here. The format follows Keep a Changelog, and the project uses Semantic Versioning. Before 1.0, a minor version may break the API.
[Unreleased]¶
Nothing is released yet. The first release will contain the following.
Added¶
- Databases on S3 (
s3://bucket/prefix), S3-compatible servers, local disk and memory, with a capability probe that refuses to write to a store that ignores conditional writes. - Tables stored as Delta Lake tables (reader version 1, writer version 2): create and open,
append,upsert,delete,delete_keys,updateandadd_column. - Group commit per table handle: one PUT per data file and one conditional PUT of the next log entry per commit, with no listing on the write path; conflict-checked rebase after a lost race, a starvation guard, and a commit deadline.
- Ambiguous commit outcomes resolved through the handle's own transaction id.
- Exactly-once writes with caller transaction ids:
Table::txn(app_id, version),WriteOutcome::AlreadyAppliedandlast_txn_version. - Reads at four consistency levels (
Strong,Eventual,Background,Manual), scans with filters and projections, point lookups by primary key, and pinned snapshots. - Data files sorted by cluster key and primary key, with 1,024-row pages, bloom filters on key columns, and pruning by file key ranges, bloom filters and the page index; per-version caches for footers, key indexes and small tables.
- Sidecar index bundles with a pluggable
IndexKind: the key index (used byget), vector search with usearch (vectorfeature) and TurboQuant through turbovec (turbovecfeature), the exact route for files without a sidecar, memory-bounded index builds,build_indexesandindex_status. - Full-text search with tantivy (
ftsfeature): theftsindex kind,Table::search_text, BM25 statistics summed over every data file so rankings equal one index over the whole table, an in-memory index for files without a sidecar (DatabaseBuilder::fts_memory), and a tokenizer registry with stemmers for 18 languages, character n-grams and custom tokenizers (DatabaseBuilder::tokenizer). - Hybrid search,
Table::search_hybrid: a vector and a full-text search against one snapshot, fused with reciprocal rank fusion or aFusionof the caller's. Consistency::Background: a task per table handle refreshes the snapshot and prewarms the caches for each new version;Table::refresh_metrics.Table::snapshot_in_hand,TableSnapshot::filesandDatabase::cache_usage.- List columns with any item field name (Arrow's default
itemincluded), stored under Delta'selement. - Credentials and bucket region from the AWS SDK (
aws-sdkfeature), and object tags on every write. - Maintenance:
Table::maintainruns the index build, the ingestor of queued writes, compaction, checkpoints and log cleanup, then vacuum and sidecar collection, each under an advisory lease;Table::spawn_maintenanceruns passes on a timer. - Tiered, sorted compaction (
Table::compact,CompactionPolicy): levels in the Deltaadd.tags, sidecars built before the commit, commits that keep the file order. - Checkpoints written and read by moraine,
_last_checkpoint, and log cleanup on server time; opening a checkpointed table reads the checkpoint and the entries after it. - Vacuum on server time (
Table::vacuum), keeping what snapshots younger than the grace period and pins need, and collection of sidecars, pin manifests and abandoned clock probes. - Named pins:
Table::pin,unpin,pinsandsnapshot_at_pin. - An atomic, conflict-checked file replace:
Table::replace_filesandReplace, with file tags and optional lease fencing. - Generation-file leases fenced through Delta transactions:
Table::try_lease,renew_lease,release_lease, andError::Fenced. - Queued writes:
Table::submit,ingestandis_ingested, applied exactly once. Table::verifyandTable::repair;Database::table_names.- Queries move to the newest version once when vacuum removed a file under an old snapshot,
then report
Error::SnapshotExpired. - The
morainecommand:tables,inspect,maintain,compact,checkpoint,vacuum,ingest,verify,repair,pins,pin,unpinand thestresswriter of T8. - Blobs (
moraine::blob): a blob column type whose values stay in the row up to the table'sblob_inline_bytes(256 KiB by default) and are stored as objects of their own above it, with multipart uploads above 16 MiB;BlobColumn,blob::refs,Table::blobandTable::blobswith whole, ranged and streamed reads checked against a CRC-32C; server-side copies of reused references (BlobReuse);Table::upload_blobfor values streamed ahead of a write; blob collection in vacuum; missing blobs inverify;Error::BlobCorrupt. - Index before commit: writers build their files' sidecars before the commit, within each
index's
before_commit,before_commit_max_rowsandbefore_commit_max_build_bytespolicy;IndexStatus::rows_exact_by_policy,rows_pendingandexact_share. - Streaming compaction (
CompactionPolicy::streaming, on by default): a k-way merge over the sorted inputs with memory bounded by row groups;vec-tqsidecars of merged files are merged from the inputs' codes;CompactionReport::streamedandsidecars_merged. - Store metrics count multipart parts and completions (
Op::Part,Op::Complete). - Purge and erasure:
Table::purgewithPurge(removed_before,erase) collects removed files at once rather than after the grace period, with their sidecars and blobs, and erases their statistics from the log, older checkpoints and stale pin manifests;PurgeReport, themoraine.purgedtable property andError::LeaseHeld. - File-level reads:
Scan::files,TableSnapshot::take(rows by file and offset,Take),Table::idandTableSnapshot::table_id, andScan::stream. - File replaces from a stream or an iterator of batches:
Replace::add_streamandadd_batches. Table::write_metrics(WriteMetrics): uploads deleted by writes that did not commit them, and uploads left for vacuum.Clock::spawn, which runs a table handle's commit task.moraine-testkit: a fault-injecting store, a simulated clock and store, a seeded scheduler, a reference model, an invariant checker, deterministic corruption fuzzing, text corpora with a merged tantivy reference index, a soak test with a counting allocator, and maintenance simulations checked against the store's full history.- Index builds on several threads:
DatabaseBuilder::build_threads(default: the machine's parallelism) for usearch graphs, and the builds of a file's several indexes at once. - Compaction steps of indexed tables capped to the rows their sidecar builds finish in half the commit deadline, at the rate this handle's builds have run.
- Blob uploads with four parts in flight, and blob streams read as 4 MiB ranged GETs with read-ahead.
- Vector searches over more files than
io_concurrencysearch every file's parts on one work-stealing pool; themoraine.poolsearch parameter forces either path. moraine-testkit: a per-seed watchdog (MORAINE_SIM_SEED_TIMEOUT_S) that fails a simulation seed which runs too long.
Changed¶
- Group commits run on a task owned by the table handle: a caller that stops waiting no longer fails the other writes of its group, and a panic in a group commit is raised in every caller of the group.
- A write that does not commit deletes the data files, sidecars and blobs it uploaded once its outcome is settled, instead of leaving them to vacuum.
- Compaction merges only files with the same caller tags, and the merged files keep them.
- Pinning a version older than the newest purge is refused with
Error::VersionNotAvailable. - A checkpoint pass deletes its own checkpoint when a purge committed after it read the log; log cleanup deletes checkpoints, and vacuum pin manifests, older than the newest purge.
Replaceis no longerClone: it may hold a stream.- Compaction builds its merged files' sidecars from the rows it writes, without reading the files back and without a HEAD per sidecar.
- Writes build the key index of large files through the same before-commit path as other indexes.
- Vector files built in parts search their graphs concurrently.
- Full-text statistics are kept per version; background prewarm builds in-memory text indexes only once the table has been searched by text; string columns are written in pages of at most 64 KiB.