Skip to content

Exactly-once writes

A write that times out may or may not have committed. A process that crashes after a write may not know whether it landed. Retrying such a write is only safe if the table can tell that it has already applied it. moraine does this with Delta's application transaction ids: the caller names each write with an application id and a version, and a write whose version the table has already seen for that application is not applied again.

The API

Table::txn(app_id, version) returns a TxnWrite with the same write methods as the table: append, append_unchecked, upsert, upsert_with, delete, delete_keys and update. Each returns a WriteOutcome:

Outcome Meaning
WriteOutcome::Committed(committed) The write was applied at table version committed.version.
WriteOutcome::AlreadyApplied { version, txn_version } The application has already committed this version or a later one (txn_version); version is a table version that holds the earlier commit. Nothing was written.

"Already applied" is not an error. The methods without an id are unchanged and return a Committed, whose version is the table version that holds the write. WriteOutcome may gain variants in a minor release, so a match on it ends with a wildcard arm.

use moraine::WriteOutcome;

// Replaying entry 7 of a write-ahead log, possibly for the second time.
match table.txn("ingest-1", 7).append(vec![batch]).await? {
    WriteOutcome::Committed(c) => println!("committed at version {}", c.version),
    WriteOutcome::AlreadyApplied { version, .. } => println!("already in version {version}"),
    other => println!("in version {}", other.version()),
}

// After a restart: where did this application get to?
let next = table.last_txn_version("ingest-1").await?.map_or(1, |v| v + 1);

Table::last_txn_version(app_id) refreshes the snapshot first (one GET when nothing is new). TableSnapshot::last_txn_version answers for a pinned version without a request.

The rule

It is Delta's rule: the newest txn action per application id, in log order, is the application's version, and a write at or below it is already applied. Versions may skip numbers but must grow. An application therefore sends its writes in order, and retries each one until it is committed or already applied before sending the next. A version sent after a higher one is reported as applied and never written.

A write with an id always commits

A delete or update that matches nothing, an upsert of no rows, and append_unchecked(vec![]) still commit an entry that carries the id. Otherwise a replay could apply a delete to rows written after the original found nothing to delete. append_unchecked(vec![]) is therefore the way to record "this application reached version n" with no data.

Where the check happens

The check costs no request of its own. It runs where the commit protocol already reads the log: against the snapshot a write prepares on, and against every entry the write loses a race to, since a write commits only after reading every entry between its snapshot and its version.

  • A write whose id the snapshot already holds is answered before it runs.
  • A write that finds its id applied in an entry it lost to stops as superseded.
  • After an ambiguous log put, the next attempt is safe whatever happened: if the earlier entry landed, it is either in the snapshot or in the way.

Across processes and restarts the ids come from the log itself. Every snapshot keeps the newest version per application id, and a handle that loads from a checkpoint reads the checkpoint's txn column (one more GET of the checkpoint, on that path only).

Group commit

One log entry can hold several applications' ids, one txn action per caller id beside the handle's own. Delta allows one txn per application id per commit, so a second write of an application that is already in the group starts the next group, which also keeps an application's writes in queue order.

Retention

Delta drops a txn action older than delta.setTransactionRetentionDuration by its lastUpdated field. moraine writes caller ids without lastUpdated, which Delta never expires, so a replay after a long outage is still recognised. The handle's own ids carry lastUpdated and expire with the grace period. The interoperability tests check through delta-rs that after a checkpoint a caller id survives and an id with an old lastUpdated does not.

How it is tested

Three simulation scenarios replay write logs under injected faults: processes that crash at a random request and resume from last_txn_version or earlier, log puts that land and lose their response followed by resends, and two processes (and two tasks of one handle) replaying the same logs at once. With the snapshot check disabled, the concurrent scenario fails on its first seed with an id applied twice. See Testing and simulation.