Skip to content

S3 and bucket requirements

A database on S3 is a bucket and a prefix. Every table, log entry, data file and sidecar lives under that prefix; nothing else is needed.

use std::collections::HashMap;
use moraine::Database;

let db = Database::builder("s3://my-bucket/moraine/prod")
    .storage_options(HashMap::from([
        ("aws_region".to_string(), "eu-central-1".to_string()),
    ]))
    .open()
    .await?;

Configuration

storage_options takes object_store's S3 configuration keys (AmazonS3ConfigKey). Anything not given is read from the environment (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN, AWS_REGION, AWS_ENDPOINT and the other variables object_store recognises), which also covers web identity and instance credentials.

Key Use
aws_region The bucket's region. Without the aws-sdk feature, object_store assumes us-east-1 when none is set.
aws_endpoint An S3-compatible server, for example http://127.0.0.1:9000.
aws_allow_http true to allow a plain-HTTP endpoint (a local test server).
aws_access_key_id, aws_secret_access_key, aws_session_token Static credentials.
aws_profile A named profile; needs the aws-sdk feature.

moraine sets object_store's aws_conditional_put to etag itself unless you set it, so that compare-and-swap (If-Match) is available; put-if-absent needs no setting.

Conditional writes are required

Every commit is a put-if-absent of the next log entry (If-None-Match: *). A store that accepts that header and ignores it would let two writers overwrite each other's commits without anyone noticing. Some S3-compatible services do exactly that, so moraine checks before the first write:

  • Once per store and Database handle, a capability probe writes a small object under the database's _meta/ prefix and checks that If-None-Match: * and If-Match with a stale ETag are both refused.
  • If either condition is ignored or rejected, writes fail with Error::ConditionalWritesUnsupported. Reading still works, so such a store can be opened read-only.

AWS S3 supports both conditional headers. Check any other S3-compatible service before relying on it; the probe will tell you, but only when a writer first opens it.

Credentials from the AWS SDK (aws-sdk feature)

object_store reads static keys, web identity and instance credentials, but not profiles, SSO sessions or credential_process, and it does not look up a bucket's region. With the cargo feature aws-sdk:

  • Credentials come from the SDK's default chain (environment, shared config and credentials files with the chosen profile, SSO, credential_process, web identity, container and instance metadata), cached until five minutes before they expire. The storage option aws_profile picks the profile; otherwise AWS_PROFILE does. Static keys or unsigned requests, when configured, take precedence.
  • The region is the bucket's own, found with an unauthenticated HeadBucket against AWS, falling back to the region the SDK resolves, unless a region or a custom endpoint is configured. An S3-compatible server keeps whatever region was given.
[dependencies]
morainedb = { version = "...", features = ["aws-sdk"] }
let db = Database::builder("s3://my-bucket/moraine/prod")
    .storage_options(HashMap::from([("aws_profile".to_string(), "analytics".to_string())]))
    .open()
    .await?;

Object tags

DatabaseBuilder::object_tags attaches up to 10 tags to every object the handle writes: data files, log entries, sidecars and probes. They are useful for cost allocation and lifecycle rules. Tags travel in the write request itself (S3's x-amz-tagging header), so they cost no request; the request-budget tests run with and without tags and assert equal counts. Keys are 1 to 128 characters and values at most 256, S3's limits, checked at open. If your bucket policy controls tagging separately, the writing identity needs permission to tag the objects it puts.

let db = Database::builder("s3://my-bucket/moraine/prod")
    .object_tags([("team", "search"), ("env", "prod")])
    .open()
    .await?;

What moraine does on the bucket

Request When
PUT data files and sidecars (unconditional, unique names)
PUT with If-None-Match: * each log entry, and the probe
PUT with If-Match the probe only (leases need no compare-and-swap)
GET, range GET log entries, footers, pages, sidecars
LIST when a handle opens a table (from the newest checkpoint on), the idx/ prefix per poll interval, and maintenance
DELETE probe objects, old lease generations, committed inbox batches; vacuum, log cleanup and collection

Old versions are removed by maintenance only (see Maintenance): until a pass runs vacuum, log cleanup and sidecar collection, the bucket keeps every file a table has written. No bucket lifecycle rule should delete objects under a table's prefix.

What has been tested

The test suite and the benchmarks run against RustFS, an S3-compatible server in Docker (see spike/s3-server.sh). Before trusting it, the spike checked its conditional writes directly: a second If-None-Match: * put and a stale If-Match both returned 412 PreconditionFailed.

Runs against a real AWS bucket are still owed. The spike's real-S3 variants are ignored tests that run only when a bucket and credentials are configured, and the 8-process commit test on real S3 is planned as a regression test, together with nightly real-cloud runs. AWS returns 409 ConditionalRequestConflict when two conditional writes to one key overlap; moraine treats that as an ambiguous outcome and resolves it like any other.

Other object stores

Only s3://, local paths, file:// and memory:// are wired today. Google Cloud Storage and Azure Blob Storage go through the same object_store crate and are planned, together with real-cloud test runs; until then their cargo features (gcs, azure) are placeholders. An existing object_store instance of any kind can be used through Database::builder_for_store, but only stores that pass the conditional-write probe accept writes.