S3 and bucket requirements¶
A database on S3 is a bucket and a prefix. Every table, log entry, data file and sidecar lives under that prefix; nothing else is needed.
use std::collections::HashMap;
use moraine::Database;
let db = Database::builder("s3://my-bucket/moraine/prod")
.storage_options(HashMap::from([
("aws_region".to_string(), "eu-central-1".to_string()),
]))
.open()
.await?;
Configuration¶
storage_options takes object_store's S3 configuration keys (AmazonS3ConfigKey). Anything not
given is read from the environment (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY,
AWS_SESSION_TOKEN, AWS_REGION, AWS_ENDPOINT and the other variables object_store
recognises), which also covers web identity and instance credentials.
| Key | Use |
|---|---|
aws_region |
The bucket's region. Without the aws-sdk feature, object_store assumes us-east-1 when none is set. |
aws_endpoint |
An S3-compatible server, for example http://127.0.0.1:9000. |
aws_allow_http |
true to allow a plain-HTTP endpoint (a local test server). |
aws_access_key_id, aws_secret_access_key, aws_session_token |
Static credentials. |
aws_profile |
A named profile; needs the aws-sdk feature. |
moraine sets object_store's aws_conditional_put to etag itself unless you set it, so that
compare-and-swap (If-Match) is available; put-if-absent needs no setting.
Conditional writes are required¶
Every commit is a put-if-absent of the next log entry (If-None-Match: *). A store that accepts
that header and ignores it would let two writers overwrite each other's commits without anyone
noticing. Some S3-compatible services do exactly that, so moraine checks before the first write:
- Once per store and
Databasehandle, a capability probe writes a small object under the database's_meta/prefix and checks thatIf-None-Match: *andIf-Matchwith a stale ETag are both refused. - If either condition is ignored or rejected, writes fail with
Error::ConditionalWritesUnsupported. Reading still works, so such a store can be opened read-only.
AWS S3 supports both conditional headers. Check any other S3-compatible service before relying on it; the probe will tell you, but only when a writer first opens it.
Credentials from the AWS SDK (aws-sdk feature)¶
object_store reads static keys, web identity and instance credentials, but not profiles, SSO
sessions or credential_process, and it does not look up a bucket's region. With the cargo
feature aws-sdk:
- Credentials come from the SDK's default chain (environment, shared config and credentials
files with the chosen profile, SSO,
credential_process, web identity, container and instance metadata), cached until five minutes before they expire. The storage optionaws_profilepicks the profile; otherwiseAWS_PROFILEdoes. Static keys or unsigned requests, when configured, take precedence. - The region is the bucket's own, found with an unauthenticated
HeadBucketagainst AWS, falling back to the region the SDK resolves, unless a region or a custom endpoint is configured. An S3-compatible server keeps whatever region was given.
[dependencies]
morainedb = { version = "...", features = ["aws-sdk"] }
let db = Database::builder("s3://my-bucket/moraine/prod")
.storage_options(HashMap::from([("aws_profile".to_string(), "analytics".to_string())]))
.open()
.await?;
Object tags¶
DatabaseBuilder::object_tags attaches up to 10 tags to every object the handle writes: data
files, log entries, sidecars and probes. They are useful for cost allocation and lifecycle
rules. Tags travel in the write request itself (S3's x-amz-tagging header), so they cost no
request; the request-budget tests run with and without tags and assert equal counts. Keys are 1
to 128 characters and values at most 256, S3's limits, checked at open. If your bucket policy
controls tagging separately, the writing identity needs permission to tag the objects it puts.
let db = Database::builder("s3://my-bucket/moraine/prod")
.object_tags([("team", "search"), ("env", "prod")])
.open()
.await?;
What moraine does on the bucket¶
| Request | When |
|---|---|
PUT |
data files and sidecars (unconditional, unique names) |
PUT with If-None-Match: * |
each log entry, and the probe |
PUT with If-Match |
the probe only (leases need no compare-and-swap) |
GET, range GET |
log entries, footers, pages, sidecars |
LIST |
when a handle opens a table (from the newest checkpoint on), the idx/ prefix per poll interval, and maintenance |
DELETE |
probe objects, old lease generations, committed inbox batches; vacuum, log cleanup and collection |
Old versions are removed by maintenance only (see Maintenance): until a pass runs vacuum, log cleanup and sidecar collection, the bucket keeps every file a table has written. No bucket lifecycle rule should delete objects under a table's prefix.
What has been tested¶
The test suite and the benchmarks run against RustFS, an S3-compatible server in Docker (see
spike/s3-server.sh). Before trusting it, the spike checked its conditional writes directly:
a second If-None-Match: * put and a stale If-Match both returned 412 PreconditionFailed.
Runs against a real AWS bucket are still owed. The spike's real-S3 variants are ignored tests
that run only when a bucket and credentials are configured, and the 8-process commit test on real
S3 is planned as a regression test, together with nightly real-cloud runs.
AWS returns 409 ConditionalRequestConflict when two conditional writes to one key overlap;
moraine treats that as an ambiguous outcome and resolves it like any other.
Other object stores¶
Only s3://, local paths, file:// and memory:// are wired today. Google Cloud Storage and
Azure Blob Storage go through the same object_store crate and are planned, together with
real-cloud test runs; until then their cargo features (gcs, azure) are
placeholders. An existing object_store instance of any kind can be used through
Database::builder_for_store, but only stores that pass the conditional-write probe accept
writes.