The moraine command¶
moraine is the operator tool for moraine databases: it inspects tables and runs the
maintenance jobs from a shell or a scheduled job, for deployments whose
application processes are too short-lived to run them.
cargo install --git https://github.com/sayef/moraine moraine-cli
moraine [--option KEY=VALUE]... LOCATION COMMAND [ARGS]
LOCATION is a database URL (s3://bucket/prefix, file:///path) or a local directory.
Storage options (credentials, endpoint, region) come from --option and from the environment,
as object_store reads it (AWS_ACCESS_KEY_ID, AWS_ENDPOINT, and so on). Every command
except stress prints what it did; errors go to standard error with exit status 1.
Commands¶
| Command | What it does |
|---|---|
tables |
Lists the tables of the database |
inspect TABLE |
Version, files, rows, bytes, files per compaction level, pins and indexes |
maintain TABLE [--no-leases] [--every SECONDS] |
One maintenance pass, or one every SECONDS until stopped; prints each pass's report |
compact TABLE [--full] |
Compaction alone, without a lease; --full merges every run of small files |
checkpoint TABLE |
Writes a checkpoint at the newest version if none exists for it |
vacuum TABLE |
Vacuum and sidecar collection alone, without a lease |
ingest TABLE |
Commits queued writes from the inbox, under the ingest lease |
verify TABLE |
Checks data files and sidecars; exit status 1 when something is wrong |
repair TABLE |
Removes the table's references to missing data files and reports the rows lost |
pins TABLE |
Lists the pins, name and version |
pin TABLE NAME [VERSION] |
Pins a version under NAME (the newest by default) |
unpin TABLE NAME |
Removes a pin |
stress TABLE --seconds N --process ID [--maintain] |
One writer of the multi-process stress test (T8); see below |
Examples¶
# A pass every minute, from a container that runs nothing else.
moraine s3://bucket/db maintain events --every 60
# Against an S3-compatible server.
moraine --option aws_endpoint=http://localhost:9000 --option aws_allow_http=true \
s3://bucket/db inspect events
# Keep a version readable through vacuum, and read what is pinned.
moraine ./data pin events release-1
moraine ./data pins events
The stress test¶
moraine LOCATION stress TABLE --seconds N --process ID [--maintain] is one writer process of
the multi-process stress test T8 of the test suites: it appends batches with
transaction ids, deletes its own junk rows, queues batches for the ingestor and reads its rows
back, and with --maintain runs a maintenance pass every two seconds. Every acknowledged write
is appended to the file MORAINE_STRESS_ACKS names before the next one starts. The test
crates/moraine-cli/tests/t8.rs runs eight of them on one table with random kill -9 and
restarts, then checks the table against the acknowledgements:
MORAINE_T8_SECONDS=600 cargo test --release -p moraine-cli --test t8 -- --ignored