Skip to content

Full-text and hybrid search

This guide adds a full-text index to a table, searches it, and combines it with a vector index. How sidecars are built, cached and found is in Sidecar indexes; this page covers what is particular to text.

Enable the kind

The full-text kind, fts, is behind a cargo feature; hybrid search needs a vector kind too:

[dependencies]
morainedb = { git = "https://github.com/sayef/moraine", features = ["fts", "vector"] }
serde_json = "1"                                                                  # index parameters

fts builds one tantivy index per data file. Every client that searches or builds a full-text index needs the feature.

Create the index

A full-text index covers one or more string columns (Utf8, LargeUtf8 or Utf8View). Null and empty values are fine: every row is a document, so document counts always match row counts.

use moraine::IndexSpec;
use serde_json::json;

let spec = IndexSpec::new("text_en", "fts", &["title", "body"])
    .params(json!({
        "tokenizer": "en_stem",
        "fields": {"title": {"record": "freqs"}}
    }))
    .search(json!({"boosts": {"title": 2.0}}));
let resolved = table.create_index(spec).await?;
println!("{}", resolved.params);

tokenizer and record at the top of params apply to every column; fields.<column> sets them for one. create_index stores every column's settings, so the stored spec never changes meaning.

Where Key Default Meaning
params tokenizer default How text becomes terms (see Tokenizers)
params record positions basic (documents only), freqs (term frequencies), positions (needed by phrase queries)
params max_indexed_chars 1,000,000 Longer values are cut for indexing; the stored value is untouched
policy min_rows_for_sidecar 1,000 Smaller files get no sidecar and are indexed in memory when searched
search boosts 1.0 per column Multiplies a column's scores
search conjunction false true makes every word required

params change what a sidecar holds and are hashed; policy and search are not, so changing a boost needs no rebuild. Column names must be usable in the query syntax: no whitespace, no :, ., ^, quotes or brackets, and no leading -, _ or +.

use moraine::expr::{col, lit};

let result = table
    .search_text("text", "glacier OR \"conditional put\"")
    .limit(10)
    .filter(col("id").gt(lit(0i64)))
    .select(["id", "title"])
    .collect()
    .await?;
// Rows best first, with `_moraine_score`: the BM25 score (larger is better).
for hit in &result.hits {
    println!("{} @ {}: {}", hit.file, hit.offset, hit.score);
}

// Search defaults can be overridden per query.
let strict = table
    .search_text("text", "glacier rock")
    .param("conjunction", true)
    .param("boosts", serde_json::json!({"title": 3.0}))
    .collect()
    .await?;
println!("{} rows hold both words", strict.batch.num_rows());

The query uses tantivy's syntax: words (any may match), "a phrase", title:word, +required, -excluded, AND, OR, parentheses and word^2 for a boost. A query that is empty, or that holds nothing once tokenised, or only excluded words, returns no rows. A syntax error, a column the index does not hold, or a phrase on a column that records no positions is Error::InvalidQuery, naming the problem.

The result has the same shape as a vector search: the rows, best first, the hits (data file, row offset, score) and how the files were searched (stats.sidecar_files, and stats.exact_files for files indexed in memory).

How ranking works

BM25 scores a row with statistics of the whole collection: how many documents there are, their average length, and how many hold each query word. moraine keeps one index per data file, but it does not score with each file's own statistics. A search first gets a searcher for every file of the snapshot, adds up their statistics, and scores every file with the sums. The scores are then those of one tantivy index over the whole table, whatever the number of files and whichever of them have sidecars.

The test suite checks this against such a merged index on random corpora: every score agrees within 1e-5 and the rankings are the same, only rows with exactly equal scores may be ordered differently (moraine breaks ties by data file, then row). See Benchmarks for the numbers.

A filter does not change the statistics, just as it would not change a merged index's: it only drops rows. A file whose statistics rule the filter out is not searched, but still counts in the statistics.

Files without a sidecar

Until build_indexes has built a file's sidecar, and always for files below min_rows_for_sidecar, a search indexes the file's text columns in memory with the same builder, so the answer is the same either way. The in-memory indexes are kept for later searches up to DatabaseBuilder::fts_memory (default 512 MiB); beyond it the least recently used are dropped, and a query never fails for it.

Sidecars are built as for vectors:

table.build_indexes(BuildOptions::default()).await?;

A build holds tantivy's writer memory (32 MiB) and spills larger files to temporary segments on disk, merged into one segment at the end. Rebuilding the same rows gives a byte-identical sidecar.

Tokenizers

Name Terms
default Words (runs of letters and digits), longer than 40 bytes dropped, lowercased
raw The whole value as one term (identifiers, tags)
whitespace Split on whitespace, lowercased
en_stem, de_stem, fr_stem, ... default, then the Snowball stemmer of the language: ar da de el en es fi fr hu it nl no pt ro ru sv ta tr
ngram2, ngram3 Lowercased character pairs or triples, for languages written without spaces

A tokenizer of your own is a tantivy TextAnalyzer registered under a name. Every client that builds or searches an index naming it must register the same tokenizer under the same name; a client that does not know a name gets Error::TokenizerNotRegistered rather than different terms. Built-in names cannot be registered again.

use moraine::Database;
use moraine::index::fts::tantivy::tokenizer::{
    LowerCaser, TextAnalyzer, WhitespaceTokenizer,
};

let words = TextAnalyzer::builder(WhitespaceTokenizer::default())
    .filter(LowerCaser)
    .build();
let db = Database::builder(location)
    .tokenizer("words", words)
    .open()
    .await?;

search_hybrid runs a vector search and a full-text search against the same snapshot and fuses their rankings:

let result = docs
    .search_hybrid("emb", vec![1.0, 0.1], "text", "storage")
    .limit(3)
    .select(["id", "body"])
    .collect()
    .await?;

Each search asks for k * oversample hits (default 4, .oversample(n)), with the same filter. The fusion is reciprocal rank fusion with the constant 60: a row's score is the sum, over the lists it appears in, of 1 / (60 + rank). I chose it because it needs no calibration between a vector distance and a BM25 score, which live on unrelated scales, and because a row both searches find outranks one that either finds alone at a similar rank. _moraine_score holds the fused score. .vector_param and .text_param override each search's defaults, and .fusion takes any moraine::index::hybrid::Fusion, for example a weighted sum of normalised scores.

Fresh results without waiting

With Consistency::Background, a task per table handle picks up new commits every interval and prepares the caches for the new files, including their full-text searchers, so the first search after another writer's commit makes no request for the snapshot and indexes nothing:

use std::time::Duration;

use moraine::{Consistency, Database};

let db = Database::builder(location)
    .consistency(Consistency::Background {
        interval: Duration::from_secs(1),
    })
    .open()
    .await?;
let notes = db.open_table("notes").await?;
// Queries use the snapshot the background task keeps current: no request for it.
let hits = notes.search_text("text", "moraine").collect().await?;
println!("{} hits at version {}", hits.hits.len(), notes.version());