snapshot of https://github.com/blevesearch/bleve.git (master, 2026-09-15; blobs > 131072B omitted)

docs/create_and_search_your_first_index.md

@@ -0,0 +1,226 @@
+# Creating a Bleve Index
+
+A simple how-to example using Bleve in Go to create an index, add documents, and run search queries with results.
+
+```go
+package main
+
+import (
+    "fmt"
+    "log"
+
+    bleve "github.com/blevesearch/bleve/v2"
+)
+
+type Document struct {
+    ID      string `json:"id"`
+    Title   string `json:"title"`
+    Content string `json:"content"`
+}
+
+func main() {
+    indexPath := "example.bleve"
+    // Create a new index
+    mapping := bleve.NewIndexMapping()
+    index, err := bleve.New(indexPath, mapping)
+    if err != nil {
+        log.Fatal(err)
+    }
+    defer index.Close()
+
+    // Add documents
+    documents := []Document{
+        {
+            ID:      "doc",
+            Title:   "Bleve documentation",
+            Content: "Bleve provides full-text search capabilities.",
+        },
+        {
+            ID:      "doc1",
+            Title:   "Elasticsearch documentation",
+            Content: "Elasticsearch provides full-text search capabilities as well.",
+        },
+    }
+
+    // Iterate and index the documents
+    batch := index.NewBatch()
+    for _, doc := range documents {
+        batch.Index(doc.ID, doc)
+    }
+    if err := index.Batch(batch); err != nil {
+        log.Fatal(err)
+    }
+
+    // Search the created index
+    query := bleve.NewQueryStringQuery("bleve")
+    searchRequest := bleve.NewSearchRequest(query)
+    searchRequest.Explain = true
+    searchRequest.Fields = []string{"title", "content"}
+    searchResult, err := index.Search(searchRequest)
+    if err != nil {
+        log.Fatal(err)
+    }
+    fmt.Println(searchResult)
+}
+
+```
+
+## Output
+
+```bash
+$ go run main.go
+
+1 matches, showing 1 through 1, took 262.333µs
+    1. doc (0.471405)
+        title
+                Bleve documentation
+        content
+                Bleve provides full-text search capabilities.
+```
+
+## Step-by-Step Breakdown
+
+### 1. Index Creation
+
+```go
+// Create a new index mapping
+mapping := bleve.NewIndexMapping()
+// Create a new index (this creates a directory on disk)
+index, err := bleve.New(indexPath, mapping)
+```
+
+**What happens:**
+
+- Creates an index mapping with default settings
+- Creates a new index directory `example.bleve/`
+- Sets up the underlying storage (Scorch engine by default)
+
+### 2. Document Indexing
+
+```go
+// Index a document with a unique ID
+err := index.Index("doc", map[string]interface{}{
+    "title":   "My Document",
+    "content": "This is the document content",
+    "author":  "John Doe",
+})
+```
+
+**What happens:**
+
+- Document gets a unique ID (`doc`)
+- Fields are automatically mapped based on their Go types
+- Text fields are analyzed (tokenized, lowercased, etc.) based on the chosen mapping (here, the default one)
+- Document is stored in the search index
+
+### 3. Searching
+
+```go
+// Create a query
+query := bleve.NewQueryStringQuery("search terms")
+request := bleve.NewSearchRequest(query)
+
+// Execute search
+results, err := index.Search(request)
+```
+
+**What happens:**
+
+- Query string is parsed and analyzed
+- Index is searched for matching documents
+- Results are scored and ranked by relevance by the algorithm used
+- Document metadata and highlights are returned
+
+## Working with Existing Indexes
+
+To open an existing index instead of creating a new one:
+
+```go
+// Open existing index
+index, err := bleve.Open("example.bleve")
+if err != nil {
+    log.Fatal(err)
+}
+defer index.Close()
+```
+
+## Different Query Types
+
+### 1. Query String Query (Simple)
+
+```go
+query := bleve.NewQueryStringQuery("golang programming")
+```
+
+### 2. Match Query (Exact Field)
+
+```go
+query := bleve.NewMatchQuery("bleve")
+query.SetField("title")  // Search only in title field
+```
+
+### 3. Boolean Query (Complex)
+
+```go
+mustQuery := bleve.NewMatchQuery("golang")
+shouldQuery := bleve.NewMatchQuery("programming")
+
+boolQuery := bleve.NewBooleanQuery()
+boolQuery.AddMust(mustQuery)
+boolQuery.AddShould(shouldQuery)
+```
+
+### 4. Range Query (Numeric/Date)
+
+```go
+minPrice := 20.50
+maxPrice := 40.75
+query := bleve.NewNumericRangeQuery(&minPrice, &maxPrice)
+query.SetField("price")
+```
+
+## Advanced Index Configuration
+
+### Custom Field Mapping
+
+```go
+// We can create customized mappings as well with configurable analyzers
+mapping := bleve.NewIndexMapping()
+
+// Text field with custom analyzer
+titleMapping := bleve.NewTextFieldMapping()
+titleMapping.Analyzer = "en"  // English analyzer
+
+// Numeric field
+priceMapping := bleve.NewNumericFieldMapping()
+
+// Date field
+dateMapping := bleve.NewDateTimeFieldMapping()
+
+// Document mapping
+docMapping := bleve.NewDocumentMapping()
+docMapping.AddFieldMappingsAt("title", titleMapping)
+docMapping.AddFieldMappingsAt("price", priceMapping)
+docMapping.AddFieldMappingsAt("created_at", dateMapping)
+
+// Add to index mapping
+mapping.AddDocumentMapping("product", docMapping)
+
+// Create index with custom mapping
+index, err := bleve.New("products.bleve", mapping)
+```
+
+### Batch Operations
+
+For better performance when indexing many documents, we can do indexing in batches:
+
+```go
+batch := index.NewBatch()
+
+for _, doc := range documents {
+    batch.Index(doc.ID, doc)
+}
+
+// Execute batch
+err := index.Batch(batch)
+```

docs/custom_query.md

@@ -0,0 +1,137 @@
+# Custom Queries: `custom_filter` and `custom_score`
+
+Bleve exposes two query nodes for embedding-defined per-hit logic:
+
+- `custom_filter`: keep or drop each candidate hit.
+- `custom_score`: mutate each candidate hit score.
+
+Bleve itself only executes callbacks that were already provided/attached by the
+embedding application. It does not interpret any
+embedder-specific payload such as callback source, "params", or requested fields.
+
+## Query Objects
+
+`CustomFilterQuery` contains:
+
+- a child query
+- a `searcher.CustomFilterFunc` callback attached by the embedder
+
+`CustomScoreQuery` contains:
+
+- a child query
+- a `searcher.CustomScoreFunc` callback attached by the embedder
+
+## Constructors
+
+Embedding applications can construct the nodes directly with callbacks:
+
+```go
+filterQuery := query.NewCustomFilterQueryWithFilter(childQuery,
+    func(d *search.DocumentMatch) bool {
+        return true
+    },
+    nil, // fields; nil when callback does not need field values
+    nil, // payload; nil when no source/params round-trip is needed
+)
+
+scoreQuery := query.NewCustomScoreQueryWithScorer(childQuery,
+    func(d *search.DocumentMatch) float64 {
+        return d.Score
+    },
+    nil, // fields
+    nil, // payload
+)
+```
+
+Constructor arguments after the callback are:
+
+- `fields`: field names to preload into `d.Fields` via doc values before the callback runs
+- `payload`: non-query custom attributes that must survive JSON round-trips (for example distributed fan-out)
+
+Pass `nil` for either when not needed:
+
+```go
+payload := map[string]interface{}{
+    "params": map[string]interface{}{"weight": 0.05},
+    "source": "function score(doc, params) { ... }",
+}
+
+scoreQuery := query.NewCustomScoreQueryWithScorer(childQuery,
+    func(d *search.DocumentMatch) float64 {
+        f := d.Fields
+        return d.Score + (f["abv"].(float64) * 0.05)
+    },
+    []string{"abv", "ibu"},
+    payload,
+)
+```
+
+Payload rules:
+
+- pass only non-query custom-node attributes like `source` and `params`
+- do not include `query` in payload; `query` always comes from the child query argument
+- do not include `fields` in payload; `fields` come from the dedicated constructor argument
+- pass `nil` as the fourth argument when payload round-trip is not needed
+
+## Field Access and Doc Values Requirement
+
+When the `fields` constructor argument is non-empty, bleve loads the listed
+field values into `DocumentMatch.Fields` before invoking the callback. Field
+values are read from **doc values** (column-oriented, memory-mapped), not
+stored fields.
+
+**Doc values must be enabled for every field listed in `"fields"`.** If a field
+does not have doc values enabled in the index mapping, its value will be silently
+absent from `DocumentMatch.Fields` at callback time — no error is raised, the
+field simply won't appear. This can cause filters to drop all hits or scores to
+compute incorrectly if the callback assumes the field is present.
+
+Doc values are enabled by default for all field types (`DocValues: true`) and
+for dynamic mappings (`DocValuesDynamic: true`), so no extra configuration is
+needed unless these defaults have been explicitly disabled. If they have been
+disabled, re-enable them:
+
+```go
+fm := mapping.NewTextFieldMapping()
+fm.DocValues = true                    // default is true; only needed if previously disabled
+imap.DefaultMapping.AddFieldMappingsAt("genre", fm)
+```
+
+For dynamic mappings, ensure `docvalues_dynamic` has not been set to `false` on
+the document mapping.
+
+**Datetime fields require an explicit datetime mapping.** Dynamic mapping indexes
+date-like strings as text, which means the standard analyzer tokenizes the value
+(e.g. `"2010-07-22 20:00:20"` becomes `["2010", "07", "22", ...]`). The doc
+value then contains tokens instead of the original date, making comparisons
+fail. To use datetime fields in custom_filter or custom_score callbacks, add an
+explicit field mapping with `"type": "datetime"` in the index definition.
+
+### Field type decoding
+
+At the bleve layer, doc values are decoded based on the field's mapping type:
+
+| Mapping type | Go type in `d.Fields` | Notes |
+| --- | --- | --- |
+| `text` / `keyword` | `string` | Raw text bytes |
+| `number` | `float64` | IEEE 754 decoded |
+| `boolean` | `bool` | `true` / `false` |
+| `datetime` | `float64` | Epoch nanoseconds stored as a floating-point value |
+
+## Runtime Behavior
+
+- Bleve builds the child query searcher first.
+- If `fields` is non-empty, a `DocValueReader` is created for those fields.
+- `custom_filter` wraps the child with `CustomFilterSearcher`.
+- `custom_score` wraps the child with `CustomScoreSearcher`.
+- On each hit, doc values are loaded into `DocumentMatch.Fields`, then the callback is invoked.
+- The callback receives only `*search.DocumentMatch`; it does not receive `SearchContext`.
+
+## Error Cases
+
+- missing child query:
+  - `"custom filter query must have a query"`
+  - `"custom score query must have a query"`
+- missing bound callback:
+  - `"custom filter query must have a filter callback"`
+  - `"custom score query must have a score callback"`

docs/fast_merge.md

@@ -0,0 +1,64 @@
+# Fast Merge
+
+* v2.6.0 comes with the support for a feature called "fast merge" where we first train the index on a vector dataset and then build the vector index using this trained information of the centroid layout.
+* This is a separate feature when compared to the existing behavior where we were performing the merge in a very naive fashion of reconstructing the participating vector indexes, re-training and then adding back the vectors into the index. Fast merge essentially merges the corresponding centroid cells' data vectors in a block-wise fashion without doing the expensive operations. For this, a centroid/trained segment template needs to be built using enough random samples from a dataset prior to building segments with the actual index data.
+* A trade-off with this feature is if new data were introduced that is not available at the time of index introduction, a data drift can develop resulting in reduced recall.
+* This feature underneath the hood utilizes the existing [`merge_from` API](https://github.com/blevesearch/faiss/blob/ffd910a91f1acf49b9898a7e514e462db89ee7b3/faiss/Index.h#L396) in our fork of the FAISS codebase.
+
+## Support
+
+* This feature is supported primarily for the IVF family of indexes. This happens when
+  * the field mapping has optimization type as `ivf,rabitq` or when using `bivf-sq8`/`bivf-flat` for binary quantization.
+  * when the above optimizations aren't used but the scale of data exceeds 10K vectors.
+
+## Usage
+
+The feature can be enabled by first passing a key-value pair in the config part while creating a new index. If the flag is false, then the behavior falls back to the more expensive naive merge.
+
+```go
+kvConfig := map[string]interface{}{
+  "vector_index_fast_merge": true,
+}
+
+index, err := bleve.NewUsing("example.bleve", bleve.NewIndexMapping(), bleve.Config.DefaultIndexType, bleve.Config.DefaultMemKVStore, kvConfig)
+if err != nil {
+  log.Fatal(err)
+}
+```
+
+The user should now "train" the index on a random sample of the vector dataset they're planning to index and search.
+
+* It's completely up to the user as to how much data they want to use for training, controlling the batch size used while training and also marking whether the training is complete.
+* NOTE: User must index their data only after marking the training as complete, otherwise the batch won't be indexed.
+
+The training is done using the new `Train()` API which takes the existing `Batch` construct we have
+
+```go
+batch := index.NewBatch()
+for _, doc := range trainingDocuments {
+  batch.Index(doc.ID, doc)
+}
+
+// train the index on the batch of data
+// NOTE: the training can be done in an incremental manner as well, by using the same Train() API but repeatedly calling it on a particular batch of data.
+if err := index.Train(batch); err != nil {
+  log.Fatal(err)
+}
+
+batch.Reset()
+batch.SetInternal(util.BoltTrainCompleteKey, []byte("true"))
+if err := index.Train(batch); err != nil {
+  log.Fatal(err)
+}
+
+// at this point the index is trained completely, and any training related constructs in the background will be cleaned up
+```
+
+## Disclaimer
+
+* This feature is primarily meant for the use case where the user is aware of how much data they want to index and also for a read-heavy workload and little to no updates on the index itself.
+* The intention of the feature is to be able to quickly index a massive scale of data on an index in an efficient manner and perform search on it.
+* Without this feature, i.e. when the index build happens without a prior training phase
+  * The user wouldn't have to worry about use cases where the dataset is continuously updated with new "type" of vector. This is because each merge cycle would do the training afresh.
+  * The user doesn't have a lag in indexing the data either, they can start ingesting the data immediately.
+* Based on what's mentioned above, when it comes to update and delete type of workloads on the dataset it's extremely difficult to detect when the data drift will occur. So we end up falling back to the naive way of reconstructing + re-training.

docs/geo.md

@@ -0,0 +1,3 @@
+# Geo spatial search
+
+Redirect to [geo/README.md](https://github.com/blevesearch/bleve/blob/master/geo/README.md)

docs/hierarchy.md

@@ -0,0 +1,376 @@
+# Hierarchical nested search
+
+* *v2.6.0* (and after) will come with support for **array indexing and hierarchical nested search**.
+* We've achieved this by embedding nested documents within our bleve (scorch) indexes.
+* Usage of zap file format: [v17](https://github.com/blevesearch/zapx/blob/master/zap.md). Here we preserve hierarchical document relationships within segments, continuing to conform to the segmented architecture of *scorch*.
+
+## Supported
+
+* Indexing `Arrays` allows specifying fields that contain arrays of objects. Each object in the array can have its own set of fields, enabling the representation of hierarchical data structures within a single document.
+
+```json
+{
+    "id": "1",
+    "name": "John Doe",
+    "addresses": [
+        {
+            "type": "home",
+            "street": "123 Main St",
+            "city": "Hometown",
+            "zip": "12345"
+        },
+        {
+            "type": "work",
+            "street": "456 Corporate Blvd",
+            "city": "Metropolis",
+            "zip": "67890"
+        }
+    ]
+}
+```
+
+* Multi-level arrays: arrays can contain objects that themselves have array fields, allowing for deeply nested structures, such as a list of projects, each with its own list of tasks.
+
+```json
+{
+    "id": "2",
+    "name": "Jane Smith",
+    "projects": [
+        {
+            "name": "Project Alpha",
+            "tasks": [
+                {"title": "Task 1", "status": "completed"},
+                {"title": "Task 2", "status": "in-progress"}
+            ]
+        },
+        {
+            "name": "Project Beta",
+            "tasks": [
+                {"title": "Task A", "status": "not-started"},
+                {"title": "Task B", "status": "completed"}
+            ]
+        }
+    ]
+}
+```
+
+* Multiple arrays: A document can have multiple fields that are arrays, each representing different hierarchical data, such as a list of phone numbers and a list of email addresses.
+
+```json
+{
+    "id": "3",
+    "name": "Alice Johnson",
+    "phones": [
+        {"type": "mobile", "number": "555-1234"},
+        {"type": "home", "number": "555-5678"}
+    ],
+    "emails": [
+        {"type": "personal", "address": "alice@example.com"},
+        {"type": "work", "address": "alice@work.com"}
+    ]
+}
+```
+
+* Hybrid arrays: Multi-level and multiple arrays can be combined within the same document to represent complex hierarchical data structures, such as a company with multiple departments, each having its own list of employees and projects.
+
+```json
+{
+    "id": "doc1",
+    "company": {
+        "id": "c1",
+        "name": "TechCorp",
+        "departments": [
+            {
+                "name": "Engineering",
+                "budget": 2000000,
+                "employees": [
+                    {"name": "Alice", "role": "Engineer"},
+                    {"name": "Bob", "role": "Manager"}
+                ],
+                "projects": [
+                    {"title": "Project X", "status": "ongoing"},
+                    {"title": "Project Y", "status": "completed"}
+                ]
+            },
+            {
+                "name": "Sales",
+                "budget": 300000,
+                "employees": [
+                    {"name": "Eve", "role": "Salesperson"},
+                    {"name": "Mallory", "role": "Manager"}
+                ],
+                "projects": [
+                    {"title": "Project A", "status": "completed"},
+                    {"title": "Project B", "status": "ongoing"}
+                ]
+            }
+        ],
+        "locations": [
+            {"city": "Athens","country": "Greece"},
+            {"city": "Berlin", "country": "USA"}
+        ]
+    }
+}
+```
+
+* Earlier versions of Bleve only supported flat arrays of primitive types (e.g., strings, numbers), and would flatten nested structures, losing the hierarchical relationships, so the above complex documents could not be accurately represented or queried. For example, the "employees" and "projects" fields within each department would be flattened, making it impossible to associate employees with their respective departments.
+
+* From v2.6.0 onwards, Bleve allows for accurate representation and querying of complex nested structures, preserving the relationships between different levels of the hierarchy, across multi-level, multiple and hybrid arrays.
+
+* The addition of `nested` document mappings enables the definition of fields that contain arrays of objects, giving the option to preserve the hierarchical relationships within the array during indexing. Having `nested` as false (default) will flatten the objects within the array, losing the hierarchy, which was the earlier behavior.
+
+```json
+{
+    "departments": {  
+        "dynamic": false,
+        "enabled": true,
+        "nested": true,
+        "properties": {
+            "employees": {
+                "dynamic": false,
+                "enabled": true,
+                "nested": true 
+            },
+            "projects": {
+                "dynamic": false,
+                "enabled": true,
+                "nested": true
+            }
+        }
+    },
+    "locations": {
+        "dynamic": false,
+        "enabled": true,
+        "nested": true
+    }
+}
+```
+
+* Any Bleve query (e.g., `match`, `phrase`, `term`, `fuzzy`, `numeric/date range`, etc.) can be executed against fields within nested documents, with no special handling required. The query processor will automatically traverse the nested structures to find matches. Additional search constructs
+like vector search, synonym search, hybrid and pre-filtered vector search integrate seamlessly with hierarchy search.
+
+* Conjunction Queries (AND queries) and other queries that depend on term co-occurrence within the same hierarchical context will respect the boundaries of nested documents. This means that terms must appear within the same nested object to be considered a match. For example, a conjunction query searching for an employee named "Alice" with the role "Engineer" within the "Engineering" department will only return results where both name and role terms are found within the same employee object, which is itself within an "Engineering" department object.
+
+* Some other search constructs will have enhanced precision with hierarchy search.
+  * Field-Level Highlighting: Only fields within the matched nested object are retrieved and highlighted, ensuring highlights appear in the correct hierarchical context. For example, a match in `departments[name=Engineering].employees` highlights only employees in that department.
+
+  * Nested Faceting / Aggregations: Facets are computed within matched nested objects, producing context-aware buckets. E.g., a facet on `departments.projects.status` returns ongoing or completed only for projects in matched departments.
+
+  * Sorting by Nested Fields: Sorting can use fields from the relevant nested object, e.g., ordering companies by `departments.budget` sorts based on the budget of the specific matched department, not unrelated departments.
+
+* Vector Search (KNN / Multi-KNN): When a document contains an array of objects with vector/multi-vector fields, the final document score and ranking are identical whether or not the array is marked as `nested`. In both cases, the highest-scoring vector is selected; either directly from the array (non-nested) or from the best-matching nested object with its score bubbled up to the parent document.
+
+* Pre-Filtered Vector Search: When vector search is combined with filters on fields inside a nested array, the filters are applied first to pick which nested items are eligible. Vector similarity is then computed only on the vector fields of those filtered nested objects. For example, if `departments.employees` is a `nested` array, a pre-filtered KNN query for employees with a `skills_vector` matching `machine learning engineer`, a role of `Manager`, and belonging to the `Sales` department will first narrow the candidate set to only employees who meet the requirement, and then compute vector similarity on the `skills_vector` of that filtered subset. This ensures that vector search results come only from the employees that satisfy the filter, and not from unrelated employees in other departments.
+
+## Indexing
+
+Below is an example of using the Bleve API to index documents with hierarchical structures, using hybrid arrays and nested mappings.
+
+```go
+// Define a document to be indexed.
+docJSON :=
+    `{
+        "company": {
+            "id": "c3",
+            "name": "WebSolutions",
+            "departments": [
+                {
+                    "name": "HR",
+                    "budget": 800000,
+                    "employees": [
+                        {"name": "Eve", "role": "Manager"},
+                        {"name": "Frank", "role": "HR"}
+                    ],
+                    "projects": [
+                        {"title": "Project Beta", "status": "completed"},
+                        {"title": "Project B", "status": "ongoing"}
+                    ]
+                },
+                {
+                    "name": "Engineering",
+                    "budget": 200000,
+                    "employees": [
+                        {"name": "Heidi", "role": "Support Engineer"},
+                        {"name": "Ivan", "role": "Manager"}
+                    ],
+                    "projects": [
+                        {"title": "Project Helpdesk", "status": "ongoing"},
+                        {"title": "Project FAQ", "status": "completed"}
+                    ]
+                }
+            ],
+            "locations": [
+                {"city": "Edinburgh", "country": "UK"},
+                {"city": "London", "country": "Canada"}
+            ]
+        }
+    }`
+
+// Define departments as a nested document mapping (since it contains arrays of objects)
+// and index name and budget fields
+departmentsMapping := bleve.NewNestedDocumentMapping()
+departmentsMapping.AddFieldMappingsAt("name", bleve.NewTextFieldMapping())
+departmentsMapping.AddFieldMappingsAt("budget", bleve.NewNumericFieldMapping())
+
+// Define employees as a nested document mapping within departments (since it contains arrays of objects)
+// and index name and role fields
+employeesMapping := bleve.NewNestedDocumentMapping()
+employeesMapping.AddFieldMappingsAt("name", bleve.NewTextFieldMapping())
+employeesMapping.AddFieldMappingsAt("role", bleve.NewTextFieldMapping())
+departmentsMapping.AddSubDocumentMapping("employees", employeesMapping)
+
+// Define projects as a nested document mapping within departments (since it contains arrays of objects)
+// and index title and status fields
+projectsMapping := bleve.NewNestedDocumentMapping()
+projectsMapping.AddFieldMappingsAt("title", bleve.NewTextFieldMapping())
+projectsMapping.AddFieldMappingsAt("status", bleve.NewTextFieldMapping())
+departmentsMapping.AddSubDocumentMapping("projects", projectsMapping)
+
+// Define locations as a nested document mapping (since it contains arrays of objects) 
+// and index city and country fields
+locationsMapping := bleve.NewNestedDocumentMapping()
+locationsMapping.AddFieldMappingsAt("city", bleve.NewTextFieldMapping())
+locationsMapping.AddFieldMappingsAt("country", bleve.NewTextFieldMapping())
+
+// Define company as a document mapping and index its name field and 
+// add departments and locations as sub-document mappings
+companyMapping := bleve.NewDocumentMapping()
+companyMapping.AddFieldMappingsAt("name", bleve.NewTextFieldMapping())
+companyMapping.AddSubDocumentMapping("departments", departmentsMapping)
+companyMapping.AddSubDocumentMapping("locations", locationsMapping)
+
+// Define the final index mapping and add company as a sub-document mapping in the default mapping
+indexMapping := bleve.NewIndexMapping()
+indexMapping.DefaultMapping.AddSubDocumentMapping("company", companyMapping)
+
+// Create the index with the defined mapping
+index, err := bleve.New("hierarchy_example.bleve", indexMapping)
+if err != nil {
+    panic(err)
+}
+
+// Unmarshal the document JSON into a map, for indexing
+var doc map[string]interface{}
+err = json.Unmarshal([]byte(docJSON), &doc)
+if err != nil {
+    panic(err)
+}
+
+// Index the document
+err = index.Index("doc1", doc)
+if err != nil {
+    panic(err)
+}
+```
+
+## Querying
+
+```go
+// Open the index
+index, err := bleve.Open("hierarchy_example.bleve")
+if err != nil {
+    panic(err)
+}
+
+var (
+    req *bleve.SearchRequest
+    res *bleve.SearchResult
+)
+
+// Example 1: Simple Match Query on a field within a nested document, should work as if it were a flat field
+q1 := bleve.NewMatchQuery("Engineer")
+q1.SetField("company.departments.employees.role")
+req = bleve.NewSearchRequest(q1)
+res, err = index.Search(req)
+if err != nil {
+    panic(err)
+}
+fmt.Println("Match Query Results:", res)
+
+// Example 2: Conjunction Query (AND) on fields within the same nested document
+// like finding employees with name "Eve" and role "Manager". This will only match
+// if both terms are in the same employee object.
+q1 = bleve.NewMatchQuery("Eve")
+q1.SetField("company.departments.employees.name")
+q2 := bleve.NewMatchQuery("Manager")
+q2.SetField("company.departments.employees.role")
+conjQuery := bleve.NewConjunctionQuery(
+    q1,
+    q2,
+)
+req = bleve.NewSearchRequest(conjQuery)
+res, err = index.Search(req)
+if err != nil {
+    panic(err)
+}
+fmt.Println("Conjunction Query Results:", res)
+
+// Example 3: Multi-level Nested Query, finding projects with status "ongoing"
+// within the "Engineering" department. This ensures both conditions are met
+// within the correct hierarchy, i.e., the ongoing project must belong to the
+// Engineering department.
+q1 = bleve.NewMatchQuery("Engineering")
+q1.SetField("company.departments.name")
+q2 = bleve.NewMatchQuery("ongoing")
+q2.SetField("company.departments.projects.status")
+multiLevelQuery := bleve.NewConjunctionQuery(
+    q1,
+    q2,
+)
+req = bleve.NewSearchRequest(multiLevelQuery)
+res, err = index.Search(req)
+if err != nil {
+    panic(err)
+}
+fmt.Println("Multi-level Nested Query Results:", res)
+
+// Example 4: Multiple Arrays Query, finding documents with a location in "London"
+// and an employee with the role "Manager". This checks conditions across different arrays.
+q1 = bleve.NewMatchQuery("London")
+q1.SetField("company.locations.city")
+q2 = bleve.NewMatchQuery("Manager")
+q2.SetField("company.departments.employees.role")
+multiArrayQuery := bleve.NewConjunctionQuery(
+    q1,
+    q2,
+)
+req = bleve.NewSearchRequest(multiArrayQuery)
+res, err = index.Search(req)
+if err != nil {
+    panic(err)
+}
+fmt.Println("Multiple Arrays Query Results:", res)
+
+// Hybrid Arrays Query, combining multi-level and multiple arrays,
+// finding documents with a Manager named Ivan working in Edinburgh, UK
+q1 = bleve.NewMatchQuery("Ivan")
+q1.SetField("company.departments.employees.name")
+q2 = bleve.NewMatchQuery("Manager")
+q2.SetField("company.departments.employees.role")
+q3 := bleve.NewMatchQuery("Edinburgh")
+q3.SetField("company.locations.city")
+q4 := bleve.NewMatchQuery("UK")
+q4.SetField("company.locations.country")
+hybridArrayQuery := bleve.NewConjunctionQuery(
+    bleve.NewConjunctionQuery(
+        q1,
+        q2,
+    ),
+    bleve.NewConjunctionQuery(
+        q3,
+        q4,
+    ),
+)
+req = bleve.NewSearchRequest(hybridArrayQuery)
+res, err = index.Search(req)
+if err != nil {
+    panic(err)
+}
+fmt.Println("Hybrid Arrays Query Results:", res)
+
+// Close the index when done
+err = index.Close()
+if err != nil {
+    panic(err)
+}
+```

docs/index_update.md

@@ -0,0 +1,45 @@
+# Ability to reduce downtime during index mapping updates
+
+* *v2.5.4* (and after) will come with support for deleting or modifying any field mapping in the index mapping without requiring a full rebuild of the index
+* We do this by storing which portions of the field that have to be deleted within zap and then lazily executing the deletion during subsequent merging of the segments
+
+## Usage
+
+While opening an index, if an updated mapping is provided as a string under the key `updated_mapping` within the `runtimeConfig` parameter of `OpenUsing`, then we open the index and try to update it to use the new mapping provided.
+
+If the update fails, the index is unchanged and an error is returned explaining why the update was unsuccessful.
+
+## What can be deleted and what can't be deleted?
+
+Fields can be partially deleted by changing their Index, Store, and DocValues parameters from true to false, or completely removed by deleting the field itself.
+
+Additionally, document mappings can be deleted either by fully removing them from the index mapping or by setting the Enabled value to false, which deletes all fields defined within that mapping.
+
+However, if any of the following conditions are met, the index is considered non-updatable.
+
+* Any additional fields or enabled document mappings in the new index mapping
+* Any changes to IncludeInAll, type, IncludeTermVectors and SkipFreqNorm
+* Any document mapping having its enabled value changing from false to true
+* Text fields with a different analyzer or date time fields with a different date time format
+* Vector and VectorBase64 fields changing dims, similarity or vectorIndexOptimizedFor
+* Any changes when field is part of `_all`
+* Full field deletions when it is covered by any dynamic setting (Index, Store or DocValues Dynamic)
+* Any changes to dynamic settings at the top level or any enabled document mapping
+* If multiple fields sharing the same field name either from different type mappings or aliases are present, then any non-compatible changes across all of these fields
+
+## How to enforce immediate deletion?
+
+Since the deletion is only done during merging, a [force merge](https://github.com/blevesearch/bleve/blob/b82baf10b205511cf12da5cb24330abd9f5b1b74/index/scorch/merge.go#L164) may be used to completely remove the stale data.
+
+## Sample code to update an existing index
+
+```go
+newMapping := `<Updated Index Mapping>`
+config := map[string]interface{}{
+    "updated_mapping": newMapping,
+}
+index, err := bleve.OpenUsing("<Path to Index>", config)
+if err != nil {
+    return err
+}
+```

docs/pagination.md

@@ -0,0 +1,109 @@
+# Pagination
+
+## Why pagination matters
+
+Search queries can match many documents. Pagination lets you fetch and display results in chunks, keeping responses small and fast.
+
+By default, Bleve returns the first 10 hits sorted by relevance (score), highest first.
+
+## Two pagination modes
+
+- `From`/`Size`: simple and stateless; cost grows with page depth.
+- `SearchAfter`/`SearchBefore`: efficient for deep paging; requires passing sort keys from the previous page.
+
+Both modes can be used with any valid sort.
+
+## `Size`/`From`
+
+Offset-based pagination uses `Size` (page length) and `From` (number of hits to skip). Bleve collects at least `Size + From` ordered results, then returns the `Size` slice starting at `From`.
+
+JSON example:
+
+```json
+{
+  "query": { "match": "California" },
+  "sort": ["-_score"],
+  "size": 5,
+  "from": 10
+}
+```
+
+The result would be 5 hits starting from the 11th hit.
+
+When to use:
+
+- Simple, stateless pagination for shallow pages.
+- Avoid for deep pages, as memory grows with `From` for deeper pages.
+
+## `SearchAfter` and `SearchBefore`
+
+This returns the next (or previous) page based on a boundary defined by the sort keys of a specific hit. This keeps resource usage proportional to the page size, even for deep pages.
+
+Rules:
+
+- Use either `SearchAfter` (forward) or `SearchBefore` (backward), not both at once.
+- The length of `SearchAfter`/`SearchBefore` must match the length of `Sort`.
+- Values are strings representing the sort keys, in the same order as `Sort`.
+- Keep the same `query` and `sort` across pages for consistent navigation.
+
+Where do sort keys come from?
+
+- Each hit includes `Sort` (and `DecodedSort` from Bleve v2.5.2). Take the last hit's sort keys for `SearchAfter`, or the first hit's sort keys for `SearchBefore`.
+- If the field/fields to be searched over are numeric, datetime or geo, the values in the `Sort` field may have garbled values; this is because of how Bleve represents such data types internally. To use such fields as sort keys, use the `DecodedSort` field, which decodes the internal representations. This feature is available from Bleve v2.5.4.
+
+> When using `DecodedSort`, the `Sort` array in the search request needs to explicitly declare the type of the field for proper decoding. Hence, the `Sort` array must contain either `SortField` objects (for numeric and datetime) or `SortGeoDistance` objects (for geo) rather than just the field names. More info on `SortField` and `SortGeoDistance` can be found in [sort_facet.md](sort_facet.md).
+
+Forward pagination over `_id` and `_score`:
+
+```json
+{
+  "query": { "match": "California" },
+  "sort": ["_id", "_score"],
+  "search_after": ["hotel_10180", "0.998"],
+  "size": 3
+}
+```
+
+Backward pagination over `_id` and `_score`:
+
+```json
+{
+  "query": { "match": "California" },
+  "sort": ["_id", "_score"],
+  "search_before": ["hotel_17595", "0.623"],
+  "size": 4
+}
+```
+
+Pagination using numeric, datetime, and geo fields. Notice how we specify the sort objects, with the "type" field explicitly declared in case of numeric and datetime:
+
+```json
+{
+  "query": {
+    "match_all": {}
+  },
+  "size": 10,
+  "sort": [
+    {"by": "field", "field": "price", "type": "number"},
+    {"by": "field", "field": "created_at", "type": "date"},
+    {"by": "geo_distance", "field": "location", "location": {"lat": 40.7128, "lon": -74.0060}}
+  ],
+  "search_after": ["99.99", "2023-10-15T10:30:00Z", "5.2"]
+}
+```
+
+## Total Sort Order
+
+Pagination is deterministic. Ensure your `Sort` defines a total order, so that documents with the same sort keys are not left out:
+
+- Sort strings can be field names (prefix with `-` for descending), `"_score"`, or `"_id"`.
+- Always include a stable tie-breaker as the last key, typically `"_id"`.
+- Examples:
+  - `["country", "-age", "_id"]`
+  - `["-_score", "_id"]` (default score desc with a tie-breaker)
+
+## Performance guidance
+
+- Offset pagination cost grows with `From` (collects at least `Size + From` results before slicing).
+- `SearchAfter`/`SearchBefore` keeps memory and network proportional to `Size`.
+- For large datasets and deep navigation, prefer using `SearchAfter` and `SearchBefore`.

docs/persister.md

@@ -0,0 +1,57 @@
+# Scorch Index Memory and File Management
+
+## Memory Management
+
+When data is indexed in Scorch — using either the `index.Index()` or `index.Batch()` API — it is added as part of an in-memory "segment". Memory management in Scorch indexing mainly relates to handling these in-memory segments during workloads that involve inserts or updates.
+
+In scenarios with a continuous stream of incoming data, a large number of in-memory segments can accumulate over time. This is where the persister component comes into play—its job is to flush these in-memory segments to disk.
+
+Starting with v2.5.0, Scorch supports parallel flushing of in-memory segments to disk, where the persister checks the total in-memory data and distributes the flush across multiple workers. This feature is disabled by default and can be enabled using two configuration options:
+
+- `NumPersisterWorkers`: This factor decides how many maximum workers can be spawned to flush out the in-memory segments. Each worker will work on a disjoint subset of segments, merge them, and flush them out to the disk. By default the persister deploys only one worker.
+- `MaxSizeInMemoryMergePerWorker`: This config decides what's the maximum amount of input data in bytes a single worker can work upon. By default this value is equal to 0 which means that this config is disabled and the worker tries to merge all the data in one shot. Also note that it's imperative that the user set this config if `NumPersisterWorkers > 1`.
+
+If the index is tuned to have a higher `NumPersisterWorkers` value, the memory can potentially drain out faster and ensure stronger consistency behavior — but there would be a lot of on-disk files, and the background merger would experience the pressure of managing this large number of files, which can be resource-intensive.
+
+- Tuning this config is very dependent on the available CPU resources. Something to keep in mind here is that the process's RSS can increase if the number of workers — and each of them working upon a large amount of data — is high.
+
+Increasing the `MaxSizeInMemoryMergePerWorker` value would mean that each worker acts upon a larger amount of data and spends more time merging and flushing it out to disk — which can be healthy behavior in terms of I/O, although it comes at the cost of time.
+
+- Changing this config is use-case dependent; for example, in use cases where the payload or per-doc size is generally large (for example vector use cases), it would be beneficial to have a larger value for this.
+
+So, having the ideal values for these two configs is definitely dependent on the use case and can involve a bunch of experiments, keeping the resource usage in mind.
+
+## File Management
+
+The persister introducing some number of file segments into the system would change the state of the system, and the merger would wake up and try to manage these on-disk files.
+
+Management of these files is crucial when it comes to query latency because a higher number of files would dictate searching through a larger number of files and also higher read amplification to some extent, because the backing data structures can potentially be compacted in size across files.
+
+The merger sees the files on disk and plans out which segments to merge so that the final layout of segment tiers (each tier having multiple files), which grow in a logarithmic way (the chances of larger tiers growing in number would decrease), is maintained. This also implies that deciding this first-tier size becomes important in deciding the number of segment files across all tiers.
+
+Starting with v2.5.0, this first-tier size is dependent on the file size using the `FloorSegmentFileSize` config, because that's a better metric to consider (unlike the legacy live doc count metric) in order to ensure that the behavior is in line with the use case and aware of the payload/doc size.
+
+- This config can also be tuned to dictate how the I/O behaviour should be within an index. While tuning this config, it should be in proportion to the `MaxSizeInMemoryMergePerWorker` since that dictates the amount of data flushed out per flush.
+- The observation here is that `FloorSegmentFileSize` is less than `MaxSizeInMemoryMergePerWorker` and for an optimal I/O during indexing, this value can be set close to `MaxSizeInMemoryMergePerWorker/6`.
+
+## Setting a Persister/Merger Config in Index
+
+The configs are set via the `kvConfig` parameter in the `NewUsing()` or `OpenUsing()` API:
+
+```go
+// setting the persister and merger configs
+kvConfig := map[string]interface{}{
+    "scorchPersisterOptions": map[string]interface{}{
+        "numPersisterWorkers":           4,
+        "maxSizeInMemoryMergePerWorker": 20000000,
+    },
+    "scorchMergePlanOptions": map[string]interface{}{
+        "floorSegmentFileSize": 10000000,
+    },
+}
+// passing the config to the index
+index, err := bleve.NewUsing("example.bleve", bleve.NewIndexMapping(), bleve.Config.DefaultIndexType, bleve.Config.DefaultMemKVStore, kvConfig)
+if err != nil {
+    panic(err)
+}
+```

docs/query-openapi-spec.yaml

@@ -0,0 +1,862 @@
+openapi: 3.0.3
+info:
+  title: Bleve JSON Query Language
+  description: |
+    A comprehensive specification for the Bleve Query JSON Language.
+
+    Bleve is a text indexing library for Go that provides full-text search capabilities.
+    This specification defines the JSON structure for all supported query types in Bleve.
+
+    ## Query Types Supported:
+
+    ### Basic Queries
+    - **Term Query**: Exact term matching
+    - **Match Query**: Full-text search with analysis
+    - **Match Phrase Query**: Phrase matching with proximity
+    - **Prefix Query**: Prefix matching
+    - **Wildcard Query**: Pattern matching with * and ?
+    - **Regexp Query**: Regular expression matching
+    - **Fuzzy Query**: Fuzzy matching with edit distance
+    - **Query String Query**: Human-readable query syntax
+
+    ### Range Queries
+    - **Numeric Range Query**: Numeric value ranges
+    - **Date Range Query**: Date/time ranges
+    - **Term Range Query**: Lexicographic term ranges
+
+    ### Boolean Queries
+    - **Boolean Query**: Must/should/must_not combinations
+    - **Conjunction Query**: All queries must match
+    - **Disjunction Query**: Any query can match
+
+    ### Special Queries
+    - **Match All Query**: Matches all documents
+    - **Match None Query**: Matches no documents
+    - **Bool Field Query**: Boolean field matching
+    - **Doc ID Query**: Document ID matching
+
+    ### Geographic Queries
+    - **Geo Distance Query**: Distance-based geographic search
+    - **Geo Bounding Box Query**: Bounding box geographic search
+    - **Geo Bounding Polygon Query**: Polygon geographic search
+    - **Geo Shape Query**: Complex geographic shape matching
+
+    ### Vector Queries
+    - **KNN Query**: K-nearest neighbors vector search
+
+    ### Network Queries
+    - **IP Range Query**: IP address range matching
+
+    ## Common Properties
+
+    All queries support:
+      - **boost**: Query boost value (float, default 1.0)
+      - **field**: Field to search (string, optional)
+
+    ## Query Parsing
+
+    Bleve automatically detects query types based on the presence of specific JSON fields.
+    The query parser uses a heuristic approach to determine the query type from the JSON structure.
+
+  version: 2.0.0
+  contact:
+    name: Bleve Search
+    url: https://github.com/blevesearch/bleve
+  license:
+    name: Apache 2.0
+    url: https://www.apache.org/licenses/LICENSE-2.0
+
+tags:
+  - name: Basic Queries
+    description: Basic text search queries
+  - name: Range Queries
+    description: Range-based queries for numeric, date, and term values
+  - name: Boolean Queries
+    description: Compound queries combining multiple sub-queries
+  - name: Special Queries
+    description: Special purpose queries like match all/none
+  - name: Geographic Queries
+    description: Geographic and spatial search queries
+  - name: Vector Queries
+    description: Vector similarity and KNN queries
+  - name: Network Queries
+    description: Network and IP address queries
+
+components:
+  schemas:
+    # Base Query Schema
+    Query:
+      type: object
+      description: Base query object - all queries extend this
+      properties:
+        boost:
+          type: number
+          format: float
+          description: Query boost value
+          default: 1.0
+          minimum: 0
+        field:
+          type: string
+          description: Field to search (optional, uses default field if not specified)
+      additionalProperties: true
+
+    # Basic Queries
+    TermQuery:
+      allOf:
+        - type: object
+          properties:
+            term:
+              type: string
+              description: Exact term to search for
+              example: "search"
+          required:
+            - term
+          example:
+            term: "search"
+            field: "title"
+            boost: 2.0
+
+    MatchQuery:
+      allOf:
+        - type: object
+          properties:
+            match:
+              type: string
+              description: Text to match (will be analyzed)
+              example: "full text search"
+            operator:
+              type: string
+              enum: [and, or]
+              description: Boolean operator for multiple terms
+              default: "or"
+            fuzziness:
+              oneOf:
+                - type: integer
+                  minimum: 0
+                  maximum: 2
+                  description: Edit distance for fuzzy matching
+                - type: string
+                  enum: ["auto"]
+                  description: Automatic fuzziness
+              default: 0
+            prefix_length:
+              type: integer
+              description: Prefix length for fuzzy matching
+              minimum: 0
+              default: 0
+            analyzer:
+              type: string
+              description: Analyzer to use (optional)
+          required:
+            - match
+          example:
+            match: "full text search"
+            field: "content"
+            operator: "and"
+            fuzziness: 1
+
+    MatchPhraseQuery:
+      allOf:
+        - type: object
+          properties:
+            match_phrase:
+              type: string
+              description: Phrase to match exactly
+              example: "exact phrase match"
+            fuzziness:
+              oneOf:
+                - type: integer
+                  minimum: 0
+                  maximum: 2
+                - type: string
+                  enum: ["auto"]
+              default: 0
+            analyzer:
+              type: string
+              description: Analyzer to use (optional)
+          required:
+            - match_phrase
+          example:
+            match_phrase: "exact phrase match"
+            field: "content"
+            fuzziness: "auto"
+
+    PrefixQuery:
+      allOf:
+        - type: object
+          properties:
+            prefix:
+              type: string
+              description: Prefix to match
+              example: "pref"
+          required:
+            - prefix
+          example:
+            prefix: "pref"
+            field: "title"
+
+    WildcardQuery:
+      allOf:
+        - type: object
+          properties:
+            wildcard:
+              type: string
+              description: Wildcard pattern (* matches any sequence, ? matches single character)
+              example: "te*t"
+          required:
+            - wildcard
+          example:
+            wildcard: "te*t"
+            field: "title"
+
+    RegexpQuery:
+      allOf:
+        - type: object
+          properties:
+            regexp:
+              type: string
+              description: Regular expression pattern (should not include ^ or $)
+              example: "te[st]"
+          required:
+            - regexp
+          example:
+            regexp: "te[st]"
+            field: "title"
+
+    FuzzyQuery:
+      allOf:
+        - type: object
+          properties:
+            term:
+              type: string
+              description: Term for fuzzy matching
+              example: "search"
+            fuzziness:
+              oneOf:
+                - type: integer
+                  minimum: 0
+                  maximum: 2
+                - type: string
+                  enum: ["auto"]
+              default: 1
+            prefix_length:
+              type: integer
+              description: Prefix length for fuzzy matching
+              minimum: 0
+              default: 0
+          required:
+            - term
+          example:
+            term: "search"
+            field: "title"
+            fuzziness: "auto"
+
+    QueryStringQuery:
+      allOf:
+        - type: object
+          properties:
+            query:
+              type: string
+              description: Query string in human-readable syntax
+              example: "title:search AND content:full text"
+          required:
+            - query
+          example:
+            query: "title:search AND content:full text"
+
+    # Range Queries
+    NumericRangeQuery:
+      allOf:
+        - type: object
+          properties:
+            min:
+              type: number
+              format: float
+              description: Minimum value (inclusive by default)
+            max:
+              type: number
+              format: float
+              description: Maximum value (exclusive by default)
+            inclusive_min:
+              type: boolean
+              description: Whether minimum is inclusive
+              default: true
+            inclusive_max:
+              type: boolean
+              description: Whether maximum is inclusive
+              default: false
+          example:
+            min: 10.5
+            max: 100.0
+            inclusive_min: true
+            inclusive_max: false
+            field: "price"
+
+    DateRangeQuery:
+      allOf:
+        - type: object
+          properties:
+            start:
+              type: string
+              format: date-time
+              description: Start date/time
+            end:
+              type: string
+              format: date-time
+              description: End date/time
+            inclusive_start:
+              type: boolean
+              description: Whether start is inclusive
+              default: true
+            inclusive_end:
+              type: boolean
+              description: Whether end is inclusive
+              default: false
+          example:
+            start: "2023-01-01T00:00:00Z"
+            end: "2023-12-31T23:59:59Z"
+            inclusive_start: true
+            inclusive_end: true
+            field: "created_at"
+
+    TermRangeQuery:
+      allOf:
+        - type: object
+          properties:
+            min:
+              type: string
+              description: Minimum term (lexicographic)
+            max:
+              type: string
+              description: Maximum term (lexicographic)
+            inclusive_min:
+              type: boolean
+              description: Whether minimum is inclusive
+              default: true
+            inclusive_max:
+              type: boolean
+              description: Whether maximum is inclusive
+              default: false
+          example:
+            min: "a"
+            max: "m"
+            inclusive_min: true
+            inclusive_max: false
+            field: "name"
+
+    # Boolean Queries
+    BooleanQuery:
+      allOf:
+        - type: object
+          properties:
+            must:
+              oneOf:
+                - type: array
+                  items:
+                    $ref: '#/components/schemas/Query'
+                - $ref: '#/components/schemas/Query'
+              description: Queries that must match (can be single query or array)
+            should:
+              oneOf:
+                - type: array
+                  items:
+                    $ref: '#/components/schemas/Query'
+                - $ref: '#/components/schemas/Query'
+              description: Queries that should match (boost score, can be single query or array)
+            must_not:
+              oneOf:
+                - type: array
+                  items:
+                    $ref: '#/components/schemas/Query'
+                - $ref: '#/components/schemas/Query'
+              description: Queries that must not match (can be single query or array)
+            filter:
+              oneOf:
+                - type: array
+                  items:
+                    $ref: '#/components/schemas/Query'
+                - $ref: '#/components/schemas/Query'
+              description: Queries that filter results (no score impact, can be single query or array)
+            min_should:
+              type: integer
+              description: Minimum number of should queries that must match
+              minimum: 0
+          example:
+            must:
+              - term: "important"
+                field: "status"
+              - match: "urgent"
+                field: "title"
+            should:
+              - match: "critical"
+                field: "content"
+              - match: "priority"
+                field: "tags"
+            must_not:
+              - term: "archived"
+                field: "status"
+            filter:
+              - term: "published"
+                field: "status"
+            boost: 1.5
+
+    ConjunctionQuery:
+      allOf:
+        - type: object
+          properties:
+            conjuncts:
+              type: array
+              items:
+                $ref: '#/components/schemas/Query'
+              description: Array of queries that must all match
+              minItems: 1
+          required:
+            - conjuncts
+          example:
+            conjuncts:
+              - term: "search"
+                field: "title"
+              - match: "full text"
+                field: "content"
+
+    DisjunctionQuery:
+      allOf:
+        - type: object
+          properties:
+            disjuncts:
+              type: array
+              items:
+                $ref: '#/components/schemas/Query'
+              description: Array of queries where at least one must match
+              minItems: 1
+            min:
+              type: number
+              format: float
+              description: Minimum number of disjuncts that must match
+              default: 1
+              minimum: 0
+          required:
+            - disjuncts
+          example:
+            disjuncts:
+              - term: "urgent"
+                field: "title"
+              - term: "critical"
+                field: "content"
+            min: 1
+
+    # Special Queries
+    MatchAllQuery:
+      allOf:
+        - type: object
+          properties:
+            match_all:
+              type: object
+              description: Empty object indicating match all
+          example:
+            match_all: {}
+            boost: 1.0
+
+    MatchNoneQuery:
+      allOf:
+        - type: object
+          properties:
+            match_none:
+              type: object
+              description: Empty object indicating match none
+          example:
+            match_none: {}
+
+    BoolFieldQuery:
+      allOf:
+        - type: object
+          properties:
+            bool:
+              type: boolean
+              description: Boolean value to match
+          required:
+            - bool
+          example:
+            bool: true
+            field: "published"
+
+    DocIDQuery:
+      allOf:
+        - type: object
+          properties:
+            ids:
+              type: array
+              items:
+                type: string
+              description: Array of document IDs to match
+              minItems: 1
+          required:
+            - ids
+          example:
+            ids: ["doc1", "doc2", "doc3"]
+
+    # Geographic Queries
+    GeoDistanceQuery:
+      allOf:
+        - type: object
+          properties:
+            location:
+              oneOf:
+                - type: array
+                  items:
+                    type: number
+                    format: float
+                  minItems: 2
+                  maxItems: 2
+                  description: [longitude, latitude]
+                - type: object
+                  properties:
+                    lon:
+                      type: number
+                      format: float
+                    lat:
+                      type: number
+                      format: float
+            distance:
+              type: string
+              description: Distance string (e.g., "10km", "5mi")
+          required:
+            - location
+            - distance
+          example:
+            location: [-122.4194, 37.7749]
+            distance: "10km"
+            field: "location"
+
+    GeoBoundingBoxQuery:
+      allOf:
+        - type: object
+          properties:
+            top_left:
+              oneOf:
+                - type: array
+                  items:
+                    type: number
+                    format: float
+                  minItems: 2
+                  maxItems: 2
+                - type: object
+                  properties:
+                    lon:
+                      type: number
+                      format: float
+                    lat:
+                      type: number
+                      format: float
+            bottom_right:
+              oneOf:
+                - type: array
+                  items:
+                    type: number
+                    format: float
+                  minItems: 2
+                  maxItems: 2
+                - type: object
+                  properties:
+                    lon:
+                      type: number
+                      format: float
+                    lat:
+                      type: number
+                      format: float
+          required:
+            - top_left
+            - bottom_right
+          example:
+            top_left: [-122.5, 37.8]
+            bottom_right: [-122.3, 37.7]
+            field: "location"
+
+    GeoBoundingPolygonQuery:
+      allOf:
+        - type: object
+          properties:
+            polygon_points:
+              type: array
+              items:
+                oneOf:
+                  - type: array
+                    items:
+                      type: number
+                      format: float
+                    minItems: 2
+                    maxItems: 2
+                  - type: object
+                    properties:
+                      lon:
+                        type: number
+                        format: float
+                      lat:
+                        type: number
+                        format: float
+              description: Array of polygon points [lon, lat]
+              minItems: 3
+          required:
+            - polygon_points
+          example:
+            polygon_points:
+              - [-122.5, 37.8]
+              - [-122.3, 37.8]
+              - [-122.3, 37.7]
+              - [-122.5, 37.7]
+            field: "location"
+
+    GeoShapeQuery:
+      allOf:
+        - type: object
+          properties:
+            geometry:
+              type: object
+              description: GeoJSON geometry object
+              properties:
+                type:
+                  type: string
+                  enum: [Point, LineString, Polygon, MultiPoint, MultiLineString, MultiPolygon, GeometryCollection]
+                coordinates:
+                  description: Geometry coordinates (format depends on geometry type)
+                  oneOf:
+                    - type: array
+                      description: Coordinates for Point, LineString, MultiPoint, etc.
+                    - type: array
+                      items:
+                        type: array
+                      description: Nested coordinates for Polygon, MultiLineString, etc.
+                    - type: array
+                      items:
+                        type: array
+                        items:
+                          type: array
+                      description: Deeply nested coordinates for MultiPolygon, etc.
+              required:
+                - type
+                - coordinates
+          required:
+            - geometry
+          example:
+            geometry:
+              type: "Polygon"
+              coordinates:
+                - [[-122.5, 37.8], [-122.3, 37.8], [-122.3, 37.7], [-122.5, 37.7], [-122.5, 37.8]]
+            field: "location"
+
+    # Vector Queries
+    KNNQuery:
+      allOf:
+        - type: object
+          properties:
+            field:
+              type: string
+              description: Vector field name
+            vector:
+              type: array
+              items:
+                type: number
+                format: float
+              description: Query vector
+              minItems: 1
+            k:
+              type: integer
+              format: int64
+              description: Number of nearest neighbors to return
+              minimum: 1
+              maximum: 10000
+            params:
+              type: object
+              description: Additional parameters for vector search
+          required:
+            - field
+            - vector
+            - k
+          example:
+            field: "embedding"
+            vector: [0.1, 0.2, 0.3, 0.4, 0.5]
+            k: 10
+            boost: 1.0
+
+    # Network Queries
+    IPRangeQuery:
+      allOf:
+        - type: object
+          properties:
+            cidr:
+              type: string
+              description: CIDR notation for IP range (e.g., "192.168.1.0/24")
+          required:
+            - cidr
+          example:
+            cidr: "192.168.1.0/24"
+            field: "ip_address"
+
+    # Phrase Query (internal use)
+    PhraseQuery:
+      allOf:
+        - type: object
+          properties:
+            terms:
+              type: array
+              items:
+                type: string
+              description: Array of terms in phrase order
+          required:
+            - terms
+          example:
+            terms: ["exact", "phrase", "match"]
+            field: "content"
+
+    MultiPhraseQuery:
+      allOf:
+        - type: object
+          properties:
+            terms:
+              type: array
+              items:
+                type: array
+                items:
+                  type: string
+              description: Array of term arrays for multi-phrase matching
+          required:
+            - terms
+          example:
+            terms:
+              - ["exact", "precise"]
+              - ["phrase"]
+              - ["match", "search"]
+
+  examples:
+    # Basic Query Examples
+    simple_term:
+      summary: Simple Term Query
+      value:
+        term: "search"
+        field: "title"
+        boost: 2.0
+
+    simple_match:
+      summary: Simple Match Query
+      value:
+        match: "full text search"
+        field: "content"
+        operator: "and"
+
+    fuzzy_search:
+      summary: Fuzzy Search
+      value:
+        term: "search"
+        field: "title"
+        fuzziness: "auto"
+        prefix_length: 2
+
+    # Boolean Query Examples
+    complex_boolean:
+      summary: Complex Boolean Query
+      value:
+        must:
+          - term: "important"
+            field: "status"
+          - match: "urgent"
+            field: "title"
+        should:
+          - match: "critical"
+            field: "content"
+          - match: "priority"
+            field: "tags"
+        must_not:
+          - term: "archived"
+            field: "status"
+        filter:
+          - term: "published"
+            field: "status"
+        boost: 1.5
+
+    # Range Query Examples
+    price_range:
+      summary: Price Range Query
+      value:
+        min: 10.0
+        max: 100.0
+        inclusive_min: true
+        inclusive_max: false
+        field: "price"
+
+    date_range:
+      summary: Date Range Query
+      value:
+        start: "2023-01-01T00:00:00Z"
+        end: "2023-12-31T23:59:59Z"
+        inclusive_start: true
+        inclusive_end: true
+        field: "created_at"
+
+    # Geographic Query Examples
+    geo_distance:
+      summary: Geographic Distance Query
+      value:
+        location: [-122.4194, 37.7749]
+        distance: "10km"
+        field: "location"
+
+    geo_bounding_box:
+      summary: Geographic Bounding Box Query
+      value:
+        top_left: [-122.5, 37.8]
+        bottom_right: [-122.3, 37.7]
+        field: "location"
+
+    # Vector Query Examples
+    knn_search:
+      summary: K-Nearest Neighbors Query
+      value:
+        field: "embedding"
+        vector: [0.1, 0.2, 0.3, 0.4, 0.5]
+        k: 10
+        boost: 1.0
+
+    # Query String Examples
+    query_string:
+      summary: Query String Query
+      value:
+        query: "title:search AND (content:full text OR content:search) AND -status:archived"
+
+  responses:
+    ValidationSuccess:
+      description: Query validation successful
+      content:
+        application/json:
+          schema:
+            type: object
+            properties:
+              valid:
+                type: boolean
+                example: true
+              query_type:
+                type: string
+                example: "term"
+              message:
+                type: string
+                example: "Query is valid"
+
+    ValidationError:
+      description: Query validation failed
+      content:
+        application/json:
+          schema:
+            type: object
+            properties:
+              valid:
+                type: boolean
+                example: false
+              error:
+                type: string
+                example: "unknown query type"
+              details:
+                type: string
+                example: "No recognized query fields found"
+
+externalDocs:
+  description: Bleve Documentation
+  url: https://github.com/blevesearch/bleve

docs/score_fusion.md

@@ -0,0 +1,250 @@
+# Score Fusion for Hybrid Search
+
+Bleve supports **hybrid search** that combines full-text search (FTS) with vector (kNN) search to leverage the strengths of both approaches:
+
+* **Full-text search** excels at exact keyword matching, filtering, and structured queries
+* **Vector search** captures semantic similarity and handles synonyms and paraphrasing naturally
+
+With *v2.5.4* onwards - when using hybrid search, you can choose different **score fusion strategies** to combine results from both search methods. This document describes the available fusion strategies and how to use them.
+
+## Fusion Strategies
+
+### Additive Score Fusion (Default)
+
+By default, Bleve combines FTS and kNN scores using a simple weighted addition. See the [Vector Search documentation](vectors.md#querying) for details on the default hybrid search behavior and examples.
+
+While this approach works well with proper boost tuning, it can be sensitive to different score scales and distributions. The fusion strategies below (RRF and RSF) provide more robust alternatives that handle score normalization automatically.
+
+### Reciprocal Rank Fusion (RRF)
+
+Reciprocal Rank Fusion is a **rank-based** algorithm that combines results based on their position in each result list, rather than their raw scores. This makes it robust to different score scales and distributions.
+
+**Algorithm:**
+
+For each document appearing in FTS or kNN results, the RRF score is calculated as:
+
+```math
+RRF\_score = w_{\text{fts}} \cdot \frac{1}{k + \text{rank}_{\text{fts}}} + \sum_{i=1}^{n} w_{\text{knn}_i} \cdot \frac{1}{k + \text{rank}_{\text{knn}_i}}
+```
+
+Where:
+
+* $\text{rank}_{\text{fts}}$: 1-indexed rank of the document in the FTS result list (or 0 if not present)
+* $\text{rank}_{\text{knn}_i}$: 1-indexed rank of the document in the i-th kNN result list (or 0 if not present)
+* $k$: rank constant (default: 60) that dampens the impact of rank differences
+* $w_{\text{fts}}$: weight from the FTS query boost value
+* $w_{\text{knn}_i}$: weight from the i-th kNN query boost value
+* $\sum_{i=1}^{n}$: summation over all kNN queries (you can add multiple kNN queries)
+
+**Advantages:**
+
+* Distribution-agnostic - no need for score normalization
+* Works out of the box with minimal tuning
+* Prioritizes documents appearing in both result lists
+* Robust to outliers since only ranks matter
+
+**Disadvantages:**
+
+* Ignores score magnitude (loses some information)
+* May be sensitive to imbalanced result list sizes
+
+**Usage:**
+
+```go
+// Create a hybrid search with RRF fusion
+searchRequest := bleve.NewSearchRequest(bleve.NewMatchQuery("dark chocolate"))
+searchRequest.Score = bleve.ScoreRRF  // Alternatively, set to "rrf"
+
+// Add first kNN component
+searchRequest.AddKNN(
+    "embedding",                             // Vector field
+    []float32{0.1, 0.2, 0.3, 0.4},          // Query vector
+    30,                                      // k neighbors
+    1.0,                                     // kNN weight (boost)
+)
+
+// Add second kNN component (optional - you can add multiple)
+searchRequest.AddKNN(
+    "image_embedding",                       // Different vector field
+    []float32{0.5, 0.3, 0.1, 0.8},          // Query vector
+    20,                                      // k neighbors
+    0.5,                                     // kNN weight (boost)
+)
+
+// Optional: Configure RRF parameters
+params := bleve.RequestParams{
+    ScoreRankConstant: 60,                   // Rank constant (default: 60)
+    ScoreWindowSize: 150                     // Window size (default: size)
+}
+searchRequest.AddParams(params)
+
+searchResult, err := index.Search(searchRequest)
+```
+
+### Relative Score Fusion (RSF)
+
+Relative Score Fusion is a **score-based** strategy that normalizes scores from both modalities into a common [0, 1] range using min-max normalization before combining them.
+
+**Algorithm:**
+
+1. **Min-max normalize** each result set independently:
+
+    ```math
+    \text{normalized\_score} = \frac{\text{score} - \text{min\_score}}{\text{max\_score} - \text{min\_score}}
+    ```
+
+2. **Combine** normalized scores using weighted addition:
+
+    ```math
+    RSF\_score = w_{\text{fts}} \cdot \text{normalized\_score\_fts} + \sum_{i=1}^{n} w_{\text{knn}_i} \cdot \text{normalized\_score\_knn}_i
+    ```
+
+Where:
+
+* $w_{\text{fts}}$: weight from the FTS query boost value
+* $w_{\text{knn}_i}$: weight from the i-th kNN query boost value
+* $\sum_{i=1}^{n}$: summation over all kNN queries (you can add multiple kNN queries)
+
+**Advantages:**
+
+* Score-aware - retains relevance magnitude information
+* Resolves incompatible score ranges
+* Easy to understand
+
+**Disadvantages:**
+
+* Sensitive to outliers - a single extreme score can skew normalization
+* Doesn't account for the shape or distribution of scores
+
+**Usage:**
+
+```go
+// Create a hybrid search with RSF fusion
+searchRequest := bleve.NewSearchRequest(bleve.NewMatchQuery("machine learning"))
+searchRequest.Score = bleve.ScoreRSF  // Or set to "rsf"
+
+// Add first kNN component
+searchRequest.AddKNN(
+    "content_vector",                        // Vector field
+    []float32{0.5, 0.3, 0.1, 0.8},          // Query vector
+    20,                                      // k neighbors
+    1.0,                                     // kNN weight (boost)
+)
+
+// Add second kNN component (optional - you can add multiple)
+searchRequest.AddKNN(
+    "title_vector",                          // Different vector field
+    []float32{0.2, 0.7, 0.4, 0.1},          // Query vector
+    15,                                      // k neighbors
+    0.8,                                     // kNN weight (boost)
+)
+
+// Optional: Configure RSF parameters
+params := bleve.RequestParams{
+    ScoreWindowSize: 150                     // Window size (default: size)
+}
+searchRequest.AddParams(params)
+
+searchResult, err := index.Search(searchRequest)
+```
+
+## Parameters
+
+### Score
+
+The `Score` field in your search request specifies which fusion strategy to use:
+
+* **`ScoreRRF ("rrf")`**: Reciprocal Rank Fusion
+* **`ScoreRSF ("rsf")`**: Relative Score Fusion  
+* **Omitted or empty**: Default additive fusion with scores returned
+
+### Params
+
+The `Params` object contains additional parameters for score fusion:
+
+#### Score Window Size
+
+`ScoreWindowSize` is the maximum number of results to consider from each result list for fusion.
+
+* **Default**: Same as `Size` parameter
+* **Minimum**: Must be ≥ `Size` and ≥ 1
+* **Purpose**: Controls the tradeoff between relevance and performance
+
+A larger window size increases the chance of finding relevant results but requires more computation. For pagination to work consistently, ensure:
+
+```text
+From + Size <= ScoreWindowSize
+```
+
+**Example:**
+
+```json
+{
+  "score": "rrf",
+  "params": {
+    "score_window_size": 150
+  },
+  "size": 10,
+  "from": 0
+}
+```
+
+With window size set to 150, you can paginate through up to 150 results. If you try to access results beyond this (e.g., `from=160`), you'll get an empty result set.
+
+#### Score Rank Constant
+
+> *Only applicable for RRF*
+
+`ScoreRankConstant` controls how much the rank position affects the reciprocal rank score.
+
+* **Default**: 60
+* **Range**: Any positive integer
+* **Effect**: Higher values dampen the impact of rank differences
+
+**Example:**
+
+```json
+{
+  "score": "rrf",
+  "params": {
+    "score_rank_constant": 60
+  }
+}
+```
+
+## Weighting Queries
+
+The boost value in your query components controls their relative importance in hybrid search:
+
+```go
+// FTS query with boost 2.0
+query := bleve.NewMatchQuery("search term")
+query.SetBoost(2.0)
+
+searchRequest := bleve.NewSearchRequest(query)
+
+// kNN query with boost 1.0
+searchRequest.AddKNN("vec", queryVector, 10, 1.0)
+```
+
+For RRF and RSF, weights determine the **relative importance** of each component's contribution, rather than scaling raw scores.
+
+**Example:** If `fts_boost = 2.0` and `knn_boost = 1.0`, the FTS contribution is twice as important as the kNN contribution in the final ranking in RRF or RSF.
+
+## Restrictions
+
+When using score fusion (`Score` set to `"rrf"` or `"rsf"`), certain features are not supported:
+
+* **SearchAfter/SearchBefore**: Not compatible with score fusion. For pagination, use `From` and `Size` only.
+* **Sort**: Only descending score sort (`-_score`) or default sorting is allowed
+* **Faceting**: Only documents included in the FTS result list are considered. Documents that appear exclusively in the kNN result list are ignored during faceting.
+
+## Choosing a Fusion Strategy
+
+| Use Case | Recommended Strategy |
+| -------- | -------------------- |
+| Different score scales (e.g., TF-IDF + L2 distance) | **RRF/RSF** |
+| Minimal tuning, out-of-the-box performance | **RRF** |
+| Want to preserve score magnitude importance | **RSF** |
+| Have well-tuned boost values already | **Additive (default)** |
+| Score distributions have extreme outliers | **RRF** |

docs/scoring.md

@@ -0,0 +1,90 @@
+# Scoring models for document hits
+
+* Search is performed on a collection of fields using compound queries such as conjunction/disjunction/boolean, etc. However, the scoring itself is done independently for each field and then aggregated to get the final score for a document hit.
+* Default scoring scheme for document hits involving text hits: `tf-idf`.
+* Nearest-neighbor/vector hits scoring depends on chosen `knn distance` metric, highlighted [here](https://github.com/blevesearch/bleve/blob/master/docs/vectors.md#supported).
+* Hybrid search scoring will combine `tf-idf` scores with `knn distance` numbers.
+* *v2.5.0* (and after) will come with support for `bm25` scoring for exact searches.
+
+## BM25
+
+When it comes to scoring a document hit for a specific field, BM25 scoring mechanism requires the following stats:
+
+* fieldLen - The number of analyzed terms in the current document's field.
+* avgFieldLen - The average number of analyzed terms in the field across all the documents.
+* docTotal - The total number of documents in the index.
+* docTerm - The total number of documents containing the query term within the index.
+
+The scoring formula followed in BM25 is
+
+```math
+\sum_{i}^n IDF(q_i) {{f(q_i,D) * (k1 + 1)}\over{f(q_i,D) + k1 * (1-b+b*{{fieldLen}\over{avgFieldLen}})}}
+```
+
+$IDF(q_i)$ here refers to Inverse Document Frequency, which talks about how rare (and hence rich in information) is a particular query term $`q_i`$ across all the documents in the index, which is calculated as
+
+```math
+    \ln(1 + {{docTotal - docTerm + 0.5}\over{docTerm + 0.5}})
+```
+
+Coming back to the BM25 scoring, $f(q_i,D)$ refers to the frequency of the query term in document $D$. The entire equation has certain multipliers
+
+* $k1$ - helps in controlling the saturation of the score with respect to query term in a document. Basically if the query term's frequency is too high, the score value gets saturated and doesn't increase beyond a certain point.
+* $b$ - controls the extent to which the $fieldLen$ normalizes the term's frequency.
+
+### How to enable and use BM25
+
+Bleve v2.5.0 updated the `indexMapping` construct with the concept of `scoringModel`. This is a global (meaning applicable to all fields) setting which drives which scoring algorithm to apply while scoring the document hits. Supported scoring models are defined [here](https://github.com/blevesearch/bleve_index_api/blob/f54d76f0a71a838837159aa44ced0404bb6ec25f/indexing_options.go#L27)
+
+For instance, while defining the index mapping for the data model that's been decided by the user, following snippet can be referred to enable BM25
+
+```go
+indexMapping := bleve.NewIndexMapping()
+indexMapping.TypeField = "type"
+indexMapping.DefaultAnalyzer = "en"
+indexMapping.ScoringModel = "bm25"
+```
+
+During search time there's no explicit change involved, unless the user wants to perform a global scoring.
+
+### Global Scoring
+
+Let's say that the user has a dataset which is quite large (let's say 3 million) and to have good throughput, they create 3 shards (with the same index mapping) for the "index". Each of these shards can be `bleve.Index` type and while performing a search over the entire dataset, a `bleve.IndexAlias` can be created over which a search can be performed. This parallelizes things pretty well, both on the indexing path and the search path.
+
+The concept of global scoring is applicable when the index is "sharded" (similar to above situation). This is because each index has data which is disjoint, and thereby while performing the scoring on document hits on each of them, the value of stats is not complete at a global level, since we're doing a search over the entire dataset using the `bleve.IndexAlias`. For eg: `docTotal` value while scoring the document hits would be 1 million which is incorrect at a global level.
+
+So in order to keep the scoring roughly same across varying count of the number of shards involved, we provide a mechanism to enable "global scoring". In this type of search, an initial roundtrip is performed to gather and aggregate the stats necessary for the scoring mechanism and in the second phase, the actual search is performed. So naturally this comes at a cost of latency. As a reference here's how the user can go about with it
+
+```go
+multiPartIndex := bleve.NewIndexAlias(shard1, shard2)
+// set the alias with the same index mapping which both the shards use.
+err = multiPartIndex.SetIndexMapping(indexMapping)
+if err != nil {
+    return err
+}
+
+ctx := context.Background()
+ctx = context.WithValue(ctx, search.SearchTypeKey, search.GlobalScoring)
+
+res, err := multiPartIndex.SearchInContext(ctx, searchRequest)
+```
+
+A note here is that, this would only matter if the relative order of the document hits vary quite a bit (vs single shard case). This would be possible when the shard count increases quite a bit, in low doc count situations or if there is a heavy skew in the data distribution amongst the shards for some reason. Ideally the shards are created when the data is quite large and each of them index the same amount of data - in which case the scores won't fluctuate much to affect the relative hit order and the user can choose to avoid the global scoring mechanism altogether.
+
+## TF-IDF
+
+TF-IDF is the default scoring mechanism involved (for backward compatibility reasons) and requires no change from the user at index or search time to avail it.
+
+The scoring formula involved is
+
+```math
+    \sum_{i}^n f(q_i, D) * {{1}\over{\sqrt{fieldLen}}} * IDF(q_i)
+```
+
+where $IDF(q_i)$ is
+
+```math
+    1 + {{docTotal}\over{1 + docTerm}}
+```
+
+Note: TF-IDF formula doesn't accommodate logic for score saturation due to term frequency or fieldLen. So, it's recommended to use BM25 scoring by explicitly setting it in the index mapping.

docs/search_autocomplete.md

@@ -0,0 +1,358 @@
+# Edge N-gram Autocomplete in Bleve
+
+Search autocomplete is a feature which we see in search boxes when suggestions appear while we type.
+So when we type `jav`, we see suggestions like: `java` `javascript` `javascript programming` etc.
+This is helpful because it saves users time in finding what they are looking for.
+
+![Alt Text](/docs/images/search_autocomplete.png "search autocomplete suggestion")
+
+## 2. How Does It Work?
+
+Autocomplete generally works in three steps:
+
+1. **Index Time**: Breaking text into searchable pieces (tokens)
+2. **Query Time**: Matching user input against indexed tokens  
+3. **Results**: Returning relevant suggestions quickly
+
+But before we jump into the flow, let's understand different methods to achieve this and why edge n-grams are the most efficient approach.
+
+## 3. Different Tokenization Methods
+
+There are several tokenization approaches, each with its own strengths and weaknesses:
+
+### 3.1 Single Token Tokenizer
+
+```go
+// analysis/tokenizer/single/single.go
+func (t *SingleTokenTokenizer) Tokenize(input []byte) analysis.TokenStream {
+    return analysis.TokenStream{
+        &analysis.Token{
+            Term:     input,    // Here entire input as one token
+            Position: 1,
+            Start:    0,
+            End:      len(input),
+            Type:     analysis.AlphaNumeric,
+        },
+    }
+}
+```
+
+**How it works**: Treats the entire input as a single token.
+
+**Example**: "JavaScript Programming" → [`"JavaScript Programming"`]
+
+**Pros**:
+
+- Simple and fast
+- Perfect for exact phrase matching
+- Minimal index size
+
+**Cons**:
+
+- No autocomplete support (can't match partial text)
+- Not flexible for search
+
+**Use case**: Keyword fields, IDs, exact phrase matching
+
+### 3.2 Whitespace Token Tokenizer
+
+```go
+// analysis/tokenizer/whitespace/whitespace.go  
+func TokenizerConstructor(config map[string]interface{}, cache *registry.Cache) (analysis.Tokenizer, error) {
+    return character.NewCharacterTokenizer(notSpace), nil
+}
+
+func notSpace(r rune) bool {
+    return !unicode.IsSpace(r)  // Split on whitespace
+}
+```
+
+**How it works**: Splits text on whitespace characters.
+
+**Example**: "JavaScript Programming" → [`"JavaScript"`, `"Programming"`]
+
+**Pros**:
+
+- Simple word-based tokenization
+- Works well for basic prefix search
+
+**Cons**:
+
+- Only matches from word beginnings
+- No support for partial word matching
+- Limited autocomplete capabilities
+
+**Use case**: Basic search, word-level indexing
+
+### 3.3 N-gram Method
+
+```go
+// analysis/token/ngram/ngram.go
+func (s *NgramFilter) Filter(input analysis.TokenStream) analysis.TokenStream {
+    rv := make(analysis.TokenStream, 0, len(input))
+    
+    for _, token := range input {
+        runeCount := utf8.RuneCount(token.Term)
+        runes := bytes.Runes(token.Term)
+        
+        // ..generate all possible n-grams
+    }
+    return rv
+}
+```
+
+**How it works**: Creates ALL possible substrings of specified lengths.
+
+**Example**: "java" with 2-3 grams → [`"ja"`, `"av"`, `"va"`, `"jav"`, `"ava"`]
+
+**Pros**:
+
+- Supports partial matching anywhere in the text
+- Coverage of all possible substrings
+
+**Cons**:
+
+- **MASSIVE index size** (exponential growth)
+- Brings more noise as irrelevant matches (e.g., "av" matching "java")
+- Poor performance for autocomplete
+- High memory usage
+
+**Use case**: Full-text substring search (not ideal for autocomplete)
+
+### 3.4 Edge N-gram Method (preferred for autocomplete)
+
+```go
+// analysis/token/edgengram/edgengram.go
+func (s *EdgeNgramFilter) Filter(input analysis.TokenStream) analysis.TokenStream {
+    rv := make(analysis.TokenStream, 0, len(input))
+    
+    for _, token := range input {
+        runeCount := utf8.RuneCount(token.Term)
+        runes := bytes.Runes(token.Term)
+        // ..builds tokens based on either end, specified in the input
+    }
+    return rv
+}
+```
+
+**How it works**: Creates substrings only from the beginning (or end) of each word.
+
+**Example**: "javascript" with front edge n-grams (1-5) → [`"j"`, `"ja"`, `"jav"`, `"java"`, `"javas"`]
+
+**Pros**:
+
+- Perfect for autocomplete (matches prefixes naturally)
+- Efficient index size (linear growth vs exponential)  
+- Fast queries (direct term matching, no complex processing)
+- Intuitive results (matches what users expect)
+- Highly scalable for large datasets
+
+**Cons**:
+
+- Only supports prefix matching (but that's preferred for autocomplete!)
+- Slightly larger index than basic tokenization
+
+**Use case**: Search autocomplete, prefix-based search
+
+## 4. Why Edge N-grams Are Most Efficient for Autocomplete
+
+Let's see what happens when a user types "java" with edge_ngram tokenizer:
+
+```text
+Index contains: ["j", "ja", "jav", "java", "javas", "javasc", "javascr", ...]
+User types: "java"  
+Query: ExactTermQuery("java")
+Process: Direct hash table lookup for term "java"
+Result: Instant match, then retrieve documents containing this term
+```
+
+**Key advantages of Edge N-gram approach:**
+
+1. **O(1) lookup**: Direct term matching
+2. **No query-time processing**: Terms are pre-computed at index time  
+3. **Better caching**: Exact term queries cache better than prefix queries
+4. **Consistent performance**: Query time doesn't increase with index size
+
+## 5. Step-by-Step Implementation: Building Search Autocomplete
+
+Now let's see how to implement this in practice using our exact configuration. We'll build a complete autocomplete system step by step.
+
+### Step 1: Create Custom Edge N-gram Token Filter
+
+First, we need to create a custom token filter for edge n-grams. Here's how it looks in our configuration:
+
+```go
+// Create a new index mapping
+indexMapping := mapping.NewIndexMapping()
+
+// 1. Define the edgeGram token filter
+edgeGramFilter := map[string]interface{}{
+    "type": edgengram.Name,
+    "min":  2.0,
+    "max":  4.0,
+    "back": false,
+}
+
+// Register the token filter
+if err := indexMapping.AddCustomTokenFilter("Engram", edgeGramFilter); err != nil {
+    log.Fatal(err)
+}
+```
+
+**What each setting does:**
+
+- `"type": "edge_ngram"` - Tells Bleve to use the edge n-gram filter
+- `"min": 2` - Start creating tokens from 2 characters ("ja", "sc", etc.)
+- `"max": 4` - Stop at 4 characters ("java", "scri", etc.)
+- `"back": false` - Create tokens from the front (beginning) of words
+
+### Step 2: Create Custom Analyzer
+
+Next, we create an analyzer that uses our custom token filter along with other helpful filters:
+
+```go
+// 2. Define a custom analyzer that uses it
+customAnalyzer := map[string]interface{}{
+    "type":      custom.Name,
+    "tokenizer": "unicode",
+    "char_filters": []string{
+         zerowidthnonjoiner.Name,
+    },
+    "token_filters": []string{
+        "Engram", // our custom edge_ngram filter
+        "to_lower",
+        "stop_en",
+    },
+}
+
+if err := indexMapping.AddCustomAnalyzer("edgeGramAnalyzer", customAnalyzer); err != nil {
+    log.Fatal(err)
+}
+```
+
+**The pipeline works like this:**
+
+1. **Input Text**: "Schaumbergfest Event"
+
+2. **Tokenizer** (`unicode`): Splits into words
+
+   ```text
+   ["Schaumbergfest", "Event"]
+   ```
+
+3. **Character Filter** (`zero_width_spaces`): Removes invisible characters
+
+   ```text
+   ["Schaumbergfest", "Event"] (cleaned)
+   ```
+
+4. **Token Filter 1** (`Engram`): Creates edge n-grams (2-4 chars)
+
+   ```text
+   ["Sc", "Sch", "Scha", "Ev", "Eve", "Even"]
+   ```
+
+5. **Token Filter 2** (`to_lower`): Makes everything lowercase
+
+   ```text
+   ["sc", "sch", "scha", "ev", "eve", "even"]
+   ```
+
+6. **Token Filter 3** (`stop_en`): Removes common words (none in this case)
+
+   ```text
+   ["sc", "sch", "scha", "ev", "eve", "even"] (final tokens)
+   ```
+
+### Step 3: Configure Field Mapping
+
+Now we tell Bleve which fields to apply our autocomplete analyzer to:
+
+```go
+    // 3. Assign analyzer to a field mapping
+    fieldMapping := mapping.NewTextFieldMapping()
+    fieldMapping.Analyzer = "edgeGramAnalyzer"
+    
+    indexMapping.DefaultMapping.AddFieldMappingsAt("title", fieldMapping)
+    
+    indexPath := "example.bleve"
+    index, err := bleve.New(indexPath, indexMapping)
+    if err != nil {
+        log.Fatal(err)
+    }
+```
+
+### Step 4: How It Works in Real Search
+
+When someone searches for "sc", here's what happens:
+
+**Index contains these tokens:**
+
+```text
+"sc" → [document1: "Schaumbergfest", document2: "Script", ...]
+"sch" → [document1: "Schaumbergfest", ...]  
+"scha" → [document1: "Schaumbergfest", ...]
+```
+
+**User types "sc":**
+
+1. Query: `name:sc`
+2. Bleve looks up exact term "sc" in the index
+3. Finds document with "Schaumbergfest" and "Script"
+4. Returns suggestion instantly
+
+```go
+
+type Document struct {
+  ID    string `json:"id"`
+  Title string `json:"title"`
+}
+// 4. Index Documents
+documents := []Document{
+  {
+    ID:    "doc1",
+    Title: "Schaumbergfest",
+  },
+  {
+    ID:    "doc2",
+    Title: "Script",
+  },
+}
+
+batch := index.NewBatch()
+for _, doc := range documents {
+  batch.Index(doc.ID, doc)
+}
+if err := index.Batch(batch); err != nil {
+  log.Fatal(err)
+}
+
+// 5. Search the created index
+query := bleve.NewMatchQuery("sc")
+query.SetField("title")
+searchRequest := bleve.NewSearchRequest(query)
+searchRequest.Explain = true
+searchRequest.Fields = []string{"title"}
+searchResult, err := index.Search(searchRequest)
+if err != nil {
+  log.Fatal(err)
+}
+fmt.Println(searchResult)
+```
+
+Output:
+
+```bash
+
+$ go run main.go
+
+2 matches, showing 1 through 2, took 189.041µs
+    1. doc2 (0.343255)
+        title
+                Script
+    2. doc1 (0.343255)
+        title
+                Schaumbergfest
+```
+
+Note: To run code, enclose code starting from Step 1 in func main.

docs/sort_facet.md

@@ -0,0 +1,785 @@
+<h2>Purpose of Docvalues</h2>
+
+<h3>Background</h3>
+
+<p align="justify">What are docValues? In the index mapping, there is an option to enable or disable docValues for a specific field mapping. However, what does it actually mean to activate or deactivate docValues, and how does it impact the end user? This document aims to address these questions.</p>
+<pre>
+	"default_mapping": {
+		"dynamic": true,
+		"enabled": true,
+		"properties": {
+			"loremIpsum": {
+			"enabled": true,
+			"dynamic": false,
+			"fields": [
+				{
+					"name": "loremIpsum",
+					"type": "text",
+					"store": false,
+					"index": true,
+					"include_term_vectors": false,
+					"include_in_all": false,
+					"docvalues": true
+				}
+			]
+		}
+	}
+</pre>
+<p align="justify">Enabling docValues will always result in an increase in the size of your Bleve index, leading to a corresponding increase in disk usage. But what advantages can you expect in return? This document also quantitatively assesses this trade-off with a test case.</p>
+
+<p align="justify">In a more general sense, we recommend enabling docValues on a field mapping if you anticipate queries that involve sorting and/or facet operations on that field. It's important to note, though, that sorting and faceting will work irrespective of whether docValues are enabled or not. This may lead you to wonder if there's any real benefit to enabling docValues since you're allocating extra disk space without an apparent return. The real advantage, however, becomes evident in enhanced query response times and reduced memory consumption during active usage. By accepting a minor increase in the disk space used by your Full-Text Search (FTS) index, you can anticipate better performance in handling search requests that involve sorting and faceting.</p>
+
+<h3>Usage</h3>
+
+<p align="justify">The initial use of docValues comes into play when sorting is involved. In the search request JSON, there is a field named "sort." This optional "sort" field can have a slice of JSON objects as its value. Each JSON object must belong to one of the following types: 
+	<ul>
+		<li>SortDocID</li>
+		<li>SortScore (which is the default if none is specified)</li>
+		<li>SortGeoDistance</li> 
+		<li>SortField</li>
+	</ul>
+</p>
+<p align="justify">DocValues are relevant only when any of the JSON objects in the "sort" field are of type SortGeoDistance or SortField. This means that if you expect queries on a field F, where the queries either do not specify a value for the "sort" field or provide a JSON object of type SortDocID or SortScore, enabling docValues will not improve sorting operations, and as a result, query latency will remain unchanged. It's worth noting that the default sorting object, SortScore, does not require docValues to be enabled for any of the field mappings. Therefore, a search request without a sorting operation will not utilize docValues at all.</p>
+<div style="overflow-x: auto;">
+<table>
+	<tr>
+		<th>No Sort Objects</th>
+		<th>SortDocID</th>
+		<th>SortScore</th>
+		<th>SortField</th>
+		<th>SortGeoDistance</th>
+	</tr>
+	<tr>
+		<td style="vertical-align: top; width: 20%;">
+			<pre>
+{
+  "explain": true,
+  "fields": [
+    "*"
+  ],
+  "highlight": {},
+  "query": {
+    "match": "lorem ipsum",
+    "field":"dolor"
+  },
+  "size": 10,
+  "from": 0
+}
+			</pre>
+		</td>
+		<td style="vertical-align: top; width: 20%;">
+			<pre>
+{
+  "explain": true,
+  "fields": [
+    "*"
+  ],
+  "highlight": {},
+  "query": {
+    "match": "lorem ipsum",
+    "field":"sit_amet"
+  },
+  "sort":[
+    {
+     "by":"id",
+     "desc":true
+    }
+    ],
+  "size": 10,
+  "from": 0
+}
+			</pre>
+		</td>
+		<td style="vertical-align: top; width: 20%;">
+			<pre>
+{
+  "explain": true,
+  "fields": [
+    "*"
+  ],
+  "highlight": {},
+  "query": {
+    "match": "lorem ipsum",
+    "field":"sit_amet"
+  },
+  "sort":[
+    {
+     "by":"score",
+    }
+    ],
+  "size": 10,
+  "from": 0
+}
+			</pre>
+		</td>
+		<td style="vertical-align: top; width: 20%;">
+			<pre>
+{
+  "explain": true,
+  "fields": [
+    "*"
+  ],
+  "highlight": {},
+  "query": {
+    "match": "lorem ipsum",
+    "field":"sit_amet"
+  },
+  "sort":[
+    {
+     "by":"field",
+     "field":"dolor",
+     "type":"auto",
+     "mode":"min",
+     "missing":"last"
+    }
+    ],
+  "size": 10,
+  "from": 0
+}
+			</pre>
+		</td>
+		<td style="vertical-align: top; width: 20%;">
+			<pre>
+{
+  "explain": true,
+  "fields": [
+    "*"
+  ],
+  "highlight": {},
+  "query": {
+    "match": "lorem ipsum",
+    "field": "dolor"
+  },
+  "sort": [
+    {
+      "by": "geo_distance",
+      "field": "sit_amet",
+      "location": [
+        123.223,
+        34.33
+      ],
+      "unit": "km"
+    }
+  ],
+  "size": 10,
+  "from": 0
+}
+			</pre>
+		</td>
+	</tr>
+	<tr align="center">
+		<td>No DocValues used</td>
+		<td>No DocValues used</td>
+		<td>No DocValues used</td>
+		<td>DocValues used for field "dolor". Field Mapping for "dolor" may enable docValues.</td>
+		<td colspan="3">DocValues used for field "dolor". Field Mapping for "dolor" may enable docValues.</td>	
+	</tr>
+</table>
+</div>
+<p align="justify">Now, let's consider faceting. The search request object also includes another field called "facets," where you can specify a collection of facet requests, with each request being associated with a unique name. Each of these facet requests can fall into one of three types:
+<ul>
+	<li>Date range</li> 
+	<li>Numeric range</li>
+	<li>Term facet</li> 
+</ul>
+Enabling docValues for the fields associated with such facet requests might provide benefits in this context.</p>
+<div style="overflow-x: auto;">
+<table>
+	<tr>
+		<th>No Facet Request</th>
+		<th>Date Range Facet</th>
+		<th>Numeric Range Facet</th>
+		<th>Term Facet</th>
+	</tr>
+	<tr>
+		<td style="vertical-align: top; width: 20%;">
+			<pre>
+{
+  "explain": true,
+  "fields": [
+    "*"
+  ],
+  "highlight": {},
+  "query": {
+    "match": "lorem ipsum",
+    "field": "dolor"
+  },
+  "size": 10,
+  "from": 0
+}
+			</pre>
+		</td>
+		<td style="vertical-align: top; width: 20%;">
+			<pre>
+{
+  "explain": true,
+  "fields": [
+    "*"
+  ],
+  "highlight": {},
+  "query": {
+    "match": "lorem ipsum",
+    "field": "sit_amet"
+  },
+  "facet": {
+    "facetA": {
+      "size": 1,
+      "field": "dolor",
+      "date_ranges": [
+        {
+          "name": "lorem",
+          "start": "20/August/2001",
+          "end": "22/August/2002",
+          "datetime_parser": "custDT"
+        }
+      ]
+    }
+  },
+  "size": 10,
+  "from": 0
+}
+			</pre>
+		</td>
+		<td style="vertical-align: top; width: 20%;">
+			<pre>
+{
+  "explain": true,
+  "fields": [
+    "*"
+  ],
+  "highlight": {},
+  "query": {
+    "match": "lorem ipsum",
+    "field": "sit_amet"
+  },
+  "facet": {
+    "facetA": {
+      "size": 1,
+      "field": "dolor",
+      "numeric_ranges":[
+          { 
+            "name":"lorem",
+            "min":22,
+            "max":34
+          }
+        ]
+    }
+  },
+  "size": 10,
+  "from": 0
+}
+			</pre>
+		</td>
+		<td style="vertical-align: top; width: 20%;">
+			<pre>
+{
+  "explain": true,
+  "fields": [
+    "*"
+  ],
+  "highlight": {},
+  "query": {
+    "match": "lorem ipsum",
+    "field": "sit_amet"
+  },
+  "facet": {
+    "facetA": {
+      "size": 1,
+      "field": "dolor"
+    }
+  },
+  "size": 10,
+  "from": 0
+}
+			</pre>
+		</td>
+	</tr>
+	<tr align="center">
+		<td>No DocValues used</td>
+		<td colspan="3">DocValues used for field "dolor". Field Mapping for "dolor" may enable docValues.</td>	
+  </tr>
+</table>
+</div>
+
+<p align="justify">In summary, when a search request is received by the Bleve index, it extracts all the fields from the sort objects and facet objects. To potentially benefit from docValues, you should consider enabling docValues for the fields mentioned in SortField and SortGeoDistance sort objects, as well as the fields associated with all the facet objects. By doing so, you can optimize sorting and faceting operations in your search queries.</p>
+
+<div style="overflow-x: auto;">
+<table>
+	<tr>
+		<th>Combo A</th>
+		<th>Combo B</th>
+	</tr>
+	<tr>
+		<td style="vertical-align: top; width: 20%;">
+			<pre>
+{
+  "explain": true,
+  "fields": [
+    "*"
+  ],
+  "highlight": {},
+  "query": {
+    "match": "lorem ipsum",
+    "field": "sit_amet"
+  },
+  "facet": {
+    "facetA": {
+      "size": 1,
+      "field": "dolor",
+      "date_ranges": [
+        {
+          "name": "lorem",
+          "start": "20/August/2001",
+          "end": "22/August/2002",
+          "datetime_parser": "custDT"
+        }
+      ]
+    }
+  },
+  "sort":[
+    {
+     "by":"field",
+     "field":"lorem",
+     "type":"auto",
+     "mode":"min",
+     "missing":"last"
+    }
+    ],
+  "size": 10,
+  "from": 0
+}
+			</pre>
+		</td>
+		<td style="vertical-align: top; width: 20%;">
+			<pre>
+{
+  "explain": true,
+  "fields": [
+    "*"
+  ],
+  "highlight": {},
+  "query": {
+    "match": "lorem ipsum",
+    "field": "sit_amet"
+  },
+  "facet": {
+    "facetA": {
+      "size": 1,
+      "field": "dolor",
+      "numeric_ranges":[
+          { 
+            "name":"lorem",
+            "min":22,
+            "max":34
+          }
+        ]
+    }
+  },
+  "sort": [
+    {
+      "by": "geo_distance",
+      "field": "ipsum",
+      "location": [
+        123.223,
+        34.33
+      ],
+      "unit": "km"
+    }
+  ],
+  "size": 10,
+  "from": 0
+}
+			</pre>
+		</td>
+	</tr>
+	<tr align="center">
+		<td>DocValues used for fields "dolor" and "lorem". Field Mapping for "dolor" and "lorem" may enable docValues.</td>
+		<td>DocValues used for fields "dolor" and "ipsum". Field Mapping for "dolor" and "ipsum" may enable docValues.</td>
+	</tr>
+</table>
+</div>
+
+<h3>Empirical Analysis</h3>
+
+<p align="justify">To evaluate our hypothesis, I've set up a sample dataset on my personal computer and I've created two Bleve indexes: one with docvalues enabled for three fields (<code>dummyDate</code>, <code>dummyNumber</code>, and <code>dummyTerm</code>), and another where I've disabled docValues for the same three fields. These field mappings were incorporated into the Default Mapping. It's important to mention that for both indexes, DocValues for dynamic fields were enabled, as the default mapping is dynamic.</p>
+
+<p align="justify">The values for <code>dummyDate</code> and <code>dummyNumber</code> were configured to increase monotonically, with <code>dummyDate</code> representing a date value and `dummyNumber` representing a numeric value. This setup was intentional to ensure that facet aggregation would consistently result in cache hits and misses, providing a useful testing scenario.</p>
+
+<div style="overflow-x: auto;">
+<table>
+	<tr>
+		<th>Index A</th>
+		<th>Index B</th>
+	</tr>
+	<tr>
+		<td style="vertical-align: top; width: 20%;">
+			<pre>
+   "default_mapping": {
+    "dynamic": true,
+    "enabled": true,
+    "properties": {
+     "dummyNumber": {
+      "enabled": true,
+      "dynamic": false,
+      "fields": [
+       {
+        "name": "dummyNumber",
+        "type": "text",
+        "store": false,
+        "index": true,
+        "include_term_vectors": false,
+        "include_in_all": false,
+        "docvalues": true
+       }
+      ]
+     },
+     "dummyTerm": {
+      "enabled": true,
+      "dynamic": false,
+      "fields": [
+       {
+        "name": "dummyTerm",
+        "type": "text",
+        "store": false,
+        "index": true,
+        "include_term_vectors": false,
+        "include_in_all": false,
+        "docvalues": true
+       }
+      ]
+     },
+     "dummyDate": {
+      "enabled": true,
+      "dynamic": false,
+      "fields": [
+       {
+        "name": "dummyDate",
+        "type": "text",
+        "store": false,
+        "index": true,
+        "include_term_vectors": false,
+        "include_in_all": false,
+        "docvalues": true
+       }
+      ]
+     }
+    }
+   }
+			</pre>
+		</td>
+		<td style="vertical-align: top; width: 20%;">
+			<pre>
+   "default_mapping": {
+    "dynamic": true,
+    "enabled": true,
+    "properties": {
+     "dummyNumber": {
+      "enabled": true,
+      "dynamic": false,
+      "fields": [
+       {
+        "name": "dummyNumber",
+        "type": "text",
+        "store": false,
+        "index": true,
+        "include_term_vectors": false,
+        "include_in_all": false,
+        "docvalues": false
+       }
+      ]
+     },
+     "dummyTerm": {
+      "enabled": true,
+      "dynamic": false,
+      "fields": [
+       {
+        "name": "dummyTerm",
+        "type": "text",
+        "store": false,
+        "index": true,
+        "include_term_vectors": false,
+        "include_in_all": false,
+        "docvalues": false
+       }
+      ]
+     },
+     "dummyDate": {
+      "enabled": true,
+      "dynamic": false,
+      "fields": [
+       {
+        "name": "dummyDate",
+        "type": "text",
+        "store": false,
+        "index": true,
+        "include_term_vectors": false,
+        "include_in_all": false,
+        "docvalues": false
+       }
+      ]
+     }
+    }
+   }
+			</pre>
+		</td>
+	</tr>
+	<tr align="center">
+		<td>Docvalues enabled across all three field mappings</td>
+		<td>Docvalues disabled across all three field mappings</td>
+	</tr>
+</table>
+</div>
+
+Document Format used for the test scenario:
+
+<div style="overflow-x: auto;">
+<table>
+	<tr>
+		<th>Document 1</th>
+		<th>Document 2</th>
+		<th>... Document i</th>
+		<th>Document 5000</th>
+	</tr>
+	<tr>
+		<td style="vertical-align: top; width: 20%;">
+			<pre>
+{
+	"dummyTerm":"Term",
+	"dummyDate":"2000-01-01T00:00:00",
+	"dummyNumber":1
+}
+			</pre>
+		</td>
+		<td style="vertical-align: top; width: 20%;">
+			<pre>
+{
+	"dummyTerm":"Term",
+	"dummyDate":"2000-01-01T01:00:00",
+	"dummyNumber":2
+}
+			</pre>
+		</td>
+		<td style="vertical-align: top; width: 20%;">
+			<pre>
+{
+	"dummyTerm":"Term",
+	"dummyDate":"2000-01-01T01:00:00"+(i hours),
+	"dummyNumber":i
+}
+			</pre>
+		</td>
+		<td style="vertical-align: top; width: 20%;">
+			<pre>
+{
+	"dummyTerm":"Term",
+	"dummyDate":"2000-01-01T01:00:00 + (5000 hours)",
+	"dummyNumber":5000
+}
+			</pre>
+		</td>
+</table>
+</div>
+
+<p align="justify">Now I ran the following set of search requests across both the indexes, while increasing the number of documents indexed from 2000 to 4000.</p>
+
+<div style="overflow-x: auto;">
+<table>
+	<tr>
+		<th>Request 1</th>
+		<th>Request 2</th>
+		<th>... Request i</th>
+		<th>Request 1000</th>
+	</tr>
+	<tr>
+		<td style="vertical-align: top; width: 20%;">
+			<pre>
+{
+  "explain": true,
+  "fields": [
+    "*"
+  ],
+  "highlight": {},
+  "query": {
+    "match": "term",
+    "field":"dummyTerm"
+  },
+  "facets":{
+    "myDate":{
+      "field":"dummyDate",
+      "size":100000,
+      "date_ranges":[
+        {
+          "start":"2000-01-01T00:00:00",
+          "end":"2000-01-01T01:00:00"
+        }
+      ]
+    },
+    "myNum":{
+      "field":"dummyNumber",
+      "size":100000,
+      "numeric_ranges":[
+        {
+          "min": 1000,
+          "max": 1001
+        }
+      ]
+    }
+  },
+  "size": 10,
+  "from": 0
+}
+			</pre>
+		</td>
+		<td style="vertical-align: top; width: 20%;">
+			<pre>
+{
+  "explain": true,
+  "fields": [
+    "*"
+  ],
+  "highlight": {},
+  "query": {
+    "match": "term",
+    "field":"dummyTerm"
+  },
+  "facets":{
+    "myDate":{
+      "field":"dummyDate",
+      "size":100000,
+      "date_ranges":[
+        {
+          "start":"2000-01-01T01:00:00",
+          "end":"2000-01-01T02:00:00"
+        }
+      ]
+    },
+    "myNum":{
+      "field":"dummyNumber",
+      "size":100000,
+      "numeric_ranges":[
+        {
+          "min": 999,
+          "max": 1000
+        }
+      ]
+    }
+  },
+  "size": 10,
+  "from": 0
+}
+			</pre>
+		</td>
+		<td style="vertical-align: top; width: 20%;">
+			<pre>
+{
+  "explain": true,
+  "fields": [
+    "*"
+  ],
+  "highlight": {},
+  "query": {
+    "match": "term",
+    "field":"dummyTerm"
+  },
+  "facets":{
+    "myDate":{
+      "field":"dummyDate",
+      "size":100000,
+      "date_ranges":[
+        {
+          "start":"2000-01-01T00:00:00" + i hour
+          "end":"2000-01-01T00:00:00" + (i+1) hour
+        }
+      ]
+    },
+    "myNum":{
+      "field":"dummyNumber",
+      "size":100000,
+      "numeric_ranges":[
+        {
+          "min": 1000-i,
+          "max": 1000-i+1
+        }
+      ]
+    }
+  },
+  "size": 10,
+  "from": 0
+}
+			</pre>
+		</td>
+		<td style="vertical-align: top; width: 20%;">
+			<pre>
+{
+  "explain": true,
+  "fields": [
+    "*"
+  ],
+  "highlight": {},
+  "query": {
+    "match": "term",
+    "field":"dummyTerm"
+  },
+  "facets":{
+    "myDate":{
+      "field":"dummyDate",
+      "size":100000,
+      "date_ranges":[
+        {
+          "start":"2000-01-01T01:00:00" + 1000 hour,
+          "end":"2000-01-01T02:00:00" + 1001 hour
+        }
+      ]
+    },
+    "myNum":{
+      "field":"dummyNumber",
+      "size":100000,
+      "numeric_ranges":[
+        {
+          "min": 0,
+          "max": 1
+        }
+      ]
+    }
+  },
+  "size": 10,
+  "from": 0
+}
+			</pre>
+		</td>
+</table>
+</div>
+
+
+<div style="overflow-x: auto;">
+<table>
+	<tr>
+		<th>Bleve index size growth with increase in indexed documents</th>
+		<th>Total query time for 1000 queries with increase in number of indexed documents</th>
+	</tr>
+		<td><img src = "images/indexSizeVsNumDocs.png" alt = "indexSizeVsNumDocs.png"/></td>
+		<td><img src = "images/queryTimevsNumDocs.png" alt = "queryTimevsNumDocs.png"/></td>
+	</tr>
+</table>
+</div>
+
+<div style="overflow-x: auto;">
+<table>
+	<tr>
+		<th style="width:50%">Average increase in index size (in bytes) by enabling DocValues</th>
+		<th style="width:50%">Average reduction in time taken to perform 1000 queries (in milliseconds) by enabling DocValues</th>
+	</tr>
+	<tr>
+		<td align="center"><code>7762.47</code></td>
+		<td align="center"><code>27.034</code></td>
+	</tr>
+</table>
+Even at this small scale, with a small document size and a very limited number of indexed documents, we still observe a noticeable tradeoff. With just a slight increase in the index size (an average of 7KB), we obtain a 20ms reduction in the total execution time, on average, for only 1000 queries.
+
+<h3>Technical Information</h3>
+
+<p align="justify">When a search request involves facet or sorting operations on a field F, these operations occur after the main search query is executed. For instance, if the main query yields a result of 200 documents, the sorting and faceting processes will be applied to these 200 documents. However, the main query result only provides a set of document IDs, not the actual document contents.</p>
+
+<p align="justify">Here's where docValues become essential. If the field mapping for F is docValue enabled, the system can directly access the values for the field from the stored docValue part in the index file. This means that for each document ID returned in the search result, the field values are readily available.</p>
+
+<p align="justify">However, if docValues are not enabled for field F, the system must take a different approach. It needs to "fetch the document" from the index file, read the value for field F, and cache this field-document pair in memory for further processing. The issue becomes apparent in the latter scenario. By not enabling docValues for field F, you essentially retrieve all the documents in the search result (at the worst case), which can be a substantial amount of data. Moreover, you have to cache this information in memory, leading to increased memory usage. As a result, query latency significantly suffers because you're essentially fetching and processing all documents, which can be both time-consuming and resource-intensive. Enabling docValues for the relevant fields is, therefore, a crucial optimization to enhance query performance and reduce memory overhead in such situations.</p>

docs/synonyms.md

@@ -0,0 +1,180 @@
+# Synonym search
+
+* *v2.5.0* (and after) will come with support for **synonym definition indexing and search**.
+* We've achieved this by embedding synonym indexes within our bleve (scorch) indexes.
+* Usage of zap file format: [v16](https://github.com/blevesearch/zapx/blob/master/zap.md). Here we co-locate text, vector, and synonym indexes as neighbors within segments, continuing to conform to the segmented architecture of *scorch*.
+
+## Supported
+
+* Indexing `Synonym Definitions` allows specifying equivalent terms that will be used to construct the synonym index. There are currently two types of `Synonym Definitions` supported:
+
+    1. Equivalent Mapping:
+
+        In this type, all terms in the *synonyms* list are considered equal and can replace one another. Any of these terms can match a query or document containing any other term in the group, ensuring full synonym coverage.
+
+        ```json
+        {
+            "synonyms": [
+                "tranquil",
+                "peaceful",
+                "calm",
+                "relaxed",
+                "unruffled"
+            ]
+        }
+        ```
+
+    2. Explicit Mapping:
+
+        In this mapping, only the terms in the *input* list ("blazing") will have the terms in *synonyms* as their synonyms. The input terms are not equivalent to each other, and the synonym relationship is explicitly directional, applying only from the *input* to the *synonyms*.
+
+        ```json
+        {
+            "input": [
+                "blazing"
+            ],
+            "synonyms": [
+                "intense",
+                "radiant",
+                "burning",
+                "fiery",
+                "glowing"
+            ]
+        }
+        ```
+
+* The addition of `Synonym Sources` in the index mapping enables associating a set of `synonym definitions` (called a `synonym collection`) with a specific analyzer. This allows for preprocessing of terms in both the "input" and "synonyms" lists before the synonym index is created. By using an analyzer, you can normalize or transform terms (e.g., case folding, stemming) to improve synonym matching.
+
+    ```json
+    {
+        "analysis": {
+            "synonym_sources": {
+                "english": {
+                    "collection": "en_thesaurus",
+                    "analyzer": "en"
+                },
+                "german": {
+                    "collection": "de_thesaurus",
+                    "analyzer": "de"
+                }
+            }
+        }
+   }
+   ```
+
+    There are two `synonym sources` named "english" and "german," each associated with its respective `synonym collection` and analyzer. In any text field mapping, a `synonym source` can be specified to enable synonym expansion when the field is queried. The analyzer of the synonym source must match the analyzer of the field mapping to which it is applied.
+
+* Any text-based Bleve query (e.g., match, phrase, term, fuzzy, etc.) will use the `synonym source` (if available) for the queried field to expand the search terms using the thesaurus created from user-defined synonym definitions. The behavior for specific query types is as follows:
+
+    1. Queries with `fuzziness` parameter: For queries like match, phrase, and match-phrase that support the `fuzziness` parameter, the queried terms are fuzzily matched with the thesaurus's LHS terms to generate candidate terms. These terms are then combined with the results of fuzzy matching against the field dictionary, which contains the terms present in the queried field.
+
+    2. Wildcard, Regexp, and Prefix queries: These queries follow a similar approach. First, the thesaurus is used to expand terms (e.g., LHS terms that match the prefix or regex). The resulting terms are then combined with candidate terms from dictionary expansion.
+
+## Indexing
+
+Below is an example of using the Bleve API to define synonym sources, index synonym definitions, and associate them with a text field mapping:
+
+```go
+// Define a document to be indexed.
+doc := struct {
+    Text string `json:"text"`
+}{
+    Text: "hardworking employee",
+}
+
+// Define a synonym definition where "hardworking" has equivalent terms.
+synDef := &bleve.SynonymDefinition{
+    Synonyms: []string{
+        "hardworking",
+        "industrious",
+        "conscientious",
+        "persistent",
+    },
+}
+
+// Define the name of the `synonym collection`.
+// This collection groups multiple synonym definitions.
+synonymCollection := "collection1"
+
+// Define the name of the `synonym source`.
+// This source will be associated with specific field mappings.
+synonymSourceName := "english"
+
+// Define the analyzer to process terms in the synonym definitions.
+// This analyzer must match the one applied to the field using the synonym source.
+analyzer := "en"
+
+// Configure the synonym source by associating it with the synonym collection and analyzer.
+synonymSourceConfig := map[string]interface{}{
+    "collection": synonymCollection,
+    "analyzer":   analyzer,
+}
+
+// Create a new index mapping.
+bleveMapping := bleve.NewIndexMapping()
+
+// Add the synonym source configuration to the index mapping.
+err := bleveMapping.AddSynonymSource(synonymSourceName, synonymSourceConfig)
+if err != nil {
+    panic(err)
+}
+
+// Create a text field mapping with the specified analyzer and synonym source.
+textFieldMapping := bleve.NewTextFieldMapping()
+textFieldMapping.Analyzer = analyzer
+textFieldMapping.SynonymSource = synonymSourceName
+
+// Associate the text field mapping with the "text" field in the default document mapping.
+bleveMapping.DefaultMapping.AddFieldMappingsAt("text", textFieldMapping)
+
+// Create a new index with the specified mapping.
+index, err := bleve.New("example.bleve", bleveMapping)
+if err != nil {
+    panic(err)
+}
+
+// Index the document into the created index.
+err = index.Index("doc1", doc)
+if err != nil {
+    panic(err)
+}
+
+// Check if the index supports synonym indexing and add the synonym definition.
+if synIndex, ok := index.(bleve.SynonymIndex); ok {
+    err = synIndex.IndexSynonym("synDoc1", synonymCollection, synDef)
+    if err != nil {
+        panic(err)
+    }
+} else {
+    // If the index does not support synonym indexing, raise an error.
+    panic("expected synonym index")
+}
+```
+
+## Querying
+
+```go
+// Query the index created above.
+// Create a match query for the term "persistent".
+query := bleve.NewMatchQuery("persistent")
+
+// Specify the field to search within, in this case, the "text" field.
+query.SetField("text")
+
+// Create a search request with the query and enable explanation to understand how results are scored.
+searchRequest := bleve.NewSearchRequest(query)
+searchRequest.Explain = true
+
+// Execute the search on the index.
+searchResult, err := index.Search(searchRequest)
+if err != nil {
+    // Handle any errors that occur during the search.
+    panic(err)
+}
+
+// The search result will contain one match: "doc1". This document includes the term "hardworking",
+// which is a synonym for the queried term "persistent". The synonym relationship is based on 
+// the user-defined thesaurus associated with the index.
+// Print the search results, which will include the explanation for the match.
+fmt.Println(searchResult)
+```

docs/vectors.md

@@ -0,0 +1,347 @@
+# Approximate Nearest Neighbor Search (over vectors)
+
+* *v2.4.0* (and after) will come with support for **vector indexing and search**.
+* We've achieved this by embedding [FAISS](https://github.com/facebookresearch/faiss) indexes within our bleve (scorch) indexes.
+* Introduction of new zap file formats: [v16](https://github.com/blevesearch/zapx/blob/v16.x/zap.md), [v17](https://github.com/blevesearch/zapx/blob/master/zap.md) .. to co-locate text and vector indexes as neighbors within segments, continuing to conform to the segmented architecture of *scorch*.
+
+## Pre-requisite(s)
+
+* Induction of [FAISS](https://github.com/blevesearch/faiss) into our ecosystem, which is a fork of the original [facebookresearch/faiss](https://github.com/facebookresearch/faiss)
+* FAISS is a C++ library that needs to be compiled and its shared libraries need to be situated at an accessible path for your application.
+* A `vectors` GO TAG needs to be set for bleve to access all the supporting code. This TAG must be set only after the FAISS shared library is made available. Failure to do either will inhibit you from using this feature.
+* Please follow these [instructions](#setup-instructions) below for any assistance in the area.
+* Releases of `blevesearch/bleve` work with select checkpoints of `blevesearch/faiss` owing to API changes and improvements (tracking over the `bleve` branch):
+
+    | bleve version(s) | blevesearch/faiss version |
+    | --- | --- |
+    | `v2.4.0` | [blevesearch/faiss@7b119f4](https://github.com/blevesearch/faiss/tree/7b119f4b9c408989b696b36f8cc53908e53de6db) (modified v1.7.4) |
+    | `v2.4.1`, `v2.4.2` | [blevesearch/faiss@d9db66a](https://github.com/blevesearch/faiss/tree/d9db66a38518d99eb334218697e1df0732f3fdf8) (modified v1.7.4) |
+    | `v2.4.3`, `v2.4.4` | [blevesearch/faiss@b747c55](https://github.com/blevesearch/faiss/tree/b747c55a93a9627039c34d44b081f375dca94e57) (modified v1.8.0) |
+    | `v2.5.0`, `v2.5.1` | [blevesearch/faiss@352484e](https://github.com/blevesearch/faiss/tree/352484e0fc9d1f8f46737841efe5f26e0f383f71) (modified v1.10.0) |
+    | `v2.5.2`, `v2.5.3`, `v2.5.4` | [blevesearch/faiss@b3d4e00](https://github.com/blevesearch/faiss/tree/b3d4e00a69425b95e0b283da7801efc9f66b580d) (modified v1.11.0) |
+    | `v2.5.5`, `v2.5.6`, `v2.5.7` | [blevesearch/faiss@8a59a0c](https://github.com/blevesearch/faiss/tree/8a59a0c552fa2d14fa871f6b6bc793de1d277f5e) (modified v1.12.0) |
+    | `v2.6.0` | [blevesearch/faiss@fff814d](https://github.com/blevesearch/faiss/tree/fff814dea2bdda020363506904979b204ee201aa) (modified v1.13.2) |
+
+## Features
+
+* The `vector` field type is an array that is to hold float32 values only.
+* The `vector_base64` field type to support base64 encoded strings using little endian byte ordering (v2.4.1+)
+* Supported similarity metrics are: [`"cosine"` (v2.4.3+), `"dot_product"`, `"l2_norm"`].
+  * `cosine` paths will additionally normalize vectors before indexing and search.
+* Supported dimensionality is between 1 and 2048 (v2.4.0), and up to **4096** (v2.4.1+).
+* Supported vector index optimizations:
+  * `recall`, `latency` (v2.4.0+)
+    * Combination of Flat and IVF indexes with SQ8 quantization.
+  * `memory_efficient` (v2.4.1+)
+    * Combination of Flat and IVF indexes with SQ4 quantization.
+  * `bivf-flat`, `bivf-sq8` (v2.6.0+)
+    * Combination of BFlat and BIVF indexes with Binary quantization.
+    * Uses an additional index which is either a Flat index (`bivf-flat`) or an SQ8 index (`bivf-sq8`) for re-ranking.
+  * `ivf,rabitq` (v2.6.0+)
+    * Combination of Flat and IVF indexes with RaBitQ quantization.
+    * Works with [Fast Merge](https://github.com/blevesearch/bleve/blob/master/docs/fast_merge.md) - where a centroid index needs to be built/trained on a sample dataset prior to building the index for segments that deploy IVF indexes to reflect.
+
+---
+
+* Vectors from documents that do not conform to the index mapping dimensionality are simply discarded at index time.
+* The dimensionality of the query vector must match the dimensionality of the indexed vectors to obtain any results.
+* Pure kNN searches can be performed, but the `query` attribute within the search request must be set -- to `{"match_none": {}}` in this case. The `query` attribute is made optional when `knn` is available with v2.4.1+.
+* Hybrid searches are supported, where results from `query` are unioned with results from `knn`. The FTS scores from exact searches are simply summed with the similarity distances to determine the aggregate scores.
+
+```text
+aggregate_score = (query_boost * query_hit_score) + (knn_boost * knn_hit_distance)
+```
+
+* Advanced score fusion strategies (v2.5.4+) are available if requested for - see [score fusion](score_fusion.md#score-fusion-for-hybrid-search).
+* Multi kNN searches are supported - the `knn` object within the search request accepts an array of requests. These sub objects are unioned by default but this behavior can be overridden by setting `knn_operator` to `"and"`.
+* Previously supported pagination settings will work as they were, with size/limit being applied over the top-K hits combined with any exact search hits.
+* Pre-filtered vector and hybrid search (v2.4.3+): Apply any Bleve filter query first to narrow down candidates before running kNN search, making vector and hybrid searches faster and more relevant.
+* Fields containing multiple vectors (v2.5.7+):
+  * A single document may contain multiple vectors within the same field, in the form of either:
+    * an array of vectors (multi-vector field)
+    * an array of objects each containing a vector (nested-vector field)
+  * For single-kNN queries, each document is scored using its single best-matching vector.
+  * For multi-kNN queries, the system selects the best-matching vector for each query vector within the document.
+
+---
+
+* GPU-Accelerated vector search (v2.6.0+):
+  * Requires FAISS built with `-DFAISS_ENABLE_GPU=ON` CMake option (needs NVIDIA CUDA toolkit).
+  * Requires the `gpu` go tag in addition to the `vectors` tag when building bleve.
+  * GPU acceleration is enabled per vector field via the field mapping's GPU option.
+  * Bleve will use any available GPUs to offload training, indexing, and kNN searches for GPU-enabled vector fields.
+  * GPUs will be used for vector fields which are optimized for `recall`, `latency`, and `memory_efficient`.
+  * Multi-GPU support: when multiple GPUs are available, a load balancer distributes vector search workloads across devices.
+  * See [GPU setup instructions](#gpu-setup-instructions-v260) below for building FAISS with GPU support.
+
+## Indexing
+
+```go
+// Example document with single-vector, multi-vector, and nested-vector fields
+doc := struct {
+    Id         string      `json:"id"`
+    Text       string      `json:"text"`
+    Vec        []float32   `json:"vec"`        // Single-vector field
+    Embeddings [][]float32 `json:"embeddings"` // Multi-vector field: array of vectors (v2.5.7+)
+    Sections   []struct {  // Nested-vector field: array of objects with vectors (v2.5.7+)
+        Text string `json:"text"`
+        Vec  []float32 `json:"vec"`
+    } `json:"sections"`
+}{
+    Id:   "example",
+    Text: "hello from united states",
+    Vec:  []float32{0, 1, 2, 3, 4, 5, 6, 7, 8, 9}, // Single-vector field of dimensionality 10
+    Embeddings: [][]float32{ // Multi-vector field containing 2 vectors of dimensionality 10
+        {10, 11, 12, 13, 14, 15, 16, 17, 18, 19}, // First vector
+        {20, 21, 22, 23, 24, 25, 26, 27, 28, 29}, // Second vector
+    },
+    Sections: []struct { // Nested-vector field containing 2 objects each with a vector of dimensionality 10
+        Text string `json:"text"`
+        Vec  []float32 `json:"vec"`
+    }{
+        {Text: "first section", Vec: []float32{30, 31, 32, 33, 34, 35, 36, 37, 38, 39}},
+        {Text: "second section", Vec: []float32{40, 41, 42, 43, 44, 45, 46, 47, 48, 49}},
+    },
+}
+
+// Field mappings
+textFieldMapping := bleve.NewTextFieldMapping()
+vectorFieldMapping := bleve.NewVectorFieldMapping()
+vectorFieldMapping.Dims = 10                                               // Set vector dimensionality
+vectorFieldMapping.Similarity = index.CosineSimilarity                     // Set similarity metric
+vectorFieldMapping.VectorIndexOptimizedFor = index.IndexOptimizedForRecall // Set vector index optimization type
+vectorFieldMapping.GPU = true                                              // Enable GPU acceleration
+
+// Sub-document mappings
+sectionsMapping := bleve.NewDocumentMapping()
+sectionsMapping.AddFieldMappingsAt("text", textFieldMapping)
+sectionsMapping.AddFieldMappingsAt("vec", vectorFieldMapping)
+
+// Index mapping
+bleveMapping := bleve.NewIndexMapping()
+bleveMapping.DefaultMapping.AddFieldMappingsAt("text", textFieldMapping)
+bleveMapping.DefaultMapping.AddFieldMappingsAt("vec", vectorFieldMapping)        // Single-vector
+bleveMapping.DefaultMapping.AddFieldMappingsAt("embeddings", vectorFieldMapping) // Multi-vector
+bleveMapping.DefaultMapping.AddSubDocumentMapping("sections", sectionsMapping)   // Nested-vector
+
+// Create the index
+index, err := bleve.New("example.bleve", bleveMapping)
+if err != nil {
+    panic(err)
+}
+
+// Index the document
+err = index.Index(doc.Id, doc)
+if err != nil {
+    panic(err)
+}
+```
+
+## Querying
+
+```go
+// ------------------------------------
+// Single-vector field search (v2.4.0+)
+// ------------------------------------
+searchRequest := bleve.NewSearchRequest(bleve.NewMatchNoneQuery())
+searchRequest.AddKNN(
+    "vec",                                   // Vector field
+    []float32{0, 1, 1, 4, 4, 5, 7, 6, 8, 9}, // Query vector
+    5,                                       // top-k
+    1,                                       // boost
+)
+searchResult, err := index.Search(searchRequest)
+if err != nil {
+    panic(err)
+}
+// Scores are 1 / squared L2 distance, e.g., score = 0.25 for squared distance of 4
+fmt.Printf("Single-vector field kNN result:\n%s\n", searchResult)
+```
+
+```go
+// -----------------------------------
+// Multi-vector field search (v2.5.7+)
+// -----------------------------------
+searchRequest = bleve.NewSearchRequest(bleve.NewMatchNoneQuery())
+searchRequest.AddKNN(
+    "embeddings",
+    []float32{0, 1, 1, 4, 4, 5, 7, 6, 8, 9},
+    5,
+    1.0,
+)
+searchResult, err = index.Search(searchRequest)
+if err != nil {
+    panic(err)
+}
+// Scores are based on the **best-matching vector** from the multi-vector field.
+// Example: distances to doc vectors {10..19} and {20..29} → pick the closer one (smaller squared L2),
+// then score = 1 / squared L2 distance.
+fmt.Printf("Multi-vector field kNN result:\n%s\n", searchResult)
+```
+
+```go
+// ------------------------------------
+// Nested-vector field search (v2.5.7+)
+// ------------------------------------
+searchRequest = bleve.NewSearchRequest(bleve.NewMatchNoneQuery())
+searchRequest.AddKNN(
+    "sections.vec",
+    []float32{0, 1, 1, 4, 4, 5, 7, 6, 8, 9},
+    5,
+    1.0,
+)
+searchResult, err = index.Search(searchRequest)
+if err != nil {
+    panic(err)
+}
+// Scores are based on the **best-matching vector** from the nested-vector field.
+// Example: distances to doc vectors {30..39} and {40..49} → pick the closer one (smaller squared L2),
+// then score = 1 / squared L2 distance.
+fmt.Printf("Nested-vector field kNN result:\n%s\n", searchResult)
+```
+
+```go
+// -----------------------------------------------------
+// Multi kNN queries on multi-vector documents (v2.5.7+)
+// -----------------------------------------------------
+searchRequest = bleve.NewSearchRequest(bleve.NewMatchNoneQuery())
+searchRequest.AddKNN(
+    "embeddings",
+    []float32{0, 1, 1, 4, 4, 5, 7, 6, 8, 9},
+    5,
+    1.0,
+)
+searchRequest.AddKNN(
+    "embeddings",
+    []float32{1, 2, 2, 5, 5, 6, 8, 7, 9, 10},
+    8,
+    1.0,
+)
+searchResult, err = index.Search(searchRequest)
+if err != nil {
+    panic(err)
+}
+// Document score explanation:
+// - For each query vector, Bleve selects the **closest vector** in the multi-vector field.
+// - Scores from multiple queries are then **normalized and summed** to get the final document score.
+// For example, if the closest vector to the first query has squared L2 distance 4 (score 0.25)
+// and the closest vector to the second query has squared L2 distance 1 (score 1.0),
+// and both queries use equal boost values of 1.0, the normalization factor is 1/√2 (where 2 is the number of kNN queries).
+// Then the total document score = 1/√2 * 0.25 + 1/√2 * 1.0 = 0.1768 + 0.7071 = 0.8839.
+// Note: If the boost values differ, or if more queries are used, the normalization factor and score calculation will change accordingly.
+fmt.Printf("Multi kNN queries result:\n%s\n", searchResult)
+```
+
+```go
+// --------------------------------------
+// Hybrid search: text + vector (v2.4.0+)
+// --------------------------------------
+searchRequest = bleve.NewSearchRequest(bleve.NewMatchQuery("united states"))
+searchRequest.AddKNN(
+    "vec",
+    []float32{0, 1, 1, 4, 4, 5, 7, 6, 8, 9},
+    5,
+    1,
+)
+searchResult, err = index.Search(searchRequest)
+if err != nil {
+    panic(err)
+}
+// Score = sum of text relevance score + kNN vector score
+// Example: text score 0.5 + vector score 0.25 = total score 0.75
+fmt.Printf("Hybrid search result:\n%s\n", searchResult)
+```
+
+## Querying with filters (v2.4.3+)
+
+```go
+// Pre-filtered vector/hybrid search: filter query narrows candidates before KNN search
+searchRequest = bleve.NewSearchRequest(bleve.NewMatchNoneQuery()) // replace with any Bleve query for Pre-filtered Hybrid Search
+filterQuery := bleve.NewTermQuery("hello")                        // Filter query to narrow candidates
+searchRequest.AddKNNWithFilter(
+    "vec",                                   // Vector field name
+    []float32{0, 1, 1, 4, 4, 5, 7, 6, 8, 9}, // Query vector (must match indexed vector dims)
+    5,                                       // Number of nearest neighbors to return (k)
+    1,                                       // Boost factor for KNN score
+    filterQuery,                             // Filter query applied before KNN search
+)
+searchResult, err = index.Search(searchRequest)
+if err != nil {
+    panic(err)
+}
+// Scores are computed only among documents matching the filter query
+// Example: if only one document matches the filter and has squared L2 distance 4 to the query vector,
+// its score will be 0.25 (1 / 4) and returned as the top result.
+fmt.Printf("Pre-filtered kNN search result:\n%s\n", searchResult)
+```
+
+## Setup Instructions
+
+* Using `cmake` is a recommended approach by FAISS authors.
+* More details here - [faiss/INSTALL](https://github.com/blevesearch/faiss/blob/main/INSTALL.md).
+
+### Linux
+
+Also documented here - [go-faiss/README](https://github.com/blevesearch/go-faiss/blob/master/README.md).
+
+```shell
+git clone https://github.com/blevesearch/faiss.git
+cd faiss
+cmake -B build -DFAISS_ENABLE_GPU=OFF -DFAISS_ENABLE_C_API=ON -DBUILD_SHARED_LIBS=ON .
+make -C build
+sudo make -C build install
+```
+
+Building will produce the dynamic library `faiss_c`. You will need to install it in a place where your system will find it (e.g. /usr/lib). You can do this with:
+
+```shell
+sudo cp build/c_api/libfaiss_c.so /usr/local/lib
+```
+
+### OSX
+
+While you shouldn't need to do anything different for macOS x86_64, with aarch64 - some instructions need adjusting (see [facebookresearch/faiss#2111](https://github.com/facebookresearch/faiss/issues/2111)) ..
+
+```shell
+LDFLAGS="-L/opt/homebrew/opt/llvm/lib" CPPFLAGS="-I/opt/homebrew/opt/llvm/include" CXX=/opt/homebrew/opt/llvm/bin/clang++ CC=/opt/homebrew/opt/llvm/bin/clang cmake -B build -DFAISS_ENABLE_GPU=OFF -DFAISS_ENABLE_C_API=ON -DBUILD_SHARED_LIBS=ON -DFAISS_ENABLE_PYTHON=OFF .
+make -C build
+sudo make -C build install
+sudo cp build/c_api/libfaiss_c.dylib /usr/local/lib
+```
+
+### Sanity check
+
+Once the supporting library is built and made available, a sanity run is recommended to make sure all unit tests and especially those accessing the vectors' code pass. Here's how ..
+
+```shell
+go test -ldflags "-r /usr/local/lib" ./... -tags=vectors
+```
+
+## GPU Setup Instructions (v2.6.0+)
+
+GPU-accelerated vector search requires FAISS to be compiled with CUDA support and the `gpu` go tag to be set when building bleve.
+
+### Pre-requisites
+
+* An NVIDIA GPU with CUDA support (Pascal architecture / CC 6.0+ recommended).
+* [NVIDIA CUDA Toolkit](https://developer.nvidia.com/cuda-toolkit) installed and available on your system.
+* The FAISS C API shared library built with GPU support.
+
+### Linux (GPU)
+
+```shell
+git clone https://github.com/blevesearch/faiss.git
+cd faiss
+cmake -B build -DFAISS_ENABLE_GPU=ON -DFAISS_ENABLE_C_API=ON -DBUILD_SHARED_LIBS=ON .
+make -C build
+sudo make -C build install
+sudo cp build/c_api/libfaiss_c.so /usr/local/lib
+```
+
+### Sanity check (GPU)
+
+Run the full test suite with both the `vectors` and `gpu` build tags:
+
+```shell
+go test -ldflags "-r /usr/local/lib" ./... -tags=vectors,gpu
+```