snapshot of https://github.com/blevesearch/bleve.git (master, 2026-09-15; blobs > 131072B omitted)

index/scorch/README.md

@@ -0,0 +1,369 @@
+# scorch
+
+## Definitions
+
+Batch
+
+- A collection of Documents to mutate in the index.
+
+Document
+
+- Has a unique identifier (arbitrary bytes).
+- Is comprised of a list of fields.
+
+Field
+
+- Has a name (string).
+- Has a type (text, number, date, geopoint).
+- Has a value (depending on type).
+- Can be indexed, stored, or both.
+- If indexed, can be analyzed.
+-m If indexed, can optionally store term vectors.
+
+## Scope
+
+Scorch *MUST* implement the bleve.index API without requiring any changes to this API.
+
+Scorch *MAY* introduce new interfaces, which can be discovered to allow use of new capabilities not in the current API.
+
+## Implementation
+
+The scorch implementation starts with the concept of a segmented index.
+
+A segment is simply a slice, subset, or portion of the entire index.  A segmented index is one which is composed of one or more segments.  Although segments are created in a particular order, knowing this ordering is not required to achieve correct semantics when querying.  Because there is no ordering, this means that when searching an index, you can (and should) search all the segments concurrently.
+
+### Internal Wrapper
+
+In order to accommodate the existing APIs while also improving the implementation, the scorch implementation includes some wrapper functionality that must be described.
+
+#### \_id field
+
+In scorch, field 0 is prearranged to be named \_id.  All documents have a value for this field, which is the documents external identifier.  In this version the field *MUST* be both indexed AND stored.  The scorch wrapper adds this field, as it will not be present in the Document from the calling bleve code.
+
+NOTE: If a document already contains a field \_id, it will be replaced.  If this is problematic, the caller must ensure such a scenario does not happen.
+
+### Proposed Structures
+
+```go
+type Segment interface {
+
+  Dictionary(field string) TermDictionary
+
+}
+
+type TermDictionary interface {
+
+  PostingsList(term string, excluding PostingsList) PostingsList
+
+}
+
+type PostingsList interface {
+
+  Next() Posting
+
+  And(other PostingsList) PostingsList
+  Or(other PostingsList) PostingsList
+
+}
+
+type Posting interface {
+  Number() uint64
+
+  Frequency() uint64
+  Norm() float64
+
+  Locations() Locations
+}
+
+type Locations interface {
+  Start() uint64
+  End() uint64
+  Pos() uint64
+  ArrayPositions() ...
+}
+
+type DeletedDocs {
+
+}
+
+type SegmentSnapshot struct {
+  segment Segment
+  deleted PostingsList
+}
+
+type IndexSnapshot struct {
+  segment []SegmentSnapshot
+}
+```
+
+**What about errors?**
+**What about memory mgmnt or context?**
+**Postings List separate iterator to separate stateful from stateless**
+
+### Mutating the Index
+
+The bleve.index API has methods for directly making individual mutations (Update/Delete/SetInternal/DeleteInternal), however for this first implementation, we assume that all of these calls can simply be turned into a Batch of size 1.  This may be highly inefficient, but it will be correct.  This decision is made based on the fact that Couchbase FTS always uses Batches.
+
+NOTE: As a side-effect of this decision, it should be clear that performance tuning may depend on the batch size, which may in-turn require changes in FTS.
+
+From this point forward, only Batch mutations will be discussed.
+
+Sequence of Operations:
+
+1. For each document in the batch, search through all existing segments.  The goal is to build up a per-segment bitset which tells us which documents in that segment are obsoleted by the addition of the new segment we're currently building.  NOTE: we're not ready for this change to take effect yet, so rather than this operation mutating anything, they simply return bitsets, which we can apply later.  Logically, this is something like:
+
+    ```go
+      foreach segment {
+        dict := segment.Dictionary("\_id")
+        postings := empty postings list
+        foreach docID {
+          postings = postings.Or(dict.PostingsList(docID, nil))
+        }
+      }
+    ```
+
+    NOTE: it is illustrated above as nested for loops, but some or all of these could be concurrently.  The end result is that for each segment, we have (possibly empty) bitset.
+
+2. Also concurrent with 1, the documents in the batch are analyzed.  This analysis proceeds using the existing analyzer pool.
+
+3. (after 2 completes) Analyzed documents are fed into a function which builds a new Segment representing this information.
+
+4. We now have everything we need to update the state of the system to include this new snapshot.
+    - Acquire a lock
+    - Create a new IndexSnapshot
+    - For each SegmentSnapshot in the IndexSnapshot, take the deleted PostingsList and OR it with the new postings list for this Segment.  Construct a new SegmentSnapshot for the segment using this new deleted PostingsList.  Append this SegmentSnapshot to the IndexSnapshot.
+    - Create a new SegmentSnapshot wrapping our new segment with nil deleted docs.
+    - Append the new SegmentSnapshot to the IndexSnapshot
+    - Release the lock
+
+An ASCII art example:
+
+```text
+  0 - Empty Index
+
+  No segments
+
+  IndexSnapshot
+    segments []
+    deleted []
+
+
+  1 - Index Batch [ A B C ]
+
+  segment       0
+  numbers   [ 1 2 3 ]
+  \_id      [ A B C ]
+
+  IndexSnapshot
+    segments [ 0 ]
+    deleted [ nil ]
+
+
+  2 - Index Batch [ B' ]
+
+  segment       0           1
+  numbers   [ 1 2 3 ]     [ 1 ]
+  \_id      [ A B C ]     [ B ]
+
+  Compute bitset segment-0-deleted-by-1:
+            [ 0 1 0 ]
+
+  OR it with previous (nil) (call it 0-1)
+            [ 0 1 0 ]
+
+  IndexSnapshot
+    segments [  0    1 ]
+    deleted  [ 0-1 nil ]
+
+  3 - Index Batch [ C' ]
+
+    segment       0           1      2
+    numbers   [ 1 2 3 ]     [ 1 ]  [ 1 ]
+    \_id      [ A B C ]     [ B ]  [ C ]
+
+    Compute bitset segment-0-deleted-by-2:
+              [ 0 0 1 ]
+
+    OR it with previous ([ 0 1 0 ]) (call it 0-12)
+              [ 0 1 1 ]
+
+  Compute bitset segment-1-deleted-by-2:
+              [ 0 ]
+
+  OR it with previous (nil)
+              still just nil
+
+
+    IndexSnapshot
+      segments [  0    1    2 ]
+      deleted  [ 0-12 nil  nil ]
+  ```
+
+**is there opportunity to stop early when doc is found in one segment**
+**also, more efficient way to find bits for long lists of ids?**
+
+### Searching
+
+In the bleve.index API all searching starts by getting an IndexReader, which represents a snapshot of the index at a point in time.
+
+As described in the section above, our index implementation maintains a pointer to the current IndexSnapshot.  When a caller gets an IndexReader, they get a copy of this pointer, and can use it as long as they like.  The IndexSnapshot contains SegmentSnapshots, which only contain pointers to immutable segments.  The deleted posting lists associated with a segment change over time, but the particular deleted posting list in YOUR snapshot is immutable.  This gives a stable view of the data.
+
+#### Term Search
+
+Term search is the only searching primitive exposed in today's bleve.index API.  This ultimately could limit our ability to take advantage of the indexing improvements, but it also means it will be easier to get a first version of this working.
+
+A term search for term T in field F will look something like this:
+
+```go
+  searchResultPostings = empty
+  foreach segment {
+    dict := segment.Dictionary(F)
+    segmentResultPostings = dict.PostingsList(T, segmentSnapshotDeleted)
+    // make segmentLocal numbers into global numbers, and flip bits in searchResultPostings
+  }
+```
+
+The searchResultPostings will be a new implementation of the TermFieldReader interface.
+
+As a reminder this interface is:
+
+```go
+// TermFieldReader is the interface exposing the enumeration of documents
+// containing a given term in a given field. Documents are returned in byte
+// lexicographic order over their identifiers.
+type TermFieldReader interface {
+  // Next returns the next document containing the term in this field, or nil
+  // when it reaches the end of the enumeration.  The preAlloced TermFieldDoc
+  // is optional, and when non-nil, will be used instead of allocating memory.
+  Next(preAlloced *TermFieldDoc) (*TermFieldDoc, error)
+
+  // Advance resets the enumeration at specified document or its immediate
+  // follower.
+  Advance(ID IndexInternalID, preAlloced *TermFieldDoc) (*TermFieldDoc, error)
+
+  // Count returns the number of documents contains the term in this field.
+  Count() uint64
+  Close() error
+}
+```
+
+At first glance this appears problematic, we have no way to return documents in order of their identifiers.  But it turns out the wording of this perhaps too strong, or a bit ambiguous.  Originally, this referred to the external identifiers, but with the introduction of a distinction between internal/external identifiers, returning them in order of their internal identifiers is also acceptable.  **ASIDE**: the reason for this is that most callers just use Next() and literally don't care what the order is, they could be in any order and it would be fine.  There is only one search that cares and that is the ConjunctionSearcher, which relies on Next/Advance having very specific semantics.  Later in this document we will have a proposal to split into multiple interfaces:
+
+- The weakest interface, only supports Next() no ordering at all.
+- Ordered, supporting Advance()
+- And/Or'able capable of internally efficiently doing these ops with like interfaces (if not capable then can always fall back to external walking)
+
+But, the good news is that we don't even have to do that for our first implementation.  As long as the global numbers we use for internal identifiers are consistent within this IndexSnapshot, then Next() will be ordered by ascending document number, and Advance() will still work correctly.
+
+NOTE: there is another place where we rely on the ordering of these hits, and that is in the "\_id" sort order.  Previously this was the natural order, and a NOOP for the collector, now it must be implemented by actually sorting on the "\_id" field.  We probably should introduce at least a marker interface to detect this.
+
+An ASCII art example:
+
+```text
+Let's start with the IndexSnapshot we ended with earlier:
+
+3 - Index Batch [ C' ]
+
+  segment       0           1      2
+  numbers   [ 1 2 3 ]     [ 1 ]  [ 1 ]
+  \_id      [ A B C ]     [ B ]  [ C ]
+
+  Compute bitset segment-0-deleted-by-2:
+            [ 0 0 1 ]
+
+  OR it with previous ([ 0 1 0 ]) (call it 0-12)
+            [ 0 1 1 ]
+
+Compute bitset segment-1-deleted-by-2:
+            [ 0 0 0 ]
+
+OR it with previous (nil)
+            still just nil
+
+
+  IndexSnapshot
+    segments [  0    1    2 ]
+    deleted  [ 0-12 nil  nil ]
+
+Now let's search for the term 'cat' in the field 'desc' and let's assume that Document C (both versions) would match it.
+
+Concurrently:
+
+  - Segment 0
+   - Get Term Dictionary For Field 'desc'
+   - From it get Postings List for term 'cat' EXCLUDING 0-12
+   - raw segment matches [ 0 0 1 ] but excluding [ 0 1 1 ] gives [ 0 0 0 ]
+  - Segment 1
+   - Get Term Dictionary For Field 'desc'
+   - From it get Postings List for term 'cat' excluding nil
+   - [ 0 ]
+  - Segment 2
+   - Get Term Dictionary For Field 'desc'
+   - From it get Postings List for term 'cat' excluding nil
+   - [ 1 ]
+
+Map local bitsets into global number space (global meaning cross-segment but still unique to this snapshot)
+
+IndexSnapshot already should have mapping something like:
+0 - Offset 0
+1 - Offset 3 (because segment 0 had 3 docs)
+2 - Offset 4 (because segment 1 had 1 doc)
+
+This maps to search result bitset:
+
+[ 0 0 0 0 1]
+
+Caller would call Next() and get doc number 5 (assuming 1 based indexing for now)
+
+Caller could then ask to get term locations, stored fields, external doc ID for document number 5.  Internally in the IndexSnapshot, we can now convert that back, and realize doc number 5 comes from segment 2, 5-4=1 so we're looking for doc number 1 in segment 2.  That happens to be C...
+
+```
+
+#### Future improvements
+
+In the future, interfaces to detect these non-serially operating TermFieldReaders could expose their own And() and Or() up to the higher level Conjunction/Disjunction searchers.  Doing this alone offers some win, but also means there would be greater burden on the Searcher code rewriting logical expressions for maximum performance.
+
+Another related topic is that of peak memory usage.  With serially operating TermFieldReaders it was necessary to start them all at the same time and operate in unison.  However, with these non-serially operating TermFieldReaders we have the option of doing a few at a time, consolidating them, dispoting the intermediaries, and then doing a few more.  For very complex queries with many clauses this could reduce peak memory usage.
+
+### Memory Tracking
+
+All segments must be able to produce two statistics, an estimate of their explicit memory usage, and their actual size on disk (if any).  For in-memory segments, disk usage could be zero, and the memory usage represents the entire information content.  For mmap-based disk segments, the memory could be as low as the size of tracking structure itself (say just a few pointers).
+
+This would allow the implementation to throttle or block incoming mutations when a threshold memory usage has (or would be) exceeded.
+
+### Persistence
+
+Obviously, we want to support (but maybe not require) asynchronous persistence of segments.  My expectation is that segments are initially built in memory.  At some point they are persisted to disk.  This poses some interesting challenges.
+
+At runtime, the state of an index (it's IndexSnapshot) is not only the contents of the segments, but also the bitmasks of deleted documents.  These bitmasks indirectly encode an ordering in which the segments were added.  The reason is that the bitmasks encode which items have been obsoleted by other (subsequent or more future) segments.  In the runtime implementation we compute bitmask deltas and then merge them at the same time we bring the new segment in.  One idea is that we could take a similar approach on disk.  When we persist a segment, we persist the bitmask deltas of segments known to exist at that time, and eventually these can get merged up into a base segment deleted bitmask.
+
+This also relates to the topic rollback, addressed next...
+
+### Rollback
+
+One desirable property in the Couchbase ecosystem is the ability to rollback to some previous (though typically not long ago) state.  One idea for keeping this property in this design is to protect some of the most recent segments from merging.  Then, if necessary, they could be "undone" to reveal previous states of the system.  In these scenarios "undone" has to properly undo the deleted bitmasks on the other segments.  Again, the current thinking is that rather than "undo" anything, it could be work that was deferred in the first place, thus making it easier to logically undo.
+
+Another possibly related approach would be to tie this into our existing snapshot mechanism.  Perhaps simulating a slow reader (holding onto index snapshots) for some period of time, can be the mechanism to achieve the desired end goal.
+
+### Internal Storage
+
+The bleve.index API has support for "internal storage".  The ability to store information under a separate name space.
+
+This is not used for high volume storage, so it is tempting to think we could just put a small k/v store alongside the rest of the index.  But, the reality is that this storage is used to maintain key information related to the rollback scenario.  Because of this, its crucial that ordering and overwriting of key/value pairs correspond with actual segment persistence in the index.  Based on this, I believe its important to put the internal key/value pairs inside the segments themselves.  But, this also means that they must follow a similar "deleted" bitmask approach to obsolete values in older segments.  But, this also seems to substantially increase the complexity of the solution because of the separate name space, it would appear to require its own bitmask.  Further keys aren't numeric, which then implies yet another mapping from internal key to number, etc.
+
+More thought is required here.
+
+### Merging
+
+The segmented index approach requires merging to prevent the number of segments from growing too large.
+
+Recent experience with LSMs has taught us that having the correct merge strategy can make a huge difference in the overall performance of the system.  In particular, a simple merge strategy which merges segments too aggressively can lead to high write amplification and unnecessarily rendering cached data useless.
+
+A few simple principles have been identified.
+
+- Roughly we merge multiple smaller segments into a single larger one.
+- The larger a segment gets the less likely we should be to ever merge it.
+- Segments with large numbers of deleted/obsoleted items are good candidates as the merge will result in a space savings.
+- Segments with all items deleted/obsoleted can be dropped.
+
+Merging of a segment should be able to proceed even if that segment is held by an ongoing snapshot, it should only delay the removal of it.

index/scorch/builder.go

@@ -0,0 +1,332 @@
+//  Copyright (c) 2019 Couchbase, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// 		http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package scorch
+
+import (
+	"fmt"
+	"os"
+	"sync"
+
+	"github.com/RoaringBitmap/roaring/v2"
+	"github.com/blevesearch/bleve/v2/util"
+	index "github.com/blevesearch/bleve_index_api"
+	segment "github.com/blevesearch/scorch_segment_api/v2"
+)
+
+const DefaultBuilderBatchSize = 1000
+const DefaultBuilderMergeMax = 10
+
+type Builder struct {
+	m         sync.Mutex
+	segCount  uint64
+	path      string
+	buildPath string
+	segPaths  []string
+	batchSize int
+	mergeMax  int
+	batch     *index.Batch
+	internal  map[string][]byte
+	segPlugin SegmentPlugin
+}
+
+func NewBuilder(config map[string]interface{}) (*Builder, error) {
+	path, ok := config["path"].(string)
+	if !ok {
+		return nil, fmt.Errorf("must specify path")
+	}
+
+	buildPathPrefix, _ := config["buildPathPrefix"].(string)
+	buildPath, err := os.MkdirTemp(buildPathPrefix, "scorch-offline-build")
+	if err != nil {
+		return nil, err
+	}
+
+	rv := &Builder{
+		path:      path,
+		buildPath: buildPath,
+		mergeMax:  DefaultBuilderMergeMax,
+		batchSize: DefaultBuilderBatchSize,
+		batch:     index.NewBatch(),
+		segPlugin: defaultSegmentPlugin,
+	}
+
+	err = rv.parseConfig(config)
+	if err != nil {
+		return nil, fmt.Errorf("error parsing builder config: %v", err)
+	}
+
+	return rv, nil
+}
+
+func (o *Builder) parseConfig(config map[string]interface{}) (err error) {
+	if v, ok := config["mergeMax"]; ok {
+		var t int
+		if t, err = parseToInteger(v); err != nil {
+			return fmt.Errorf("mergeMax parse err: %v", err)
+		}
+		if t > 0 {
+			o.mergeMax = t
+		}
+	}
+
+	if v, ok := config["batchSize"]; ok {
+		var t int
+		if t, err = parseToInteger(v); err != nil {
+			return fmt.Errorf("batchSize parse err: %v", err)
+		}
+		if t > 0 {
+			o.batchSize = t
+		}
+	}
+
+	if v, ok := config["internal"]; ok {
+		if vinternal, ok := v.(map[string][]byte); ok {
+			o.internal = vinternal
+		}
+	}
+
+	forcedSegmentType, forcedSegmentVersion, err := configForceSegmentTypeVersion(config)
+	if err != nil {
+		return err
+	}
+	if forcedSegmentType != "" && forcedSegmentVersion != 0 {
+		segPlugin, err := chooseSegmentPlugin(forcedSegmentType,
+			uint32(forcedSegmentVersion))
+		if err != nil {
+			return err
+		}
+		o.segPlugin = segPlugin
+	}
+
+	return nil
+}
+
+// Index will place the document into the index.
+// It is invalid to index the same document multiple times.
+func (o *Builder) Index(doc index.Document) error {
+	o.m.Lock()
+	defer o.m.Unlock()
+
+	o.batch.Update(doc)
+
+	return o.maybeFlushBatchLOCKED(o.batchSize)
+}
+
+func (o *Builder) maybeFlushBatchLOCKED(moreThan int) error {
+	if len(o.batch.IndexOps) >= moreThan {
+		defer o.batch.Reset()
+		return o.executeBatchLOCKED(o.batch)
+	}
+	return nil
+}
+
+func (o *Builder) executeBatchLOCKED(batch *index.Batch) (err error) {
+	analysisResults := make([]index.Document, 0, len(batch.IndexOps))
+	for _, doc := range batch.IndexOps {
+		if doc != nil {
+			// insert _id field
+			doc.AddIDField()
+			// perform analysis directly
+			analyze(doc, nil)
+			analysisResults = append(analysisResults, doc)
+		}
+	}
+
+	seg, _, err := o.segPlugin.New(analysisResults)
+	if err != nil {
+		return fmt.Errorf("error building segment base: %v", err)
+	}
+
+	filename := zapFileName(o.segCount)
+	o.segCount++
+	path := o.buildPath + string(os.PathSeparator) + filename
+
+	if segUnpersisted, ok := seg.(segment.UnpersistedSegment); ok {
+		err = segUnpersisted.Persist(path)
+		if err != nil {
+			return fmt.Errorf("error persisting segment base to %s: %v", path, err)
+		}
+
+		o.segPaths = append(o.segPaths, path)
+		return nil
+	}
+
+	return fmt.Errorf("new segment does not implement unpersisted: %T", seg)
+}
+
+func (o *Builder) doMerge() error {
+	// as long as we have more than 1 segment, keep merging
+	for len(o.segPaths) > 1 {
+
+		// merge the next <mergeMax> number of segments into one new one
+		// or, if there are fewer than <mergeMax> remaining, merge them all
+		mergeCount := o.mergeMax
+		if mergeCount > len(o.segPaths) {
+			mergeCount = len(o.segPaths)
+		}
+
+		mergePaths := o.segPaths[0:mergeCount]
+		o.segPaths = o.segPaths[mergeCount:]
+
+		// open each of the segments to be merged
+		mergeSegs := make([]segment.Segment, 0, mergeCount)
+
+		// closeOpenedSegs attempts to close all opened
+		// segments even if an error occurs, in which case
+		// the first error is returned
+		closeOpenedSegs := func() error {
+			var err error
+			for _, seg := range mergeSegs {
+				clErr := seg.Close()
+				if clErr != nil && err == nil {
+					err = clErr
+				}
+			}
+			return err
+		}
+
+		for _, mergePath := range mergePaths {
+			seg, err := o.segPlugin.Open(mergePath)
+			if err != nil {
+				_ = closeOpenedSegs()
+				return fmt.Errorf("error opening segment (%s) for merge: %v", mergePath, err)
+			}
+			mergeSegs = append(mergeSegs, seg)
+		}
+
+		// do the merge
+		mergedSegPath := o.buildPath + string(os.PathSeparator) + zapFileName(o.segCount)
+		drops := make([]*roaring.Bitmap, mergeCount)
+		_, _, err := o.segPlugin.Merge(mergeSegs, drops, mergedSegPath, nil, nil)
+		if err != nil {
+			_ = closeOpenedSegs()
+			return fmt.Errorf("error merging segments (%v): %v", mergePaths, err)
+		}
+		o.segCount++
+		o.segPaths = append(o.segPaths, mergedSegPath)
+
+		// close segments opened for merge
+		err = closeOpenedSegs()
+		if err != nil {
+			return fmt.Errorf("error closing opened segments: %v", err)
+		}
+
+		// remove merged segments
+		for _, mergePath := range mergePaths {
+			err = os.RemoveAll(mergePath)
+			if err != nil {
+				return fmt.Errorf("error removing segment %s after merge: %v", mergePath, err)
+			}
+		}
+	}
+
+	return nil
+}
+
+func (o *Builder) Close() error {
+	o.m.Lock()
+	defer o.m.Unlock()
+
+	// see if there is a partial batch
+	err := o.maybeFlushBatchLOCKED(1)
+	if err != nil {
+		return fmt.Errorf("error flushing batch before close: %v", err)
+	}
+
+	// perform all the merging
+	err = o.doMerge()
+	if err != nil {
+		return fmt.Errorf("error while merging: %v", err)
+	}
+
+	// ensure the store path exists
+	err = os.MkdirAll(o.path, 0700)
+	if err != nil {
+		return err
+	}
+
+	// move final segment into place
+	// segment id 2 is chosen to match the behavior of a scorch
+	// index which indexes a single batch of data
+	finalSegPath := o.path + string(os.PathSeparator) + zapFileName(2)
+	err = os.Rename(o.segPaths[0], finalSegPath)
+	if err != nil {
+		return fmt.Errorf("error moving final segment into place: %v", err)
+	}
+
+	// remove the buildPath, as it is no longer needed
+	err = os.RemoveAll(o.buildPath)
+	if err != nil {
+		return fmt.Errorf("error removing build path: %v", err)
+	}
+
+	// prepare wrapping
+	seg, err := o.segPlugin.Open(finalSegPath)
+	if err != nil {
+		return fmt.Errorf("error opening final segment")
+	}
+
+	// create a segment snapshot for this segment
+	ss := &SegmentSnapshot{
+		segment: seg,
+	}
+	is := &IndexSnapshot{
+		epoch:    3, // chosen to match scorch behavior when indexing a single batch
+		segment:  []*SegmentSnapshot{ss},
+		creator:  "scorch-builder",
+		internal: o.internal,
+	}
+
+	// create the root bolt
+	rootBoltPath := o.path + string(os.PathSeparator) + "root.bolt"
+	rootBolt, err := util.OpenBolt(rootBoltPath, 0600, nil)
+	if err != nil {
+		return err
+	}
+
+	// start a write transaction
+	tx, err := rootBolt.Begin(true)
+	if err != nil {
+		return err
+	}
+
+	// fill the root bolt with this fake index snapshot
+	_, _, err = prepareBoltSnapshot(is, tx, o.path, o.segPlugin, nil)
+	if err != nil {
+		_ = tx.Rollback()
+		_ = rootBolt.Close()
+		return fmt.Errorf("error preparing bolt snapshot in root.bolt: %v", err)
+	}
+
+	// commit bolt data
+	err = tx.Commit()
+	if err != nil {
+		_ = rootBolt.Close()
+		return fmt.Errorf("error committing bolt tx in root.bolt: %v", err)
+	}
+
+	// close bolt
+	err = rootBolt.Close()
+	if err != nil {
+		return fmt.Errorf("error closing root.bolt: %v", err)
+	}
+
+	// close final segment
+	err = seg.Close()
+	if err != nil {
+		return fmt.Errorf("error closing final segment: %v", err)
+	}
+	return nil
+}

index/scorch/builder_test.go

@@ -0,0 +1,159 @@
+//  Copyright (c) 2019 Couchbase, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// 		http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package scorch
+
+import (
+	"context"
+	"fmt"
+	"os"
+	"testing"
+
+	"github.com/blevesearch/bleve/v2/document"
+	index "github.com/blevesearch/bleve_index_api"
+)
+
+func TestBuilder(t *testing.T) {
+	tmpDir, err := os.MkdirTemp("", "scorch-builder-test")
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer func() {
+		err = os.RemoveAll(tmpDir)
+		if err != nil {
+			t.Fatalf("error cleaning up test index: %v", err)
+		}
+	}()
+	options := map[string]interface{}{
+		"path":      tmpDir,
+		"batchSize": 2,
+		"mergeMax":  2,
+	}
+	b, err := NewBuilder(options)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	for i := 0; i < 10; i++ {
+		doc := document.NewDocument(fmt.Sprintf("%d", i))
+		doc.AddField(document.NewTextField("name", nil, []byte("hello")))
+		err = b.Index(doc)
+		if err != nil {
+			t.Fatal(err)
+		}
+	}
+
+	err = b.Close()
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	checkIndex(t, tmpDir, []byte("hello"), "name", 10)
+}
+
+func checkIndex(t *testing.T, path string, term []byte, field string, expectCount int) {
+	cfg := make(map[string]interface{})
+	cfg["path"] = path
+	analysisQueue := index.NewAnalysisQueue(1)
+	idx, err := NewScorch(Name, cfg, analysisQueue)
+	if err != nil {
+		t.Fatal(err)
+	}
+	err = idx.Open()
+	if err != nil {
+		t.Fatalf("error opening index: %v", err)
+	}
+	defer func() {
+		err = idx.Close()
+		if err != nil {
+			t.Fatalf("error closing index: %v", err)
+		}
+	}()
+
+	r, err := idx.Reader()
+	if err != nil {
+		t.Fatalf("error accessing index reader: %v", err)
+	}
+	defer func() {
+		err = r.Close()
+		if err != nil {
+			t.Fatalf("error closing reader: %v", err)
+		}
+	}()
+
+	// check the count, expect 10 docs
+	count, err := r.DocCount()
+	if err != nil {
+		t.Errorf("error accessing index doc count: %v", err)
+	} else if count != uint64(expectCount) {
+		t.Errorf("expected %d docs, got %d", expectCount, count)
+	}
+
+	// run a search for hello
+	tfr, err := r.TermFieldReader(context.TODO(), term, field, false, false, false)
+	if err != nil {
+		t.Errorf("error accessing term field reader: %v", err)
+	} else {
+		var rows int
+		tfd, err := tfr.Next(nil)
+		for err == nil && tfd != nil {
+			rows++
+			tfd, err = tfr.Next(nil)
+		}
+		if err != nil {
+			t.Errorf("error calling next on term field reader: %v", err)
+		}
+		if rows != expectCount {
+			t.Errorf("expected %d rows for term hello, field name, got %d", expectCount, rows)
+		}
+	}
+}
+
+func TestBuilderFlushFinalBatch(t *testing.T) {
+	tmpDir, err := os.MkdirTemp("", "scorch-builder-test")
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer func() {
+		err = os.RemoveAll(tmpDir)
+		if err != nil {
+			t.Fatalf("error cleaning up test index: %v", err)
+		}
+	}()
+	options := map[string]interface{}{
+		"path":      tmpDir,
+		"batchSize": 2,
+		"mergeMax":  2,
+	}
+	b, err := NewBuilder(options)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	for i := 0; i < 9; i++ {
+		doc := document.NewDocument(fmt.Sprintf("%d", i))
+		doc.AddField(document.NewTextField("name", nil, []byte("hello")))
+		err = b.Index(doc)
+		if err != nil {
+			t.Fatal(err)
+		}
+	}
+
+	err = b.Close()
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	checkIndex(t, tmpDir, []byte("hello"), "name", 9)
+}

index/scorch/empty.go

@@ -0,0 +1,50 @@
+//  Copyright (c) 2020 Couchbase, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// 		http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package scorch
+
+import (
+	"github.com/RoaringBitmap/roaring/v2"
+	segment "github.com/blevesearch/scorch_segment_api/v2"
+)
+
+type emptyPostingsIterator struct{}
+
+func (e *emptyPostingsIterator) Next() (segment.Posting, error) {
+	return nil, nil
+}
+
+func (e *emptyPostingsIterator) Advance(uint64) (segment.Posting, error) {
+	return nil, nil
+}
+
+func (e *emptyPostingsIterator) Size() int {
+	return 0
+}
+
+func (e *emptyPostingsIterator) BytesRead() uint64 {
+	return 0
+}
+
+func (e *emptyPostingsIterator) ResetBytesRead(uint64) {}
+
+func (e *emptyPostingsIterator) BytesWritten() uint64 { return 0 }
+
+func (e *emptyPostingsIterator) ActualBitmap() *roaring.Bitmap { return nil }
+
+func (e *emptyPostingsIterator) DocNum1Hit() (uint64, bool) { return 0, false }
+
+func (e *emptyPostingsIterator) ReplaceActual(*roaring.Bitmap) {}
+
+var anEmptyPostingsIterator = &emptyPostingsIterator{}

index/scorch/event.go

@@ -0,0 +1,79 @@
+//  Copyright (c) 2018 Couchbase, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// 		http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package scorch
+
+import "time"
+
+// RegistryAsyncErrorCallbacks should be treated as read-only after
+// process init()'ialization.
+var RegistryAsyncErrorCallbacks = map[string]func(error, string){}
+
+// RegistryEventCallbacks should be treated as read-only after
+// process init()'ialization.
+// In the event of not having a callback, these return true.
+var RegistryEventCallbacks = map[string]func(Event) bool{}
+
+// Event represents the information provided in an OnEvent() callback.
+type Event struct {
+	Kind     EventKind
+	Scorch   *Scorch
+	Duration time.Duration
+}
+
+// EventKind represents an event code for OnEvent() callbacks.
+type EventKind int
+
+const (
+	// EventKindCloseStart is fired when a Scorch.Close() has begun.
+	EventKindCloseStart EventKind = iota
+
+	// EventKindClose is fired when a scorch index has been fully closed.
+	EventKindClose
+
+	// EventKindMergerProgress is fired when the merger has completed a
+	// round of merge processing.
+	EventKindMergerProgress
+
+	// EventKindPersisterProgress is fired when the persister has completed
+	// a round of persistence processing.
+	EventKindPersisterProgress
+
+	// EventKindBatchIntroductionStart is fired when Batch() is invoked which
+	// introduces a new segment.
+	EventKindBatchIntroductionStart
+
+	// EventKindBatchIntroduction is fired when Batch() completes.
+	EventKindBatchIntroduction
+
+	// EventKindMergeTaskIntroductionStart is fired when the merger is about to
+	// start the introduction of merged segment from a single merge task.
+	EventKindMergeTaskIntroductionStart
+
+	// EventKindMergeTaskIntroduction is fired when the merger has completed
+	// the introduction of merged segment from a single merge task.
+	EventKindMergeTaskIntroduction
+
+	// EventKindPreMergeCheck is fired before the merge begins to check if
+	// the caller should proceed with the merge.
+	EventKindPreMergeCheck
+
+	// EventKindIndexStart is fired when Index() is invoked which
+	// creates a new Document object from an interface using the index mapping.
+	EventKindIndexStart
+
+	// EventKindPurgerCheck is fired before the purge code is invoked and decides
+	// whether to execute or not. For unit test purposes
+	EventKindPurgerCheck
+)

index/scorch/event_test.go

@@ -0,0 +1,79 @@
+//  Copyright (c) 2018 Couchbase, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// 		http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package scorch
+
+import (
+	"testing"
+
+	"github.com/blevesearch/bleve/v2/document"
+	index "github.com/blevesearch/bleve_index_api"
+)
+
+func TestEventBatchIntroductionStart(t *testing.T) {
+	testConfig := CreateConfig("TestEventBatchIntroductionStart")
+	err := InitTest(testConfig)
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer func() {
+		err := DestroyTest(testConfig)
+		if err != nil {
+			t.Fatal(err)
+		}
+	}()
+
+	var count int
+	RegistryEventCallbacks["test"] = func(e Event) bool {
+		if e.Kind == EventKindBatchIntroductionStart {
+			count++
+		}
+		return true
+	}
+
+	ourConfig := make(map[string]interface{}, len(testConfig))
+	for k, v := range testConfig {
+		ourConfig[k] = v
+	}
+	ourConfig["eventCallbackName"] = "test"
+
+	analysisQueue := index.NewAnalysisQueue(1)
+	idx, err := NewScorch(Name, ourConfig, analysisQueue)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	err = idx.Open()
+	if err != nil {
+		t.Fatalf("error opening index: %v", err)
+	}
+
+	doc := document.NewDocument("1")
+	doc.AddField(document.NewTextField("name", []uint64{}, []byte("test")))
+	err = idx.Update(doc)
+	if err != nil {
+		t.Errorf("Error updating index: %v", err)
+	}
+
+	defer func() {
+		err := idx.Close()
+		if err != nil {
+			t.Fatal(err)
+		}
+	}()
+
+	if count != 1 {
+		t.Fatalf("expected to see 1 batch introduction event event, saw %d", count)
+	}
+}

index/scorch/field_dict_test.go

@@ -0,0 +1,186 @@
+//  Copyright (c) 2017 Couchbase, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// 		http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package scorch
+
+import (
+	"reflect"
+	"testing"
+
+	"github.com/blevesearch/bleve/v2/document"
+	index "github.com/blevesearch/bleve_index_api"
+)
+
+func TestIndexFieldDict(t *testing.T) {
+	cfg := CreateConfig("TestIndexFieldDict")
+	err := InitTest(cfg)
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer func() {
+		err := DestroyTest(cfg)
+		if err != nil {
+			t.Log(err)
+		}
+	}()
+
+	analysisQueue := index.NewAnalysisQueue(1)
+	idx, err := NewScorch(Name, cfg, analysisQueue)
+	if err != nil {
+		t.Fatal(err)
+	}
+	err = idx.Open()
+	if err != nil {
+		t.Fatalf("error opening index: %v", err)
+	}
+	defer func() {
+		cerr := idx.Close()
+		if cerr != nil {
+			t.Fatal(cerr)
+		}
+	}()
+
+	doc := document.NewDocument("1")
+	doc.AddField(document.NewTextField("name", []uint64{}, []byte("test")))
+	err = idx.Update(doc)
+	if err != nil {
+		t.Errorf("Error updating index: %v", err)
+	}
+
+	doc = document.NewDocument("2")
+	doc.AddField(document.NewTextFieldWithAnalyzer("name", []uint64{}, []byte("test test test"), testAnalyzer))
+	doc.AddField(document.NewTextFieldCustom("desc", []uint64{}, []byte("eat more rice"), index.IndexField|index.IncludeTermVectors, testAnalyzer))
+	doc.AddField(document.NewTextFieldCustom("prefix", []uint64{}, []byte("bob cat cats catting dog doggy zoo"), index.IndexField|index.IncludeTermVectors, testAnalyzer))
+	err = idx.Update(doc)
+	if err != nil {
+		t.Errorf("Error updating index: %v", err)
+	}
+
+	indexReader, err := idx.Reader()
+	if err != nil {
+		t.Error(err)
+	}
+	defer func() {
+		err := indexReader.Close()
+		if err != nil {
+			t.Fatal(err)
+		}
+	}()
+
+	dict, err := indexReader.FieldDict("name")
+	if err != nil {
+		t.Errorf("error creating reader: %v", err)
+	}
+	defer func() {
+		err := dict.Close()
+		if err != nil {
+			t.Fatal(err)
+		}
+	}()
+
+	termCount := 0
+	curr, err := dict.Next()
+	for err == nil && curr != nil {
+		termCount++
+		if curr.Term != "test" {
+			t.Errorf("expected term to be 'test', got '%s'", curr.Term)
+		}
+		curr, err = dict.Next()
+	}
+	if termCount != 1 {
+		t.Errorf("expected 1 term for this field, got %d", termCount)
+	}
+
+	dict2, err := indexReader.FieldDict("desc")
+	if err != nil {
+		t.Fatalf("error creating reader: %v", err)
+	}
+	defer func() {
+		err := dict2.Close()
+		if err != nil {
+			t.Fatal(err)
+		}
+	}()
+
+	termCount = 0
+	terms := make([]string, 0)
+	curr, err = dict2.Next()
+	for err == nil && curr != nil {
+		termCount++
+		terms = append(terms, curr.Term)
+		curr, err = dict2.Next()
+	}
+	if termCount != 3 {
+		t.Errorf("expected 3 term for this field, got %d", termCount)
+	}
+	expectedTerms := []string{"eat", "more", "rice"}
+	if !reflect.DeepEqual(expectedTerms, terms) {
+		t.Errorf("expected %#v, got %#v", expectedTerms, terms)
+	}
+	// test start and end range
+	dict3, err := indexReader.FieldDictRange("desc", []byte("fun"), []byte("nice"))
+	if err != nil {
+		t.Errorf("error creating reader: %v", err)
+	}
+	defer func() {
+		err := dict3.Close()
+		if err != nil {
+			t.Fatal(err)
+		}
+	}()
+
+	termCount = 0
+	terms = make([]string, 0)
+	curr, err = dict3.Next()
+	for err == nil && curr != nil {
+		termCount++
+		terms = append(terms, curr.Term)
+		curr, err = dict3.Next()
+	}
+	if termCount != 1 {
+		t.Errorf("expected 1 term for this field, got %d", termCount)
+	}
+	expectedTerms = []string{"more"}
+	if !reflect.DeepEqual(expectedTerms, terms) {
+		t.Errorf("expected %#v, got %#v", expectedTerms, terms)
+	}
+
+	// test use case for prefix
+	dict4, err := indexReader.FieldDictPrefix("prefix", []byte("cat"))
+	if err != nil {
+		t.Errorf("error creating reader: %v", err)
+	}
+	defer func() {
+		err := dict4.Close()
+		if err != nil {
+			t.Fatal(err)
+		}
+	}()
+
+	termCount = 0
+	terms = make([]string, 0)
+	curr, err = dict4.Next()
+	for err == nil && curr != nil {
+		termCount++
+		terms = append(terms, curr.Term)
+		curr, err = dict4.Next()
+	}
+	if termCount != 3 {
+		t.Errorf("expected 3 term for this field, got %d", termCount)
+	}
+	expectedTerms = []string{"cat", "cats", "catting"}
+	if !reflect.DeepEqual(expectedTerms, terms) {
+		t.Errorf("expected %#v, got %#v", expectedTerms, terms)
+	}
+}

index/scorch/fuzzy_omitcount_test.go

@@ -0,0 +1,98 @@
+//	Copyright (c) 2026 Couchbase, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//	http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+package scorch
+
+import (
+	"fmt"
+	"testing"
+
+	"github.com/blevesearch/bleve/v2/document"
+	index "github.com/blevesearch/bleve_index_api"
+)
+
+// TestFieldDictFuzzyAutomatonOmitsCount verifies that the fuzzy automaton field
+// dictionary is wired to the count-omitting iterator: the segment dictionary
+// implements the optional interface, and the collected candidate entries have
+// their (discarded) Count left at zero rather than incurring a postings read.
+func TestFieldDictFuzzyAutomatonOmitsCount(t *testing.T) {
+	cfg := CreateConfig("TestFieldDictFuzzyAutomatonOmitsCount")
+	if err := InitTest(cfg); err != nil {
+		t.Fatal(err)
+	}
+	defer func() { _ = DestroyTest(cfg) }()
+	analysisQueue := index.NewAnalysisQueue(1)
+	idx, err := NewScorch(Name, cfg, analysisQueue)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if err = idx.Open(); err != nil {
+		t.Fatal(err)
+	}
+	defer func() { _ = idx.Close() }()
+
+	// Each doc's "desc" field is a single token (nil analyzer => whole value is
+	// one term). Repeating terms across docs yields multi-doc postings lists.
+	b := index.NewBatch()
+	for d := 0; d < 300; d++ {
+		doc := document.NewDocument(fmt.Sprintf("%d", d))
+		doc.AddField(document.NewTextField("desc", nil, []byte(fmt.Sprintf("term%04d", d%100))))
+		b.Update(doc)
+	}
+	if err = idx.Batch(b); err != nil {
+		t.Fatal(err)
+	}
+
+	r, err := idx.Reader()
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer func() { _ = r.Close() }()
+	is := r.(*IndexSnapshot)
+
+	// the segment dictionary must implement the count-omit optional interface,
+	// otherwise the wiring silently falls back to the count-reading path.
+	seg := is.segment[0].segment
+	dict, err := seg.Dictionary("desc")
+	if err != nil {
+		t.Fatal(err)
+	}
+	if _, ok := dict.(termDictionaryOmitCount); !ok {
+		// The omit-count wiring only activates once the zapx dependency ships
+		// AutomatonIteratorOmitCount; until then the code falls back to the
+		// count-reading path, so there is nothing to assert.
+		t.Skipf("segment dict %T does not implement termDictionaryOmitCount "+
+			"(zapx dependency predates the omit-count iterator)", dict)
+	}
+
+	fd, _, err := is.FieldDictFuzzyAutomaton("desc", "term0050", 2, "")
+	if err != nil {
+		t.Fatal(err)
+	}
+	n := 0
+	tfd, err := fd.Next()
+	for err == nil && tfd != nil {
+		n++
+		if tfd.Count != 0 {
+			t.Fatalf("expected omitted (zero) count for %q, got %d", tfd.Term, tfd.Count)
+		}
+		tfd, err = fd.Next()
+	}
+	if err != nil {
+		t.Fatal(err)
+	}
+	if n == 0 {
+		t.Fatal("expected at least one fuzzy candidate term")
+	}
+	t.Logf("collected %d candidate terms, all with omitted count", n)
+}

index/scorch/int.go

@@ -0,0 +1,92 @@
+// Copyright 2014 The Cockroach Authors.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+// implied. See the License for the specific language governing
+// permissions and limitations under the License.
+
+// This code originated from:
+// https://github.com/cockroachdb/cockroach/blob/2dd65dde5d90c157f4b93f92502ca1063b904e1d/pkg/util/encoding/encoding.go
+
+// Modified to not use pkg/errors
+
+package scorch
+
+import "fmt"
+
+const (
+	// intMin is chosen such that the range of int tags does not overlap the
+	// ascii character set that is frequently used in testing.
+	intMin      = 0x80 // 128
+	intMaxWidth = 8
+	intZero     = intMin + intMaxWidth           // 136
+	intSmall    = intMax - intZero - intMaxWidth // 109
+	// intMax is the maximum int tag value.
+	intMax = 0xfd // 253
+)
+
+// encodeUvarintAscending encodes the uint64 value using a variable length
+// (length-prefixed) representation. The length is encoded as a single
+// byte indicating the number of encoded bytes (-8) to follow. See
+// EncodeVarintAscending for rationale. The encoded bytes are appended to the
+// supplied buffer and the final buffer is returned.
+func encodeUvarintAscending(b []byte, v uint64) []byte {
+	switch {
+	case v <= intSmall:
+		return append(b, intZero+byte(v))
+	case v <= 0xff:
+		return append(b, intMax-7, byte(v))
+	case v <= 0xffff:
+		return append(b, intMax-6, byte(v>>8), byte(v))
+	case v <= 0xffffff:
+		return append(b, intMax-5, byte(v>>16), byte(v>>8), byte(v))
+	case v <= 0xffffffff:
+		return append(b, intMax-4, byte(v>>24), byte(v>>16), byte(v>>8), byte(v))
+	case v <= 0xffffffffff:
+		return append(b, intMax-3, byte(v>>32), byte(v>>24), byte(v>>16), byte(v>>8),
+			byte(v))
+	case v <= 0xffffffffffff:
+		return append(b, intMax-2, byte(v>>40), byte(v>>32), byte(v>>24), byte(v>>16),
+			byte(v>>8), byte(v))
+	case v <= 0xffffffffffffff:
+		return append(b, intMax-1, byte(v>>48), byte(v>>40), byte(v>>32), byte(v>>24),
+			byte(v>>16), byte(v>>8), byte(v))
+	default:
+		return append(b, intMax, byte(v>>56), byte(v>>48), byte(v>>40), byte(v>>32),
+			byte(v>>24), byte(v>>16), byte(v>>8), byte(v))
+	}
+}
+
+// decodeUvarintAscending decodes a varint encoded uint64 from the input
+// buffer. The remainder of the input buffer and the decoded uint64
+// are returned.
+func decodeUvarintAscending(b []byte) ([]byte, uint64, error) {
+	if len(b) == 0 {
+		return nil, 0, fmt.Errorf("insufficient bytes to decode uvarint value")
+	}
+	length := int(b[0]) - intZero
+	b = b[1:] // skip length byte
+	if length <= intSmall {
+		return b, uint64(length), nil
+	}
+	length -= intSmall
+	if length < 0 || length > 8 {
+		return nil, 0, fmt.Errorf("invalid uvarint length of %d", length)
+	} else if len(b) < length {
+		return nil, 0, fmt.Errorf("insufficient bytes to decode uvarint value: %q", b)
+	}
+	var v uint64
+	// It is faster to range over the elements in a slice than to index
+	// into the slice on each loop iteration.
+	for _, t := range b[:length] {
+		v = (v << 8) | uint64(t)
+	}
+	return b[length:], v, nil
+}

index/scorch/int_test.go

@@ -0,0 +1,96 @@
+// Copyright 2014 The Cockroach Authors.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+// implied. See the License for the specific language governing
+// permissions and limitations under the License.
+
+// This code originated from:
+// https://github.com/cockroachdb/cockroach/blob/2dd65dde5d90c157f4b93f92502ca1063b904e1d/pkg/util/encoding/encoding_test.go
+
+// Modified to only test the parts we borrowed
+
+package scorch
+
+import (
+	"bytes"
+	"math"
+	"testing"
+)
+
+type testCaseUint64 struct {
+	value  uint64
+	expEnc []byte
+}
+
+func TestEncodeDecodeUvarint(t *testing.T) {
+	testBasicEncodeDecodeUint64(encodeUvarintAscending, decodeUvarintAscending, false, t)
+	testCases := []testCaseUint64{
+		{0, []byte{0x88}},
+		{1, []byte{0x89}},
+		{109, []byte{0xf5}},
+		{110, []byte{0xf6, 0x6e}},
+		{1 << 8, []byte{0xf7, 0x01, 0x00}},
+		{math.MaxUint64, []byte{0xfd, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff}},
+	}
+	testCustomEncodeUint64(testCases, encodeUvarintAscending, t)
+}
+
+func testBasicEncodeDecodeUint64(
+	encFunc func([]byte, uint64) []byte,
+	decFunc func([]byte) ([]byte, uint64, error),
+	descending bool, t *testing.T,
+) {
+	testCases := []uint64{
+		0, 1,
+		1<<8 - 1, 1 << 8,
+		1<<16 - 1, 1 << 16,
+		1<<24 - 1, 1 << 24,
+		1<<32 - 1, 1 << 32,
+		1<<40 - 1, 1 << 40,
+		1<<48 - 1, 1 << 48,
+		1<<56 - 1, 1 << 56,
+		math.MaxUint64 - 1, math.MaxUint64,
+	}
+
+	var lastEnc []byte
+	for i, v := range testCases {
+		enc := encFunc(nil, v)
+		if i > 0 {
+			if (descending && bytes.Compare(enc, lastEnc) >= 0) ||
+				(!descending && bytes.Compare(enc, lastEnc) < 0) {
+				t.Errorf("ordered constraint violated for %d: [% x] vs. [% x]", v, enc, lastEnc)
+			}
+		}
+		b, decode, err := decFunc(enc)
+		if err != nil {
+			t.Error(err)
+			continue
+		}
+		if len(b) != 0 {
+			t.Errorf("leftover bytes: [% x]", b)
+		}
+		if decode != v {
+			t.Errorf("decode yielded different value than input: %d vs. %d", decode, v)
+		}
+		lastEnc = enc
+	}
+}
+
+func testCustomEncodeUint64(
+	testCases []testCaseUint64, encFunc func([]byte, uint64) []byte, t *testing.T,
+) {
+	for _, test := range testCases {
+		enc := encFunc(nil, test.value)
+		if !bytes.Equal(enc, test.expEnc) {
+			t.Errorf("expected [% x]; got [% x] (value: %d)", test.expEnc, enc, test.value)
+		}
+	}
+}

index/scorch/introducer.go

@@ -0,0 +1,532 @@
+//  Copyright (c) 2017 Couchbase, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// 		http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package scorch
+
+import (
+	"fmt"
+	"path/filepath"
+	"sync/atomic"
+
+	"github.com/RoaringBitmap/roaring/v2"
+	index "github.com/blevesearch/bleve_index_api"
+	segment "github.com/blevesearch/scorch_segment_api/v2"
+)
+
+const introducer = "introducer"
+
+type segmentIntroduction struct {
+	id        uint64
+	data      segment.Segment
+	obsoletes map[uint64]*roaring.Bitmap
+	ids       []string
+	internal  map[string][]byte
+
+	applied           chan error
+	persisted         chan error
+	persistedCallback index.BatchCallback
+}
+
+type persistIntroduction struct {
+	persisted map[uint64]segment.Segment
+	applied   notificationChan
+}
+
+type epochWatcher struct {
+	epoch    uint64
+	notifyCh notificationChan
+}
+
+func (s *Scorch) introducerLoop() {
+	defer func() {
+		if r := recover(); r != nil {
+			s.fireAsyncError(NewScorchError(
+				introducer,
+				fmt.Sprintf("panic: %v, path: %s", r, s.path),
+				ErrAsyncPanic,
+			))
+		}
+
+		s.asyncTasks.Done()
+	}()
+
+	var epochWatchers []*epochWatcher
+OUTER:
+	for {
+		atomic.AddUint64(&s.stats.TotIntroduceLoop, 1)
+
+		select {
+		case <-s.closeCh:
+			break OUTER
+
+		case epochWatcher := <-s.introducerNotifier:
+			epochWatchers = append(epochWatchers, epochWatcher)
+
+		case nextMerge := <-s.merges:
+			s.introduceMerge(nextMerge)
+
+		case next := <-s.introductions:
+			err := s.introduceSegment(next)
+			if err != nil {
+				continue OUTER
+			}
+
+		case persist := <-s.persists:
+			s.introducePersist(persist)
+
+		}
+
+		var epochCurr uint64
+		s.rootLock.RLock()
+		if s.root != nil {
+			epochCurr = s.root.epoch
+		}
+		s.rootLock.RUnlock()
+		var epochWatchersNext []*epochWatcher
+		for _, w := range epochWatchers {
+			if w.epoch < epochCurr {
+				close(w.notifyCh)
+			} else {
+				epochWatchersNext = append(epochWatchersNext, w)
+			}
+		}
+		epochWatchers = epochWatchersNext
+	}
+}
+
+func (s *Scorch) introduceSegment(next *segmentIntroduction) error {
+	atomic.AddUint64(&s.stats.TotIntroduceSegmentBeg, 1)
+	defer atomic.AddUint64(&s.stats.TotIntroduceSegmentEnd, 1)
+
+	s.rootLock.RLock()
+	root := s.root
+	root.AddRef()
+	s.rootLock.RUnlock()
+
+	defer func() { _ = root.DecRef() }()
+
+	nsegs := len(root.segment)
+
+	// prepare new index snapshot
+	newSnapshot := &IndexSnapshot{
+		parent:   s,
+		segment:  make([]*SegmentSnapshot, 0, nsegs+1),
+		offsets:  make([]uint64, 0, nsegs+1),
+		internal: make(map[string][]byte, len(root.internal)),
+		refs:     1,
+		creator:  "introduceSegment",
+	}
+
+	// iterate through current segments
+	var running uint64
+	var docsToPersistCount, memSegments, fileSegments uint64
+	var droppedSegmentFiles []string
+	for i := range root.segment {
+		// see if optimistic work included this segment
+		delta, ok := next.obsoletes[root.segment[i].id]
+		if !ok {
+			var err error
+			delta, err = root.segment[i].segment.DocNumbers(next.ids)
+			if err != nil {
+				next.applied <- fmt.Errorf("error computing doc numbers: %v", err)
+				close(next.applied)
+				_ = newSnapshot.DecRef()
+				return err
+			}
+		}
+
+		newss := &SegmentSnapshot{
+			id:         root.segment[i].id,
+			segment:    root.segment[i].segment,
+			stats:      root.segment[i].stats,
+			cachedDocs: root.segment[i].cachedDocs,
+			cachedMeta: root.segment[i].cachedMeta,
+			creator:    root.segment[i].creator,
+		}
+
+		// apply new obsoletions
+		if root.segment[i].deleted == nil {
+			newss.deleted = delta
+		} else {
+			newss.deleted = roaring.Or(root.segment[i].deleted, delta)
+		}
+		if newss.deleted.IsEmpty() {
+			newss.deleted = nil
+		}
+
+		// update the deleted bitmap to include any nested/sub-documents as well
+		// if the segment supports that
+		if ns, ok := newss.segment.(segment.NestedSegment); ok {
+			newss.deleted = ns.AddNestedDocuments(newss.deleted)
+		}
+		// check for live size before copying
+		if newss.LiveSize() > 0 {
+			newSnapshot.segment = append(newSnapshot.segment, newss)
+			root.segment[i].segment.AddRef()
+			newSnapshot.offsets = append(newSnapshot.offsets, running)
+			running += newss.segment.Count()
+		} else if seg, ok := newss.segment.(segment.PersistedSegment); ok {
+			droppedSegmentFiles = append(droppedSegmentFiles,
+				filepath.Base(seg.Path()))
+		}
+
+		if isMemorySegment(root.segment[i]) {
+			docsToPersistCount += root.segment[i].Count()
+			memSegments++
+		} else {
+			fileSegments++
+		}
+	}
+
+	atomic.StoreUint64(&s.stats.TotItemsToPersist, docsToPersistCount)
+	atomic.StoreUint64(&s.stats.TotMemorySegmentsAtRoot, memSegments)
+	atomic.StoreUint64(&s.stats.TotFileSegmentsAtRoot, fileSegments)
+
+	// append new segment, if any, to end of the new index snapshot
+	if next.data != nil {
+		stats := newFieldStats()
+		if fsr, ok := next.data.(segment.FieldStatsReporter); ok {
+			fsr.UpdateFieldStats(stats)
+		}
+		newSegmentSnapshot := &SegmentSnapshot{
+			id:         next.id,
+			segment:    next.data, // take ownership of next.data's ref-count
+			stats:      stats,
+			cachedDocs: &cachedDocs{cache: nil},
+			cachedMeta: newCachedMeta(),
+			creator:    "introduceSegment",
+		}
+		newSnapshot.segment = append(newSnapshot.segment, newSegmentSnapshot)
+		newSnapshot.offsets = append(newSnapshot.offsets, running)
+
+		// increment numItemsIntroduced which tracks the number of items
+		// queued for persistence.
+		atomic.AddUint64(&s.stats.TotIntroducedItems, newSegmentSnapshot.Count())
+		atomic.AddUint64(&s.stats.TotIntroducedSegmentsBatch, 1)
+	}
+	// copy old values
+	for key, oldVal := range root.internal {
+		newSnapshot.internal[key] = oldVal
+	}
+	// set new values and apply deletes
+	for key, newVal := range next.internal {
+		if newVal != nil {
+			newSnapshot.internal[key] = newVal
+		} else {
+			delete(newSnapshot.internal, key)
+		}
+	}
+
+	newSnapshot.updateSize()
+	s.rootLock.Lock()
+	if next.persisted != nil {
+		s.rootPersisted = append(s.rootPersisted, next.persisted)
+	}
+	if next.persistedCallback != nil {
+		s.persistedCallbacks = append(s.persistedCallbacks, next.persistedCallback)
+	}
+	// swap in new index snapshot
+	newSnapshot.epoch = s.nextSnapshotEpoch
+	s.nextSnapshotEpoch++
+	rootPrev := s.root
+	s.root = newSnapshot
+	atomic.StoreUint64(&s.stats.CurRootEpoch, s.root.epoch)
+	// release lock
+	s.rootLock.Unlock()
+
+	if rootPrev != nil {
+		_ = rootPrev.DecRef()
+	}
+
+	// update the removal eligibility for those segment files
+	// that are not a part of the latest root.
+	for _, filename := range droppedSegmentFiles {
+		s.unmarkIneligibleForRemoval(filename)
+	}
+
+	close(next.applied)
+
+	return nil
+}
+
+func (s *Scorch) introducePersist(persist *persistIntroduction) {
+	atomic.AddUint64(&s.stats.TotIntroducePersistBeg, 1)
+	defer atomic.AddUint64(&s.stats.TotIntroducePersistEnd, 1)
+
+	s.rootLock.Lock()
+	root := s.root
+	root.AddRef()
+	nextSnapshotEpoch := s.nextSnapshotEpoch
+	s.nextSnapshotEpoch++
+	s.rootLock.Unlock()
+
+	defer func() { _ = root.DecRef() }()
+
+	newIndexSnapshot := &IndexSnapshot{
+		parent:   s,
+		epoch:    nextSnapshotEpoch,
+		segment:  make([]*SegmentSnapshot, len(root.segment)),
+		offsets:  make([]uint64, len(root.offsets)),
+		internal: make(map[string][]byte, len(root.internal)),
+		refs:     1,
+		creator:  "introducePersist",
+	}
+
+	var docsToPersistCount, memSegments, fileSegments uint64
+	for i, segmentSnapshot := range root.segment {
+		// see if this segment has been replaced
+		if replacement, ok := persist.persisted[segmentSnapshot.id]; ok {
+			newSegmentSnapshot := &SegmentSnapshot{
+				id:         segmentSnapshot.id,
+				segment:    replacement,
+				deleted:    segmentSnapshot.deleted,
+				stats:      segmentSnapshot.stats,
+				cachedDocs: segmentSnapshot.cachedDocs,
+				cachedMeta: segmentSnapshot.cachedMeta,
+				creator:    "introducePersist",
+				mmaped:     1,
+			}
+			newIndexSnapshot.segment[i] = newSegmentSnapshot
+			delete(persist.persisted, segmentSnapshot.id)
+
+			// update items persisted in case of a new segment snapshot
+			atomic.AddUint64(&s.stats.TotPersistedItems, newSegmentSnapshot.Count())
+			atomic.AddUint64(&s.stats.TotPersistedSegments, 1)
+			fileSegments++
+		} else {
+			newIndexSnapshot.segment[i] = root.segment[i]
+			newIndexSnapshot.segment[i].segment.AddRef()
+
+			if isMemorySegment(root.segment[i]) {
+				docsToPersistCount += root.segment[i].Count()
+				memSegments++
+			} else {
+				fileSegments++
+			}
+		}
+		newIndexSnapshot.offsets[i] = root.offsets[i]
+	}
+
+	for k, v := range root.internal {
+		newIndexSnapshot.internal[k] = v
+	}
+
+	atomic.StoreUint64(&s.stats.TotItemsToPersist, docsToPersistCount)
+	atomic.StoreUint64(&s.stats.TotMemorySegmentsAtRoot, memSegments)
+	atomic.StoreUint64(&s.stats.TotFileSegmentsAtRoot, fileSegments)
+	newIndexSnapshot.updateSize()
+	s.rootLock.Lock()
+	rootPrev := s.root
+	s.root = newIndexSnapshot
+	atomic.StoreUint64(&s.stats.CurRootEpoch, s.root.epoch)
+	s.rootLock.Unlock()
+
+	if rootPrev != nil {
+		_ = rootPrev.DecRef()
+	}
+
+	close(persist.applied)
+}
+
+// The introducer should definitely handle the segmentMerge.notify
+// channel before exiting the introduceMerge.
+func (s *Scorch) introduceMerge(nextMerge *segmentMerge) {
+	atomic.AddUint64(&s.stats.TotIntroduceMergeBeg, 1)
+	defer atomic.AddUint64(&s.stats.TotIntroduceMergeEnd, 1)
+
+	s.rootLock.RLock()
+	root := s.root
+	root.AddRef()
+	s.rootLock.RUnlock()
+
+	defer func() { _ = root.DecRef() }()
+
+	newSnapshot := &IndexSnapshot{
+		parent:   s,
+		internal: root.internal,
+		refs:     1,
+		creator:  "introduceMerge",
+	}
+
+	var running, docsToPersistCount, memSegments, fileSegments uint64
+	var droppedSegmentFiles []string
+	newSegmentsDeleted := make([]*roaring.Bitmap, len(nextMerge.newSegments))
+	for i := range newSegmentsDeleted {
+		// create a bitmaps to track the obsoletes per newly merged segments
+		newSegmentsDeleted[i] = roaring.NewBitmap()
+	}
+
+	// iterate through current segments
+	for i := range root.segment {
+		segmentID := root.segment[i].id
+		if history, ok := nextMerge.mergedSegHistory[segmentID]; ok {
+			// this segment is going away, see if anything else was deleted since we started the merge
+			if history != nil && root.segment[i].deleted != nil {
+				// assume all these deletes are new
+				deletedSince := root.segment[i].deleted
+				// if we already knew about some of them, remove
+				if history.oldSegment.deleted != nil {
+					deletedSince = roaring.AndNot(root.segment[i].deleted, history.oldSegment.deleted)
+				}
+				deletedSinceItr := deletedSince.Iterator()
+				for deletedSinceItr.HasNext() {
+					oldDocNum := deletedSinceItr.Next()
+					newDocNum := history.oldNewDocIDs[oldDocNum]
+					newSegmentsDeleted[history.batchID].Add(uint32(newDocNum))
+				}
+			}
+
+			// clean up the old segment map to figure out the
+			// obsolete segments wrt root in meantime, whatever
+			// segments left behind in old map after processing
+			// the root segments would be the obsolete segment set
+			delete(nextMerge.mergedSegHistory, segmentID)
+		} else if root.segment[i].LiveSize() > 0 {
+			// this segment is staying
+			newSnapshot.segment = append(newSnapshot.segment, &SegmentSnapshot{
+				id:         root.segment[i].id,
+				segment:    root.segment[i].segment,
+				deleted:    root.segment[i].deleted,
+				stats:      root.segment[i].stats,
+				cachedDocs: root.segment[i].cachedDocs,
+				cachedMeta: root.segment[i].cachedMeta,
+				creator:    root.segment[i].creator,
+			})
+			root.segment[i].segment.AddRef()
+			newSnapshot.offsets = append(newSnapshot.offsets, running)
+			running += root.segment[i].segment.Count()
+
+			if isMemorySegment(root.segment[i]) {
+				docsToPersistCount += root.segment[i].Count()
+				memSegments++
+			} else {
+				fileSegments++
+			}
+		} else if root.segment[i].LiveSize() == 0 {
+			if seg, ok := root.segment[i].segment.(segment.PersistedSegment); ok {
+				droppedSegmentFiles = append(droppedSegmentFiles,
+					filepath.Base(seg.Path()))
+			}
+		}
+	}
+	// before the newMerge introduction, need to clean the newly
+	// merged segment wrt the current root segments, hence
+	// applying the obsolete segment contents to newly merged segment
+	for _, ss := range nextMerge.mergedSegHistory {
+		obsoleted := ss.oldSegment.DocNumbersLive()
+		if obsoleted != nil {
+			obsoletedIter := obsoleted.Iterator()
+			for obsoletedIter.HasNext() {
+				oldDocNum := obsoletedIter.Next()
+				newDocNum := ss.oldNewDocIDs[oldDocNum]
+				newSegmentsDeleted[ss.batchID].Add(uint32(newDocNum))
+			}
+		}
+	}
+
+	skipped := make([]bool, len(nextMerge.newSegments))
+	// make the newly merged segments part of the newSnapshot being constructed
+	for i, newMergedSegment := range nextMerge.newSegments {
+		if newMergedSegment == nil {
+			if nextMerge.fileMerge {
+				atomic.AddUint64(&s.stats.TotFileMergeIntroductionsSkipped, 1)
+			} else {
+				atomic.AddUint64(&s.stats.TotMemMergeIntroductionsSkipped, 1)
+			}
+			skipped[i] = true
+			continue
+		}
+		// checking if this newly merged segment is worth keeping based on
+		// obsoleted doc count since the merge intro started
+		if newMergedSegment.Count() > newSegmentsDeleted[i].GetCardinality() {
+			stats := newFieldStats()
+			if fsr, ok := newMergedSegment.(segment.FieldStatsReporter); ok {
+				fsr.UpdateFieldStats(stats)
+			}
+
+			// put the merged segment at the end of newSnapshot
+			newSnapshot.segment = append(newSnapshot.segment, &SegmentSnapshot{
+				id:         nextMerge.newSegmentIDs[i],
+				segment:    newMergedSegment, // take ownership for newMergedSegment's ref-count
+				deleted:    newSegmentsDeleted[i],
+				stats:      stats,
+				cachedDocs: &cachedDocs{cache: nil},
+				cachedMeta: newCachedMeta(),
+				creator:    "introduceMerge",
+				mmaped:     nextMerge.mmaped,
+			})
+			newSnapshot.offsets = append(newSnapshot.offsets, running)
+			running += newMergedSegment.Count()
+
+			switch newMergedSegment.(type) {
+			case segment.PersistedSegment:
+				fileSegments++
+			default:
+				docsToPersistCount += newMergedSegment.Count() - newSegmentsDeleted[i].GetCardinality()
+				memSegments++
+			}
+			atomic.AddUint64(&s.stats.TotIntroducedSegmentsMerge, 1)
+			skipped[i] = false
+		} else {
+			if nextMerge.fileMerge {
+				atomic.AddUint64(&s.stats.TotFileMergeIntroductionsObsoleted, 1)
+			} else {
+				atomic.AddUint64(&s.stats.TotMemMergeIntroductionsObsoleted, 1)
+			}
+			skipped[i] = true
+		}
+	}
+
+	atomic.StoreUint64(&s.stats.TotItemsToPersist, docsToPersistCount)
+	atomic.StoreUint64(&s.stats.TotMemorySegmentsAtRoot, memSegments)
+	atomic.StoreUint64(&s.stats.TotFileSegmentsAtRoot, fileSegments)
+
+	newSnapshot.AddRef() // 1 ref for the nextMerge.notify response
+
+	newSnapshot.updateSize()
+	s.rootLock.Lock()
+	// swap in new index snapshot
+	newSnapshot.epoch = s.nextSnapshotEpoch
+	s.nextSnapshotEpoch++
+	rootPrev := s.root
+	s.root = newSnapshot
+	atomic.StoreUint64(&s.stats.CurRootEpoch, s.root.epoch)
+	// release lock
+	s.rootLock.Unlock()
+
+	if rootPrev != nil {
+		_ = rootPrev.DecRef()
+	}
+
+	// update the removal eligibility for those segment files
+	// that are not a part of the latest root.
+	for _, filename := range droppedSegmentFiles {
+		s.unmarkIneligibleForRemoval(filename)
+	}
+
+	// notify requester that we incorporated this
+	nextMerge.notifyCh <- &mergeTaskIntroStatus{
+		indexSnapshot: newSnapshot,
+		skipped:       skipped}
+	close(nextMerge.notifyCh)
+}
+
+func isMemorySegment(s *SegmentSnapshot) bool {
+	switch s.segment.(type) {
+	case segment.PersistedSegment:
+		return false
+	default:
+		return true
+	}
+}

index/scorch/merge.go

@@ -0,0 +1,730 @@
+//  Copyright (c) 2017 Couchbase, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// 		http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package scorch
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"os"
+	"strings"
+	"sync"
+	"sync/atomic"
+	"time"
+
+	"github.com/RoaringBitmap/roaring/v2"
+	"github.com/blevesearch/bleve/v2/index/scorch/mergeplan"
+	"github.com/blevesearch/bleve/v2/util"
+	segment "github.com/blevesearch/scorch_segment_api/v2"
+)
+
+const merger = "merger"
+
+// used in the context of mergerCtrl to provide a way to verify
+// the completion of a merge operation
+const mergeDoneKey = "mergeDone"
+
+type mergeDoneChan chan error
+
+// used in the context of mergerCtrl to provide a way to use
+// a custom merge plan instead of the one generated by the
+// default merge planner
+const mergePlanFuncKey = "mergePlanFunc"
+
+type mergePlanFunc func(*IndexSnapshot) (*mergeplan.MergePlan, error)
+
+func (s *Scorch) mergerLoop() {
+	defer func() {
+		if r := recover(); r != nil {
+			s.fireAsyncError(NewScorchError(
+				merger,
+				fmt.Sprintf("panic: %v, path: %s", r, s.path),
+				ErrAsyncPanic,
+			))
+		}
+
+		s.asyncTasks.Done()
+	}()
+
+	var lastEpochMergePlanned uint64
+	var ctrlMsg *mergerCtrl
+	ctrlMsgDflt := &mergerCtrl{ctx: context.Background(),
+		options: s.mergePlannerOptions,
+		doneCh:  nil}
+
+OUTER:
+	for {
+		atomic.AddUint64(&s.stats.TotFileMergeLoopBeg, 1)
+
+		select {
+		case <-s.closeCh:
+			break OUTER
+
+		default:
+			// check to see if there is a new snapshot to persist
+			s.rootLock.Lock()
+			ourSnapshot := s.root
+			ourSnapshot.AddRef()
+			atomic.StoreUint64(&s.iStats.mergeSnapshotSize, uint64(ourSnapshot.Size()))
+			atomic.StoreUint64(&s.iStats.mergeEpoch, ourSnapshot.epoch)
+			s.rootLock.Unlock()
+
+			if ctrlMsg == nil && ourSnapshot.epoch != lastEpochMergePlanned {
+				ctrlMsg = ctrlMsgDflt
+			}
+			if ctrlMsg != nil {
+				continueMerge := s.fireEvent(EventKindPreMergeCheck, 0)
+				// The default, if there's no handler, is to continue the merge.
+				if !continueMerge {
+					// If it's decided that this merge can't take place now,
+					// begin the merge process all over again.
+					// Retry instead of blocking/waiting here since a long wait
+					// can result in more segments introduced i.e. s.root will
+					// be updated.
+
+					// decrement the ref count since its no longer needed in this
+					// iteration
+					_ = ourSnapshot.DecRef()
+					continue OUTER
+				}
+
+				// lets get started
+				startTime := time.Now()
+				err := s.planMergeAtSnapshot(ctrlMsg, ourSnapshot)
+				if err != nil {
+					atomic.StoreUint64(&s.iStats.mergeEpoch, 0)
+					if err == segment.ErrClosed {
+						// index has been closed
+						_ = ourSnapshot.DecRef()
+
+						// continue the workloop on a user triggered cancel
+						if ctrlMsg.doneCh != nil {
+							close(ctrlMsg.doneCh)
+							ctrlMsg = nil
+							continue OUTER
+						}
+
+						// exit the workloop on index closure
+						ctrlMsg = nil
+						break OUTER
+					}
+
+					s.fireAsyncError(NewScorchError(
+						merger,
+						fmt.Sprintf("merging err: %v", err),
+						ErrPersist,
+					))
+					_ = ourSnapshot.DecRef()
+					atomic.AddUint64(&s.stats.TotFileMergeLoopErr, 1)
+					continue OUTER
+				}
+
+				if ctrlMsg.doneCh != nil {
+					close(ctrlMsg.doneCh)
+				}
+				ctrlMsg = nil
+
+				lastEpochMergePlanned = ourSnapshot.epoch
+
+				atomic.StoreUint64(&s.stats.LastMergedEpoch, ourSnapshot.epoch)
+
+				s.fireEvent(EventKindMergerProgress, time.Since(startTime))
+			}
+			_ = ourSnapshot.DecRef()
+
+			// tell the persister we're waiting for changes
+			// first make a epochWatcher chan
+			ew := &epochWatcher{
+				epoch:    lastEpochMergePlanned,
+				notifyCh: make(notificationChan, 1),
+			}
+
+			// give it to the persister
+			select {
+			case <-s.closeCh:
+				break OUTER
+			case s.persisterNotifier <- ew:
+			case ctrlMsg = <-s.forceMergeRequestCh:
+				continue OUTER
+			}
+
+			// now wait for persister (but also detect close)
+			select {
+			case <-s.closeCh:
+				break OUTER
+			case <-ew.notifyCh:
+			case ctrlMsg = <-s.forceMergeRequestCh:
+			}
+		}
+
+		atomic.AddUint64(&s.stats.TotFileMergeLoopEnd, 1)
+	}
+}
+
+type mergerCtrl struct {
+	ctx     context.Context
+	options *mergeplan.MergePlanOptions
+	doneCh  chan struct{}
+}
+
+// ForceMerge helps users trigger a merge operation on
+// an online scorch index.
+func (s *Scorch) ForceMerge(ctx context.Context,
+	mo *mergeplan.MergePlanOptions) error {
+	// check whether force merge is already under processing
+	s.rootLock.Lock()
+	if atomic.LoadUint64(&s.stats.TotFileMergeForceOpsStarted) >
+		atomic.LoadUint64(&s.stats.TotFileMergeForceOpsCompleted) {
+		s.rootLock.Unlock()
+		return fmt.Errorf("force merge already in progress")
+	}
+
+	atomic.AddUint64(&s.stats.TotFileMergeForceOpsStarted, 1)
+	s.rootLock.Unlock()
+
+	if mo != nil {
+		err := mergeplan.ValidateMergePlannerOptions(mo)
+		if err != nil {
+			return err
+		}
+	} else {
+		// assume the default single segment merge policy
+		mo = &mergeplan.SingleSegmentMergePlanOptions
+	}
+	msg := &mergerCtrl{options: mo,
+		doneCh: make(chan struct{}),
+		ctx:    ctx,
+	}
+
+	// request the merger perform a force merge
+	select {
+	case s.forceMergeRequestCh <- msg:
+	case <-s.closeCh:
+		return nil
+	}
+
+	// wait for the force merge operation completion
+	select {
+	case <-msg.doneCh:
+		atomic.AddUint64(&s.stats.TotFileMergeForceOpsCompleted, 1)
+	case <-s.closeCh:
+	}
+
+	return nil
+}
+
+func (s *Scorch) parseMergePlannerOptions(po *persisterOptions) (*mergeplan.MergePlanOptions,
+	error) {
+	mergePlannerOptions := mergeplan.DefaultMergePlanOptions
+	// by default use the MaxSizeInMemoryMergePerWorker from the persister option
+	// as the FloorSegmentFileSize for the merge planner which would be the
+	// first tier size in the planning. If the value is 0, then we don't use the
+	// file size in the planning.
+	mergePlannerOptions.FloorSegmentFileSize = int64(po.MaxSizeInMemoryMergePerWorker)
+
+	if v, ok := s.config["scorchMergePlanOptions"]; ok {
+		b, err := util.MarshalJSON(v)
+		if err != nil {
+			return &mergePlannerOptions, err
+		}
+
+		err = util.UnmarshalJSON(b, &mergePlannerOptions)
+		if err != nil {
+			return &mergePlannerOptions, err
+		}
+
+		err = mergeplan.ValidateMergePlannerOptions(&mergePlannerOptions)
+		if err != nil {
+			return nil, err
+		}
+	}
+	return &mergePlannerOptions, nil
+}
+
+type closeChWrapper struct {
+	ch1      chan struct{}
+	ctx      context.Context
+	closeCh  chan struct{}
+	cancelCh chan struct{}
+}
+
+func newCloseChWrapper(ch1 chan struct{},
+	ctx context.Context) *closeChWrapper {
+	return &closeChWrapper{
+		ch1:      ch1,
+		ctx:      ctx,
+		closeCh:  make(chan struct{}),
+		cancelCh: make(chan struct{}),
+	}
+}
+
+func (w *closeChWrapper) close() {
+	close(w.closeCh)
+}
+
+func (w *closeChWrapper) listen() {
+	select {
+	case <-w.ch1:
+		close(w.cancelCh)
+	case <-w.ctx.Done():
+		close(w.cancelCh)
+	case <-w.closeCh:
+	}
+}
+
+// mergeBatch represents a batch of segments to be merged together,
+// along with the new segment that will be created as a result of the merge.
+type mergeBatch struct {
+	snapshots []*SegmentSnapshot
+	filenames []string
+	segments  []segment.Segment
+	drops     []*roaring.Bitmap
+
+	new         segment.Segment
+	newID       uint64
+	newDocNums  [][]uint64
+	newFilename string
+	newTime     uint64
+}
+
+// planMergeAtSnapshot plans and executes the merge operations for a given snapshot
+// if there is a custom merge plan function provided, it uses that to get the merge plan
+// otherwise, it builds the merge plan using the default planner and executes the merge tasks in the plan.
+func (s *Scorch) planMergeAtSnapshot(ctrlMsg *mergerCtrl, ourSnapshot *IndexSnapshot) error {
+	var mergePlan *mergeplan.MergePlan
+	// if a merge plan function is provided in the context, use it to get the merge plan
+	if mergePlanFunc, ok := ctrlMsg.ctx.Value(mergePlanFuncKey).(mergePlanFunc); ok {
+		var err error
+		mergePlan, err = mergePlanFunc(ourSnapshot)
+		if err != nil {
+			atomic.AddUint64(&s.stats.TotFileMergePlanErr, 1)
+			return fmt.Errorf("merge planning err: %v", err)
+		}
+	}
+
+	// default to making a merge plan if a custom one is not provided
+	if mergePlan == nil || len(mergePlan.Tasks) == 0 {
+		// build list of persisted segments in this snapshot
+		var onlyPersistedSnapshots []mergeplan.Segment
+		for _, segmentSnapshot := range ourSnapshot.segment {
+			if _, ok := segmentSnapshot.segment.(segment.PersistedSegment); ok {
+				onlyPersistedSnapshots = append(onlyPersistedSnapshots, segmentSnapshot)
+			}
+		}
+
+		atomic.AddUint64(&s.stats.TotFileMergePlan, 1)
+
+		// give this list to the planner
+		var err error
+		mergePlan, err = mergeplan.Plan(onlyPersistedSnapshots, ctrlMsg.options)
+		if err != nil {
+			atomic.AddUint64(&s.stats.TotFileMergePlanErr, 1)
+			return fmt.Errorf("merge planning err: %v", err)
+		}
+
+		if mergePlan == nil || len(mergePlan.Tasks) == 0 {
+			// nothing to do
+			atomic.AddUint64(&s.stats.TotFileMergePlanNone, 1)
+			return nil
+		}
+	}
+
+	atomic.AddUint64(&s.stats.TotFileMergePlanOk, 1)
+	atomic.AddUint64(&s.stats.TotFileMergePlanTasks, uint64(len(mergePlan.Tasks)))
+
+	cw := newCloseChWrapper(s.closeCh, ctrlMsg.ctx)
+	defer cw.close()
+	go cw.listen()
+
+	var err error
+	defer func() {
+		// send error to done channel if present
+		if done, ok := cw.ctx.Value(mergeDoneKey).(chan error); ok {
+			done <- err
+		}
+	}()
+
+	numBatches := len(mergePlan.Tasks)
+	mergeBatches := make([]*mergeBatch, numBatches)
+	defer func() {
+		for batchID := 0; batchID < numBatches; batchID++ {
+			batch := mergeBatches[batchID]
+			if batch == nil {
+				continue
+			}
+			if err == nil {
+				s.rootLock.Lock()
+				for _, filename := range batch.filenames {
+					delete(s.ineligibleForRemoval, filename)
+				}
+				s.rootLock.Unlock()
+			} else {
+				if batch.new != nil {
+					_ = batch.new.Close()
+				}
+				s.unmarkIneligibleForRemoval(batch.newFilename)
+			}
+		}
+	}()
+
+	numMergedSegments := 0
+	for batchID := 0; batchID < numBatches; batchID++ {
+		task := mergePlan.Tasks[batchID]
+		if len(task.Segments) == 0 {
+			atomic.AddUint64(&s.stats.TotFileMergePlanTasksSegmentsEmpty, 1)
+		}
+		atomic.AddUint64(&s.stats.TotFileMergePlanTasksSegments, uint64(len(task.Segments)))
+
+		newSegmentID := atomic.AddUint64(&s.nextSegmentID, 1)
+		batch := &mergeBatch{
+			snapshots: make([]*SegmentSnapshot, 0, len(task.Segments)),
+			filenames: make([]string, 0, len(task.Segments)),
+			segments:  make([]segment.Segment, 0, len(task.Segments)),
+			drops:     make([]*roaring.Bitmap, 0, len(task.Segments)),
+
+			new:         nil,
+			newID:       newSegmentID,
+			newDocNums:  nil,
+			newFilename: zapFileName(newSegmentID),
+			newTime:     0,
+		}
+		mergeBatches[batchID] = batch
+
+		for _, planSegment := range task.Segments {
+			if segSnapshot, ok := planSegment.(*SegmentSnapshot); ok {
+				if persistedSeg, ok := segSnapshot.segment.(segment.PersistedSegment); ok {
+					if segSnapshot.LiveSize() == 0 {
+						atomic.AddUint64(&s.stats.TotFileMergeSegmentsEmpty, 1)
+					} else {
+						batch.snapshots = append(batch.snapshots, segSnapshot)
+						batch.segments = append(batch.segments, segSnapshot.segment)
+						batch.drops = append(batch.drops, segSnapshot.deleted)
+					}
+					// track the files getting merged for unsetting the
+					// removal ineligibility. This helps to unflip files
+					// even with fast merger, slow persister work flows.
+					path := persistedSeg.Path()
+					batch.filenames = append(batch.filenames,
+						strings.TrimPrefix(path, s.path+string(os.PathSeparator)))
+				}
+			}
+		}
+
+		if len(batch.segments) == 0 {
+			continue
+		}
+
+		numMergedSegments += len(batch.snapshots)
+		s.markIneligibleForRemoval(batch.newFilename)
+		path := s.path + string(os.PathSeparator) + batch.newFilename
+
+		prevBytesReadTotal := cumulateBytesRead(batch.segments)
+		fileMergeZapStartTime := time.Now()
+		atomic.AddUint64(&s.stats.TotFileMergeZapBeg, 1)
+		batch.newDocNums, _, err = s.segPlugin.MergeUsing(batch.segments, batch.drops, path,
+			cw.cancelCh, s, s.segmentConfig)
+		atomic.AddUint64(&s.stats.TotFileMergeZapEnd, 1)
+		batch.newTime = uint64(time.Since(fileMergeZapStartTime))
+		atomic.AddUint64(&s.stats.TotFileMergeZapTime, batch.newTime)
+		if atomic.LoadUint64(&s.stats.MaxFileMergeZapTime) < batch.newTime {
+			atomic.StoreUint64(&s.stats.MaxFileMergeZapTime, batch.newTime)
+		}
+		if err != nil {
+			atomic.AddUint64(&s.stats.TotFileMergePlanTasksErr, 1)
+			return err
+		}
+
+		batch.new, err = s.segPlugin.OpenUsing(path, s.segmentConfig)
+		if err != nil {
+			atomic.AddUint64(&s.stats.TotFileMergePlanTasksErr, 1)
+			return err
+		}
+
+		totalBytesRead := batch.new.BytesRead() + prevBytesReadTotal
+		batch.new.ResetBytesRead(totalBytesRead)
+		atomic.AddUint64(&s.stats.TotFileMergeSegments, uint64(len(batch.segments)))
+	}
+
+	newSegmentIDs := make([]uint64, numBatches)
+	newSegments := make([]segment.Segment, numBatches)
+	mergedSegHistory := make(map[uint64]*mergedSegmentHistory, numMergedSegments)
+	for batchID := 0; batchID < numBatches; batchID++ {
+		batch := mergeBatches[batchID]
+		newSegmentIDs[batchID] = batch.newID
+		newSegments[batchID] = batch.new
+		for j, ss := range batch.snapshots {
+			mergedSegHistory[ss.id] = &mergedSegmentHistory{
+				batchID:      batchID,
+				oldNewDocIDs: batch.newDocNums[j],
+				oldSegment:   ss,
+			}
+		}
+	}
+
+	sm := &segmentMerge{
+		newSegmentIDs:    newSegmentIDs,
+		newSegments:      newSegments,
+		mergedSegHistory: mergedSegHistory,
+		notifyCh:         make(chan *mergeTaskIntroStatus),
+		mmaped:           1,
+		fileMerge:        true,
+	}
+
+	s.fireEvent(EventKindMergeTaskIntroductionStart, 0)
+	// give it to the introducer
+	select {
+	case <-s.closeCh:
+		err = segment.ErrClosed
+		return err
+	case s.merges <- sm:
+		atomic.AddUint64(&s.stats.TotFileMergeIntroductions, uint64(numBatches))
+	}
+
+	introStartTime := time.Now()
+	// it is safe to blockingly wait for the merge introduction
+	// here as the introducer is bound to handle the notify channel.
+	introStatus := <-sm.notifyCh
+	introTime := uint64(time.Since(introStartTime))
+	atomic.AddUint64(&s.stats.TotFileMergeZapIntroductionTime, introTime)
+	if atomic.LoadUint64(&s.stats.MaxFileMergeZapIntroductionTime) < introTime {
+		atomic.StoreUint64(&s.stats.MaxFileMergeZapIntroductionTime, introTime)
+	}
+	atomic.AddUint64(&s.stats.TotFileMergeIntroductionsDone, uint64(numBatches))
+
+	_ = introStatus.indexSnapshot.DecRef()
+	for batchID, skipped := range introStatus.skipped {
+		if skipped {
+			batch := mergeBatches[batchID]
+			if batch.new != nil {
+				_ = batch.new.Close()
+			}
+			s.unmarkIneligibleForRemoval(batch.newFilename)
+		}
+	}
+
+	atomic.AddUint64(&s.stats.TotFileMergePlanTasksDone, uint64(numBatches))
+	s.fireEvent(EventKindMergeTaskIntroduction, 0)
+	return nil
+}
+
+type mergeTaskIntroStatus struct {
+	indexSnapshot *IndexSnapshot
+	skipped       []bool
+}
+
+// this is important when it comes to introducing multiple merged segments in a
+// single introducer channel push. That way there is a check to ensure that the
+// file count doesn't explode during the index's lifetime.
+type mergedSegmentHistory struct {
+	batchID      int
+	oldNewDocIDs []uint64
+	oldSegment   *SegmentSnapshot
+}
+
+type segmentMerge struct {
+	newSegmentIDs    []uint64
+	newSegments      []segment.Segment
+	mergedSegHistory map[uint64]*mergedSegmentHistory
+	notifyCh         chan *mergeTaskIntroStatus
+	mmaped           uint32
+	fileMerge        bool
+}
+
+func cumulateBytesRead(sbs []segment.Segment) uint64 {
+	var rv uint64
+	for _, seg := range sbs {
+		rv += seg.BytesRead()
+	}
+	return rv
+}
+
+func (s *Scorch) mergeAndPersistInMemorySegments(flushes []*flushable, po *persisterOptions) (*IndexSnapshot, map[uint64]struct{}, error) {
+	atomic.AddUint64(&s.stats.TotMemMergeBeg, 1)
+
+	var err error
+	numBatches := len(flushes)
+	mergeBatches := make([]*mergeBatch, numBatches)
+	defer func() {
+		if err != nil {
+			for batchID := 0; batchID < numBatches; batchID++ {
+				batch := mergeBatches[batchID]
+				if batch.new != nil {
+					_ = batch.new.Close()
+				}
+				s.unmarkIneligibleForRemoval(batch.newFilename)
+			}
+		}
+	}()
+
+	var numMergedSegments uint64
+	sem := make(chan struct{}, po.NumPersisterWorkers)
+	var wg sync.WaitGroup
+	var errM sync.Mutex
+	var errs []error
+	for batchID := 0; batchID < numBatches; batchID++ {
+		wg.Add(1)
+		sem <- struct{}{}
+		go func(batchID int) {
+			var err error
+			defer func() {
+				if err != nil {
+					atomic.AddUint64(&s.stats.TotMemMergeErr, 1)
+					errM.Lock()
+					errs = append(errs, err)
+					errM.Unlock()
+				}
+				<-sem
+				wg.Done()
+			}()
+
+			flush := flushes[batchID]
+			newSegmentID := atomic.AddUint64(&s.nextSegmentID, 1)
+			batch := &mergeBatch{
+				snapshots: flush.sbsBatchSnapshots,
+				filenames: nil,
+				segments:  flush.sbsBatch,
+				drops:     flush.sbsBatchDrops,
+
+				new:         nil,
+				newID:       newSegmentID,
+				newDocNums:  nil,
+				newFilename: zapFileName(newSegmentID),
+				newTime:     0,
+			}
+			mergeBatches[batchID] = batch
+
+			s.markIneligibleForRemoval(batch.newFilename)
+			path := s.path + string(os.PathSeparator) + batch.newFilename
+
+			memMergeZapStartTime := time.Now()
+			atomic.AddUint64(&s.stats.TotMemMergeZapBeg, 1)
+			batch.newDocNums, _, err = s.segPlugin.MergeUsing(batch.segments, batch.drops, path,
+				s.closeCh, s, s.segmentConfig)
+			atomic.AddUint64(&s.stats.TotMemMergeZapEnd, 1)
+			batch.newTime = uint64(time.Since(memMergeZapStartTime))
+			atomic.AddUint64(&s.stats.TotMemMergeZapTime, batch.newTime)
+			if err != nil {
+				return
+			}
+			batch.new, err = s.segPlugin.OpenUsing(path, s.segmentConfig)
+			if err != nil {
+				return
+			}
+			atomic.AddUint64(&numMergedSegments, uint64(len(batch.segments)))
+		}(batchID)
+	}
+	wg.Wait()
+	close(sem)
+
+	var maxZapTime uint64
+	for _, batch := range mergeBatches {
+		if batch.newTime > maxZapTime {
+			maxZapTime = batch.newTime
+		}
+	}
+	if maxZapTime > atomic.LoadUint64(&s.stats.MaxMemMergeZapTime) {
+		atomic.StoreUint64(&s.stats.MaxMemMergeZapTime, maxZapTime)
+	}
+	atomic.AddUint64(&s.stats.TotMemMergeSegments, numMergedSegments)
+
+	if len(errs) > 0 {
+		var errf error
+		numClosed := 0
+		for _, e := range errs {
+			if e == segment.ErrClosed {
+				numClosed++
+			}
+			errf = errors.Join(errf, e)
+		}
+		if numClosed == len(errs) {
+			// the index snapshot was closed which will be handled gracefully
+			// by retrying the whole merge+flush operation in a later iteration
+			// so its safe to early exit the same error.
+			err = segment.ErrClosed
+		} else {
+			err = errf
+		}
+		return nil, nil, err
+	}
+
+	newSegmentIDs := make([]uint64, numBatches)
+	newSegments := make([]segment.Segment, numBatches)
+	mergedSegHistory := make(map[uint64]*mergedSegmentHistory, numMergedSegments)
+	for batchID := 0; batchID < numBatches; batchID++ {
+		batch := mergeBatches[batchID]
+		newSegmentIDs[batchID] = batch.newID
+		newSegments[batchID] = batch.new
+		for j, ss := range batch.snapshots {
+			mergedSegHistory[ss.id] = &mergedSegmentHistory{
+				batchID:      batchID,
+				oldNewDocIDs: batch.newDocNums[j],
+				oldSegment:   ss,
+			}
+		}
+	}
+
+	sm := &segmentMerge{
+		newSegmentIDs:    newSegmentIDs,
+		newSegments:      newSegments,
+		mergedSegHistory: mergedSegHistory,
+		notifyCh:         make(chan *mergeTaskIntroStatus),
+		mmaped:           1,
+		fileMerge:        false,
+	}
+
+	select {
+	case <-s.closeCh:
+		err = segment.ErrClosed
+		return nil, nil, err
+	case s.merges <- sm:
+		atomic.AddUint64(&s.stats.TotMemMergeIntroductions, uint64(numBatches))
+	}
+
+	introStartTime := time.Now()
+	// blockingly wait for the introduction to complete
+	introStatus := <-sm.notifyCh
+	introTime := uint64(time.Since(introStartTime))
+	atomic.AddUint64(&s.stats.TotMemMergeZapIntroductionTime, introTime)
+	if atomic.LoadUint64(&s.stats.MaxMemMergeZapIntroductionTime) < introTime {
+		atomic.StoreUint64(&s.stats.MaxMemMergeZapIntroductionTime, introTime)
+	}
+	atomic.AddUint64(&s.stats.TotMemMergeIntroductionsDone, uint64(numBatches))
+	introducedSnapshot := introStatus.indexSnapshot
+	introducedSegmentIDs := make(map[uint64]struct{}, numBatches)
+	for batchID, skipped := range introStatus.skipped {
+		batch := mergeBatches[batchID]
+		if skipped {
+			_ = batch.new.Close()
+			s.unmarkIneligibleForRemoval(batch.newFilename)
+		} else {
+			introducedSegmentIDs[batch.newID] = struct{}{}
+		}
+	}
+
+	atomic.AddUint64(&s.stats.TotMemMergeDone, 1)
+
+	// if we could not introduce all of the newly merged segments,
+	// then we cannot persist a snapshot with any merged segments at all
+	if len(introducedSegmentIDs) != numBatches {
+		_ = introducedSnapshot.DecRef()
+		return nil, nil, nil
+	}
+
+	return introducedSnapshot, introducedSegmentIDs, nil
+}
+
+func (s *Scorch) ReportBytesWritten(bytesWritten uint64) {
+	atomic.AddUint64(&s.stats.TotFileMergeWrittenBytes, bytesWritten)
+}

index/scorch/merge_test.go

@@ -0,0 +1,157 @@
+//  Copyright (c) 2020 Couchbase, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// 		http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package scorch
+
+import (
+	"sync"
+	"sync/atomic"
+	"testing"
+
+	"github.com/blevesearch/bleve/v2/document"
+	index "github.com/blevesearch/bleve_index_api"
+)
+
+func TestObsoleteSegmentMergeIntroduction(t *testing.T) {
+	testConfig := CreateConfig("TestObsoleteSegmentMergeIntroduction")
+	err := InitTest(testConfig)
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer func() {
+		err := DestroyTest(testConfig)
+		if err != nil {
+			t.Fatal(err)
+		}
+	}()
+
+	var introComplete, mergeIntroStart, mergeIntroComplete sync.WaitGroup
+	introComplete.Add(1)
+	mergeIntroStart.Add(1)
+	mergeIntroComplete.Add(1)
+	var segIntroCompleted int
+	RegistryEventCallbacks["test"] = func(e Event) bool {
+		switch e.Kind {
+		case EventKindBatchIntroduction:
+			segIntroCompleted++
+			if segIntroCompleted == 3 {
+				// all 3 segments introduced
+				introComplete.Done()
+			}
+		case EventKindMergeTaskIntroductionStart:
+			// signal the start of merge task introduction so that
+			// we can introduce a new batch which obsoletes the
+			// merged segment's contents.
+			mergeIntroStart.Done()
+			// hold the merge task introduction until the merged segment contents
+			// are obsoleted with the next batch/segment introduction.
+			introComplete.Wait()
+		case EventKindMergeTaskIntroduction:
+			// signal the completion of the merge task introduction.
+			mergeIntroComplete.Done()
+
+		}
+
+		return true
+	}
+
+	ourConfig := make(map[string]interface{}, len(testConfig))
+	for k, v := range testConfig {
+		ourConfig[k] = v
+	}
+	ourConfig["eventCallbackName"] = "test"
+
+	analysisQueue := index.NewAnalysisQueue(1)
+	idx, err := NewScorch(Name, ourConfig, analysisQueue)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	err = idx.Open()
+	if err != nil {
+		t.Fatalf("error opening index: %v", err)
+	}
+	defer func() {
+		err := idx.Close()
+		if err != nil {
+			t.Fatal(err)
+		}
+	}()
+
+	// first introduce two documents over two batches.
+	batch := index.NewBatch()
+	doc := document.NewDocument("1")
+	doc.AddField(document.NewTextField("name", []uint64{}, []byte("test3")))
+	batch.Update(doc)
+	err = idx.Batch(batch)
+	if err != nil {
+		t.Error(err)
+	}
+
+	batch.Reset()
+	doc = document.NewDocument("2")
+	doc.AddField(document.NewTextField("name", []uint64{}, []byte("test2updated")))
+	batch.Update(doc)
+	err = idx.Batch(batch)
+	if err != nil {
+		t.Error(err)
+	}
+
+	// wait until the merger trying to introduce the new merged segment.
+	mergeIntroStart.Wait()
+
+	// execute another batch which obsoletes the contents of the new merged
+	// segment awaiting introduction.
+	batch.Reset()
+	batch.Delete("1")
+	batch.Delete("2")
+	doc = document.NewDocument("3")
+	doc.AddField(document.NewTextField("name", []uint64{}, []byte("test3updated")))
+	batch.Update(doc)
+	err = idx.Batch(batch)
+	if err != nil {
+		t.Error(err)
+	}
+
+	// wait until the merge task introduction complete.
+	mergeIntroComplete.Wait()
+
+	idxr, err := idx.Reader()
+	if err != nil {
+		t.Error(err)
+	}
+
+	numSegments := len(idxr.(*IndexSnapshot).segment)
+	if numSegments != 1 {
+		t.Errorf("expected one segment at the root, got: %d", numSegments)
+	}
+
+	skipIntroCount := atomic.LoadUint64(&idxr.(*IndexSnapshot).parent.stats.TotFileMergeIntroductionsObsoleted)
+	if skipIntroCount != 1 {
+		t.Errorf("expected one obsolete merge segment skipping the introduction, got: %d", skipIntroCount)
+	}
+
+	docCount, err := idxr.DocCount()
+	if err != nil {
+		t.Fatal(err)
+	}
+	if docCount != 1 {
+		t.Errorf("Expected document count to be %d got %d", 1, docCount)
+	}
+
+	err = idxr.Close()
+	if err != nil {
+		t.Fatal(err)
+	}
+}

index/scorch/mergeplan/merge_plan.go

@@ -0,0 +1,542 @@
+//  Copyright (c) 2017 Couchbase, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// 		http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package mergeplan provides a segment merge planning approach that's
+// inspired by Lucene's TieredMergePolicy.java and descriptions like
+// http://blog.mikemccandless.com/2011/02/visualizing-lucenes-segment-merges.html
+package mergeplan
+
+import (
+	"errors"
+	"fmt"
+	"math"
+	"sort"
+	"strings"
+)
+
+// A Segment represents the information that the planner needs to
+// calculate segment merging.
+type Segment interface {
+	// Unique id of the segment -- used for sorting.
+	Id() uint64
+
+	// Full segment size (the size before any logical deletions).
+	FullSize() int64
+
+	// Size of the live data of the segment; i.e., FullSize() minus
+	// any logical deletions.
+	LiveSize() int64
+
+	HasVector() bool
+
+	// Size of the persisted segment file.
+	FileSize() int64
+
+	// Size of the persisted segment file
+	// accounting for any logical deletions.
+	LiveFileSize() int64
+}
+
+// Plan() will functionally compute a merge plan.  A segment will be
+// assigned to at most a single MergeTask in the output MergePlan.  A
+// segment not assigned to any MergeTask means the segment should
+// remain unmerged.
+func Plan(segments []Segment, o *MergePlanOptions) (*MergePlan, error) {
+	return plan(segments, o)
+}
+
+// A MergePlan is the result of the Plan() API.
+//
+// The planner doesn't know how or whether these tasks are executed --
+// that's up to a separate merge execution system, which might execute
+// these tasks concurrently or not, and which might execute all the
+// tasks or not.
+type MergePlan struct {
+	Tasks []*MergeTask
+}
+
+// A MergeTask represents several segments that should be merged
+// together into a single segment.
+type MergeTask struct {
+	Segments []Segment
+}
+
+// The MergePlanOptions is designed to be reusable between planning calls.
+type MergePlanOptions struct {
+	// Max # segments per logarithmic tier, or max width of any
+	// logarithmic "step".  Smaller values mean more merging but fewer
+	// segments.  Should be >= SegmentsPerMergeTask, else you'll have
+	// too much merging.
+	MaxSegmentsPerTier int
+
+	// Max size of any segment produced after merging.  Actual
+	// merging, however, may produce segment sizes different than the
+	// planner's predicted sizes.
+	MaxSegmentSize int64
+
+	// Max size (in bytes) of the persisted segment file that contains the
+	// vectors.  This is used to prevent merging of segments that
+	// contain vectors that are too large.
+	MaxSegmentFileSize int64
+
+	// The growth factor for each tier in a staircase of idealized
+	// segments computed by CalcBudget().
+	TierGrowth float64
+
+	// The number of segments in any resulting MergeTask.  e.g.,
+	// len(result.Tasks[ * ].Segments) == SegmentsPerMergeTask.
+	SegmentsPerMergeTask int
+
+	// Small segments are rounded up to this size, i.e., treated as
+	// equal (floor) size for consideration.  This is to prevent lots
+	// of tiny segments from resulting in a long tail in the index.
+	FloorSegmentSize int64
+
+	// Small segments' file size are rounded up to this size to prevent lot
+	// of tiny segments causing a long tail in the index.
+	FloorSegmentFileSize int64
+
+	// Controls how aggressively merges that reclaim more deletions
+	// are favored.  Higher values will more aggressively target
+	// merges that reclaim deletions, but be careful not to go so high
+	// that way too much merging takes place; a value of 3.0 is
+	// probably nearly too high.  A value of 0.0 means deletions don't
+	// impact merge selection.
+	ReclaimDeletesWeight float64
+
+	// Optional, defaults to mergeplan.CalcBudget().
+	CalcBudget func(totalSize int64, firstTierSize int64,
+		o *MergePlanOptions) (budgetNumSegments int)
+
+	// Optional, defaults to mergeplan.ScoreSegments().
+	ScoreSegments func(segments []Segment, o *MergePlanOptions) float64
+
+	// Optional.
+	Logger func(string)
+}
+
+// Returns the higher of the input or FloorSegmentSize.
+func (o *MergePlanOptions) RaiseToFloorSegmentSize(s int64) int64 {
+	if s > o.FloorSegmentSize {
+		return s
+	}
+	return o.FloorSegmentSize
+}
+
+func (o *MergePlanOptions) RaiseToFloorSegmentFileSize(s int64) int64 {
+	if s > o.FloorSegmentFileSize {
+		return s
+	}
+	return o.FloorSegmentFileSize
+}
+
+func (o *MergePlanOptions) BudgetCurrency() BudgetCurrency {
+	if o.FloorSegmentFileSize > 0 {
+		return FileSizeBudget
+	}
+	return LiveSizeBudget
+}
+
+// MaxSegmentSizeLimit represents the maximum size of a segment,
+// this limit comes with hit-1 optimisation/max encoding limit uint31.
+const MaxSegmentSizeLimit = 1<<31 - 1
+
+// ErrMaxSegmentSizeTooLarge is returned when the size of the segment
+// exceeds the MaxSegmentSizeLimit
+var ErrMaxSegmentSizeTooLarge = errors.New("MaxSegmentSize exceeds the size limit")
+
+// DefaultMergePlanOptions suggests the default options.
+var DefaultMergePlanOptions = MergePlanOptions{
+	MaxSegmentsPerTier:   10,
+	MaxSegmentSize:       5000000,
+	MaxSegmentFileSize:   4 * 1024 * 1024 * 1024, // 4GiB
+	TierGrowth:           10.0,
+	SegmentsPerMergeTask: 10,
+	FloorSegmentSize:     2000,
+	ReclaimDeletesWeight: 2.0,
+}
+
+// SingleSegmentMergePlanOptions helps in creating a
+// single segment index.
+var SingleSegmentMergePlanOptions = MergePlanOptions{
+	MaxSegmentsPerTier:   1,
+	MaxSegmentSize:       1 << 30,
+	MaxSegmentFileSize:   1 << 40,
+	TierGrowth:           1.0,
+	SegmentsPerMergeTask: 10,
+	FloorSegmentSize:     1 << 30,
+	ReclaimDeletesWeight: 2.0,
+	FloorSegmentFileSize: 1 << 40,
+}
+
+// -------------------------------------------
+
+// BudgetCurrency is the currency used for budget
+// calculations in the merge planner.
+type BudgetCurrency int
+
+const (
+	// FileSizeBudget uses the file size
+	// of the segments to determine the budget.
+	FileSizeBudget BudgetCurrency = iota
+
+	// LiveSizeBudget uses the live size
+	// of the segments to determine the budget.
+	LiveSizeBudget
+)
+
+var ErrUnknownCurrency = errors.New("unknown budget currency")
+
+// -------------------------------------------
+
+// rosterCandidate is a potential roster of segments considered while
+// planning, along with its computed score.
+type rosterCandidate struct {
+	segments []Segment
+	score    float64 // Lower is better.
+}
+
+func (c *rosterCandidate) betterThan(other *rosterCandidate) bool {
+	ourSegmentCount := c.count()
+	otherSegmentCount := other.count()
+	// if we have more than one segment and the other has only one, and that
+	// one segment is fully live, then we are better than the other.
+	if ourSegmentCount > 1 && otherSegmentCount == 1 {
+		solo := other.segments[0]
+		if solo.LiveSize() == solo.FullSize() {
+			return true
+		}
+	}
+	// if we have only one segment and the other has more than one, and that
+	// one segment is fully live, then we are worse than the other.
+	if ourSegmentCount == 1 && otherSegmentCount > 1 {
+		solo := c.segments[0]
+		if solo.LiveSize() == solo.FullSize() {
+			return false
+		}
+	}
+	return c.score < other.score
+}
+
+func (c *rosterCandidate) count() int {
+	return len(c.segments)
+}
+
+// -------------------------------------------
+
+func plan(segmentsIn []Segment, o *MergePlanOptions) (*MergePlan, error) {
+	if len(segmentsIn) <= 1 {
+		return nil, nil
+	}
+
+	if o == nil {
+		o = &DefaultMergePlanOptions
+	}
+
+	segments := append([]Segment(nil), segmentsIn...) // Copy.
+
+	sort.Sort(byLiveSizeDescending(segments))
+
+	var minLiveSize int64 = math.MaxInt64
+	var minLiveFileSize int64 = math.MaxInt64
+
+	var eligibles []Segment
+	var eligiblesLiveSize int64
+	var eligiblesLiveFileSize int64
+
+	for _, segment := range segments {
+		liveSize := segment.LiveSize()
+		liveFileSize := segment.LiveFileSize()
+
+		if minLiveSize > liveSize {
+			minLiveSize = liveSize
+		}
+
+		if minLiveFileSize > liveFileSize {
+			minLiveFileSize = liveFileSize
+		}
+
+		isEligible := liveSize < o.MaxSegmentSize/2
+		// An eligible segment (based on #documents) may be too large
+		// and thus need a stricter check based on the file size.
+		// This is particularly important for segments that contain
+		// vectors.
+		if segment.HasVector() {
+			isEligible = isEligible && liveFileSize < o.MaxSegmentFileSize/2
+		}
+
+		// Only small-enough segments are eligible.
+		if isEligible {
+			eligibles = append(eligibles, segment)
+			eligiblesLiveSize += liveSize
+			eligiblesLiveFileSize += liveFileSize
+		}
+	}
+
+	calcBudget := o.CalcBudget
+	if calcBudget == nil {
+		calcBudget = CalcBudget
+	}
+
+	currency := o.BudgetCurrency()
+
+	var budgetNumSegments int
+	switch currency {
+	case FileSizeBudget:
+		minLiveFileSize = o.RaiseToFloorSegmentFileSize(minLiveFileSize)
+		budgetNumSegments = calcBudget(eligiblesLiveFileSize, minLiveFileSize, o)
+	case LiveSizeBudget:
+		minLiveSize = o.RaiseToFloorSegmentSize(minLiveSize)
+		budgetNumSegments = calcBudget(eligiblesLiveSize, minLiveSize, o)
+	default:
+		return nil, ErrUnknownCurrency
+	}
+
+	scoreSegments := o.ScoreSegments
+	if scoreSegments == nil {
+		scoreSegments = ScoreSegments
+	}
+
+	rv := &MergePlan{}
+
+	var empties []Segment
+	for _, eligible := range eligibles {
+		if eligible.LiveSize() <= 0 {
+			empties = append(empties, eligible)
+		}
+	}
+	if len(empties) > 0 {
+		rv.Tasks = append(rv.Tasks, &MergeTask{Segments: empties})
+		eligibles = removeSegments(eligibles, empties)
+	}
+
+	numMergeTasks := 0
+
+	// While we're over budget, keep looping, which might produce
+	// another MergeTask.
+	for len(eligibles) > 0 && (len(eligibles)+numMergeTasks) > budgetNumSegments {
+		// Track the current best candidate as we examine and score
+		// potential rosters of merges.
+		var best *rosterCandidate
+
+		for startIdx := 0; startIdx < len(eligibles); startIdx++ {
+			var roster []Segment
+			var rosterLiveSize int64
+			var rosterLiveFileSize int64
+
+			for idx := startIdx; idx < len(eligibles) && len(roster) < o.SegmentsPerMergeTask; idx++ {
+				eligible := eligibles[idx]
+
+				liveSize := eligible.LiveSize()
+
+				if rosterLiveSize+liveSize >= o.MaxSegmentSize {
+					continue
+				}
+
+				if eligible.HasVector() {
+					liveFileSize := eligible.LiveFileSize()
+
+					if rosterLiveFileSize+liveFileSize >= o.MaxSegmentFileSize {
+						continue
+					}
+
+					rosterLiveFileSize += liveFileSize
+				}
+
+				roster = append(roster, eligible)
+
+				rosterLiveSize += liveSize
+			}
+
+			if len(roster) > 0 {
+				rosterScore := scoreSegments(roster, o)
+
+				candidate := &rosterCandidate{
+					segments: roster,
+					score:    rosterScore,
+				}
+
+				if best == nil || candidate.betterThan(best) {
+					best = candidate
+				}
+			}
+		}
+
+		if best == nil {
+			return rv, nil
+		}
+
+		bestRoster := best.segments
+
+		// create tasks with valid merges - i.e. there should be at least 2 non-empty segments
+		// or the bestRoster should have 1 segment with deletes to reclaim.
+		if len(bestRoster) > 1 || bestRoster[0].LiveSize() < bestRoster[0].FullSize() {
+			rv.Tasks = append(rv.Tasks, &MergeTask{Segments: bestRoster})
+			numMergeTasks++
+		}
+
+		eligibles = removeSegments(eligibles, bestRoster)
+	}
+
+	return rv, nil
+}
+
+// Compute the number of segments that would be needed to cover the
+// totalSize, by climbing up a logarithmically growing staircase of
+// segment tiers.
+func CalcBudget(totalSize int64, firstTierSize int64, o *MergePlanOptions) (
+	budgetNumSegments int) {
+	tierSize := firstTierSize
+	if tierSize < 1 {
+		tierSize = 1
+	}
+
+	maxSegmentsPerTier := o.MaxSegmentsPerTier
+	if maxSegmentsPerTier < 1 {
+		maxSegmentsPerTier = 1
+	}
+
+	tierGrowth := o.TierGrowth
+	if tierGrowth < 1.0 {
+		tierGrowth = 1.0
+	}
+
+	for totalSize > 0 {
+		segmentsInTier := float64(totalSize) / float64(tierSize)
+		if segmentsInTier < float64(maxSegmentsPerTier) {
+			budgetNumSegments += int(math.Ceil(segmentsInTier))
+			break
+		}
+
+		budgetNumSegments += maxSegmentsPerTier
+		totalSize -= int64(maxSegmentsPerTier) * tierSize
+		tierSize = int64(float64(tierSize) * tierGrowth)
+	}
+
+	return budgetNumSegments
+}
+
+// Of note, removeSegments() keeps the ordering of the results stable.
+func removeSegments(segments []Segment, toRemove []Segment) []Segment {
+	rv := make([]Segment, 0, len(segments)-len(toRemove))
+OUTER:
+	for _, segment := range segments {
+		for _, r := range toRemove {
+			if segment == r {
+				continue OUTER
+			}
+		}
+		rv = append(rv, segment)
+	}
+	return rv
+}
+
+// Smaller result score is better.
+func ScoreSegments(segments []Segment, o *MergePlanOptions) float64 {
+	var totBeforeSize int64
+	var totAfterSize int64
+	var totAfterSizeFloored int64
+
+	for _, segment := range segments {
+		totBeforeSize += segment.FullSize()
+		totAfterSize += segment.LiveSize()
+		totAfterSizeFloored += o.RaiseToFloorSegmentSize(segment.LiveSize())
+	}
+
+	if totBeforeSize <= 0 || totAfterSize <= 0 || totAfterSizeFloored <= 0 {
+		return 0
+	}
+
+	// Roughly guess the "balance" of the segments -- whether the
+	// segments are about the same size.
+	balance :=
+		float64(o.RaiseToFloorSegmentSize(segments[0].LiveSize())) /
+			float64(totAfterSizeFloored)
+
+	// Gently favor smaller merges over bigger ones.  We don't want to
+	// make the exponent too large else we end up with poor merges of
+	// small segments in order to avoid the large merges.
+	score := balance * math.Pow(float64(totAfterSize), 0.05)
+
+	// Strongly favor merges that reclaim deletes.
+	nonDelRatio := float64(totAfterSize) / float64(totBeforeSize)
+
+	score *= math.Pow(nonDelRatio, o.ReclaimDeletesWeight)
+
+	return score
+}
+
+// ------------------------------------------
+
+// ToBarChart returns an ASCII rendering of the segments and the plan.
+// The barMax is the max width of the bars in the bar chart.
+func ToBarChart(prefix string, barMax int, segments []Segment, plan *MergePlan) string {
+	rv := make([]string, 0, len(segments))
+
+	var maxFullSize int64
+	for _, segment := range segments {
+		if maxFullSize < segment.FullSize() {
+			maxFullSize = segment.FullSize()
+		}
+	}
+	if maxFullSize < 0 {
+		maxFullSize = 1
+	}
+
+	for _, segment := range segments {
+		barFull := int(segment.FullSize())
+		barLive := int(segment.LiveSize())
+
+		if maxFullSize > int64(barMax) {
+			barFull = int(float64(barMax) * float64(barFull) / float64(maxFullSize))
+			barLive = int(float64(barMax) * float64(barLive) / float64(maxFullSize))
+		}
+
+		barKind := " "
+		barChar := "."
+
+		if plan != nil {
+		TASK_LOOP:
+			for taski, task := range plan.Tasks {
+				for _, taskSegment := range task.Segments {
+					if taskSegment == segment {
+						barKind = "*"
+						barChar = fmt.Sprintf("%d", taski)
+						break TASK_LOOP
+					}
+				}
+			}
+		}
+
+		bar :=
+			strings.Repeat(barChar, barLive)[0:barLive] +
+				strings.Repeat("x", barFull-barLive)[0:barFull-barLive]
+
+		rv = append(rv, fmt.Sprintf("%s %5d: %5d /%5d - %s %s", prefix,
+			segment.Id(),
+			segment.LiveSize(),
+			segment.FullSize(),
+			barKind, bar))
+	}
+
+	return strings.Join(rv, "\n")
+}
+
+// ValidateMergePlannerOptions validates the merge planner options
+func ValidateMergePlannerOptions(options *MergePlanOptions) error {
+	if options.MaxSegmentSize > MaxSegmentSizeLimit {
+		return ErrMaxSegmentSizeTooLarge
+	}
+	return nil
+}

index/scorch/mergeplan/sort.go

@@ -0,0 +1,28 @@
+//  Copyright (c) 2017 Couchbase, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// 		http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package mergeplan
+
+type byLiveSizeDescending []Segment
+
+func (a byLiveSizeDescending) Len() int { return len(a) }
+
+func (a byLiveSizeDescending) Swap(i, j int) { a[i], a[j] = a[j], a[i] }
+
+func (a byLiveSizeDescending) Less(i, j int) bool {
+	if a[i].LiveSize() != a[j].LiveSize() {
+		return a[i].LiveSize() > a[j].LiveSize()
+	}
+	return a[i].Id() < a[j].Id()
+}

index/scorch/optimize.go

@@ -0,0 +1,391 @@
+//  Copyright (c) 2018 Couchbase, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// 		http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package scorch
+
+import (
+	"fmt"
+	"sync/atomic"
+
+	"github.com/RoaringBitmap/roaring/v2"
+	index "github.com/blevesearch/bleve_index_api"
+	segment "github.com/blevesearch/scorch_segment_api/v2"
+)
+
+var OptimizeConjunction = true
+var OptimizeConjunctionUnadorned = true
+var OptimizeDisjunctionUnadorned = true
+
+func (s *IndexSnapshotTermFieldReader) Optimize(kind string,
+	octx index.OptimizableContext) (index.OptimizableContext, error) {
+	if OptimizeConjunction && kind == "conjunction" {
+		return s.optimizeConjunction(octx)
+	}
+
+	if OptimizeConjunctionUnadorned && kind == "conjunction:unadorned" {
+		return s.optimizeConjunctionUnadorned(octx)
+	}
+
+	if OptimizeDisjunctionUnadorned && kind == "disjunction:unadorned" {
+		return s.optimizeDisjunctionUnadorned(octx)
+	}
+
+	return nil, nil
+}
+
+var OptimizeDisjunctionUnadornedMinChildCardinality = uint64(256)
+
+// ----------------------------------------------------------------
+
+func (s *IndexSnapshotTermFieldReader) optimizeConjunction(
+	octx index.OptimizableContext) (index.OptimizableContext, error) {
+	if octx == nil {
+		octx = &OptimizeTFRConjunction{snapshot: s.snapshot}
+	}
+
+	o, ok := octx.(*OptimizeTFRConjunction)
+	if !ok {
+		return octx, nil
+	}
+
+	if o.snapshot != s.snapshot {
+		return nil, fmt.Errorf("tried to optimize conjunction across different snapshots")
+	}
+
+	o.tfrs = append(o.tfrs, s)
+
+	return o, nil
+}
+
+type OptimizeTFRConjunction struct {
+	snapshot *IndexSnapshot
+
+	tfrs []*IndexSnapshotTermFieldReader
+}
+
+func (o *OptimizeTFRConjunction) Finish() (index.Optimized, error) {
+	if len(o.tfrs) <= 1 {
+		return nil, nil
+	}
+
+	for i := range o.snapshot.segment {
+		itr0, ok := o.tfrs[0].iterators[i].(segment.OptimizablePostingsIterator)
+		if !ok || itr0.ActualBitmap() == nil {
+			continue
+		}
+
+		itr1, ok := o.tfrs[1].iterators[i].(segment.OptimizablePostingsIterator)
+		if !ok || itr1.ActualBitmap() == nil {
+			continue
+		}
+
+		bm := roaring.And(itr0.ActualBitmap(), itr1.ActualBitmap())
+
+		for _, tfr := range o.tfrs[2:] {
+			itr, ok := tfr.iterators[i].(segment.OptimizablePostingsIterator)
+			if !ok || itr.ActualBitmap() == nil {
+				continue
+			}
+
+			bm.And(itr.ActualBitmap())
+		}
+
+		// in this conjunction optimization, the postings iterators
+		// will all share the same AND'ed together actual bitmap.  The
+		// regular conjunction searcher machinery will still be used,
+		// but the underlying bitmap will be smaller.
+		for _, tfr := range o.tfrs {
+			itr, ok := tfr.iterators[i].(segment.OptimizablePostingsIterator)
+			if ok && itr.ActualBitmap() != nil {
+				itr.ReplaceActual(bm)
+			}
+		}
+	}
+
+	return nil, nil
+}
+
+// ----------------------------------------------------------------
+
+// An "unadorned" conjunction optimization is appropriate when
+// additional or subsidiary information like freq-norm's and
+// term-vectors are not required, and instead only the internal-id's
+// are needed.
+func (s *IndexSnapshotTermFieldReader) optimizeConjunctionUnadorned(
+	octx index.OptimizableContext) (index.OptimizableContext, error) {
+	if octx == nil {
+		octx = &OptimizeTFRConjunctionUnadorned{snapshot: s.snapshot}
+	}
+
+	o, ok := octx.(*OptimizeTFRConjunctionUnadorned)
+	if !ok {
+		return nil, nil
+	}
+
+	if o.snapshot != s.snapshot {
+		return nil, fmt.Errorf("tried to optimize unadorned conjunction across different snapshots")
+	}
+
+	o.tfrs = append(o.tfrs, s)
+
+	return o, nil
+}
+
+type OptimizeTFRConjunctionUnadorned struct {
+	snapshot *IndexSnapshot
+
+	tfrs []*IndexSnapshotTermFieldReader
+}
+
+var OptimizeTFRConjunctionUnadornedTerm = []byte("<conjunction:unadorned>")
+var OptimizeTFRConjunctionUnadornedField = "*"
+
+// Finish of an unadorned conjunction optimization will compute a
+// termFieldReader with an "actual" bitmap that represents the
+// constituent bitmaps AND'ed together.  This termFieldReader cannot
+// provide any freq-norm or termVector associated information.
+func (o *OptimizeTFRConjunctionUnadorned) Finish() (rv index.Optimized, err error) {
+	if len(o.tfrs) <= 1 {
+		return nil, nil
+	}
+
+	// We use an artificial term and field because the optimized
+	// termFieldReader can represent multiple terms and fields.
+	oTFR := o.snapshot.unadornedTermFieldReader(
+		OptimizeTFRConjunctionUnadornedTerm, OptimizeTFRConjunctionUnadornedField)
+
+	var actualBMs []*roaring.Bitmap // Collected from regular posting lists.
+
+OUTER:
+	for i := range o.snapshot.segment {
+		actualBMs = actualBMs[:0]
+
+		var docNum1HitLast uint64
+		var docNum1HitLastOk bool
+
+		for _, tfr := range o.tfrs {
+			if _, ok := tfr.iterators[i].(*emptyPostingsIterator); ok {
+				// An empty postings iterator means the entire AND is empty.
+				oTFR.iterators[i] = anEmptyPostingsIterator
+				continue OUTER
+			}
+
+			itr, ok := tfr.iterators[i].(segment.OptimizablePostingsIterator)
+			if !ok {
+				// We only optimize postings iterators that support this operation.
+				return nil, nil
+			}
+
+			// If the postings iterator is "1-hit" optimized, then we
+			// can perform several optimizations up-front here.
+			docNum1Hit, ok := itr.DocNum1Hit()
+			if ok {
+				if docNum1HitLastOk && docNum1HitLast != docNum1Hit {
+					// The docNum1Hit doesn't match the previous
+					// docNum1HitLast, so the entire AND is empty.
+					oTFR.iterators[i] = anEmptyPostingsIterator
+					continue OUTER
+				}
+
+				docNum1HitLast = docNum1Hit
+				docNum1HitLastOk = true
+
+				continue
+			}
+
+			if itr.ActualBitmap() == nil {
+				// An empty actual bitmap means the entire AND is empty.
+				oTFR.iterators[i] = anEmptyPostingsIterator
+				continue OUTER
+			}
+
+			// Collect the actual bitmap for more processing later.
+			actualBMs = append(actualBMs, itr.ActualBitmap())
+		}
+
+		if docNum1HitLastOk {
+			// We reach here if all the 1-hit optimized posting
+			// iterators had the same 1-hit docNum, so we can check if
+			// our collected actual bitmaps also have that docNum.
+			for _, bm := range actualBMs {
+				if !bm.Contains(uint32(docNum1HitLast)) {
+					// The docNum1Hit isn't in one of our actual
+					// bitmaps, so the entire AND is empty.
+					oTFR.iterators[i] = anEmptyPostingsIterator
+					continue OUTER
+				}
+			}
+
+			// The actual bitmaps and docNum1Hits all contain or have
+			// the same 1-hit docNum, so that's our AND'ed result.
+			oTFR.iterators[i] = newUnadornedPostingsIteratorFrom1Hit(docNum1HitLast)
+
+			continue OUTER
+		}
+
+		if len(actualBMs) == 0 {
+			// If we've collected no actual bitmaps at this point,
+			// then the entire AND is empty.
+			oTFR.iterators[i] = anEmptyPostingsIterator
+			continue OUTER
+		}
+
+		if len(actualBMs) == 1 {
+			// If we've only 1 actual bitmap, then that's our result.
+			oTFR.iterators[i] = newUnadornedPostingsIteratorFromBitmap(actualBMs[0])
+
+			continue OUTER
+		}
+
+		// Else, AND together our collected bitmaps as our result.
+		bm := roaring.And(actualBMs[0], actualBMs[1])
+
+		for _, actualBM := range actualBMs[2:] {
+			bm.And(actualBM)
+		}
+
+		oTFR.iterators[i] = newUnadornedPostingsIteratorFromBitmap(bm)
+	}
+
+	atomic.AddUint64(&o.snapshot.parent.stats.TotTermSearchersStarted, uint64(1))
+	return oTFR, nil
+}
+
+// ----------------------------------------------------------------
+
+// An "unadorned" disjunction optimization is appropriate when
+// additional or subsidiary information like freq-norm's and
+// term-vectors are not required, and instead only the internal-id's
+// are needed.
+func (s *IndexSnapshotTermFieldReader) optimizeDisjunctionUnadorned(
+	octx index.OptimizableContext) (index.OptimizableContext, error) {
+	if octx == nil {
+		octx = &OptimizeTFRDisjunctionUnadorned{
+			snapshot: s.snapshot,
+		}
+	}
+
+	o, ok := octx.(*OptimizeTFRDisjunctionUnadorned)
+	if !ok {
+		return nil, nil
+	}
+
+	if o.snapshot != s.snapshot {
+		return nil, fmt.Errorf("tried to optimize unadorned disjunction across different snapshots")
+	}
+
+	o.tfrs = append(o.tfrs, s)
+
+	return o, nil
+}
+
+type OptimizeTFRDisjunctionUnadorned struct {
+	snapshot *IndexSnapshot
+
+	tfrs []*IndexSnapshotTermFieldReader
+}
+
+var OptimizeTFRDisjunctionUnadornedTerm = []byte("<disjunction:unadorned>")
+var OptimizeTFRDisjunctionUnadornedField = "*"
+
+// Finish of an unadorned disjunction optimization will compute a
+// termFieldReader with an "actual" bitmap that represents the
+// constituent bitmaps OR'ed together.  This termFieldReader cannot
+// provide any freq-norm or termVector associated information.
+func (o *OptimizeTFRDisjunctionUnadorned) Finish() (rv index.Optimized, err error) {
+	if len(o.tfrs) <= 1 {
+		return nil, nil
+	}
+
+	// We use an artificial term and field because the optimized
+	// termFieldReader can represent multiple terms and fields.
+	oTFR := o.snapshot.unadornedTermFieldReader(
+		OptimizeTFRDisjunctionUnadornedTerm, OptimizeTFRDisjunctionUnadornedField)
+
+	var docNums []uint32            // Collected docNum's from 1-hit posting lists.
+	var actualBMs []*roaring.Bitmap // Collected from regular posting lists.
+
+	for i := range o.snapshot.segment {
+		docNums = docNums[:0]
+		actualBMs = actualBMs[:0]
+
+		for _, tfr := range o.tfrs {
+			itr, ok := tfr.iterators[i].(segment.OptimizablePostingsIterator)
+			if !ok {
+				return nil, nil
+			}
+
+			docNum, ok := itr.DocNum1Hit()
+			if ok {
+				docNums = append(docNums, uint32(docNum))
+				continue
+			}
+
+			if itr.ActualBitmap() != nil {
+				actualBMs = append(actualBMs, itr.ActualBitmap())
+			}
+		}
+
+		var bm *roaring.Bitmap
+		if len(actualBMs) > 2 {
+			bm = roaring.HeapOr(actualBMs...)
+		} else if len(actualBMs) == 2 {
+			bm = roaring.Or(actualBMs[0], actualBMs[1])
+		} else if len(actualBMs) == 1 {
+			bm = actualBMs[0].Clone()
+		} else {
+			if len(docNums) == 0 {
+				// no hits, reuse the zero-alloc empty sentinel
+				oTFR.iterators[i] = anEmptyPostingsIterator
+				continue
+			} else if len(docNums) == 1 {
+				// 1-hit optimized
+				oTFR.iterators[i] = newUnadornedPostingsIteratorFrom1Hit(uint64(docNums[0]))
+				continue
+			} else {
+				bm = roaring.New()
+			}
+		}
+
+		bm.AddMany(docNums)
+
+		oTFR.iterators[i] = newUnadornedPostingsIteratorFromBitmap(bm)
+	}
+
+	atomic.AddUint64(&o.snapshot.parent.stats.TotTermSearchersStarted, uint64(1))
+	return oTFR, nil
+}
+
+// ----------------------------------------------------------------
+
+func (i *IndexSnapshot) unadornedTermFieldReader(
+	term []byte, field string) *IndexSnapshotTermFieldReader {
+	// This IndexSnapshotTermFieldReader will not be recycled, more
+	// conversation here: https://github.com/blevesearch/bleve/pull/1438
+	return &IndexSnapshotTermFieldReader{
+		term:               term,
+		field:              field,
+		snapshot:           i,
+		iterators:          make([]segment.PostingsIterator, len(i.segment)),
+		segmentOffset:      0,
+		includeFreq:        false,
+		includeNorm:        false,
+		includeTermVectors: false,
+		recycle:            false,
+		// signal downstream that this is a special unadorned termFieldReader
+		unadorned: true,
+		// unadorned TFRs do not require bytes read tracking
+		updateBytesRead: false,
+	}
+}

index/scorch/optimize_knn.go

@@ -0,0 +1,207 @@
+//  Copyright (c) 2023 Couchbase, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// 		http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//go:build vectors
+// +build vectors
+
+package scorch
+
+import (
+	"context"
+	"fmt"
+	"sync"
+
+	"github.com/blevesearch/bleve/v2/search"
+	index "github.com/blevesearch/bleve_index_api"
+	segment_api "github.com/blevesearch/scorch_segment_api/v2"
+)
+
+type OptimizeVR struct {
+	ctx       context.Context
+	snapshot  *IndexSnapshot
+	totalCost uint64
+	// maps field to vector readers
+	vrs map[string][]*IndexSnapshotVectorReader
+}
+
+func (o *OptimizeVR) invokeSearcherEndCallback() {
+	if o.ctx != nil {
+		if cb := o.ctx.Value(search.SearcherEndCallbackKey); cb != nil {
+			if cbF, ok := cb.(search.SearcherEndCallbackFn); ok {
+				if o.totalCost > 0 {
+					// notify the callback that the searcher creation etc. is finished
+					// and report back the total cost for it to track and take actions
+					// appropriately.
+					_ = cbF(o.totalCost)
+				}
+			}
+		}
+	}
+}
+
+// search runs the configured kNN searches for every vector
+// reader against a single segment, populating the per-segment postings and
+// iterators on each reader.
+func (o *OptimizeVR) search(segID int) error {
+	snapshot := o.snapshot.segment[segID]
+	vecSeg, ok := snapshot.segment.(segment_api.VectorSegment)
+	if !ok {
+		return nil
+	}
+	meta := snapshot.cachedMeta
+	// for each field, get the vector index --> invoke the zap func.
+	// for each VR, populate postings list and iterators
+	// by passing the obtained vector index and getting similar vectors.
+	for field, vrs := range o.vrs {
+		// Early exit if the field is supposed to be completely deleted or
+		// if it's index data has been deleted
+		if info, ok := o.snapshot.updatedFields[field]; ok && (info.Deleted || info.Index) {
+			continue
+		}
+		vecIndex, err := vecSeg.InterpretVectorIndex(field, snapshot.deleted)
+		if err != nil {
+			return err
+		}
+		// update the vector index size as a meta value in the segment snapshot
+		// if not already present
+		if !meta.contains(field) {
+			meta.store(field, vecIndex.Size())
+		}
+		for _, vr := range vrs {
+			var pl segment_api.VecPostingsList
+			var err error
+			// for each VR, populate postings list and iterators
+			// by passing the obtained vector index and getting similar vectors.
+
+			// check if the vector reader is configured to use a pre-filter
+			// to filter out ineligible documents before performing
+			// kNN search.
+			if vr.eligibleSelector != nil {
+				pl, err = vecIndex.SearchWithFilter(
+					vr.vector,
+					vr.k,
+					vr.eligibleSelector.SegmentEligibleDocuments(segID),
+					vr.searchParams,
+				)
+			} else {
+				pl, err = vecIndex.Search(
+					vr.vector,
+					vr.k,
+					vr.searchParams,
+				)
+			}
+			if err != nil {
+				vecIndex.Close()
+				return err
+			}
+			// postings and iterators are already alloc'ed when
+			// IndexSnapshotVectorReader is created, here we are just
+			// populating them with the obtained postings list.
+			vr.postings[segID] = pl
+			vr.iterators[segID] = pl.Iterator(vr.iterators[segID])
+		}
+		go vecIndex.Close()
+	}
+	return nil
+}
+
+func (o *OptimizeVR) Finish() error {
+	defer o.invokeSearcherEndCallback()
+	numSegments := len(o.snapshot.segment)
+
+	var wg sync.WaitGroup
+	wg.Add(numSegments)
+	var errM sync.Mutex
+	var searchErr error
+	// launch goroutines to search the vector index for each segment
+	for i := 0; i < numSegments; i++ {
+		go func(segID int) {
+			defer wg.Done()
+			if err := o.search(segID); err != nil {
+				errM.Lock()
+				searchErr = err
+				errM.Unlock()
+			}
+		}(i)
+	}
+	// wait until all the launched goroutines finish and collect errors if any
+	wg.Wait()
+	return searchErr
+}
+
+func (s *IndexSnapshotVectorReader) VectorOptimize(ctx context.Context,
+	octx index.VectorOptimizableContext,
+) (index.VectorOptimizableContext, error) {
+	if s.snapshot.parent.segPlugin.Version() < VectorSearchSupportedSegmentVersion {
+		return nil, fmt.Errorf("vector search not supported for this index, "+
+			"index's segment version %v, supported segment version for vector search %v",
+			s.snapshot.parent.segPlugin.Version(), VectorSearchSupportedSegmentVersion)
+	}
+
+	if octx == nil {
+		octx = &OptimizeVR{
+			snapshot: s.snapshot,
+			vrs:      make(map[string][]*IndexSnapshotVectorReader),
+		}
+	}
+
+	o, ok := octx.(*OptimizeVR)
+	if !ok {
+		return octx, nil
+	}
+	o.ctx = ctx
+
+	if o.snapshot != s.snapshot {
+		o.invokeSearcherEndCallback()
+		return nil, fmt.Errorf("tried to optimize KNN across different snapshots")
+	}
+
+	// for every searcher creation, consult the segment snapshot to see
+	// what's the vector index size and since you're anyways going
+	// to use this vector index to perform the search etc. as part of the Finish()
+	// perform a check as to whether we allow the searcher creation (the downstream)
+	// Finish() logic to even occur or not.
+	var sumVectorIndexSize uint64
+	for _, seg := range o.snapshot.segment {
+		if vecIndexSize, ok := seg.cachedMeta.load(s.field); ok {
+			sumVectorIndexSize += vecIndexSize.(uint64)
+		}
+	}
+
+	if o.ctx != nil {
+		if cb := o.ctx.Value(search.SearcherStartCallbackKey); cb != nil {
+			if cbF, ok := cb.(search.SearcherStartCallbackFn); ok {
+				err := cbF(sumVectorIndexSize)
+				if err != nil {
+					// it's important to invoke the end callback at this point since
+					// if the earlier searchers of this optimize struct were successful
+					// the cost corresponding to it would be incremented and if the
+					// current searcher fails the check then we end up erroring out
+					// the overall optimized searcher creation, the cost needs to be
+					// handled appropriately.
+					o.invokeSearcherEndCallback()
+					return nil, err
+				}
+			}
+		}
+	}
+
+	// total cost is essentially the sum of the vector indexes' size across all the
+	// searchers - all of them end up reading and maintaining a vector index.
+	// misacconting this value would end up calling the "end" callback with a value
+	// not equal to the value passed to "start" callback.
+	o.totalCost += sumVectorIndexSize
+	o.vrs[s.field] = append(o.vrs[s.field], s)
+	return o, nil
+}

index/scorch/reader_test.go

@@ -0,0 +1,697 @@
+//  Copyright (c) 2017 Couchbase, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// 		http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package scorch
+
+import (
+	"context"
+	"encoding/binary"
+	"reflect"
+	"testing"
+
+	"github.com/blevesearch/bleve/v2/document"
+	index "github.com/blevesearch/bleve_index_api"
+)
+
+func TestIndexReader(t *testing.T) {
+	cfg := CreateConfig("TestIndexReader")
+	err := InitTest(cfg)
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer func() {
+		err := DestroyTest(cfg)
+		if err != nil {
+			t.Log(err)
+		}
+	}()
+
+	analysisQueue := index.NewAnalysisQueue(1)
+	idx, err := NewScorch(Name, cfg, analysisQueue)
+	if err != nil {
+		t.Fatal(err)
+	}
+	err = idx.Open()
+	if err != nil {
+		t.Fatalf("error opening index: %v", err)
+	}
+	defer func() {
+		err := idx.Close()
+		if err != nil {
+			t.Fatal(err)
+		}
+	}()
+
+	var expectedCount uint64
+	doc := document.NewDocument("1")
+	doc.AddField(document.NewTextField("name", []uint64{}, []byte("test")))
+	err = idx.Update(doc)
+	if err != nil {
+		t.Errorf("Error updating index: %v", err)
+	}
+	expectedCount++
+
+	doc = document.NewDocument("2")
+	doc.AddField(document.NewTextFieldWithAnalyzer("name", []uint64{}, []byte("test test test"), testAnalyzer))
+	doc.AddField(document.NewTextFieldCustom("desc", []uint64{}, []byte("eat more rice"), index.IndexField|index.IncludeTermVectors, testAnalyzer))
+	err = idx.Update(doc)
+	if err != nil {
+		t.Errorf("Error updating index: %v", err)
+	}
+	expectedCount++
+
+	indexReader, err := idx.Reader()
+	if err != nil {
+		t.Error(err)
+	}
+	defer func() {
+		err := indexReader.Close()
+		if err != nil {
+			t.Fatal(err)
+		}
+	}()
+
+	// first look for a term that doesn't exist
+	reader, err := indexReader.TermFieldReader(context.TODO(), []byte("nope"), "name", true, true, true)
+	if err != nil {
+		t.Errorf("Error accessing term field reader: %v", err)
+	}
+	count := reader.Count()
+	if count != 0 {
+		t.Errorf("Expected doc count to be: %d got: %d", 0, count)
+	}
+	err = reader.Close()
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	reader, err = indexReader.TermFieldReader(context.TODO(), []byte("test"), "name", true, true, true)
+	if err != nil {
+		t.Errorf("Error accessing term field reader: %v", err)
+	}
+
+	count = reader.Count()
+	if count != expectedCount {
+		t.Errorf("Expected doc count to be: %d got: %d", expectedCount, count)
+	}
+
+	var match *index.TermFieldDoc
+	var actualCount uint64
+	match, err = reader.Next(nil)
+	for err == nil && match != nil {
+		match, err = reader.Next(nil)
+		if err != nil {
+			t.Errorf("unexpected error reading next")
+		}
+		actualCount++
+	}
+	if actualCount != count {
+		t.Errorf("count was 2, but only saw %d", actualCount)
+	}
+
+	internalIDBogus, err := indexReader.InternalID("a-bogus-docId")
+	if err != nil {
+		t.Fatal(err)
+	}
+	if internalIDBogus != nil {
+		t.Errorf("expected bogus docId to have nil InternalID")
+	}
+
+	internalID2, err := indexReader.InternalID("2")
+	if err != nil {
+		t.Fatal(err)
+	}
+	expectedMatch := &index.TermFieldDoc{
+		ID:   internalID2,
+		Freq: 1,
+		Norm: 0.5773502588272095,
+		Vectors: []*index.TermFieldVector{
+			{
+				Field: "desc",
+				Pos:   3,
+				Start: 9,
+				End:   13,
+			},
+		},
+	}
+	tfr, err := indexReader.TermFieldReader(context.TODO(), []byte("rice"), "desc", true, true, true)
+	if err != nil {
+		t.Errorf("unexpected error: %v", err)
+	}
+	match, err = tfr.Next(nil)
+	if err != nil {
+		t.Errorf("unexpected error: %v", err)
+	}
+
+	if !reflect.DeepEqual(expectedMatch, match) {
+		t.Errorf("got %#v, expected %#v", match, expectedMatch)
+	}
+	err = reader.Close()
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	// now test usage of advance
+	reader, err = indexReader.TermFieldReader(context.TODO(), []byte("test"), "name", true, true, true)
+	if err != nil {
+		t.Errorf("Error accessing term field reader: %v", err)
+	}
+
+	match, err = reader.Advance(internalID2, nil)
+	if err != nil {
+		t.Errorf("unexpected error: %v", err)
+	}
+	if match == nil {
+		t.Fatalf("Expected match, got nil")
+	}
+	if !match.ID.Equals(internalID2) {
+		t.Errorf("Expected ID '2', got '%s'", match.ID)
+	}
+	// have to manually construct bogus id, because it doesn't exist
+	internalID3 := make([]byte, 8)
+	binary.BigEndian.PutUint64(internalID3, 3)
+	match, err = reader.Advance(index.IndexInternalID(internalID3), nil)
+	if err != nil {
+		t.Errorf("unexpected error: %v", err)
+	}
+	if match != nil {
+		t.Errorf("expected nil, got %v", match)
+	}
+	err = reader.Close()
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	// now test creating a reader for a field that doesn't exist
+	reader, err = indexReader.TermFieldReader(context.TODO(), []byte("water"), "doesnotexist", true, true, true)
+	if err != nil {
+		t.Errorf("Error accessing term field reader: %v", err)
+	}
+	count = reader.Count()
+	if count != 0 {
+		t.Errorf("expected count 0 for reader of non-existent field")
+	}
+	match, err = reader.Next(nil)
+	if err != nil {
+		t.Errorf("unexpected error: %v", err)
+	}
+	if match != nil {
+		t.Errorf("expected nil, got %v", match)
+	}
+	match, err = reader.Advance(index.IndexInternalID("anywhere"), nil)
+	if err != nil {
+		t.Errorf("unexpected error: %v", err)
+	}
+	if match != nil {
+		t.Errorf("expected nil, got %v", match)
+	}
+}
+
+func TestIndexDocIdReader(t *testing.T) {
+	cfg := CreateConfig("TestIndexDocIdReader")
+	err := InitTest(cfg)
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer func() {
+		err := DestroyTest(cfg)
+		if err != nil {
+			t.Log(err)
+		}
+	}()
+
+	analysisQueue := index.NewAnalysisQueue(1)
+	idx, err := NewScorch(Name, cfg, analysisQueue)
+	if err != nil {
+		t.Fatal(err)
+	}
+	err = idx.Open()
+	if err != nil {
+		t.Fatalf("error opening index: %v", err)
+	}
+	defer func() {
+		err := idx.Close()
+		if err != nil {
+			t.Fatal(err)
+		}
+	}()
+
+	var expectedCount uint64
+	doc := document.NewDocument("1")
+	doc.AddField(document.NewTextField("name", []uint64{}, []byte("test")))
+	err = idx.Update(doc)
+	if err != nil {
+		t.Errorf("Error updating index: %v", err)
+	}
+	expectedCount++
+
+	doc = document.NewDocument("2")
+	doc.AddField(document.NewTextField("name", []uint64{}, []byte("test test test")))
+	doc.AddField(document.NewTextFieldWithIndexingOptions("desc", []uint64{}, []byte("eat more rice"), index.IndexField|index.IncludeTermVectors))
+	err = idx.Update(doc)
+	if err != nil {
+		t.Errorf("Error updating index: %v", err)
+	}
+	expectedCount++
+
+	indexReader, err := idx.Reader()
+	if err != nil {
+		t.Error(err)
+	}
+	defer func() {
+		err := indexReader.Close()
+		if err != nil {
+			t.Error(err)
+		}
+	}()
+
+	// first get all doc ids
+	reader, err := indexReader.DocIDReaderAll()
+	if err != nil {
+		t.Errorf("Error accessing doc id reader: %v", err)
+	}
+	defer func() {
+		err := reader.Close()
+		if err != nil {
+			t.Fatal(err)
+		}
+	}()
+
+	id, err := reader.Next()
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	count := uint64(0)
+	for id != nil {
+		count++
+		id, err = reader.Next()
+		if err != nil {
+			t.Fatal(err)
+		}
+	}
+	if count != expectedCount {
+		t.Errorf("expected %d, got %d", expectedCount, count)
+	}
+
+	// try it again, but jump to the second doc this time
+	reader2, err := indexReader.DocIDReaderAll()
+	if err != nil {
+		t.Errorf("Error accessing doc id reader: %v", err)
+	}
+	defer func() {
+		err := reader2.Close()
+		if err != nil {
+			t.Error(err)
+		}
+	}()
+
+	internalID2, err := indexReader.InternalID("2")
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	id, err = reader2.Advance(internalID2)
+	if err != nil {
+		t.Error(err)
+	}
+	if !id.Equals(internalID2) {
+		t.Errorf("expected to find id '2', got '%s'", id)
+	}
+
+	// again 3 doesn't exist cannot use internal id for 3 as there is none
+	// the important aspect is that this id doesn't exist, so its ok
+	id, err = reader2.Advance(index.IndexInternalID("3"))
+	if err != nil {
+		t.Error(err)
+	}
+	if id != nil {
+		t.Errorf("expected to find id '', got '%s'", id)
+	}
+}
+
+func TestIndexDocIdOnlyReader(t *testing.T) {
+	cfg := CreateConfig("TestIndexDocIdOnlyReader")
+	err := InitTest(cfg)
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer func() {
+		err := DestroyTest(cfg)
+		if err != nil {
+			t.Log(err)
+		}
+	}()
+
+	analysisQueue := index.NewAnalysisQueue(1)
+	idx, err := NewScorch(Name, cfg, analysisQueue)
+	if err != nil {
+		t.Fatal(err)
+	}
+	err = idx.Open()
+	if err != nil {
+		t.Fatalf("error opening index: %v", err)
+	}
+	defer func() {
+		err := idx.Close()
+		if err != nil {
+			t.Fatal(err)
+		}
+	}()
+
+	doc := document.NewDocument("1")
+	err = idx.Update(doc)
+	if err != nil {
+		t.Errorf("Error updating index: %v", err)
+	}
+
+	doc = document.NewDocument("3")
+	err = idx.Update(doc)
+	if err != nil {
+		t.Errorf("Error updating index: %v", err)
+	}
+
+	doc = document.NewDocument("5")
+	err = idx.Update(doc)
+	if err != nil {
+		t.Errorf("Error updating index: %v", err)
+	}
+
+	doc = document.NewDocument("7")
+	err = idx.Update(doc)
+	if err != nil {
+		t.Errorf("Error updating index: %v", err)
+	}
+
+	doc = document.NewDocument("9")
+	err = idx.Update(doc)
+	if err != nil {
+		t.Errorf("Error updating index: %v", err)
+	}
+
+	indexReader, err := idx.Reader()
+	if err != nil {
+		t.Error(err)
+	}
+	defer func() {
+		err := indexReader.Close()
+		if err != nil {
+			t.Error(err)
+		}
+	}()
+
+	onlyIds := []string{"1", "5", "9"}
+	reader, err := indexReader.DocIDReaderOnly(onlyIds)
+	if err != nil {
+		t.Errorf("Error accessing doc id reader: %v", err)
+	}
+	defer func() {
+		err := reader.Close()
+		if err != nil {
+			t.Fatal(err)
+		}
+	}()
+
+	id, err := reader.Next()
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	count := uint64(0)
+	for id != nil {
+		count++
+		id, err = reader.Next()
+		if err != nil {
+			t.Fatal(err)
+		}
+	}
+	if count != 3 {
+		t.Errorf("expected 3, got %d", count)
+	}
+
+	// commented out because advance works with internal ids
+	// this test presumes we see items in external doc id order
+	// which is no longer the case, so simply converting external ids
+	// to internal ones is not logically correct
+	// not removing though because we need some way to test Advance()
+
+	// // try it again, but jump
+	// reader2, err := indexReader.DocIDReaderOnly(onlyIds)
+	// if err != nil {
+	// 	t.Errorf("Error accessing doc id reader: %v", err)
+	// }
+	// defer func() {
+	// 	err := reader2.Close()
+	// 	if err != nil {
+	// 		t.Error(err)
+	// 	}
+	// }()
+	//
+	// id, err = reader2.Advance(index.IndexInternalID("5"))
+	// if err != nil {
+	// 	t.Error(err)
+	// }
+	// if !id.Equals(index.IndexInternalID("5")) {
+	// 	t.Errorf("expected to find id '5', got '%s'", id)
+	// }
+	//
+	// id, err = reader2.Advance(index.IndexInternalID("a"))
+	// if err != nil {
+	// 	t.Error(err)
+	// }
+	// if id != nil {
+	// 	t.Errorf("expected to find id '', got '%s'", id)
+	// }
+
+	// some keys aren't actually there
+	onlyIds = []string{"0", "2", "4", "5", "6", "8", "a"}
+	reader3, err := indexReader.DocIDReaderOnly(onlyIds)
+	if err != nil {
+		t.Errorf("Error accessing doc id reader: %v", err)
+	}
+	defer func() {
+		err := reader3.Close()
+		if err != nil {
+			t.Error(err)
+		}
+	}()
+
+	id, err = reader3.Next()
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	count = uint64(0)
+	for id != nil {
+		count++
+		id, err = reader3.Next()
+		if err != nil {
+			t.Fatal(err)
+		}
+	}
+	if count != 1 {
+		t.Errorf("expected 1, got %d", count)
+	}
+
+	// commented out because advance works with internal ids
+	// this test presumes we see items in external doc id order
+	// which is no longer the case, so simply converting external ids
+	// to internal ones is not logically correct
+	// not removing though because we need some way to test Advance()
+
+	// // mix advance and next
+	// onlyIds = []string{"0", "1", "3", "5", "6", "9"}
+	// reader4, err := indexReader.DocIDReaderOnly(onlyIds)
+	// if err != nil {
+	// 	t.Errorf("Error accessing doc id reader: %v", err)
+	// }
+	// defer func() {
+	// 	err := reader4.Close()
+	// 	if err != nil {
+	// 		t.Error(err)
+	// 	}
+	// }()
+	//
+	// // first key is "1"
+	// id, err = reader4.Next()
+	// if err != nil {
+	// 	t.Error(err)
+	// }
+	// if !id.Equals(index.IndexInternalID("1")) {
+	// 	t.Errorf("expected to find id '1', got '%s'", id)
+	// }
+	//
+	// // advancing to key we dont have gives next
+	// id, err = reader4.Advance(index.IndexInternalID("2"))
+	// if err != nil {
+	// 	t.Error(err)
+	// }
+	// if !id.Equals(index.IndexInternalID("3")) {
+	// 	t.Errorf("expected to find id '3', got '%s'", id)
+	// }
+	//
+	// // next after advance works
+	// id, err = reader4.Next()
+	// if err != nil {
+	// 	t.Error(err)
+	// }
+	// if !id.Equals(index.IndexInternalID("5")) {
+	// 	t.Errorf("expected to find id '5', got '%s'", id)
+	// }
+	//
+	// // advancing to key we do have works
+	// id, err = reader4.Advance(index.IndexInternalID("9"))
+	// if err != nil {
+	// 	t.Error(err)
+	// }
+	// if !id.Equals(index.IndexInternalID("9")) {
+	// 	t.Errorf("expected to find id '9', got '%s'", id)
+	// }
+	//
+	// // advance backwards at end
+	// id, err = reader4.Advance(index.IndexInternalID("4"))
+	// if err != nil {
+	// 	t.Error(err)
+	// }
+	// if !id.Equals(index.IndexInternalID("5")) {
+	// 	t.Errorf("expected to find id '5', got '%s'", id)
+	// }
+	//
+	// // next after advance works
+	// id, err = reader4.Next()
+	// if err != nil {
+	// 	t.Error(err)
+	// }
+	// if !id.Equals(index.IndexInternalID("9")) {
+	// 	t.Errorf("expected to find id '9', got '%s'", id)
+	// }
+	//
+	// // advance backwards to key that exists, but not in only set
+	// id, err = reader4.Advance(index.IndexInternalID("7"))
+	// if err != nil {
+	// 	t.Error(err)
+	// }
+	// if !id.Equals(index.IndexInternalID("9")) {
+	// 	t.Errorf("expected to find id '9', got '%s'", id)
+	// }
+}
+
+func TestSegmentIndexAndLocalDocNumFromGlobal(t *testing.T) {
+	tests := []struct {
+		offsets      []uint64
+		globalDocNum uint64
+		segmentIndex int
+		localDocNum  uint64
+	}{
+		// just 1 segment
+		{
+			offsets:      []uint64{0},
+			globalDocNum: 0,
+			segmentIndex: 0,
+			localDocNum:  0,
+		},
+		{
+			offsets:      []uint64{0},
+			globalDocNum: 1,
+			segmentIndex: 0,
+			localDocNum:  1,
+		},
+		{
+			offsets:      []uint64{0},
+			globalDocNum: 25,
+			segmentIndex: 0,
+			localDocNum:  25,
+		},
+		// now 2 segments, 30 docs in first
+		{
+			offsets:      []uint64{0, 30},
+			globalDocNum: 0,
+			segmentIndex: 0,
+			localDocNum:  0,
+		},
+		{
+			offsets:      []uint64{0, 30},
+			globalDocNum: 1,
+			segmentIndex: 0,
+			localDocNum:  1,
+		},
+		{
+			offsets:      []uint64{0, 30},
+			globalDocNum: 25,
+			segmentIndex: 0,
+			localDocNum:  25,
+		},
+		{
+			offsets:      []uint64{0, 30},
+			globalDocNum: 30,
+			segmentIndex: 1,
+			localDocNum:  0,
+		},
+		{
+			offsets:      []uint64{0, 30},
+			globalDocNum: 35,
+			segmentIndex: 1,
+			localDocNum:  5,
+		},
+		// lots of segments
+		{
+			offsets:      []uint64{0, 30, 40, 70, 99, 172, 800, 25000},
+			globalDocNum: 0,
+			segmentIndex: 0,
+			localDocNum:  0,
+		},
+		{
+			offsets:      []uint64{0, 30, 40, 70, 99, 172, 800, 25000},
+			globalDocNum: 25,
+			segmentIndex: 0,
+			localDocNum:  25,
+		},
+		{
+			offsets:      []uint64{0, 30, 40, 70, 99, 172, 800, 25000},
+			globalDocNum: 35,
+			segmentIndex: 1,
+			localDocNum:  5,
+		},
+		{
+			offsets:      []uint64{0, 30, 40, 70, 99, 172, 800, 25000},
+			globalDocNum: 100,
+			segmentIndex: 4,
+			localDocNum:  1,
+		},
+		{
+			offsets:      []uint64{0, 30, 40, 70, 99, 172, 800, 25000},
+			globalDocNum: 825,
+			segmentIndex: 6,
+			localDocNum:  25,
+		},
+	}
+
+	for _, test := range tests {
+		i := &IndexSnapshot{
+			offsets: test.offsets,
+			refs:    1,
+		}
+		gotSegmentIndex, gotLocalDocNum := i.segmentIndexAndLocalDocNumFromGlobal(test.globalDocNum)
+		if gotSegmentIndex != test.segmentIndex {
+			t.Errorf("got segment index %d expected %d for offsets %v globalDocNum %d", gotSegmentIndex, test.segmentIndex, test.offsets, test.globalDocNum)
+		}
+		if gotLocalDocNum != test.localDocNum {
+			t.Errorf("got localDocNum %d expected %d for offsets %v globalDocNum %d", gotLocalDocNum, test.localDocNum, test.offsets, test.globalDocNum)
+		}
+		err := i.DecRef()
+		if err != nil {
+			t.Errorf("expected no err, got: %v", err)
+		}
+	}
+}

index/scorch/regexp.go

@@ -0,0 +1,63 @@
+//  Copyright (c) 2020 Couchbase, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// 		http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package scorch
+
+import (
+	"regexp/syntax"
+
+	"github.com/blevesearch/vellum/regexp"
+)
+
+func parseRegexp(pattern string) (a *regexp.Regexp, prefixBeg, prefixEnd []byte, err error) {
+	// TODO: potential optimization where syntax.Regexp supports a Simplify() API?
+
+	parsed, err := syntax.Parse(pattern, syntax.Perl)
+	if err != nil {
+		return nil, nil, nil, err
+	}
+
+	re, err := regexp.NewParsedWithLimit(pattern, parsed, regexp.DefaultLimit)
+	if err != nil {
+		return nil, nil, nil, err
+	}
+
+	prefix := literalPrefix(parsed)
+	if prefix != "" {
+		prefixBeg := []byte(prefix)
+		prefixEnd := calculateExclusiveEndFromPrefix(prefixBeg)
+		return re, prefixBeg, prefixEnd, nil
+	}
+
+	return re, nil, nil, nil
+}
+
+// Returns the literal prefix given the parse tree for a regexp
+func literalPrefix(s *syntax.Regexp) string {
+	// traverse the left-most branch in the parse tree as long as the
+	// node represents a concatenation
+	for s != nil && s.Op == syntax.OpConcat {
+		if len(s.Sub) < 1 {
+			return ""
+		}
+
+		s = s.Sub[0]
+	}
+
+	if s.Op == syntax.OpLiteral && (s.Flags&syntax.FoldCase == 0) {
+		return string(s.Rune)
+	}
+
+	return "" // no literal prefix
+}

index/scorch/regexp_test.go

@@ -0,0 +1,57 @@
+//  Copyright (c) 2020 Couchbase, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// 		http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package scorch
+
+import (
+	"regexp/syntax"
+	"testing"
+)
+
+func TestLiteralPrefix(t *testing.T) {
+	tests := []struct {
+		input, expected string
+	}{
+		{"", ""},
+		{"hello", "hello"},
+		{"hello.?", "hello"},
+		{"hello$", "hello"},
+		{`[h][e][l][l][o].*world`, "hello"},
+		{`[h-h][e-e][l-l][l-l][o-o].*world`, "hello"},
+		{".*", ""},
+		{"h.*", "h"},
+		{"h.?", "h"},
+		{"h[a-z]", "h"},
+		{`h\s`, "h"},
+		{`(hello)world`, ""},
+		{`日本語`, "日本語"},
+		{`日本語\w`, "日本語"},
+		{`^hello`, ""},
+		{`^`, ""},
+		{`$`, ""},
+		{`(?i)mArTy`, ""},
+	}
+
+	for i, test := range tests {
+		s, err := syntax.Parse(test.input, syntax.Perl)
+		if err != nil {
+			t.Fatalf("expected no syntax.Parse error, got: %v", err)
+		}
+
+		got := literalPrefix(s)
+		if test.expected != got {
+			t.Fatalf("test: %d, %+v, got: %s", i, test, got)
+		}
+	}
+}

index/scorch/rollback.go

@@ -0,0 +1,230 @@
+//  Copyright (c) 2017 Couchbase, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// 		http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package scorch
+
+import (
+	"fmt"
+	"log"
+	"os"
+
+	"github.com/blevesearch/bleve/v2/util"
+	bolt "go.etcd.io/bbolt"
+)
+
+type RollbackPoint struct {
+	epoch uint64
+	meta  map[string][]byte
+}
+
+func (r *RollbackPoint) GetInternal(key []byte) []byte {
+	return r.meta[string(key)]
+}
+
+// RollbackPoints returns an array of rollback points available for
+// the application to rollback to, with more recent rollback points
+// (higher epochs) coming first.
+func RollbackPoints(path string) ([]*RollbackPoint, error) {
+	if len(path) == 0 {
+		return nil, fmt.Errorf("RollbackPoints: invalid path")
+	}
+
+	rootBoltPath := path + string(os.PathSeparator) + "root.bolt"
+	rootBoltOpt := &bolt.Options{
+		ReadOnly: true,
+	}
+	rootBolt, err := util.OpenBolt(rootBoltPath, 0600, rootBoltOpt)
+	if err != nil || rootBolt == nil {
+		return nil, err
+	}
+
+	// start a read-only bolt transaction
+	tx, err := rootBolt.Begin(false)
+	if err != nil {
+		return nil, fmt.Errorf("RollbackPoints: failed to start" +
+			" read-only transaction")
+	}
+
+	// read-only bolt transactions to be rolled back
+	defer func() {
+		_ = tx.Rollback()
+		_ = rootBolt.Close()
+	}()
+
+	snapshots := tx.Bucket(util.BoltSnapshotsBucket)
+	if snapshots == nil {
+		return nil, nil
+	}
+
+	rollbackPoints := []*RollbackPoint{}
+
+	c1 := snapshots.Cursor()
+	for k, _ := c1.Last(); k != nil; k, _ = c1.Prev() {
+		_, snapshotEpoch, err := decodeUvarintAscending(k)
+		if err != nil {
+			log.Printf("RollbackPoints:"+
+				" unable to parse segment epoch %x, continuing", k)
+			continue
+		}
+
+		snapshot := snapshots.GetBucket(k)
+		if snapshot == nil {
+			log.Printf("RollbackPoints:"+
+				" snapshot key, but bucket missing %x, continuing", k)
+			continue
+		}
+
+		metaBucket := snapshot.GetBucket(util.BoltMetaDataKey)
+		if metaBucket == nil {
+			return nil, fmt.Errorf("meta-data bucket missing")
+		}
+
+		fileWriterID, err := metaBucket.Get(util.BoltMetaDataFileWriterIDKey, nil)
+		if err != nil {
+			return nil, fmt.Errorf("unable to load file writer ID: %v", err)
+		}
+		reader, err := util.NewFileReader(string(fileWriterID), []byte(rootBoltPath))
+		if err != nil {
+			return nil, fmt.Errorf("unable to load correct reader: %v", err)
+		}
+
+		meta := map[string][]byte{}
+		c2 := snapshot.Cursor()
+		for j, _ := c2.First(); j != nil; j, _ = c2.Next() {
+			if j[0] == util.BoltInternalKey[0] {
+				internalBucket := snapshot.GetBucket(j)
+				if internalBucket == nil {
+					err = fmt.Errorf("internal bucket missing")
+					break
+				}
+				err = internalBucket.ForEach(func(key []byte, val []byte) error {
+					meta[string(key)] = val
+					return nil
+				}, reader)
+				if err != nil {
+					break
+				}
+			}
+		}
+
+		if err != nil {
+			log.Printf("RollbackPoints:"+
+				" failed in fetching internal data: %v", err)
+			continue
+		}
+
+		rollbackPoints = append(rollbackPoints, &RollbackPoint{
+			epoch: snapshotEpoch,
+			meta:  meta,
+		})
+	}
+
+	return rollbackPoints, nil
+}
+
+// Rollback atomically and durably brings the store back to the point
+// in time as represented by the RollbackPoint.
+// Rollback() should only be passed a RollbackPoint that came from the
+// same store using the RollbackPoints() API along with the index path.
+func Rollback(path string, to *RollbackPoint) error {
+	if to == nil {
+		return fmt.Errorf("Rollback: RollbackPoint is nil")
+	}
+	if len(path) == 0 {
+		return fmt.Errorf("Rollback: index path is empty")
+	}
+
+	rootBoltPath := path + string(os.PathSeparator) + "root.bolt"
+	rootBoltOpt := &bolt.Options{
+		ReadOnly: false,
+	}
+	rootBolt, err := util.OpenBolt(rootBoltPath, 0600, rootBoltOpt)
+	if err != nil || rootBolt == nil {
+		return err
+	}
+	defer func() {
+		err1 := rootBolt.Close()
+		if err1 != nil && err == nil {
+			err = err1
+		}
+	}()
+
+	// pick all the younger persisted epochs in bolt store
+	// including the target one.
+	var found bool
+	var eligibleEpochs []uint64
+	err = rootBolt.View(func(tx *util.BoltTxImpl) error {
+		snapshots := tx.Bucket(util.BoltSnapshotsBucket)
+		if snapshots == nil {
+			return nil
+		}
+		sc := snapshots.Cursor()
+		for sk, _ := sc.Last(); sk != nil && !found; sk, _ = sc.Prev() {
+			_, snapshotEpoch, err := decodeUvarintAscending(sk)
+			if err != nil {
+				continue
+			}
+			if snapshotEpoch == to.epoch {
+				found = true
+			}
+			eligibleEpochs = append(eligibleEpochs, snapshotEpoch)
+		}
+		return nil
+	})
+
+	if len(eligibleEpochs) == 0 {
+		return fmt.Errorf("Rollback: no persisted epochs found in bolt")
+	}
+	if !found {
+		return fmt.Errorf("Rollback: target epoch %d not found in bolt", to.epoch)
+	}
+
+	// start a write transaction
+	tx, err := rootBolt.Begin(true)
+	if err != nil {
+		return err
+	}
+
+	defer func() {
+		if err == nil {
+			err = tx.Commit()
+		} else {
+			_ = tx.Rollback()
+		}
+		if err == nil {
+			err = rootBolt.Sync()
+		}
+	}()
+
+	snapshots := tx.Bucket(util.BoltSnapshotsBucket)
+	if snapshots == nil {
+		return nil
+	}
+	for _, epoch := range eligibleEpochs {
+		k := encodeUvarintAscending(nil, epoch)
+		if err != nil {
+			continue
+		}
+		if epoch == to.epoch {
+			// return here as it already processed until the given epoch
+			return nil
+		}
+		err = snapshots.DeleteBucket(k)
+		if err == bolt.ErrBucketNotFound {
+			err = nil
+		}
+	}
+
+	return err
+}

index/scorch/scorch_knn_test.go

@@ -0,0 +1,165 @@
+//  Copyright (c) 2026 Couchbase, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// 		http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//go:build vectors
+// +build vectors
+
+package scorch
+
+import (
+	"fmt"
+	"math/rand"
+	"os"
+	"sync/atomic"
+	"testing"
+
+	"github.com/blevesearch/bleve/v2/document"
+	index "github.com/blevesearch/bleve_index_api"
+)
+
+const (
+	testVectorDims              = 3
+	testVectorSimilarity        = index.CosineSimilarity
+	testVectorIndexOptimizedFor = index.IndexOptimizedForRecall
+)
+
+func genVectorDoc(t *testing.T, fieldName string) index.Document {
+	t.Helper()
+	randNum := rand.Intn(1000000)
+	doc := document.NewDocument(fmt.Sprintf("vectest%d", randNum))
+	vector := make([]float32, testVectorDims)
+	for i := range vector {
+		vector[i] = float32(i+1) * float32(randNum+1)
+	}
+	doc.AddField(document.NewVectorField(fieldName, nil, vector, testVectorDims, testVectorSimilarity, testVectorIndexOptimizedFor))
+	doc.AddIDField()
+	doc.VisitFields(func(field index.Field) {
+		field.Analyze()
+	})
+	return doc
+}
+
+func TestFieldStatPersistence(t *testing.T) {
+	cfg := CreateConfig("TestFieldStatPersistence")
+	err := InitTest(cfg)
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer func() { _ = DestroyTest(cfg) }()
+	dirPath := cfg["path"].(string)
+	if err = os.Mkdir(dirPath, 0o755); err != nil {
+		t.Fatal(err)
+	}
+	analysisQueue := index.NewAnalysisQueue(1)
+	idx, err := NewScorch(Name, cfg, analysisQueue)
+	if err != nil {
+		t.Fatalf("failed to create Scorch: %v", err)
+	}
+	s, ok := idx.(*Scorch)
+	if !ok {
+		t.Fatalf("expected *Scorch, got %T", idx)
+	}
+	s.path = dirPath
+	if err = s.openBolt(); err != nil {
+		t.Fatalf("failed to open bolt: %v", err)
+	}
+	const (
+		numSegments = 3
+		fieldName   = "vector"
+		statName    = "field:" + fieldName + ":num_vectors"
+	)
+	docs := make([]index.Document, numSegments)
+	ids := make([]string, numSegments)
+	for i := 0; i < numSegments; i++ {
+		docs[i] = genVectorDoc(t, fieldName)
+		ids[i] = docs[i].ID()
+		seg, _, err := s.segPlugin.New([]index.Document{docs[i]})
+		if err != nil {
+			t.Fatalf("failed to create segment %d: %v", i, err)
+		}
+		intro := &segmentIntroduction{
+			id:      atomic.AddUint64(&s.nextSegmentID, 1),
+			data:    seg,
+			ids:     []string{ids[i]},
+			applied: make(chan error),
+		}
+		if err = s.introduceSegment(intro); err != nil {
+			t.Fatalf("introduceSegment %d failed: %v", i, err)
+		}
+	}
+
+	errCh := make(chan error, 1)
+	doneCh := make(chan struct{})
+	go func() {
+		snapshot := s.root
+		snapshot.AddRef()
+		defer snapshot.DecRef()
+		err := s.persistSnapshot(snapshot, s.persisterOptions)
+		if err != nil {
+			errCh <- err
+			return
+		}
+		doneCh <- struct{}{}
+	}()
+	select {
+	case merge := <-s.merges:
+		s.introduceMerge(merge)
+	case err := <-errCh:
+		t.Fatalf("unexpected error during persist (merge phase): %v", err)
+	}
+	<-doneCh
+	snapshot := s.root
+	if len(snapshot.segment) != 1 {
+		t.Fatalf("expected 1 segment after introduce, got %d", len(snapshot.segment))
+	}
+	if !snapshot.segment[0].HasVector() {
+		t.Fatalf("expected HasVector() == true after index persist")
+	}
+	scorchStats := s.StatsMap()
+	numVecs := scorchStats[statName].(uint64)
+	if numVecs != uint64(numSegments) {
+		t.Fatalf("expected %d vectors in stats, got %d", numSegments, numVecs)
+	}
+	if err = idx.Close(); err != nil {
+		t.Fatalf("failed to close index: %v", err)
+	}
+	idx, err = NewScorch(Name, cfg, analysisQueue)
+	if err != nil {
+		t.Fatalf("failed to recreate Scorch: %v", err)
+	}
+	s, ok = idx.(*Scorch)
+	if !ok {
+		t.Fatalf("expected *Scorch, got %T", idx)
+	}
+	s.path = dirPath
+	if err = s.openBolt(); err != nil {
+		t.Fatalf("failed to open bolt on reopen: %v", err)
+	}
+	snapshot = s.root
+	if len(snapshot.segment) != 1 {
+		t.Fatalf("expected 1 segment after reopen, got %d", len(snapshot.segment))
+	}
+	if !snapshot.segment[0].HasVector() {
+		t.Fatalf("reopened segment: expected HasVector() == true after index reopen")
+	}
+	scorchStats = s.StatsMap()
+	numVecs = scorchStats[statName].(uint64)
+	if numVecs != uint64(numSegments) {
+		t.Fatalf("expected %d vectors in stats, got %d", numSegments, numVecs)
+	}
+	err = idx.Close()
+	if err != nil {
+		t.Fatalf("failed to close index on reopen: %v", err)
+	}
+}

index/scorch/segment_plugin.go

@@ -0,0 +1,154 @@
+//  Copyright (c) 2019 Couchbase, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// 		http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package scorch
+
+import (
+	"fmt"
+
+	"github.com/RoaringBitmap/roaring/v2"
+	"github.com/blevesearch/bleve/v2/geo"
+	index "github.com/blevesearch/bleve_index_api"
+	segment "github.com/blevesearch/scorch_segment_api/v2"
+
+	zapv11 "github.com/blevesearch/zapx/v11"
+	zapv12 "github.com/blevesearch/zapx/v12"
+	zapv13 "github.com/blevesearch/zapx/v13"
+	zapv14 "github.com/blevesearch/zapx/v14"
+	zapv15 "github.com/blevesearch/zapx/v15"
+	zapv16 "github.com/blevesearch/zapx/v16"
+	zapv17 "github.com/blevesearch/zapx/v17"
+)
+
+// SegmentPlugin represents the essential functions required by a package to plug in
+// it's segment implementation
+type SegmentPlugin interface {
+
+	// Type is the name for this segment plugin
+	Type() string
+
+	// Version is a numeric value identifying a specific version of this type.
+	// When incompatible changes are made to a particular type of plugin, the
+	// version must be incremented.
+	Version() uint32
+
+	// New takes a set of Documents and turns them into a new Segment
+	New(results []index.Document) (segment.Segment, uint64, error)
+
+	NewUsing(results []index.Document, config map[string]interface{}) (segment.Segment, uint64, error)
+
+	// Open attempts to open the file at the specified path and
+	// return the corresponding Segment
+	Open(path string) (segment.Segment, error)
+
+	OpenUsing(path string, config map[string]interface{}) (segment.Segment, error)
+
+	// Merge takes a set of Segments, and creates a new segment on disk at
+	// the specified path.
+	// Drops is a set of bitmaps (one for each segment) indicating which
+	// documents can be dropped from the segments during the merge.
+	// If the closeCh channel is closed, Merge will cease doing work at
+	// the next opportunity, and return an error (closed).
+	// StatsReporter can optionally be provided, in which case progress
+	// made during the merge is reported while operation continues.
+	// Returns:
+	// A slice of new document numbers (one for each input segment),
+	// this allows the caller to know a particular document's new
+	// document number in the newly merged segment.
+	// The number of bytes written to the new segment file.
+	// An error, if any occurred.
+	Merge(segments []segment.Segment, drops []*roaring.Bitmap, path string,
+		closeCh chan struct{}, s segment.StatsReporter) (
+		[][]uint64, uint64, error)
+
+	MergeUsing(segments []segment.Segment, drops []*roaring.Bitmap, path string,
+		closeCh chan struct{}, s segment.StatsReporter, config map[string]interface{}) (
+		[][]uint64, uint64, error)
+}
+
+var supportedSegmentPlugins map[string]map[uint32]SegmentPlugin
+var defaultSegmentPlugin SegmentPlugin
+
+func init() {
+	ResetSegmentPlugins()
+	RegisterSegmentPlugin(&zapv17.ZapPlugin{}, true)
+	RegisterSegmentPlugin(&zapv16.ZapPlugin{}, false)
+	RegisterSegmentPlugin(&zapv15.ZapPlugin{}, false)
+	RegisterSegmentPlugin(&zapv14.ZapPlugin{}, false)
+	RegisterSegmentPlugin(&zapv13.ZapPlugin{}, false)
+	RegisterSegmentPlugin(&zapv12.ZapPlugin{}, false)
+	RegisterSegmentPlugin(&zapv11.ZapPlugin{}, false)
+}
+
+func ResetSegmentPlugins() {
+	supportedSegmentPlugins = map[string]map[uint32]SegmentPlugin{}
+}
+
+func RegisterSegmentPlugin(plugin SegmentPlugin, makeDefault bool) {
+	if _, ok := supportedSegmentPlugins[plugin.Type()]; !ok {
+		supportedSegmentPlugins[plugin.Type()] = map[uint32]SegmentPlugin{}
+	}
+	supportedSegmentPlugins[plugin.Type()][plugin.Version()] = plugin
+	if makeDefault {
+		defaultSegmentPlugin = plugin
+	}
+}
+
+func SupportedSegmentTypes() (rv []string) {
+	for k := range supportedSegmentPlugins {
+		rv = append(rv, k)
+	}
+	return
+}
+
+func SupportedSegmentTypeVersions(typ string) (rv []uint32) {
+	for k := range supportedSegmentPlugins[typ] {
+		rv = append(rv, k)
+	}
+	return rv
+}
+
+func chooseSegmentPlugin(forcedSegmentType string,
+	forcedSegmentVersion uint32) (SegmentPlugin, error) {
+	if versions, ok := supportedSegmentPlugins[forcedSegmentType]; ok {
+		if segPlugin, ok := versions[uint32(forcedSegmentVersion)]; ok {
+			return segPlugin, nil
+		}
+		return nil, fmt.Errorf(
+			"unsupported version %d for segment type: %s, supported: %v",
+			forcedSegmentVersion, forcedSegmentType,
+			SupportedSegmentTypeVersions(forcedSegmentType))
+	}
+	return nil, fmt.Errorf("unsupported segment type: %s, supported: %v",
+		forcedSegmentType, SupportedSegmentTypes())
+}
+
+func (s *Scorch) loadSegmentPlugin(forcedSegmentType string,
+	forcedSegmentVersion uint32) error {
+	segPlugin, err := chooseSegmentPlugin(forcedSegmentType,
+		forcedSegmentVersion)
+	if err != nil {
+		return err
+	}
+	s.segPlugin = segPlugin
+	return nil
+}
+
+func (s *Scorch) loadSpatialAnalyzerPlugin(typ string) error {
+	s.spatialPlugin = geo.GetSpatialAnalyzerPlugin(typ)
+	if s.spatialPlugin == nil {
+		return fmt.Errorf("unsupported spatial plugin type: %s", typ)
+	}
+	return nil
+}

index/scorch/snapshot_index_dict.go

@@ -0,0 +1,139 @@
+//  Copyright (c) 2017 Couchbase, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// 		http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package scorch
+
+import (
+	"container/heap"
+
+	index "github.com/blevesearch/bleve_index_api"
+	segment "github.com/blevesearch/scorch_segment_api/v2"
+)
+
+// termDictionaryOmitCount is an optional interface a segment's TermDictionary
+// may implement to return an automaton iterator that skips the per-term
+// postings read used to populate DictEntry.Count. It is used by candidate-term
+// collection (fuzzy/regexp) where the count is discarded, and falls back to the
+// regular AutomatonIterator when the segment does not implement it.
+type termDictionaryOmitCount interface {
+	AutomatonIteratorOmitCount(a segment.Automaton,
+		startKeyInclusive, endKeyExclusive []byte) segment.DictionaryIterator
+}
+
+// automatonIteratorOmitCount returns a count-omitting automaton iterator when
+// the dictionary supports it, otherwise the standard one.
+func automatonIteratorOmitCount(dict segment.TermDictionary, a segment.Automaton,
+	startKeyInclusive, endKeyExclusive []byte) segment.DictionaryIterator {
+	if oc, ok := dict.(termDictionaryOmitCount); ok {
+		return oc.AutomatonIteratorOmitCount(a, startKeyInclusive, endKeyExclusive)
+	}
+	return dict.AutomatonIterator(a, startKeyInclusive, endKeyExclusive)
+}
+
+type segmentDictCursor struct {
+	dict segment.TermDictionary
+	itr  segment.DictionaryIterator
+	curr index.DictEntry
+}
+
+type IndexSnapshotFieldDict struct {
+	cardinality int
+	bytesRead   uint64
+
+	snapshot *IndexSnapshot
+	cursors  []*segmentDictCursor
+	entry    index.DictEntry
+}
+
+func (i *IndexSnapshotFieldDict) BytesRead() uint64 {
+	return i.bytesRead
+}
+
+func (i *IndexSnapshotFieldDict) Len() int { return len(i.cursors) }
+func (i *IndexSnapshotFieldDict) Less(a, b int) bool {
+	return i.cursors[a].curr.Term < i.cursors[b].curr.Term
+}
+func (i *IndexSnapshotFieldDict) Swap(a, b int) {
+	i.cursors[a], i.cursors[b] = i.cursors[b], i.cursors[a]
+}
+
+func (i *IndexSnapshotFieldDict) Push(x interface{}) {
+	i.cursors = append(i.cursors, x.(*segmentDictCursor))
+}
+
+func (i *IndexSnapshotFieldDict) Pop() interface{} {
+	n := len(i.cursors)
+	x := i.cursors[n-1]
+	i.cursors = i.cursors[0 : n-1]
+	return x
+}
+
+func (i *IndexSnapshotFieldDict) Next() (*index.DictEntry, error) {
+	if len(i.cursors) == 0 {
+		return nil, nil
+	}
+	i.entry = i.cursors[0].curr
+	next, err := i.cursors[0].itr.Next()
+	if err != nil {
+		return nil, err
+	}
+	if next == nil {
+		// at end of this cursor, remove it
+		heap.Pop(i)
+	} else {
+		// modified heap, fix it
+		i.cursors[0].curr = *next
+		heap.Fix(i, 0)
+	}
+	// look for any other entries with the exact same term
+	for len(i.cursors) > 0 && i.cursors[0].curr.Term == i.entry.Term {
+		i.entry.Count += i.cursors[0].curr.Count
+		next, err := i.cursors[0].itr.Next()
+		if err != nil {
+			return nil, err
+		}
+		if next == nil {
+			// at end of this cursor, remove it
+			heap.Pop(i)
+		} else {
+			// modified heap, fix it
+			i.cursors[0].curr = *next
+			heap.Fix(i, 0)
+		}
+	}
+
+	return &i.entry, nil
+}
+
+func (i *IndexSnapshotFieldDict) Cardinality() int {
+	return i.cardinality
+}
+
+func (i *IndexSnapshotFieldDict) Close() error {
+	return nil
+}
+
+func (i *IndexSnapshotFieldDict) Contains(key []byte) (bool, error) {
+	if len(i.cursors) == 0 {
+		return false, nil
+	}
+
+	for _, cursor := range i.cursors {
+		if found, _ := cursor.dict.Contains(key); found {
+			return true, nil
+		}
+	}
+
+	return false, nil
+}

index/scorch/snapshot_index_doc.go

@@ -0,0 +1,79 @@
+//  Copyright (c) 2017 Couchbase, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// 		http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package scorch
+
+import (
+	"reflect"
+
+	"github.com/RoaringBitmap/roaring/v2"
+	"github.com/blevesearch/bleve/v2/size"
+	index "github.com/blevesearch/bleve_index_api"
+)
+
+var reflectStaticSizeIndexSnapshotDocIDReader int
+
+func init() {
+	var isdr IndexSnapshotDocIDReader
+	reflectStaticSizeIndexSnapshotDocIDReader = int(reflect.TypeOf(isdr).Size())
+}
+
+type IndexSnapshotDocIDReader struct {
+	snapshot      *IndexSnapshot
+	iterators     []roaring.IntIterable
+	segmentOffset int
+}
+
+func (i *IndexSnapshotDocIDReader) Size() int {
+	return reflectStaticSizeIndexSnapshotDocIDReader + size.SizeOfPtr
+}
+
+func (i *IndexSnapshotDocIDReader) Next() (index.IndexInternalID, error) {
+	for i.segmentOffset < len(i.iterators) {
+		if !i.iterators[i.segmentOffset].HasNext() {
+			i.segmentOffset++
+			continue
+		}
+		next := i.iterators[i.segmentOffset].Next()
+		// make segment number into global number by adding offset
+		globalOffset := i.snapshot.offsets[i.segmentOffset]
+		return index.NewIndexInternalID(nil, uint64(next)+globalOffset), nil
+	}
+	return nil, nil
+}
+
+func (i *IndexSnapshotDocIDReader) Advance(ID index.IndexInternalID) (index.IndexInternalID, error) {
+	// FIXME do something better
+	next, err := i.Next()
+	if err != nil {
+		return nil, err
+	}
+	if next == nil {
+		return nil, nil
+	}
+	for next.Compare(ID) < 0 {
+		next, err = i.Next()
+		if err != nil {
+			return nil, err
+		}
+		if next == nil {
+			break
+		}
+	}
+	return next, nil
+}
+
+func (i *IndexSnapshotDocIDReader) Close() error {
+	return nil
+}

index/scorch/snapshot_index_str.go

@@ -0,0 +1,79 @@
+//  Copyright (c) 2024 Couchbase, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// 		http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package scorch
+
+import (
+	"reflect"
+
+	"github.com/blevesearch/bleve/v2/size"
+	segment "github.com/blevesearch/scorch_segment_api/v2"
+)
+
+var reflectStaticSizeIndexSnapshotThesaurusTermReader int
+
+func init() {
+	var istr IndexSnapshotThesaurusTermReader
+	reflectStaticSizeIndexSnapshotThesaurusTermReader = int(reflect.TypeOf(istr).Size())
+}
+
+type IndexSnapshotThesaurusTermReader struct {
+	name          string
+	snapshot      *IndexSnapshot
+	thesauri      []segment.Thesaurus
+	postings      []segment.SynonymsList
+	iterators     []segment.SynonymsIterator
+	segmentOffset int
+}
+
+func (i *IndexSnapshotThesaurusTermReader) Size() int {
+	sizeInBytes := reflectStaticSizeIndexSnapshotThesaurusTermReader + size.SizeOfPtr +
+		len(i.name) + size.SizeOfString
+
+	for _, postings := range i.postings {
+		if postings != nil {
+			sizeInBytes += postings.Size()
+		}
+	}
+
+	for _, iterator := range i.iterators {
+		if iterator != nil {
+			sizeInBytes += iterator.Size()
+		}
+	}
+
+	return sizeInBytes
+}
+
+func (i *IndexSnapshotThesaurusTermReader) Next() (string, error) {
+	// find the next hit
+	for i.segmentOffset < len(i.iterators) {
+		if i.iterators[i.segmentOffset] != nil {
+			next, err := i.iterators[i.segmentOffset].Next()
+			if err != nil {
+				return "", err
+			}
+			if next != nil {
+				synTerm := next.Term()
+				return synTerm, nil
+			}
+		}
+		i.segmentOffset++
+	}
+	return "", nil
+}
+
+func (i *IndexSnapshotThesaurusTermReader) Close() error {
+	return nil
+}

index/scorch/snapshot_index_test.go

@@ -0,0 +1,90 @@
+package scorch
+
+import (
+	"testing"
+
+	"github.com/blevesearch/vellum"
+)
+
+func TestIndexSnapshot_getLevAutomaton(t *testing.T) {
+	// Create a dummy IndexSnapshot (parent doesn't matter for this method)
+	is := &IndexSnapshot{}
+
+	tests := []struct {
+		name        string
+		term        string
+		fuzziness   uint8
+		expectError bool
+		errorMsg    string // Optional: check specific error message
+	}{
+		{
+			name:        "fuzziness 1",
+			term:        "test",
+			fuzziness:   1,
+			expectError: false,
+		},
+		{
+			name:        "fuzziness 2",
+			term:        "another",
+			fuzziness:   2,
+			expectError: false,
+		},
+		{
+			name:        "fuzziness 0",
+			term:        "zero",
+			fuzziness:   0,
+			expectError: true,
+			errorMsg:    "fuzziness exceeds the max limit",
+		},
+		{
+			name:        "fuzziness 3",
+			term:        "three",
+			fuzziness:   3,
+			expectError: true,
+			errorMsg:    "fuzziness exceeds the max limit",
+		},
+		{
+			name:        "empty term fuzziness 1",
+			term:        "",
+			fuzziness:   1,
+			expectError: false,
+		},
+		{
+			name:        "empty term fuzziness 2",
+			term:        "",
+			fuzziness:   2,
+			expectError: false,
+		},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			gotAutomaton, err := is.getLevAutomaton(tt.term, tt.fuzziness)
+
+			if tt.expectError {
+				if err == nil {
+					t.Errorf("getLevAutomaton() expected an error but got nil")
+				} else if tt.errorMsg != "" && err.Error() != tt.errorMsg {
+					t.Errorf("getLevAutomaton() expected error msg %q but got %q", tt.errorMsg, err.Error())
+				}
+				if gotAutomaton != nil {
+					t.Errorf("getLevAutomaton() expected nil automaton on error but got %v", gotAutomaton)
+				}
+			} else {
+				if err != nil {
+					t.Errorf("getLevAutomaton() got unexpected error: %v", err)
+				}
+				if gotAutomaton == nil {
+					t.Errorf("getLevAutomaton() expected a valid automaton but got nil")
+				}
+				// Optional: Check type if needed, though non-nil is usually sufficient
+				_, ok := gotAutomaton.(vellum.Automaton)
+				if !ok {
+					t.Errorf("getLevAutomaton() returned type is not vellum.Automaton")
+				}
+			}
+		})
+	}
+}
+
+// Add other tests for snapshot_index.go below if needed...

index/scorch/snapshot_index_tfr.go

@@ -0,0 +1,239 @@
+//  Copyright (c) 2017 Couchbase, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// 		http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package scorch
+
+import (
+	"context"
+	"fmt"
+	"reflect"
+	"sync/atomic"
+
+	"github.com/blevesearch/bleve/v2/search"
+	"github.com/blevesearch/bleve/v2/size"
+	index "github.com/blevesearch/bleve_index_api"
+	segment "github.com/blevesearch/scorch_segment_api/v2"
+)
+
+var reflectStaticSizeIndexSnapshotTermFieldReader int
+
+func init() {
+	var istfr IndexSnapshotTermFieldReader
+	reflectStaticSizeIndexSnapshotTermFieldReader = int(reflect.TypeOf(istfr).Size())
+}
+
+type IndexSnapshotTermFieldReader struct {
+	term               []byte
+	field              string
+	snapshot           *IndexSnapshot
+	dicts              []segment.TermDictionary
+	postings           []segment.PostingsList
+	iterators          []segment.PostingsIterator
+	segmentOffset      int
+	includeFreq        bool
+	includeNorm        bool
+	includeTermVectors bool
+	currPosting        segment.Posting
+	currID             index.IndexInternalID
+	recycle            bool
+	bytesRead          uint64
+	ctx                context.Context
+	unadorned          bool
+	// flag to indicate whether to increment our bytesRead
+	// value after creation of the TFR while iterating our postings
+	// lists
+	updateBytesRead bool
+}
+
+func (i *IndexSnapshotTermFieldReader) incrementBytesRead(val uint64) {
+	i.bytesRead += val
+}
+
+func (i *IndexSnapshotTermFieldReader) Size() int {
+	sizeInBytes := reflectStaticSizeIndexSnapshotTermFieldReader + size.SizeOfPtr +
+		len(i.term) +
+		len(i.field) +
+		len(i.currID)
+
+	for _, entry := range i.postings {
+		sizeInBytes += entry.Size()
+	}
+
+	for _, entry := range i.iterators {
+		sizeInBytes += entry.Size()
+	}
+
+	if i.currPosting != nil {
+		sizeInBytes += i.currPosting.Size()
+	}
+
+	return sizeInBytes
+}
+
+func (i *IndexSnapshotTermFieldReader) Next(preAlloced *index.TermFieldDoc) (*index.TermFieldDoc, error) {
+	rv := preAlloced
+	if rv == nil {
+		rv = &index.TermFieldDoc{}
+	}
+	var prevBytesRead uint64
+	// find the next hit
+	for i.segmentOffset < len(i.iterators) {
+		// get our current postings iterator
+		curItr := i.iterators[i.segmentOffset]
+		if i.updateBytesRead {
+			prevBytesRead = curItr.BytesRead()
+		}
+		next, err := curItr.Next()
+		if err != nil {
+			return nil, err
+		}
+		if next != nil {
+			// make segment number into global number by adding offset
+			globalOffset := i.snapshot.offsets[i.segmentOffset]
+			nnum := next.Number()
+			rv.ID = index.NewIndexInternalID(rv.ID, nnum+globalOffset)
+			i.postingToTermFieldDoc(next, rv)
+
+			i.currID = rv.ID
+			i.currPosting = next
+			if i.updateBytesRead {
+				// postingsIterators maintains the bytesRead stat in a cumulative fashion.
+				// this is because there are chances of having a series of loadChunk calls,
+				// and they have to be added together before sending the bytesRead at this point
+				// upstream.
+				bytesRead := curItr.BytesRead()
+				if bytesRead > prevBytesRead {
+					i.incrementBytesRead(bytesRead - prevBytesRead)
+				}
+			}
+			return rv, nil
+		}
+		i.segmentOffset++
+	}
+	return nil, nil
+}
+
+func (i *IndexSnapshotTermFieldReader) postingToTermFieldDoc(next segment.Posting, rv *index.TermFieldDoc) {
+	if i.includeFreq {
+		rv.Freq = next.Frequency()
+	}
+	if i.includeNorm {
+		rv.Norm = next.Norm()
+	}
+	if i.includeTermVectors {
+		locs := next.Locations()
+		if cap(rv.Vectors) < len(locs) {
+			rv.Vectors = make([]*index.TermFieldVector, len(locs))
+			backing := make([]index.TermFieldVector, len(locs))
+			for i := range backing {
+				rv.Vectors[i] = &backing[i]
+			}
+		}
+		rv.Vectors = rv.Vectors[:len(locs)]
+		for i, loc := range locs {
+			*rv.Vectors[i] = index.TermFieldVector{
+				Start:          loc.Start(),
+				End:            loc.End(),
+				Pos:            loc.Pos(),
+				ArrayPositions: loc.ArrayPositions(),
+				Field:          loc.Field(),
+			}
+		}
+	}
+}
+
+func (i *IndexSnapshotTermFieldReader) Advance(ID index.IndexInternalID, preAlloced *index.TermFieldDoc) (*index.TermFieldDoc, error) {
+	// FIXME do something better
+	// for now, if we need to seek backwards, then restart from the beginning
+	if i.currPosting != nil && i.currID.Compare(ID) >= 0 {
+		// Check if the TFR is a special unadorned composite optimization.
+		// Such a TFR will NOT have a valid `term` or `field` set, making it
+		// impossible for the TFR to replace itself with a new one.
+		if !i.unadorned {
+			i2, err := i.snapshot.TermFieldReader(context.TODO(), i.term, i.field,
+				i.includeFreq, i.includeNorm, i.includeTermVectors)
+			if err != nil {
+				return nil, err
+			}
+			// close the current term field reader before replacing it with a new one
+			_ = i.Close()
+			*i = *(i2.(*IndexSnapshotTermFieldReader))
+		} else {
+			// unadorned composite optimization
+			// we need to reset all the iterators
+			// back to the beginning, which effectively
+			// achieves the same thing as the above
+			for _, iter := range i.iterators {
+				if optimizedIterator, ok := iter.(ResetablePostingsIterator); ok {
+					optimizedIterator.ResetIterator()
+				}
+			}
+		}
+	}
+	num := ID.Value()
+	segIndex, ldocNum := i.snapshot.segmentIndexAndLocalDocNumFromGlobal(num)
+	if segIndex >= len(i.snapshot.segment) {
+		return nil, fmt.Errorf("computed segment index %d out of bounds %d",
+			segIndex, len(i.snapshot.segment))
+	}
+	// skip directly to the target segment
+	i.segmentOffset = segIndex
+	next, err := i.iterators[i.segmentOffset].Advance(ldocNum)
+	if err != nil {
+		return nil, err
+	}
+	if next == nil {
+		// we jumped directly to the segment that should have contained it
+		// but it wasn't there, so reuse Next() which should correctly
+		// get the next hit after it (we moved i.segmentOffset)
+		return i.Next(preAlloced)
+	}
+
+	if preAlloced == nil {
+		preAlloced = &index.TermFieldDoc{}
+	}
+	preAlloced.ID = index.NewIndexInternalID(preAlloced.ID, next.Number()+
+		i.snapshot.offsets[segIndex])
+	i.postingToTermFieldDoc(next, preAlloced)
+	i.currID = preAlloced.ID
+	i.currPosting = next
+	return preAlloced, nil
+}
+
+func (i *IndexSnapshotTermFieldReader) Count() uint64 {
+	var rv uint64
+	for _, posting := range i.postings {
+		rv += posting.Count()
+	}
+	return rv
+}
+
+func (i *IndexSnapshotTermFieldReader) Close() error {
+	if i.ctx != nil {
+		statsCallbackFn := i.ctx.Value(search.SearchIOStatsCallbackKey)
+		if statsCallbackFn != nil {
+			// essentially before you close the TFR, you must report this
+			// reader's bytesRead value
+			statsCallbackFn.(search.SearchIOStatsCallbackFunc)(i.bytesRead)
+		}
+
+		search.RecordSearchCost(i.ctx, search.AddM, i.bytesRead)
+	}
+
+	if i.snapshot != nil {
+		atomic.AddUint64(&i.snapshot.parent.stats.TotTermSearchersFinished, uint64(1))
+		i.snapshot.recycleTermFieldReader(i)
+	}
+	return nil
+}

index/scorch/snapshot_index_thes.go

@@ -0,0 +1,107 @@
+//  Copyright (c) 2024 Couchbase, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// 		http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package scorch
+
+import (
+	"container/heap"
+
+	index "github.com/blevesearch/bleve_index_api"
+	segment "github.com/blevesearch/scorch_segment_api/v2"
+)
+
+type segmentThesCursor struct {
+	thes segment.Thesaurus
+	itr  segment.ThesaurusIterator
+	curr index.ThesaurusEntry
+}
+
+type IndexSnapshotThesaurusKeys struct {
+	snapshot *IndexSnapshot
+	cursors  []*segmentThesCursor
+	entry    index.ThesaurusEntry
+}
+
+func (i *IndexSnapshotThesaurusKeys) Len() int { return len(i.cursors) }
+func (i *IndexSnapshotThesaurusKeys) Less(a, b int) bool {
+	return i.cursors[a].curr.Term < i.cursors[b].curr.Term
+}
+func (i *IndexSnapshotThesaurusKeys) Swap(a, b int) {
+	i.cursors[a], i.cursors[b] = i.cursors[b], i.cursors[a]
+}
+
+func (i *IndexSnapshotThesaurusKeys) Push(x interface{}) {
+	i.cursors = append(i.cursors, x.(*segmentThesCursor))
+}
+
+func (i *IndexSnapshotThesaurusKeys) Pop() interface{} {
+	n := len(i.cursors)
+	x := i.cursors[n-1]
+	i.cursors = i.cursors[0 : n-1]
+	return x
+}
+
+func (i *IndexSnapshotThesaurusKeys) Next() (*index.ThesaurusEntry, error) {
+	if len(i.cursors) == 0 {
+		return nil, nil
+	}
+	i.entry = i.cursors[0].curr
+	next, err := i.cursors[0].itr.Next()
+	if err != nil {
+		return nil, err
+	}
+	if next == nil {
+		// at end of this cursor, remove it
+		heap.Pop(i)
+	} else {
+		// modified heap, fix it
+		i.cursors[0].curr = *next
+		heap.Fix(i, 0)
+	}
+	// look for any other entries with the exact same term
+	for len(i.cursors) > 0 && i.cursors[0].curr.Term == i.entry.Term {
+		next, err := i.cursors[0].itr.Next()
+		if err != nil {
+			return nil, err
+		}
+		if next == nil {
+			// at end of this cursor, remove it
+			heap.Pop(i)
+		} else {
+			// modified heap, fix it
+			i.cursors[0].curr = *next
+			heap.Fix(i, 0)
+		}
+	}
+
+	return &i.entry, nil
+}
+
+func (i *IndexSnapshotThesaurusKeys) Close() error {
+	return nil
+}
+
+func (i *IndexSnapshotThesaurusKeys) Contains(key []byte) (bool, error) {
+	if len(i.cursors) == 0 {
+		return false, nil
+	}
+
+	for _, cursor := range i.cursors {
+		if found, _ := cursor.thes.Contains(key); found {
+			return true, nil
+		}
+	}
+
+	return false, nil
+}

index/scorch/snapshot_index_vr.go

@@ -0,0 +1,217 @@
+//  Copyright (c) 2023 Couchbase, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// 		http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//go:build vectors
+// +build vectors
+
+package scorch
+
+import (
+	"context"
+	"encoding/json"
+	"fmt"
+	"reflect"
+	"sort"
+	"sync/atomic"
+
+	"github.com/blevesearch/bleve/v2/size"
+	index "github.com/blevesearch/bleve_index_api"
+	segment_api "github.com/blevesearch/scorch_segment_api/v2"
+)
+
+const VectorSearchSupportedSegmentVersion = 16
+
+var reflectStaticSizeIndexSnapshotVectorReader int
+
+func init() {
+	var istfr IndexSnapshotVectorReader
+	reflectStaticSizeIndexSnapshotVectorReader = int(reflect.TypeOf(istfr).Size())
+}
+
+type IndexSnapshotVectorReader struct {
+	vector        []float32
+	field         string
+	k             int64
+	snapshot      *IndexSnapshot
+	postings      []segment_api.VecPostingsList
+	iterators     []segment_api.VecPostingsIterator
+	segmentOffset int
+	currPosting   segment_api.VecPosting
+	currID        index.IndexInternalID
+	ctx           context.Context
+
+	searchParams     json.RawMessage
+	eligibleSelector index.EligibleDocumentSelector
+}
+
+func (i *IndexSnapshotVectorReader) Size() int {
+	sizeInBytes := reflectStaticSizeIndexSnapshotVectorReader + size.SizeOfPtr +
+		len(i.vector)*size.SizeOfFloat32 +
+		len(i.field) +
+		len(i.currID)
+
+	for _, entry := range i.postings {
+		sizeInBytes += entry.Size()
+	}
+
+	for _, entry := range i.iterators {
+		sizeInBytes += entry.Size()
+	}
+
+	if i.currPosting != nil {
+		sizeInBytes += i.currPosting.Size()
+	}
+
+	return sizeInBytes
+}
+
+func (i *IndexSnapshotVectorReader) Next(preAlloced *index.VectorDoc) (
+	*index.VectorDoc, error) {
+	rv := preAlloced
+	if rv == nil {
+		rv = &index.VectorDoc{}
+	}
+
+	for i.segmentOffset < len(i.iterators) {
+		if i.iterators[i.segmentOffset] == nil {
+			i.segmentOffset++
+			continue
+		}
+		next, err := i.iterators[i.segmentOffset].Next()
+		if err != nil {
+			return nil, err
+		}
+		if next != nil {
+			// make segment number into global number by adding offset
+			globalOffset := i.snapshot.offsets[i.segmentOffset]
+			nnum := next.Number()
+			rv.ID = index.NewIndexInternalID(rv.ID, nnum+globalOffset)
+			rv.Score = float64(next.Score())
+
+			i.currID = rv.ID
+			i.currPosting = next
+
+			return rv, nil
+		}
+		i.segmentOffset++
+	}
+
+	return nil, nil
+}
+
+func (i *IndexSnapshotVectorReader) Advance(ID index.IndexInternalID,
+	preAlloced *index.VectorDoc) (*index.VectorDoc, error) {
+
+	if i.currPosting != nil && i.currID.Compare(ID) >= 0 {
+		i2, err := i.snapshot.VectorReader(i.ctx, i.vector, i.field, i.k,
+			i.searchParams, i.eligibleSelector)
+		if err != nil {
+			return nil, err
+		}
+		// close the current term field reader before replacing it with a new one
+		_ = i.Close()
+		*i = *(i2.(*IndexSnapshotVectorReader))
+	}
+
+	num := ID.Value()
+	segIndex, ldocNum := i.snapshot.segmentIndexAndLocalDocNumFromGlobal(num)
+	if segIndex >= len(i.snapshot.segment) {
+		return nil, fmt.Errorf("computed segment index %d out of bounds %d",
+			segIndex, len(i.snapshot.segment))
+	}
+	// skip directly to the target segment
+	i.segmentOffset = segIndex
+	next, err := i.iterators[i.segmentOffset].Advance(ldocNum)
+	if err != nil {
+		return nil, err
+	}
+	if next == nil {
+		// we jumped directly to the segment that should have contained it
+		// but it wasn't there, so reuse Next() which should correctly
+		// get the next hit after it (we moved i.segmentOffset)
+		return i.Next(preAlloced)
+	}
+
+	if preAlloced == nil {
+		preAlloced = &index.VectorDoc{}
+	}
+	preAlloced.ID = index.NewIndexInternalID(preAlloced.ID, next.Number()+
+		i.snapshot.offsets[segIndex])
+	i.currID = preAlloced.ID
+	i.currPosting = next
+	return preAlloced, nil
+}
+
+func (i *IndexSnapshotVectorReader) Count() uint64 {
+	var rv uint64
+	for _, posting := range i.postings {
+		rv += posting.Count()
+	}
+	return rv
+}
+
+func (i *IndexSnapshotVectorReader) Close() error {
+	if i.snapshot != nil {
+		atomic.AddUint64(&i.snapshot.parent.stats.TotKNNSearches, 1)
+	}
+	return nil
+}
+
+func (i *IndexSnapshot) CentroidCardinalities(field string, limit int, descending bool) (
+	[]index.CentroidCardinality, error) {
+	if len(i.segment) == 0 {
+		return nil, nil
+	}
+
+	if limit <= 0 {
+		return nil, fmt.Errorf("limit must be positive")
+	}
+
+	centroids := make([]index.CentroidCardinality, 0, limit*len(i.segment))
+
+	for _, segment := range i.segment {
+		if sv, ok := segment.segment.(segment_api.VectorSegment); ok {
+			vecIndex, err := sv.InterpretVectorIndex(field, segment.deleted)
+			if err != nil {
+				return nil, fmt.Errorf("failed to interpret vector index for field %s in segment: %v", field, err)
+			}
+
+			centroidCardinalities, err := vecIndex.ObtainKCentroidCardinalitiesFromIVFIndex(limit, descending)
+			if err != nil {
+				return nil, fmt.Errorf("failed to obtain top k centroid cardinalities for field %s in segment: %v", field, err)
+			}
+
+			if len(centroidCardinalities) > 0 {
+				centroids = append(centroids, centroidCardinalities...)
+			}
+		}
+	}
+
+	if len(centroids) == 0 {
+		return nil, nil
+	}
+
+	sort.Slice(centroids, func(i, j int) bool {
+		if descending {
+			return centroids[i].Cardinality > centroids[j].Cardinality
+		}
+		return centroids[i].Cardinality < centroids[j].Cardinality
+	})
+
+	if limit >= len(centroids) {
+		return centroids, nil
+	}
+
+	return centroids[:limit], nil
+}

index/scorch/snapshot_segment.go

@@ -0,0 +1,408 @@
+//  Copyright (c) 2017 Couchbase, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// 		http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package scorch
+
+import (
+	"bytes"
+	"os"
+	"sync"
+	"sync/atomic"
+
+	"github.com/RoaringBitmap/roaring/v2"
+	"github.com/blevesearch/bleve/v2/size"
+	index "github.com/blevesearch/bleve_index_api"
+	segment "github.com/blevesearch/scorch_segment_api/v2"
+)
+
+type SegmentSnapshot struct {
+	// this flag is needed to identify whether this
+	// segment was mmaped recently, in which case
+	// we consider the loading cost of the metadata
+	// as part of IO stats.
+	mmaped  uint32
+	id      uint64
+	segment segment.Segment
+	deleted *roaring.Bitmap
+	creator string
+	stats   *fieldStats
+
+	updatedFields map[string]*index.UpdateFieldInfo
+
+	cachedMeta *cachedMeta
+
+	cachedDocs *cachedDocs
+
+	rootCountOnce sync.Once
+	rootCount     uint64
+}
+
+func (s *SegmentSnapshot) Segment() segment.Segment {
+	return s.segment
+}
+
+func (s *SegmentSnapshot) Deleted() *roaring.Bitmap {
+	return s.deleted
+}
+
+func (s *SegmentSnapshot) Id() uint64 {
+	return s.id
+}
+
+func (s *SegmentSnapshot) FullSize() int64 {
+	return int64(s.segment.Count())
+}
+
+func (s *SegmentSnapshot) LiveSize() int64 {
+	return int64(s.Count())
+}
+
+func (s *SegmentSnapshot) HasVector() bool {
+	// number of vectors, for each vector field in the segment
+	numVecs := s.stats.Fetch()["num_vectors"]
+	return len(numVecs) > 0
+}
+
+func (s *SegmentSnapshot) FileSize() int64 {
+	ps, ok := s.segment.(segment.PersistedSegment)
+	if !ok {
+		return 0
+	}
+
+	path := ps.Path()
+	if path == "" {
+		return 0
+	}
+
+	fi, err := os.Stat(path)
+	if err != nil {
+		return 0
+	}
+
+	return fi.Size()
+}
+
+func (s *SegmentSnapshot) LiveFileSize() int64 {
+	fullSize := float64(s.FullSize())
+	if fullSize <= 0 {
+		return 0
+	}
+
+	liveSize := float64(s.LiveSize())
+	if liveSize <= 0 {
+		return 0
+	}
+
+	fileSize := float64(s.FileSize())
+	if fileSize <= 0 {
+		return 0
+	}
+
+	liveRatio := liveSize / fullSize
+
+	return int64(fileSize * liveRatio)
+}
+
+func (s *SegmentSnapshot) Close() error {
+	return s.segment.Close()
+}
+
+func (s *SegmentSnapshot) VisitDocument(num uint64, visitor segment.StoredFieldValueVisitor) error {
+	return s.segment.VisitStoredFields(num, visitor)
+}
+
+func (s *SegmentSnapshot) DocID(num uint64) ([]byte, error) {
+	return s.segment.DocID(num)
+}
+
+func (s *SegmentSnapshot) Count() uint64 {
+	rv := s.segment.Count()
+	if s.deleted != nil {
+		rv -= s.deleted.GetCardinality()
+	}
+	return rv
+}
+
+// this counts the root documents in the segment this differs from Count() in that
+// Count() counts all live documents including nested children, whereas this method
+// counts only root live documents
+func (s *SegmentSnapshot) CountRoot() uint64 {
+	s.rootCountOnce.Do(func() {
+		if nsb, ok := s.segment.(segment.NestedSegment); ok {
+			s.rootCount = nsb.CountRoot(s.deleted)
+		} else {
+			s.rootCount = s.Count()
+		}
+	})
+	return s.rootCount
+}
+
+func (s *SegmentSnapshot) DocNumbers(docIDs []string) (*roaring.Bitmap, error) {
+	rv, err := s.segment.DocNumbers(docIDs)
+	if err != nil {
+		return nil, err
+	}
+	if s.deleted != nil {
+		rv.AndNot(s.deleted)
+	}
+	return rv, nil
+}
+
+// DocNumbersLive returns a bitmap containing doc numbers for all live docs
+func (s *SegmentSnapshot) DocNumbersLive() *roaring.Bitmap {
+	rv := roaring.NewBitmap()
+	rv.AddRange(0, s.segment.Count())
+	if s.deleted != nil {
+		rv.AndNot(s.deleted)
+	}
+	return rv
+}
+
+func (s *SegmentSnapshot) Fields() []string {
+	return s.segment.Fields()
+}
+
+func (s *SegmentSnapshot) Size() (rv int) {
+	rv = s.segment.Size()
+	if s.deleted != nil {
+		rv += int(s.deleted.GetSizeInBytes())
+	}
+	rv += s.cachedDocs.Size()
+	return
+}
+
+// Merge given updated field information with existing and pass it on to the segment base
+func (s *SegmentSnapshot) UpdateFieldsInfo(updatedFields map[string]*index.UpdateFieldInfo) {
+	if s.updatedFields == nil {
+		s.updatedFields = updatedFields
+	} else {
+		for fieldName, info := range updatedFields {
+			if val, ok := s.updatedFields[fieldName]; ok {
+				val.Deleted = val.Deleted || info.Deleted
+				val.Index = val.Index || info.Index
+				val.DocValues = val.DocValues || info.DocValues
+				val.Store = val.Store || info.Store
+			} else {
+				s.updatedFields[fieldName] = info
+			}
+		}
+	}
+
+	if segment, ok := s.segment.(segment.UpdatableSegment); ok {
+		segment.SetUpdatedFields(s.updatedFields)
+	}
+}
+
+type cachedFieldDocs struct {
+	m       sync.Mutex
+	readyCh chan struct{}     // closed when the cachedFieldDocs.docs is ready to be used.
+	err     error             // Non-nil if there was an error when preparing this cachedFieldDocs.
+	docs    map[uint64][]byte // Keyed by localDocNum, value is a list of terms delimited by 0xFF.
+	size    uint64
+}
+
+func (cfd *cachedFieldDocs) Size() int {
+	var rv int
+	cfd.m.Lock()
+	for _, entry := range cfd.docs {
+		rv += 8 /* size of uint64 */ + len(entry)
+	}
+	cfd.m.Unlock()
+	return rv
+}
+
+func (cfd *cachedFieldDocs) prepareField(field string, ss *SegmentSnapshot) {
+	cfd.m.Lock()
+	defer func() {
+		close(cfd.readyCh)
+		cfd.m.Unlock()
+	}()
+
+	cfd.size += uint64(size.SizeOfUint64) /* size field */
+	dict, err := ss.segment.Dictionary(field)
+	if err != nil {
+		cfd.err = err
+		return
+	}
+
+	var postings segment.PostingsList
+	var postingsItr segment.PostingsIterator
+
+	dictItr := dict.AutomatonIterator(nil, nil, nil)
+	next, err := dictItr.Next()
+	for err == nil && next != nil {
+		var err1 error
+		postings, err1 = dict.PostingsList([]byte(next.Term), nil, postings)
+		if err1 != nil {
+			cfd.err = err1
+			return
+		}
+
+		cfd.size += uint64(size.SizeOfUint64) /* map key */
+		postingsItr = postings.Iterator(false, false, false, postingsItr)
+		nextPosting, err2 := postingsItr.Next()
+		for err2 == nil && nextPosting != nil {
+			docNum := nextPosting.Number()
+			cfd.docs[docNum] = append(cfd.docs[docNum], []byte(next.Term)...)
+			cfd.docs[docNum] = append(cfd.docs[docNum], index.DocValueTermSeparator)
+			cfd.size += uint64(len(next.Term) + 1) // map value
+			nextPosting, err2 = postingsItr.Next()
+		}
+
+		if err2 != nil {
+			cfd.err = err2
+			return
+		}
+
+		next, err = dictItr.Next()
+	}
+
+	if err != nil {
+		cfd.err = err
+		return
+	}
+}
+
+type cachedDocs struct {
+	size  uint64
+	m     sync.RWMutex                // As the cache is asynchronously prepared, need a lock
+	cache map[string]*cachedFieldDocs // Keyed by field
+}
+
+func (c *cachedDocs) prepareFields(wantedFields []string, ss *SegmentSnapshot) error {
+	c.m.Lock()
+
+	if c.cache == nil {
+		c.cache = make(map[string]*cachedFieldDocs, len(ss.Fields()))
+	}
+
+	for _, field := range wantedFields {
+		_, exists := c.cache[field]
+		if !exists {
+			c.cache[field] = &cachedFieldDocs{
+				readyCh: make(chan struct{}),
+				docs:    make(map[uint64][]byte),
+			}
+
+			go c.cache[field].prepareField(field, ss)
+		}
+	}
+
+	for _, field := range wantedFields {
+		cachedFieldDocs := c.cache[field]
+		c.m.Unlock()
+		<-cachedFieldDocs.readyCh
+
+		if cachedFieldDocs.err != nil {
+			return cachedFieldDocs.err
+		}
+		c.m.Lock()
+	}
+
+	c.updateSizeLOCKED()
+
+	c.m.Unlock()
+	return nil
+}
+
+// hasFields returns true if the cache has all the given fields
+func (c *cachedDocs) hasFields(fields []string) bool {
+	c.m.RLock()
+	for _, field := range fields {
+		if _, exists := c.cache[field]; !exists {
+			c.m.RUnlock()
+			return false // found a field not in cache
+		}
+	}
+	c.m.RUnlock()
+	return true
+}
+
+func (c *cachedDocs) Size() int {
+	return int(atomic.LoadUint64(&c.size))
+}
+
+func (c *cachedDocs) updateSizeLOCKED() {
+	sizeInBytes := 0
+	for k, v := range c.cache { // cachedFieldDocs
+		sizeInBytes += len(k)
+		if v != nil {
+			sizeInBytes += v.Size()
+		}
+	}
+	atomic.StoreUint64(&c.size, uint64(sizeInBytes))
+}
+
+func (c *cachedDocs) visitDoc(localDocNum uint64,
+	fields []string, visitor index.DocValueVisitor) {
+	c.m.RLock()
+
+	for _, field := range fields {
+		if cachedFieldDocs, exists := c.cache[field]; exists {
+			c.m.RUnlock()
+			<-cachedFieldDocs.readyCh
+			c.m.RLock()
+
+			if tlist, exists := cachedFieldDocs.docs[localDocNum]; exists {
+				for {
+					i := bytes.IndexByte(tlist, index.DocValueTermSeparator)
+					if i < 0 {
+						break
+					}
+					visitor(field, tlist[0:i])
+					tlist = tlist[i+1:]
+				}
+			}
+		}
+	}
+
+	c.m.RUnlock()
+}
+
+// cachedMeta is a simple wrapper around sync.Map to provide typed
+// access to cached metadata values for segments.
+type cachedMeta struct {
+	meta sync.Map
+}
+
+func newCachedMeta() *cachedMeta {
+	return &cachedMeta{
+		meta: sync.Map{},
+	}
+}
+
+// store the value for a field in the cache, overwriting any existing value.
+func (c *cachedMeta) store(field string, val interface{}) {
+	c.meta.Store(field, val)
+}
+
+// load the value for a field from the cache, returning the value
+// and a boolean indicating whether the value was present.
+func (c *cachedMeta) load(field string) (rv interface{}, ok bool) {
+	return c.meta.Load(field)
+}
+
+// contains reports whether the cache has an entry for the given field.
+func (c *cachedMeta) contains(field string) bool {
+	_, ok := c.meta.Load(field)
+	return ok
+}
+
+func (s *SegmentSnapshot) Ancestors(docNum uint64, prealloc []index.AncestorID) []index.AncestorID {
+	nsb, ok := s.segment.(segment.NestedSegment)
+	if !ok {
+		return append(prealloc, index.NewAncestorID(docNum))
+	}
+	return nsb.Ancestors(docNum, prealloc)
+}

index/scorch/snapshot_vector_index.go

@@ -0,0 +1,152 @@
+//  Copyright (c) 2023 Couchbase, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// 		http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//go:build vectors
+// +build vectors
+
+package scorch
+
+import (
+	"context"
+	"encoding/json"
+	"fmt"
+
+	"github.com/bits-and-blooms/bitset"
+	index "github.com/blevesearch/bleve_index_api"
+	segment_api "github.com/blevesearch/scorch_segment_api/v2"
+)
+
+func (is *IndexSnapshot) VectorReader(ctx context.Context, vector []float32,
+	field string, k int64, searchParams json.RawMessage,
+	eligibleSelector index.EligibleDocumentSelector) (
+	index.VectorReader, error) {
+	rv := &IndexSnapshotVectorReader{
+		vector:           vector,
+		field:            field,
+		k:                k,
+		snapshot:         is,
+		searchParams:     searchParams,
+		eligibleSelector: eligibleSelector,
+		postings:         make([]segment_api.VecPostingsList, len(is.segment)),
+		iterators:        make([]segment_api.VecPostingsIterator, len(is.segment)),
+	}
+
+	// initialize postings and iterators within the OptimizeVR's Finish()
+	return rv, nil
+}
+
+// eligibleDocumentList represents the list of eligible documents within a segment.
+type eligibleDocumentList struct {
+	bs *bitset.BitSet
+}
+
+// Iterator returns an iterator for the eligible document IDs.
+func (edl *eligibleDocumentList) Iterator() index.EligibleDocumentIterator {
+	if edl.bs == nil {
+		// no eligible documents
+		return emptyEligibleIterator
+	}
+	// return the iterator
+	return &eligibleDocumentIterator{
+		bs: edl.bs,
+	}
+}
+
+// Count returns the number of eligible document IDs.
+func (edl *eligibleDocumentList) Count() uint64 {
+	if edl.bs == nil {
+		return 0
+	}
+	return uint64(edl.bs.Count())
+}
+
+// emptyEligibleDocumentList is a reusable empty eligible document list.
+var emptyEligibleDocumentList = &eligibleDocumentList{}
+
+// eligibleDocumentIterator iterates over eligible document IDs within a segment.
+type eligibleDocumentIterator struct {
+	bs      *bitset.BitSet
+	current uint
+}
+
+// Next returns the next eligible document ID and whether it exists.
+func (it *eligibleDocumentIterator) Next() (id uint64, ok bool) {
+	next, found := it.bs.NextSet(it.current)
+	if !found {
+		return 0, false
+	}
+	it.current = next + 1
+	return uint64(next), true
+}
+
+// emptyEligibleIterator is a reusable empty eligible document iterator.
+var emptyEligibleIterator = &emptyEligibleDocumentIterator{}
+
+// emptyEligibleDocumentIterator is an iterator that always returns no documents.
+type emptyEligibleDocumentIterator struct{}
+
+// Next always returns false for empty iterator.
+func (it *emptyEligibleDocumentIterator) Next() (id uint64, ok bool) {
+	return 0, false
+}
+
+// eligibleDocumentSelector is used to filter out documents that are eligible for
+// the KNN search from a pre-filter query.
+type eligibleDocumentSelector struct {
+	// segment ID -> segment local doc nums in a bitset
+	eligibleDocNums []*bitset.BitSet
+	is              *IndexSnapshot
+}
+
+// SegmentEligibleDocuments returns an EligibleDocumentList for the specified segment ID.
+func (eds *eligibleDocumentSelector) SegmentEligibleDocuments(segmentID int) index.EligibleDocumentList {
+	if eds.eligibleDocNums == nil || segmentID < 0 || segmentID >= len(eds.eligibleDocNums) {
+		return emptyEligibleDocumentList
+	}
+	bs := eds.eligibleDocNums[segmentID]
+	if bs == nil {
+		// no eligible documents for this segment
+		return emptyEligibleDocumentList
+	}
+	return &eligibleDocumentList{
+		bs: bs,
+	}
+}
+
+// AddEligibleDocumentMatch adds a document match to the list of eligible documents.
+func (eds *eligibleDocumentSelector) AddEligibleDocumentMatch(id index.IndexInternalID) error {
+	if eds.is == nil {
+		return fmt.Errorf("eligibleDocumentSelector is not initialized with IndexSnapshot")
+	}
+	// Get the segment number and the local doc number for this document.
+	segIdx, docNum, err := eds.is.segmentIndexAndLocalDocNum(id)
+	if err != nil {
+		return err
+	}
+	// allocate a bitset for this segment if needed
+	if eds.eligibleDocNums[segIdx] == nil {
+		// the size of the bitset is the full size of the segment (which is the max local doc num + 1)
+		eds.eligibleDocNums[segIdx] = bitset.New(uint(eds.is.segment[segIdx].FullSize()))
+	}
+	// Add the local doc number to the list of eligible doc numbers for this segment.
+	eds.eligibleDocNums[segIdx].Set(uint(docNum))
+	return nil
+}
+
+func (is *IndexSnapshot) NewEligibleDocumentSelector() index.EligibleDocumentSelector {
+	return &eligibleDocumentSelector{
+		eligibleDocNums: make([]*bitset.BitSet, len(is.segment)),
+		is:              is,
+	}
+}

index/scorch/stats.go

@@ -0,0 +1,172 @@
+//  Copyright (c) 2017 Couchbase, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// 		http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package scorch
+
+import (
+	"reflect"
+	"sync/atomic"
+
+	"github.com/blevesearch/bleve/v2/util"
+)
+
+// Stats tracks statistics about the index, fields that are
+// prefixed like CurXxxx are gauges (can go up and down),
+// and fields that are prefixed like TotXxxx are monotonically
+// increasing counters.
+type Stats struct {
+	TotUpdates uint64
+	TotDeletes uint64
+
+	TotBatches        uint64
+	TotBatchesEmpty   uint64
+	TotBatchIntroTime uint64
+	MaxBatchIntroTime uint64
+
+	CurRootEpoch       uint64
+	LastPersistedEpoch uint64
+	LastMergedEpoch    uint64
+
+	TotOnErrors uint64
+
+	TotAnalysisTime uint64
+	TotIndexTime    uint64
+
+	TotIndexedPlainTextBytes uint64
+
+	TotBytesReadAtQueryTime    uint64
+	TotBytesWrittenAtIndexTime uint64
+
+	TotTermSearchersStarted  uint64
+	TotTermSearchersFinished uint64
+
+	TotKNNSearches     uint64
+	TotSynonymSearches uint64
+
+	TotEventTriggerStarted   uint64
+	TotEventTriggerCompleted uint64
+
+	TotIntroduceLoop       uint64
+	TotIntroduceSegmentBeg uint64
+	TotIntroduceSegmentEnd uint64
+	TotIntroducePersistBeg uint64
+	TotIntroducePersistEnd uint64
+	TotIntroduceMergeBeg   uint64
+	TotIntroduceMergeEnd   uint64
+
+	TotIntroducedItems         uint64
+	TotIntroducedSegmentsBatch uint64
+	TotIntroducedSegmentsMerge uint64
+
+	TotPersistLoopBeg          uint64
+	TotPersistLoopErr          uint64
+	TotPersistLoopProgress     uint64
+	TotPersistLoopWait         uint64
+	TotPersistLoopWaitNotified uint64
+	TotPersistLoopEnd          uint64
+
+	TotPersistedItems    uint64
+	TotItemsToPersist    uint64
+	TotPersistedSegments uint64
+	TotMutationsFiltered uint64
+
+	TotPersisterSlowMergerPause  uint64
+	TotPersisterSlowMergerResume uint64
+
+	TotPersisterNapPauseCompleted uint64
+	TotPersisterMergerNapBreak    uint64
+
+	TotFileMergeLoopBeg uint64
+	TotFileMergeLoopErr uint64
+	TotFileMergeLoopEnd uint64
+
+	TotFileMergeForceOpsStarted   uint64
+	TotFileMergeForceOpsCompleted uint64
+
+	TotFileMergePlan     uint64
+	TotFileMergePlanErr  uint64
+	TotFileMergePlanNone uint64
+	TotFileMergePlanOk   uint64
+
+	TotFileMergePlanTasks              uint64
+	TotFileMergePlanTasksDone          uint64
+	TotFileMergePlanTasksErr           uint64
+	TotFileMergePlanTasksSegments      uint64
+	TotFileMergePlanTasksSegmentsEmpty uint64
+
+	TotFileMergeSegmentsEmpty uint64
+	TotFileMergeSegments      uint64
+	TotFileSegmentsAtRoot     uint64
+	TotFileMergeWrittenBytes  uint64
+
+	TotFileMergeZapBeg              uint64
+	TotFileMergeZapEnd              uint64
+	TotFileMergeZapTime             uint64
+	MaxFileMergeZapTime             uint64
+	TotFileMergeZapIntroductionTime uint64
+	MaxFileMergeZapIntroductionTime uint64
+
+	TotFileMergeIntroductions          uint64
+	TotFileMergeIntroductionsDone      uint64
+	TotFileMergeIntroductionsSkipped   uint64
+	TotFileMergeIntroductionsObsoleted uint64
+
+	CurFilesIneligibleForRemoval     uint64
+	TotSnapshotsRemovedFromMetaStore uint64
+
+	TotMemMergeBeg      uint64
+	TotMemMergeErr      uint64
+	TotMemMergeDone     uint64
+	TotMemMergeSegments uint64
+
+	TotMemMergeZapBeg              uint64
+	TotMemMergeZapEnd              uint64
+	TotMemMergeZapTime             uint64
+	MaxMemMergeZapTime             uint64
+	TotMemMergeZapIntroductionTime uint64
+	MaxMemMergeZapIntroductionTime uint64
+
+	TotMemMergeIntroductions          uint64
+	TotMemMergeIntroductionsDone      uint64
+	TotMemMergeIntroductionsSkipped   uint64
+	TotMemMergeIntroductionsObsoleted uint64
+
+	TotMemorySegmentsAtRoot uint64
+
+	TotTrainedSamples          uint64
+	TotTrainTime               uint64
+	TotTrainFireIndexEventTime uint64
+	TotTrainFireIndexEvents    uint64
+}
+
+// atomically populates the returned map
+func (s *Stats) ToMap() map[string]interface{} {
+	m := map[string]interface{}{}
+	sve := reflect.ValueOf(s).Elem()
+	svet := sve.Type()
+	for i := 0; i < svet.NumField(); i++ {
+		svef := sve.Field(i)
+		if svef.CanAddr() {
+			svefp := svef.Addr().Interface()
+			m[svet.Field(i).Name] = atomic.LoadUint64(svefp.(*uint64))
+		}
+	}
+	return m
+}
+
+// MarshalJSON implements json.Marshaler, and in contrast to standard
+// json marshaling provides atomic safety
+func (s *Stats) MarshalJSON() ([]byte, error) {
+	return util.MarshalJSON(s.ToMap())
+}

index/scorch/train_noop.go

@@ -0,0 +1,63 @@
+//  Copyright (c) 2026 Couchbase, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// 		http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//go:build !vectors
+// +build !vectors
+
+package scorch
+
+import (
+	"fmt"
+
+	"github.com/blevesearch/bleve/v2/util"
+	index "github.com/blevesearch/bleve_index_api"
+)
+
+func initTrainer(s *Scorch, config map[string]interface{}) *noopTrainer {
+	return nil
+}
+
+type noopTrainer struct {
+}
+
+func (t *noopTrainer) trainLoop() {}
+
+func (t *noopTrainer) train(batch *index.Batch) error {
+	return fmt.Errorf("training is not supported with this build")
+}
+
+func (t *noopTrainer) loadTrainedData(bucket *util.BoltBucketImpl) error {
+	// noop
+	return nil
+}
+
+func (t *noopTrainer) getInternal(key []byte) ([]byte, error) {
+	return nil, nil
+}
+
+func (t *noopTrainer) copyFileLOCKED(file string, d index.IndexDirectory) error {
+	return nil
+}
+
+func (t *noopTrainer) updateBolt(snapshotsBucket *util.BoltBucketImpl, key []byte, value []byte) error {
+	return nil
+}
+
+func (t *noopTrainer) dropFileWriterIDs(ids map[string]struct{}) error {
+	return nil
+}
+
+func (t *noopTrainer) fileWriterIDsInUse() (map[string]struct{}, error) {
+	return nil, nil
+}

index/scorch/train_vector.go

@@ -0,0 +1,626 @@
+//  Copyright (c) 2026 Couchbase, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// 		http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//go:build vectors
+// +build vectors
+
+package scorch
+
+import (
+	"encoding/binary"
+	"fmt"
+	"maps"
+	"os"
+	"path/filepath"
+	"strconv"
+	"strings"
+	"sync"
+	"sync/atomic"
+	"time"
+
+	"github.com/RoaringBitmap/roaring/v2"
+	"github.com/blevesearch/bleve/v2/util"
+	index "github.com/blevesearch/bleve_index_api"
+	segment "github.com/blevesearch/scorch_segment_api/v2"
+)
+
+type trainRequest struct {
+	finalSample          bool
+	sampleSize           int
+	ackCh                chan error
+	sample               segment.Segment
+	trainingParams       *index.TrainingParams
+	removedFileWriterIDs map[string]struct{}
+}
+
+type vectorTrainer struct {
+	trainingComplete atomic.Bool
+	trainedSamples   uint64
+	parent           *Scorch
+	config           map[string]interface{}
+
+	m sync.RWMutex
+	// not a searchable segment in the sense that it won't return
+	// the data vectors, returns trained centroid layout
+	trainedIndex *SegmentSnapshot
+	trainCh      chan *trainRequest
+	// closed by trainLoop when it exits (via close, training completion, or
+	// error). Producers select on it so they never block on a request that the
+	// trainer will never consume/ack.
+	doneCh chan struct{}
+}
+
+const IndexTrainedWithFastMerge = "vector_index_fast_merge"
+
+func initTrainer(s *Scorch, config map[string]interface{}) *vectorTrainer {
+	if f, ok := config[IndexTrainedWithFastMerge]; ok {
+		feature, ok := f.(bool)
+		if ok && feature {
+			trainer := vectorTrainer{
+				parent:  s,
+				config:  maps.Clone(s.segmentConfig),
+				trainCh: make(chan *trainRequest, 1),
+				doneCh:  make(chan struct{}),
+			}
+			// update the parent scorch config with the trainer's callback to fetch the trained index
+			s.segmentConfig[index.TrainedIndexCallback] = index.TrainedIndexCallbackFn(trainer.getTrainedIndex)
+			return &trainer
+		}
+	}
+	return nil
+}
+
+func moveFile(sourcePath, destPath string) error {
+	// rename is supposed to be atomic on the same filesystem
+	err := os.Rename(sourcePath, destPath)
+	if err != nil {
+		return fmt.Errorf("error renaming file: %v", err)
+	}
+	return nil
+}
+
+func (t *vectorTrainer) persistToBolt(trainReq *trainRequest) error {
+	tx, err := t.parent.rootBolt.Begin(true)
+	if err != nil {
+		return fmt.Errorf("error starting bolt transaction: %v", err)
+	}
+	defer tx.Rollback()
+
+	snapshotsBucket, err := tx.CreateBucketIfNotExists(util.BoltSnapshotsBucket)
+	if err != nil {
+		return fmt.Errorf("error creating snapshots bucket: %v", err)
+	}
+
+	trainerBucket, err := snapshotsBucket.CreateBucketIfNotExists(util.BoltTrainerKey)
+	if err != nil {
+		return fmt.Errorf("error creating trained index bucket: %v", err)
+	}
+
+	segmentPath := t.parent.path + string(os.PathSeparator) + index.TrainedIndexFileName
+	if _, err := os.Stat(segmentPath); !os.IsNotExist(err) {
+		err = trainerBucket.Put(util.BoltPathKey, []byte(index.TrainedIndexFileName), nil)
+		if err != nil {
+			return fmt.Errorf("error updating trained index bucket: %v", err)
+		}
+	}
+
+	t.trainingComplete.Store(trainReq.finalSample)
+	err = trainerBucket.Put(util.BoltTrainCompleteKey, []byte(strconv.FormatBool(trainReq.finalSample)), nil)
+	if err != nil {
+		return fmt.Errorf("error updating train complete key: %v", err)
+	}
+
+	totSamples := atomic.AddUint64(&t.trainedSamples, uint64(trainReq.sampleSize))
+	err = trainerBucket.Put(util.BoltTrainedSamplesKey, binary.LittleEndian.AppendUint64(nil, totSamples), nil)
+	if err != nil {
+		return fmt.Errorf("error updating trained samples key: %v", err)
+	}
+
+	err = tx.Commit()
+	if err != nil {
+		return fmt.Errorf("error committing bolt transaction: %v", err)
+	}
+
+	return t.parent.rootBolt.Sync()
+}
+
+// trainLoop is not a routine that runs throughout the lifetime of the index.
+// Its purpose is only to train the vector index before data ingestion starts.
+// It is the sole consumer of trainCh and the sole owner of trainedIndex
+// mutations originating from training.
+func (t *vectorTrainer) trainLoop() {
+	defer t.parent.asyncTasks.Done()
+	// signal any blocked producers that no further requests will be serviced.
+	defer close(t.doneCh)
+
+	start := time.Now()
+	path := filepath.Join(t.parent.path, index.TrainedIndexFileName)
+	// config is owned exclusively by this goroutine. t.config is read
+	// concurrently by train(), so it must not be mutated here.
+	config := maps.Clone(t.config)
+
+	for {
+		// exit once the final sample set has been ingested and training is complete.
+		if t.trainingComplete.Load() {
+			t.recordTrainingStats(start)
+			return
+		}
+		select {
+		case <-t.parent.closeCh:
+			// the deferred close(doneCh) releases any producer blocked on
+			// trainCh or its ackCh, so there is no need to drain here.
+			return
+		case req := <-t.trainCh:
+			shutdown, err := t.handleRequest(req, path, config)
+			ackRequest(req, err)
+			if shutdown {
+				return
+			}
+		}
+	}
+}
+
+// recordTrainingStats publishes the final training counters once the loop exits.
+func (t *vectorTrainer) recordTrainingStats(start time.Time) {
+	atomic.StoreUint64(&t.parent.stats.TotTrainedSamples, atomic.LoadUint64(&t.trainedSamples))
+	atomic.StoreUint64(&t.parent.stats.TotTrainTime, uint64(time.Since(start).Milliseconds()))
+}
+
+// ackRequest delivers a request's result to the producer waiting in submit and
+// then releases it. A non-nil err is sent before the close so submit observes
+// it; the send blocks until submit receives, which the shutdown handshake in
+// submit relies on.
+func ackRequest(req *trainRequest, err error) {
+	if err != nil {
+		req.ackCh <- err
+	}
+	close(req.ackCh)
+}
+
+// handleRequest processes a single training request. The returned err, if
+// non-nil, is already formatted for the producer. shutdown reports whether the
+// trainLoop must exit: every failure is fatal except a failed initial persist,
+// which leaves the trainer able to retry on the next sample.
+func (t *vectorTrainer) handleRequest(req *trainRequest, path string, config map[string]interface{}) (shutdown bool, err error) {
+	if req.trainingParams != nil {
+		// mutate the goroutine-local config only; t.config is read concurrently
+		// by train() and must stay immutable after init.
+		config[index.TrainingKey] = req.trainingParams
+	}
+
+	// remove any file writer ids that are no longer in use
+	if req.removedFileWriterIDs != nil {
+		if err := t.removeFileWriterIDs(req.removedFileWriterIDs); err != nil {
+			return true, fmt.Errorf("error removing file writer ids: %v", err)
+		}
+		return false, nil
+	}
+
+	// no sample segment: just persist state if this is the final sample.
+	if req.sample == nil {
+		if req.finalSample {
+			if err := t.persistToBolt(req); err != nil {
+				return true, fmt.Errorf("error persisting to bolt: %v", err)
+			}
+		}
+		return false, nil
+	}
+
+	// snapshot the current trained segment under the read lock; MergeUsing below
+	// only reads it, so concurrent getTrainedIndex readers stay safe.
+	t.m.RLock()
+	prev := t.trainedIndex
+	t.m.RUnlock()
+
+	if prev == nil {
+		if seg, ok := req.sample.(segment.UnpersistedSegment); ok {
+			if err := persistToDirectory(seg, nil, path); err != nil {
+				// recoverable: retry on the next sample rather than tear down.
+				return false, fmt.Errorf("error persisting segment: %v", err)
+			}
+		}
+	} else {
+		// merge the new sample with the existing index into a .tmp file; it is
+		// renamed into place by publishTrainedIndex (Os.Open on the live path is
+		// unsafe during the merge).
+		_, _, mErr := t.parent.segPlugin.MergeUsing([]segment.Segment{prev.segment, req.sample},
+			[]*roaring.Bitmap{nil, nil}, path+".tmp", t.parent.closeCh, nil, config)
+		if mErr != nil {
+			return true, fmt.Errorf("error merging trained index: %v", mErr)
+		}
+	}
+
+	if err := t.publishTrainedIndex(req, prev, path, config); err != nil {
+		return true, err
+	}
+	return false, nil
+}
+
+// publishTrainedIndex makes the freshly written index visible to readers. The
+// old segment handle must be closed before moveFile can replace the backing
+// file, so the close -> rename -> open -> pointer-swap sequence runs under the
+// write lock; otherwise a reader holding RLock could observe a closed segment
+// or a missing file. On any error past the close, the pointer is reset to nil
+// so readers never see the freed prev segment (callers fall back to naive
+// merge). The bolt write acts as a failover-recovery checkpoint: callers
+// downstream can rely on the trained index being available once this completes.
+// todo: rethink the frequency of bolt writes
+func (t *vectorTrainer) publishTrainedIndex(req *trainRequest, prev *SegmentSnapshot, path string, config map[string]interface{}) error {
+	t.m.Lock()
+	defer t.m.Unlock()
+
+	if prev != nil {
+		if err := prev.segment.Close(); err != nil {
+			t.trainedIndex = nil
+			return fmt.Errorf("error closing previous trained index: %v", err)
+		}
+		if err := moveFile(path+".tmp", path); err != nil {
+			t.trainedIndex = nil
+			return fmt.Errorf("error renaming trained index: %v", err)
+		}
+	}
+
+	if err := t.persistToBolt(req); err != nil {
+		if prev != nil {
+			t.trainedIndex = nil
+		}
+		return fmt.Errorf("error persisting to bolt: %v", err)
+	}
+
+	trainedIndex, err := t.parent.segPlugin.OpenUsing(path, config)
+	if err != nil {
+		t.trainedIndex = nil
+		return fmt.Errorf("error opening trained index: %v", err)
+	}
+	t.trainedIndex = &SegmentSnapshot{segment: trainedIndex}
+	return nil
+}
+
+// loadTrainedData loads the trained-index metadata from boltdb during init. The
+// trainedIndex write is guarded because callback-driven readers may already be
+// registered by the time this runs.
+func (t *vectorTrainer) loadTrainedData(bucket *util.BoltBucketImpl) error {
+	if bucket == nil {
+		return nil
+	}
+
+	reader, err := util.NewFileReader("", nil)
+	if err != nil {
+		return fmt.Errorf("error creating file reader: %v", err)
+	}
+
+	// segmentSnapshot is constructed only if the trained index file path exists. If
+	// application layer fails to create one and the underlying path doesn't exist, we'll
+	// just skip loading the trained index and fallback to naive merge
+	var segmentSnapshot *SegmentSnapshot
+	pathBytes, err := bucket.Get(util.BoltPathKey, nil)
+	if pathBytes != nil {
+		segmentSnapshot, err = t.parent.loadSegment(bucket, reader)
+		if err != nil {
+			return err
+		}
+	}
+
+	// get the training status out of bolt
+	trainComplete, err := bucket.Get(util.BoltTrainCompleteKey, nil)
+	if err != nil {
+		return fmt.Errorf("error getting train complete: %v", err)
+	}
+	trainedSamples, err := bucket.Get(util.BoltTrainedSamplesKey, nil)
+	if err != nil {
+		return fmt.Errorf("error getting trained samples: %v", err)
+	}
+	if len(trainedSamples) == 8 {
+		atomic.StoreUint64(&t.trainedSamples, binary.LittleEndian.Uint64(trainedSamples))
+	}
+	if len(trainComplete) > 0 {
+		comp, err := strconv.ParseBool(string(trainComplete))
+		if err != nil {
+			return fmt.Errorf("error parsing train complete: %v", err)
+		}
+		t.trainingComplete.Store(comp)
+	}
+
+	t.m.Lock()
+	defer t.m.Unlock()
+	t.trainedIndex = segmentSnapshot
+	return nil
+}
+
+func (t *vectorTrainer) train(batch *index.Batch) error {
+	// regulate the Train function
+	start := time.Now()
+	t.parent.FireIndexEvent()
+	atomic.AddUint64(&t.parent.stats.TotTrainFireIndexEventTime, uint64(time.Since(start)))
+	atomic.AddUint64(&t.parent.stats.TotTrainFireIndexEvents, 1)
+	if t.trainingComplete.Load() {
+		return fmt.Errorf("training is already complete, cannot accept more training data")
+	}
+
+	var trainData []index.Document
+	for _, doc := range batch.IndexOps {
+		if doc != nil {
+			// insert _id field
+			// no need to track updates/deletes over here since
+			// the API is singleton
+			doc.AddIDField()
+		}
+		trainData = append(trainData, doc)
+	}
+
+	trainComplete := batch.InternalOps[string(util.BoltTrainCompleteKey)]
+	if trainComplete == nil {
+		trainComplete = []byte("false")
+	}
+	fin, err := strconv.ParseBool(string(trainComplete))
+	if err != nil {
+		return fmt.Errorf("error parsing train complete: %v", err)
+	}
+
+	trainReq := &trainRequest{
+		finalSample: fin,
+		sampleSize:  len(trainData),
+		ackCh:       make(chan error),
+	}
+	// setting the training params using the internal value before the actual
+	// training has started
+	config := t.config
+	if atomic.LoadUint64(&t.trainedSamples) == 0 {
+		trainingParamsBytes := batch.InternalOps[index.TrainingKey]
+		var trainingParams index.TrainingParams
+		if trainingParamsBytes != nil {
+			err = util.UnmarshalJSON(trainingParamsBytes, &trainingParams)
+			if err != nil {
+				return fmt.Errorf("error parsing training params: %v", err)
+			}
+			trainReq.trainingParams = &trainingParams
+			config = maps.Clone(t.config)
+			config[index.TrainingKey] = &trainingParams
+		}
+	}
+
+	// just builds a new vector index out of the train data provided
+	// this is not necessarily the final train data since this is submitted
+	// as a request to the trainer component to be merged. once the training
+	// is complete, the template will be used for other operations down the line
+	// like merge and search.
+	//
+	// todo: updates/deletes -> data drift detection
+	if len(trainData) > 0 {
+		trainReq.sample, _, err = t.parent.segPlugin.NewUsing(trainData, config)
+		if err != nil {
+			return err
+		}
+	}
+
+	if err := t.submit(trainReq); err != nil {
+		return fmt.Errorf("train_vector: train() err'd out with: %w", err)
+	}
+
+	return nil
+}
+
+// submit hands a request to the trainLoop and waits for its ack. It never
+// blocks past trainer shutdown: if the trainLoop has already exited (via close,
+// training completion, or error) the doneCh releases the caller instead of
+// letting it hang on a request that will never be consumed or acked.
+func (t *vectorTrainer) submit(req *trainRequest) error {
+	select {
+	case t.trainCh <- req:
+	case <-t.doneCh:
+		return fmt.Errorf("trainer is closed")
+	}
+
+	select {
+	case err := <-req.ackCh:
+		return err
+	case <-t.doneCh:
+		// trainLoop acks every request it consumes before returning, so if our
+		// request was processed the ack is already available — prefer it over
+		// reporting a shutdown that didn't actually drop this request.
+		select {
+		case err := <-req.ackCh:
+			return err
+		default:
+			return fmt.Errorf("trainer is closed")
+		}
+	}
+}
+
+func (t *vectorTrainer) getInternal(key []byte) ([]byte, error) {
+	switch string(key) {
+	case string(util.BoltTrainCompleteKey):
+		return []byte(strconv.FormatBool(t.trainingComplete.Load())), nil
+	}
+	return nil, nil
+}
+
+func (t *vectorTrainer) getTrainedIndex(field string) (interface{}, error) {
+	// return the coarse quantizer of the trained faiss index belonging to the field
+	// if its not available then zap performs naive merge
+	t.m.RLock()
+	defer t.m.RUnlock()
+	if t.trainedIndex != nil {
+		trainedSegment, ok := t.trainedIndex.segment.(segment.TrainedSegment)
+		if !ok {
+			return nil, fmt.Errorf("segment is not a trained index segment")
+		}
+
+		coarseQuantizer, err := trainedSegment.GetCoarseQuantizer(field)
+		if err != nil {
+			return nil, err
+		}
+		return coarseQuantizer, nil
+	}
+	return nil, nil
+}
+
+func (t *vectorTrainer) copyFileLOCKED(file string, d index.IndexDirectory) error {
+	if strings.HasSuffix(file, index.TrainedIndexFileName) {
+		// trained index file - this is outside the snapshots domain so the bolt update is different
+		err := d.SetPathInBolt(util.BoltTrainerKey, []byte(file))
+		if err != nil {
+			return fmt.Errorf("error updating dest index bolt: %w", err)
+		}
+
+		// update the stats in destination
+		err = d.SetPathInBolt(util.BoltTrainedSamplesKey, binary.LittleEndian.AppendUint64(nil, atomic.LoadUint64(&t.trainedSamples)))
+		if err != nil {
+			return fmt.Errorf("error updating trained samples key: %w", err)
+		}
+	}
+
+	return nil
+}
+
+func (t *vectorTrainer) updateBolt(snapshotsBucket *util.BoltBucketImpl, key []byte, value []byte) error {
+	switch string(key) {
+	case string(util.BoltTrainerKey):
+		trainerBucket, err := snapshotsBucket.CreateBucketIfNotExists(util.BoltTrainerKey)
+		if err != nil {
+			return err
+		}
+		if trainerBucket == nil {
+			return fmt.Errorf("trainer bucket not found")
+		}
+
+		// guard against duplicate updates
+		existingValue, err := trainerBucket.Get(util.BoltPathKey, nil)
+		if err != nil {
+			return fmt.Errorf("error checking existing value: %v", err)
+		}
+		if existingValue != nil {
+			return fmt.Errorf("key already exists %v %v", t.parent.path, string(existingValue))
+		}
+
+		err = trainerBucket.Put(util.BoltPathKey, value, nil)
+		if err != nil {
+			return err
+		}
+
+		reader, err := util.NewFileReader("", nil)
+		if err != nil {
+			return fmt.Errorf("error creating file reader: %v", err)
+		}
+
+		// update the centroid index pointer under the write lock; getTrainedIndex
+		// and trainLoop may read t.trainedIndex concurrently.
+		seg, err := t.parent.loadSegment(trainerBucket, reader)
+		if err != nil {
+			return err
+		}
+		t.m.Lock()
+		t.trainedIndex = seg
+		t.m.Unlock()
+
+	case string(util.BoltTrainedSamplesKey):
+		trainedSamples := binary.LittleEndian.Uint64(value)
+		atomic.StoreUint64(&t.trainedSamples, trainedSamples)
+		trainerBucket, err := snapshotsBucket.CreateBucketIfNotExists(util.BoltTrainerKey)
+		if err != nil {
+			return err
+		}
+		err = trainerBucket.Put(util.BoltTrainedSamplesKey, value, nil)
+		if err != nil {
+			return err
+		}
+	}
+
+	return nil
+}
+
+// dropFileWriterIDs removes the given file writer ids from the trained index.
+// This is a lifecycle operation (key rotation) that must work for the entire
+// life of the trained index, not just while training is running.
+func (t *vectorTrainer) dropFileWriterIDs(ids map[string]struct{}) error {
+	if t.trainingComplete.Load() {
+		if err := t.removeFileWriterIDs(ids); err != nil {
+			return fmt.Errorf("train_vector: dropFileWriterIDs() err'd out with: %w", err)
+		}
+		return nil
+	}
+
+	trainReq := &trainRequest{
+		ackCh:                make(chan error),
+		removedFileWriterIDs: ids,
+	}
+	if err := t.submit(trainReq); err != nil {
+		return fmt.Errorf("train_vector: dropFileWriterIDs() err'd out with: %w", err)
+	}
+
+	return nil
+}
+
+func (t *vectorTrainer) removeFileWriterIDs(ids map[string]struct{}) error {
+	// trainer bucket in the bolt file doesn't have any sensitive data to be encrypted,
+	// so we don't have to update it with new writer ids.
+	// invoke the merge API and let the zap side of things handle the update of writer IDs and also
+	// the data in the trained index.
+	t.m.Lock()
+	defer t.m.Unlock()
+	trainedIndexPath := filepath.Join(t.parent.path, index.TrainedIndexFileName)
+	if t.trainedIndex != nil {
+		if encryptedIndex, ok := t.trainedIndex.segment.(segment.SegmentWithCallbacks); ok {
+			if _, ok := ids[encryptedIndex.CallbackId()]; ok {
+				_, _, err := t.parent.segPlugin.MergeUsing([]segment.Segment{t.trainedIndex.segment},
+					[]*roaring.Bitmap{nil}, trainedIndexPath+"temp", t.parent.closeCh, nil, t.config)
+				if err != nil {
+					if serr := os.Remove(trainedIndexPath + "temp"); serr != nil {
+						return fmt.Errorf("error removing temp trained index: %v", serr)
+					}
+					return err
+				}
+
+				err = t.trainedIndex.segment.Close()
+				if err != nil {
+					if serr := os.Remove(trainedIndexPath + "temp"); serr != nil {
+						return fmt.Errorf("error removing temp trained index: %v", serr)
+					}
+					t.trainedIndex = nil
+					return err
+				}
+
+				if err := moveFile(trainedIndexPath+"temp", trainedIndexPath); err != nil {
+					if serr := os.Remove(trainedIndexPath + "temp"); serr != nil {
+						return fmt.Errorf("error removing temp trained index: %v", serr)
+					}
+					t.trainedIndex = nil
+					return err
+				}
+
+				trainedIndex, err := t.parent.segPlugin.OpenUsing(trainedIndexPath, t.config)
+				if err != nil {
+					t.trainedIndex = nil
+					return err
+				}
+
+				t.trainedIndex = &SegmentSnapshot{segment: trainedIndex}
+			}
+		}
+	}
+	return nil
+}
+
+func (t *vectorTrainer) fileWriterIDsInUse() (map[string]struct{}, error) {
+	t.m.RLock()
+	defer t.m.RUnlock()
+	writerIDs := make(map[string]struct{})
+	if t.trainedIndex != nil {
+		if seg, ok := t.trainedIndex.segment.(segment.SegmentWithCallbacks); ok {
+			writerIDs[seg.CallbackId()] = struct{}{}
+		}
+	}
+
+	return writerIDs, nil
+}

index/scorch/train_vector_test.go

@@ -0,0 +1,849 @@
+//  Copyright (c) 2026 Couchbase, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// 		http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//go:build vectors
+// +build vectors
+
+package scorch
+
+import (
+	"fmt"
+	"io"
+	"os"
+	"path/filepath"
+	"strings"
+	"sync"
+	"sync/atomic"
+	"testing"
+	"time"
+
+	"github.com/blevesearch/bleve/v2/document"
+	"github.com/blevesearch/bleve/v2/util"
+	index "github.com/blevesearch/bleve_index_api"
+)
+
+// newMinimalScorchForTrainer returns a *Scorch with just the fields required by
+// vectorTrainer tests that do not touch BoltDB or the segment plugin.
+func newMinimalScorchForTrainer(t *testing.T) *Scorch {
+	t.Helper()
+	return &Scorch{
+		closeCh:       make(chan struct{}),
+		segmentConfig: make(map[string]interface{}),
+		config:        make(map[string]interface{}),
+		path:          t.TempDir(),
+	}
+}
+
+// waitGroupDone blocks until wg.Wait() returns or d elapses, returning true
+// iff the wait completed in time.
+func waitGroupDone(wg *sync.WaitGroup, d time.Duration) bool {
+	done := make(chan struct{})
+	go func() { wg.Wait(); close(done) }()
+	select {
+	case <-done:
+		return true
+	case <-time.After(d):
+		return false
+	}
+}
+
+// -----------------------------------------------------------------------
+// initTrainer
+// -----------------------------------------------------------------------
+
+func TestInitTrainerFlagAbsent(t *testing.T) {
+	s := newMinimalScorchForTrainer(t)
+	if tr := initTrainer(s, s.config); tr != nil {
+		t.Fatal("expected nil trainer when flag is absent, got non-nil")
+	}
+}
+
+func TestInitTrainerFlagFalse(t *testing.T) {
+	s := newMinimalScorchForTrainer(t)
+	cfg := map[string]interface{}{IndexTrainedWithFastMerge: false}
+	if tr := initTrainer(s, cfg); tr != nil {
+		t.Fatal("expected nil trainer when flag is false, got non-nil")
+	}
+}
+
+func TestInitTrainerReturnsTrainer(t *testing.T) {
+	s := newMinimalScorchForTrainer(t)
+	cfg := map[string]interface{}{IndexTrainedWithFastMerge: true}
+	if tr := initTrainer(s, cfg); tr == nil {
+		t.Fatal("expected non-nil trainer when flag is true")
+	}
+}
+
+func TestInitTrainerClonesConfig(t *testing.T) {
+	s := newMinimalScorchForTrainer(t)
+	s.config["pre-existing"] = "value"
+	cfg := map[string]interface{}{IndexTrainedWithFastMerge: true}
+	tr := initTrainer(s, cfg)
+	if tr == nil {
+		t.Fatal("expected non-nil trainer")
+	}
+	// Mutations to the trainer's config must not affect the parent's config.
+	tr.config["canary"] = true
+	if _, leaked := s.config["canary"]; leaked {
+		t.Error("trainer config is the same map as parent config — expected a clone")
+	}
+}
+
+// -----------------------------------------------------------------------
+// getInternal
+// -----------------------------------------------------------------------
+
+func TestGetInternalDefaultFalse(t *testing.T) {
+	s := newMinimalScorchForTrainer(t)
+	tr := &vectorTrainer{parent: s, config: map[string]interface{}{}}
+
+	got, err := tr.getInternal(util.BoltTrainCompleteKey)
+	if err != nil {
+		t.Fatalf("unexpected error: %v", err)
+	}
+	if string(got) != "false" {
+		t.Errorf("expected 'false', got %q", got)
+	}
+}
+
+func TestGetInternalTrueAfterSet(t *testing.T) {
+	s := newMinimalScorchForTrainer(t)
+	tr := &vectorTrainer{parent: s, config: map[string]interface{}{}}
+	tr.trainingComplete.Store(true)
+
+	got, err := tr.getInternal(util.BoltTrainCompleteKey)
+	if err != nil {
+		t.Fatalf("unexpected error: %v", err)
+	}
+	if string(got) != "true" {
+		t.Errorf("expected 'true', got %q", got)
+	}
+}
+
+func TestGetInternalUnknownKey(t *testing.T) {
+	s := newMinimalScorchForTrainer(t)
+	tr := &vectorTrainer{parent: s, config: map[string]interface{}{}}
+
+	got, err := tr.getInternal([]byte("no-such-key"))
+	if err != nil {
+		t.Fatalf("unexpected error: %v", err)
+	}
+	if got != nil {
+		t.Errorf("expected nil for unknown key, got %v", got)
+	}
+}
+
+// -----------------------------------------------------------------------
+// getTrainedIndex
+// -----------------------------------------------------------------------
+
+func TestGetTrainedIndexNil(t *testing.T) {
+	s := newMinimalScorchForTrainer(t)
+	tr := &vectorTrainer{parent: s, config: map[string]interface{}{}}
+
+	got, err := tr.getTrainedIndex("vec")
+	if err != nil {
+		t.Fatalf("unexpected error: %v", err)
+	}
+	if got != nil {
+		t.Errorf("expected nil, got %v", got)
+	}
+}
+
+func TestGetTrainedIndexNotTrainedSegment(t *testing.T) {
+	s := newMinimalScorchForTrainer(t)
+	tr := &vectorTrainer{
+		parent: s,
+		config: map[string]interface{}{},
+		// mockSegmentBase does NOT implement segment.TrainedSegment.
+		trainedIndex: &SegmentSnapshot{segment: &mockSegmentBase{}},
+	}
+
+	_, err := tr.getTrainedIndex("vec")
+	if err == nil {
+		t.Fatal("expected error for segment that does not implement TrainedSegment")
+	}
+}
+
+// mockTrainedSeg embeds mockSegmentBase and additionally satisfies
+// segment.TrainedSegment by implementing GetCoarseQuantizer.
+type mockTrainedSeg struct {
+	mockSegmentBase
+	coarseQuantizerFn func(string) (interface{}, error)
+}
+
+func (m *mockTrainedSeg) GetCoarseQuantizer(field string) (interface{}, error) {
+	if m.coarseQuantizerFn != nil {
+		return m.coarseQuantizerFn(field)
+	}
+	return nil, nil
+}
+
+func TestGetTrainedIndexReturnsCQ(t *testing.T) {
+	sentinel := struct{ label string }{"centroid"}
+	s := newMinimalScorchForTrainer(t)
+	tr := &vectorTrainer{
+		parent: s,
+		config: map[string]interface{}{},
+		trainedIndex: &SegmentSnapshot{
+			segment: &mockTrainedSeg{
+				coarseQuantizerFn: func(string) (interface{}, error) {
+					return sentinel, nil
+				},
+			},
+		},
+	}
+
+	got, err := tr.getTrainedIndex("vec")
+	if err != nil {
+		t.Fatalf("unexpected error: %v", err)
+	}
+	if got != sentinel {
+		t.Errorf("expected sentinel coarse quantizer, got %v", got)
+	}
+}
+
+func TestGetTrainedIndexPropagatesError(t *testing.T) {
+	s := newMinimalScorchForTrainer(t)
+	tr := &vectorTrainer{
+		parent: s,
+		config: map[string]interface{}{},
+		trainedIndex: &SegmentSnapshot{
+			segment: &mockTrainedSeg{
+				coarseQuantizerFn: func(string) (interface{}, error) {
+					return nil, fmt.Errorf("coarse quantizer unavailable")
+				},
+			},
+		},
+	}
+
+	_, err := tr.getTrainedIndex("vec")
+	if err == nil {
+		t.Fatal("expected error from GetCoarseQuantizer, got nil")
+	}
+}
+
+// -----------------------------------------------------------------------
+// trainLoop — exit paths that do not require BoltDB
+// -----------------------------------------------------------------------
+
+// TestTrainLoopExitsWhenComplete verifies that the loop returns
+// immediately (without touching any channel) when trainingComplete is already
+// set, and that it flushes the sample count into parent stats.
+func TestTrainLoopExitsWhenComplete(t *testing.T) {
+	s := newMinimalScorchForTrainer(t)
+	tr := &vectorTrainer{
+		parent:  s,
+		config:  map[string]interface{}{},
+		trainCh: make(chan *trainRequest, 1),
+	}
+	tr.trainedSamples = 42
+	tr.trainingComplete.Store(true)
+
+	s.asyncTasks.Add(1)
+	go tr.trainLoop()
+
+	if !waitGroupDone(&s.asyncTasks, 2*time.Second) {
+		t.Fatal("trainLoop did not exit within deadline")
+	}
+	if got := atomic.LoadUint64(&s.stats.TotTrainedSamples); got != 42 {
+		t.Errorf("expected TotTrainedSamples=42, got %d", got)
+	}
+}
+
+// TestTrainLoopCloseChNoRequest verifies that closing the
+// parent's closeCh causes the loop to exit cleanly when there is no buffered
+// train request.
+func TestTrainLoopCloseChNoRequest(t *testing.T) {
+	s := newMinimalScorchForTrainer(t)
+	tr := &vectorTrainer{
+		parent:  s,
+		config:  map[string]interface{}{},
+		trainCh: make(chan *trainRequest, 1),
+	}
+
+	s.asyncTasks.Add(1)
+	go tr.trainLoop()
+
+	close(s.closeCh)
+	if !waitGroupDone(&s.asyncTasks, 2*time.Second) {
+		t.Fatal("trainLoop did not exit within deadline")
+	}
+}
+
+// TestTrainLoopCloseChAcksRequest verifies that the goroutine
+// exits after closeCh is closed, and that any ackCh buffered in trainCh is
+// handled before the goroutine returns (no leaked ackCh).
+//
+// Due to Go's non-deterministic select, the loop may either:
+//
+//	(a) react to closeCh first → drain the request and ack it with an error, OR
+//	(b) consume the request first (nil-sample, non-final → close ackCh) and
+//	    then exit via closeCh on the next iteration.
+//
+// Either outcome is correct. The invariant under test is that ackCh is
+// drained/closed before the goroutine exits, preventing a caller goroutine leak.
+func TestTrainLoopCloseChAcksRequest(t *testing.T) {
+	s := newMinimalScorchForTrainer(t)
+	tr := &vectorTrainer{
+		parent:  s,
+		config:  map[string]interface{}{},
+		trainCh: make(chan *trainRequest, 1),
+	}
+
+	// Buffered so that the loop's non-blocking write (if it takes closeCh path)
+	// does not deadlock.
+	ackCh := make(chan error, 1)
+	tr.trainCh <- &trainRequest{finalSample: false, sample: nil, ackCh: ackCh}
+
+	s.asyncTasks.Add(1)
+	go tr.trainLoop()
+
+	close(s.closeCh)
+	if !waitGroupDone(&s.asyncTasks, 2*time.Second) {
+		t.Fatal("trainLoop did not exit within deadline")
+	}
+
+	// ackCh must be drainable without blocking regardless of which select path
+	// was taken.
+	select {
+	case <-ackCh:
+		// received error (closeCh drain path) or nil (closed by normal processing)
+	case <-time.After(100 * time.Millisecond):
+		t.Fatal("ackCh was not handled after trainLoop exited — possible goroutine leak")
+	}
+}
+
+// TestTrainLoopNilSampleAcks verifies that a nil-sample,
+// non-final request is acknowledged without error, and that the loop continues
+// running afterwards (cleaned up by closing closeCh).
+func TestTrainLoopNilSampleAcks(t *testing.T) {
+	s := newMinimalScorchForTrainer(t)
+	tr := &vectorTrainer{
+		parent:  s,
+		config:  map[string]interface{}{},
+		trainCh: make(chan *trainRequest, 1),
+	}
+
+	ackCh := make(chan error, 1)
+	tr.trainCh <- &trainRequest{finalSample: false, sample: nil, ackCh: ackCh}
+
+	s.asyncTasks.Add(1)
+	go tr.trainLoop()
+
+	// The loop should close ackCh with no value written to it.
+	select {
+	case err := <-ackCh:
+		if err != nil {
+			t.Errorf("expected nil error for nil-sample non-final request, got: %v", err)
+		}
+	case <-time.After(2 * time.Second):
+		t.Fatal("timed out waiting for ackCh to be closed")
+	}
+
+	// Stop the loop cleanly.
+	close(s.closeCh)
+	if !waitGroupDone(&s.asyncTasks, 2*time.Second) {
+		t.Fatal("trainLoop did not exit within deadline after closeCh")
+	}
+}
+
+// mockUnpersistedSeg satisfies both segment.Segment (via mockSegmentBase) and
+// segment.UnpersistedSegment.  persistFn controls the return value of Persist.
+type mockUnpersistedSeg struct {
+	mockSegmentBase
+	persistFn func(path string) error
+}
+
+func (m *mockUnpersistedSeg) Persist(path string) error {
+	if m.persistFn != nil {
+		return m.persistFn(path)
+	}
+	return nil
+}
+
+// TestTrainLoopPersistErrorAcks verifies that
+// when persist-to-directory fails for a non-nil UnpersistedSegment, the loop:
+//  1. sends an error on ackCh and closes it, and
+//  2. continues running (does not return), so a subsequent closeCh can stop it.
+func TestTrainLoopPersistErrorAcks(t *testing.T) {
+	s := newMinimalScorchForTrainer(t)
+	tr := &vectorTrainer{
+		parent:  s,
+		config:  map[string]interface{}{},
+		trainCh: make(chan *trainRequest, 1),
+	}
+
+	ackCh := make(chan error, 1)
+	tr.trainCh <- &trainRequest{
+		finalSample: false,
+		sample: &mockUnpersistedSeg{
+			persistFn: func(string) error { return fmt.Errorf("disk full") },
+		},
+		ackCh: ackCh,
+	}
+
+	s.asyncTasks.Add(1)
+	go tr.trainLoop()
+
+	// The loop should write an error and close ackCh before continuing.
+	select {
+	case err, ok := <-ackCh:
+		if !ok {
+			t.Fatal("ackCh was closed before the error value could be read")
+		}
+		if err == nil {
+			t.Fatal("expected non-nil error from failed persist, got nil")
+		}
+	case <-time.After(2 * time.Second):
+		t.Fatal("timed out waiting for error on ackCh")
+	}
+
+	// Loop must still be running; stop it cleanly.
+	close(s.closeCh)
+	if !waitGroupDone(&s.asyncTasks, 2*time.Second) {
+		t.Fatal("trainLoop did not exit within deadline after closeCh")
+	}
+}
+
+func TestTrainerSampleIngestion(t *testing.T) {
+	cfg := CreateConfig("TestTrainerSampleIngestion")
+	cfg[IndexTrainedWithFastMerge] = true
+	err := InitTest(cfg)
+	if err != nil {
+		t.Fatal(err)
+	}
+	analysisQueue := index.NewAnalysisQueue(1)
+	s, err := NewScorch(Name, cfg, analysisQueue)
+	if err != nil {
+		t.Fatalf("failed to create Scorch: %v", err)
+	}
+	defer s.Close()
+
+	sc, ok := s.(*Scorch)
+	if !ok {
+		t.Fatal("expected Scorch instance")
+	}
+
+	batch := index.NewBatch()
+	for i := 0; i < 100; i++ {
+		doc := document.NewDocument(fmt.Sprintf("doc-%d", i))
+		stubVal := float32(i)
+		doc.AddField(document.NewVectorField("vec", nil, []float32{stubVal, stubVal + 1, stubVal + 2}, 3, "cosine", index.IndexOptimizedForRecall))
+		batch.Update(doc)
+	}
+
+	tr, ok := sc.trainer.(*vectorTrainer)
+	if !ok {
+		t.Fatal("expected vectorTrainer instance")
+	}
+	if sc.rootBolt == nil {
+		err := sc.openBolt()
+		if err != nil {
+			t.Fatalf("opening bolt failed %v", err)
+		}
+	}
+	sc.asyncTasks.Add(1)
+	go tr.trainLoop()
+
+	err = sc.Train(batch)
+	if err != nil {
+		t.Fatalf("training failed: %v", err)
+	}
+
+	val, err := tr.getInternal(util.BoltTrainCompleteKey)
+	if err != nil {
+		t.Fatalf("failed to get internal value: %v", err)
+	}
+
+	if string(val) != "false" {
+		t.Errorf("expected 'false' for training complete key, got %q", val)
+	}
+
+	if tr.trainedSamples != 100 {
+		t.Errorf("expected 100 trained samples, got %d", tr.trainedSamples)
+	}
+
+	b := index.NewBatch()
+	doc := document.NewDocument("doc-998")
+	doc.AddField(document.NewVectorField("vec", nil, []float32{998, 998, 998}, 3, "cosine", index.IndexOptimizedForRecall))
+	b.Update(doc)
+	err = sc.Train(b)
+	if err != nil {
+		t.Fatalf("training failed: %v", err)
+	}
+	val, err = tr.getInternal(util.BoltTrainCompleteKey)
+	if err != nil {
+		t.Fatalf("failed to get internal value: %v", err)
+	}
+
+	if string(val) != "false" {
+		t.Errorf("expected 'false' for training complete key, got %q", val)
+	}
+
+	if tr.trainedSamples != 101 {
+		t.Errorf("expected 101 trained samples, got %d", tr.trainedSamples)
+	}
+
+	b.Reset()
+	b.SetInternal(util.BoltTrainCompleteKey, []byte("true"))
+	sc.Train(b)
+	if err != nil {
+		t.Fatalf("training failed: %v", err)
+	}
+
+	val, err = tr.getInternal(util.BoltTrainCompleteKey)
+	if err != nil {
+		t.Fatalf("failed to get internal value: %v", err)
+	}
+
+	if string(val) != "true" {
+		t.Errorf("expected 'true' for training complete key, got %q", val)
+	}
+
+	if tr.trainedSamples != 101 {
+		t.Errorf("expected 101 trained samples (no increment after training complete), got %d", tr.trainedSamples)
+	}
+}
+
+func TestTrainerBlocksDataIngestion(t *testing.T) {
+	cfg := CreateConfig("TestTrainerBlocksDataIngestion")
+	cfg[IndexTrainedWithFastMerge] = true
+	err := InitTest(cfg)
+	if err != nil {
+		t.Fatal(err)
+	}
+	analysisQueue := index.NewAnalysisQueue(1)
+	s, err := NewScorch(Name, cfg, analysisQueue)
+	if err != nil {
+		t.Fatalf("failed to create Scorch: %v", err)
+	}
+	defer s.Close()
+
+	sc, ok := s.(*Scorch)
+	if !ok {
+		t.Fatal("expected Scorch instance")
+	}
+
+	batch := index.NewBatch()
+	for i := 0; i < 100; i++ {
+		doc := document.NewDocument(fmt.Sprintf("doc-%d", i))
+		stubVal := float32(i)
+		doc.AddField(document.NewVectorField("vec", nil, []float32{stubVal, stubVal + 1, stubVal + 2}, 3, "cosine", index.IndexOptimizedForRecall))
+		batch.Update(doc)
+	}
+
+	tr, ok := sc.trainer.(*vectorTrainer)
+	if !ok {
+		t.Fatal("expected vectorTrainer instance")
+	}
+	if sc.rootBolt == nil {
+		err := sc.openBolt()
+		if err != nil {
+			t.Fatalf("opening bolt failed %v", err)
+		}
+	}
+
+	// spawn all the routines to check the data ingest as well
+	err = sc.Open()
+	if err != nil {
+		t.Fatalf("failed to open Scorch: %v", err)
+	}
+
+	err = sc.Train(batch)
+	if err != nil {
+		t.Fatalf("training failed: %v", err)
+	}
+
+	val, err := tr.getInternal(util.BoltTrainCompleteKey)
+	if err != nil {
+		t.Fatalf("failed to get internal value: %v", err)
+	}
+
+	if string(val) != "false" {
+		t.Errorf("expected 'false' for training complete key, got %q", val)
+	}
+
+	if tr.trainedSamples != 100 {
+		t.Errorf("expected 100 trained samples, got %d", tr.trainedSamples)
+	}
+
+	batch.Reset()
+	doc := document.NewDocument("doc-998")
+	doc.AddField(document.NewVectorField("vec", nil, []float32{998, 998, 998}, 3, "cosine", index.IndexOptimizedForRecall))
+	batch.Update(doc)
+	err = sc.Batch(batch)
+	if err == nil {
+		t.Fatalf("data ingestion should be blocked during training")
+	}
+
+	batch.Reset()
+	batch.SetInternal(util.BoltTrainCompleteKey, []byte("true"))
+	err = sc.Train(batch)
+	if err != nil {
+		t.Fatalf("training failed: %v", err)
+	}
+
+	val, err = tr.getInternal(util.BoltTrainCompleteKey)
+	if err != nil {
+		t.Fatalf("failed to get internal value: %v", err)
+	}
+
+	if string(val) != "true" {
+		t.Errorf("expected 'true' for training complete key, got %q", val)
+	}
+
+	batch.Reset()
+	doc = document.NewDocument("doc-998")
+	doc.AddField(document.NewVectorField("vec", nil, []float32{998, 998, 998}, 3, "cosine", index.IndexOptimizedForRecall))
+	batch.Update(doc)
+	err = sc.Batch(batch)
+	if err != nil {
+		t.Fatalf("data ingestion expected to suceed after training is complete")
+	}
+
+	err = sc.Train(batch)
+	if err == nil {
+		t.Fatalf("re-training is not allowed on the index")
+	}
+
+}
+
+type dummyFileCopyAPI interface {
+	CopyFile(fileName string, dst index.IndexDirectory) error
+	SetPathInBolt(key []byte, value []byte) error
+}
+
+// trainedIndexWriter implements the writer used when copying the trained index
+// file from the source partition into this partition's index directory.
+type trainedIndexWriter struct {
+	rootpath  string
+	destIndex dummyFileCopyAPI
+}
+
+func (c *trainedIndexWriter) GetWriter(path string) (io.WriteCloser, error) {
+	if !(strings.HasSuffix(path, index.TrainedIndexFileName)) {
+		return nil, fmt.Errorf("write not allowed on path %s", path)
+	}
+	return os.OpenFile(filepath.Join(c.rootpath, filepath.Base(path)), os.O_CREATE|os.O_WRONLY, 0600)
+}
+func (c *trainedIndexWriter) SetPathInBolt(key []byte, value []byte) error {
+	return c.destIndex.SetPathInBolt(key, value)
+}
+
+func TestTrainerIndexCopy(t *testing.T) {
+	cfg := CreateConfig("TestTrainerIndexCopy")
+	cfg[IndexTrainedWithFastMerge] = true
+	err := InitTest(cfg)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	cfg["path"] = filepath.Clean(cfg["path"].(string))
+	analysisQueue := index.NewAnalysisQueue(1)
+	s, err := NewScorch(Name, cfg, analysisQueue)
+	if err != nil {
+		t.Fatalf("failed to create Scorch: %v", err)
+	}
+	defer s.Close()
+
+	sc, ok := s.(*Scorch)
+	if !ok {
+		t.Fatal("expected Scorch instance")
+	}
+
+	batch := index.NewBatch()
+	for i := 0; i < 100; i++ {
+		doc := document.NewDocument(fmt.Sprintf("doc-%d", i))
+		stubVal := float32(i)
+		doc.AddField(document.NewVectorField("vec", nil, []float32{stubVal, stubVal + 1, stubVal + 2}, 3, "cosine", index.IndexOptimizedForRecall))
+		batch.Update(doc)
+	}
+
+	tr, ok := sc.trainer.(*vectorTrainer)
+	if !ok {
+		t.Fatal("expected vectorTrainer instance")
+	}
+	if sc.rootBolt == nil {
+		err := sc.openBolt()
+		if err != nil {
+			t.Fatalf("opening bolt failed %v", err)
+		}
+	}
+	sc.asyncTasks.Add(1)
+	go tr.trainLoop()
+
+	err = sc.Train(batch)
+	if err != nil {
+		t.Fatalf("training failed: %v", err)
+	}
+
+	val, err := tr.getInternal(util.BoltTrainCompleteKey)
+	if err != nil {
+		t.Fatalf("failed to get internal value: %v", err)
+	}
+
+	if string(val) != "false" {
+		t.Errorf("expected 'false' for training complete key, got %q", val)
+	}
+
+	// -----------------------------------------------------------------------
+	// At this point the source trainer has been trained with 100 samples, and the
+	// trained index file should be available on disk. Now we will create a new
+	// trainer in a new Scorch instance, and copy the trained index file from the
+	// source trainer into the destination trainer's index directory, simulating
+	// the file copy between partitions to avoid re-training.
+	// -----------------------------------------------------------------------
+	dstCfg := CreateConfig("TestTrainerIndexCopyDest")
+	dstCfg[IndexTrainedWithFastMerge] = true
+	err = InitTest(dstCfg)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	dstCfg["path"] = filepath.Clean(dstCfg["path"].(string))
+
+	dstWriter := &trainedIndexWriter{
+		rootpath: dstCfg["path"].(string),
+	}
+
+	dstScorch, err := NewScorch(Name, dstCfg, analysisQueue)
+	if err != nil {
+		t.Fatalf("failed to create new Scorch: %v", err)
+	}
+	defer dstScorch.Close()
+
+	dst, ok := dstScorch.(*Scorch)
+	if !ok {
+		t.Fatal("expected Scorch instance")
+	}
+	dstWriter.destIndex = dstScorch.(dummyFileCopyAPI)
+
+	dstTrainer, ok := dst.trainer.(*vectorTrainer)
+	if !ok {
+		t.Fatal("expected vectorTrainer instance")
+	}
+	if dst.rootBolt == nil {
+		err := dst.openBolt()
+		if err != nil {
+			t.Fatalf("opening bolt failed %v", err)
+		}
+	}
+	dst.asyncTasks.Add(1)
+	go dstTrainer.trainLoop()
+
+	src, ok := s.(dummyFileCopyAPI)
+	if !ok {
+		t.Fatal("expected dummyFileCopyAPI instance")
+	}
+
+	err = src.CopyFile(index.TrainedIndexFileName, dstWriter)
+	if err != nil {
+		t.Fatalf("CopyFile failed: %v", err)
+	}
+
+	val, err = dstTrainer.getInternal(util.BoltTrainCompleteKey)
+	if err != nil {
+		t.Fatalf("failed to get internal value: %v", err)
+	}
+
+	if string(val) != "false" {
+		t.Errorf("expected 'false' for training complete key in destination trainer, got %q", val)
+	}
+
+	if trainedIndex, err := dstTrainer.getTrainedIndex("vec"); err != nil || trainedIndex == nil {
+		t.Errorf("expected trained index in destination trainer, got nil")
+	}
+
+}
+
+func TestTrainerFallbackNaiveMerge(t *testing.T) {
+	cfg := CreateConfig("TestTrainerFallbackNaiveMerge")
+	cfg[IndexTrainedWithFastMerge] = true
+	err := InitTest(cfg)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	cfg["path"] = filepath.Clean(cfg["path"].(string))
+	analysisQueue := index.NewAnalysisQueue(1)
+	s, err := NewScorch(Name, cfg, analysisQueue)
+	if err != nil {
+		t.Fatalf("failed to create Scorch: %v", err)
+	}
+
+	sc, ok := s.(*Scorch)
+	if !ok {
+		t.Fatal("expected Scorch instance")
+	}
+
+	tr, ok := sc.trainer.(*vectorTrainer)
+	if !ok {
+		t.Fatal("expected vectorTrainer instance")
+	}
+	if sc.rootBolt == nil {
+		err := sc.openBolt()
+		if err != nil {
+			t.Fatalf("opening bolt failed %v", err)
+		}
+	}
+	// spawn all the routines to check the data ingest as well
+	err = sc.Open()
+	if err != nil {
+		t.Fatalf("failed to open Scorch: %v", err)
+	}
+
+	// mocking the scenario where the application would've failed to create the trained index file
+	// due to some fail of operation but expects the underlying the indexing layer
+	// to continue ingestion and fallback to naive merge and complete the training
+	// without the trained index file.
+	batch := index.NewBatch()
+	batch.SetInternal(util.BoltTrainCompleteKey, []byte("true"))
+	err = sc.Train(batch)
+	if err != nil {
+		t.Fatalf("training failed: %v", err)
+	}
+
+	batch.Reset()
+	doc := document.NewDocument("doc-998")
+	doc.AddField(document.NewVectorField("vec", nil, []float32{998, 998, 998}, 3, "cosine", index.IndexOptimizedForRecall))
+	batch.Update(doc)
+	err = sc.Batch(batch)
+	if err != nil {
+		t.Fatalf("data ingestion expected to suceed after training is complete")
+	}
+
+	val, err := tr.getInternal(util.BoltTrainCompleteKey)
+	if err != nil {
+		t.Fatalf("failed to get internal value: %v", err)
+	}
+
+	if string(val) != "true" {
+		t.Errorf("expected 'true' for training complete key, got %q", val)
+	}
+
+	s.Close()
+
+	// Re-open the index to verify that it can be loaded without the trained index file.
+	sn, err := NewScorch(Name, cfg, analysisQueue)
+	if err != nil {
+		t.Fatalf("failed to re-open Scorch: %v", err)
+	}
+
+	err = sn.Open()
+	if err != nil {
+		t.Fatalf("failed to open Scorch after fallback: %v", err)
+	}
+	defer sn.Close()
+}

index/scorch/unadorned.go

@@ -0,0 +1,219 @@
+//  Copyright (c) 2020 Couchbase, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// 		http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package scorch
+
+import (
+	"math"
+	"reflect"
+
+	"github.com/RoaringBitmap/roaring/v2"
+	segment "github.com/blevesearch/scorch_segment_api/v2"
+)
+
+var reflectStaticSizeUnadornedPostingsIteratorBitmap int
+var reflectStaticSizeUnadornedPostingsIterator1Hit int
+var reflectStaticSizeUnadornedPosting int
+
+func init() {
+	var pib unadornedPostingsIteratorBitmap
+	reflectStaticSizeUnadornedPostingsIteratorBitmap = int(reflect.TypeOf(pib).Size())
+	var pi1h unadornedPostingsIterator1Hit
+	reflectStaticSizeUnadornedPostingsIterator1Hit = int(reflect.TypeOf(pi1h).Size())
+	var up UnadornedPosting
+	reflectStaticSizeUnadornedPosting = int(reflect.TypeOf(up).Size())
+}
+
+type unadornedPostingsIteratorBitmap struct {
+	actual   roaring.IntPeekable
+	actualBM *roaring.Bitmap
+	next     UnadornedPosting // reused across Next() calls
+}
+
+func (i *unadornedPostingsIteratorBitmap) Next() (segment.Posting, error) {
+	return i.nextAtOrAfter(0)
+}
+
+func (i *unadornedPostingsIteratorBitmap) Advance(docNum uint64) (segment.Posting, error) {
+	return i.nextAtOrAfter(docNum)
+}
+
+func (i *unadornedPostingsIteratorBitmap) nextAtOrAfter(atOrAfter uint64) (segment.Posting, error) {
+	docNum, exists := i.nextDocNumAtOrAfter(atOrAfter)
+	if !exists {
+		return nil, nil
+	}
+	i.next = UnadornedPosting{} // clear the struct
+	rv := &i.next
+	rv.docNum = docNum
+	return rv, nil
+}
+
+func (i *unadornedPostingsIteratorBitmap) nextDocNumAtOrAfter(atOrAfter uint64) (uint64, bool) {
+	if i.actual == nil || !i.actual.HasNext() {
+		return 0, false
+	}
+	i.actual.AdvanceIfNeeded(uint32(atOrAfter))
+
+	if !i.actual.HasNext() {
+		return 0, false // couldn't find anything
+	}
+
+	return uint64(i.actual.Next()), true
+}
+
+func (i *unadornedPostingsIteratorBitmap) Size() int {
+	return reflectStaticSizeUnadornedPostingsIteratorBitmap
+}
+
+func (i *unadornedPostingsIteratorBitmap) BytesRead() uint64 {
+	return 0
+}
+
+func (i *unadornedPostingsIteratorBitmap) BytesWritten() uint64 {
+	return 0
+}
+
+func (i *unadornedPostingsIteratorBitmap) ResetBytesRead(uint64) {}
+
+func (i *unadornedPostingsIteratorBitmap) ActualBitmap() *roaring.Bitmap {
+	return i.actualBM
+}
+
+func (i *unadornedPostingsIteratorBitmap) DocNum1Hit() (uint64, bool) {
+	return 0, false
+}
+
+func (i *unadornedPostingsIteratorBitmap) ReplaceActual(actual *roaring.Bitmap) {
+	i.actualBM = actual
+	i.actual = actual.Iterator()
+}
+
+// Resets the iterator to the beginning of the postings list.
+// by resetting the actual iterator.
+func (i *unadornedPostingsIteratorBitmap) ResetIterator() {
+	i.actual = i.actualBM.Iterator()
+}
+
+func newUnadornedPostingsIteratorFromBitmap(bm *roaring.Bitmap) segment.PostingsIterator {
+	return &unadornedPostingsIteratorBitmap{
+		actualBM: bm,
+		actual:   bm.Iterator(),
+	}
+}
+
+const docNum1HitFinished = math.MaxUint64
+
+type unadornedPostingsIterator1Hit struct {
+	docNumOrig uint64           // original 1-hit docNum used to create this iterator
+	docNum     uint64           // current docNum
+	next       UnadornedPosting // reused across Next() calls
+}
+
+func (i *unadornedPostingsIterator1Hit) Next() (segment.Posting, error) {
+	return i.nextAtOrAfter(0)
+}
+
+func (i *unadornedPostingsIterator1Hit) Advance(docNum uint64) (segment.Posting, error) {
+	return i.nextAtOrAfter(docNum)
+}
+
+func (i *unadornedPostingsIterator1Hit) nextAtOrAfter(atOrAfter uint64) (segment.Posting, error) {
+	docNum, exists := i.nextDocNumAtOrAfter(atOrAfter)
+	if !exists {
+		return nil, nil
+	}
+	i.next = UnadornedPosting{} // clear the struct
+	rv := &i.next
+	rv.docNum = docNum
+	return rv, nil
+}
+
+func (i *unadornedPostingsIterator1Hit) nextDocNumAtOrAfter(atOrAfter uint64) (uint64, bool) {
+	if i.docNum == docNum1HitFinished {
+		return 0, false
+	}
+	if i.docNum < atOrAfter {
+		// advanced past our 1-hit
+		i.docNum = docNum1HitFinished // consume our 1-hit docNum
+		return 0, false
+	}
+	docNum := i.docNum
+	i.docNum = docNum1HitFinished // consume our 1-hit docNum
+	return docNum, true
+}
+
+func (i *unadornedPostingsIterator1Hit) Size() int {
+	return reflectStaticSizeUnadornedPostingsIterator1Hit
+}
+
+func (i *unadornedPostingsIterator1Hit) BytesRead() uint64 {
+	return 0
+}
+
+func (i *unadornedPostingsIterator1Hit) BytesWritten() uint64 {
+	return 0
+}
+
+func (i *unadornedPostingsIterator1Hit) ResetBytesRead(uint64) {}
+
+func (i *unadornedPostingsIterator1Hit) ActualBitmap() *roaring.Bitmap { return nil }
+
+func (i *unadornedPostingsIterator1Hit) DocNum1Hit() (uint64, bool) {
+	return i.docNumOrig, true
+}
+
+// ReplaceActual is a no-op: a 1-hit iterator has no actual bitmap, and callers
+// only invoke ReplaceActual on iterators whose ActualBitmap is non-nil.
+func (i *unadornedPostingsIterator1Hit) ReplaceActual(*roaring.Bitmap) {}
+
+// ResetIterator resets the iterator to the original state.
+func (i *unadornedPostingsIterator1Hit) ResetIterator() {
+	i.docNum = i.docNumOrig
+}
+
+func newUnadornedPostingsIteratorFrom1Hit(docNum1Hit uint64) segment.PostingsIterator {
+	return &unadornedPostingsIterator1Hit{
+		docNumOrig: docNum1Hit,
+		docNum:     docNum1Hit,
+	}
+}
+
+type ResetablePostingsIterator interface {
+	ResetIterator()
+}
+
+type UnadornedPosting struct {
+	docNum uint64
+}
+
+func (p *UnadornedPosting) Number() uint64 {
+	return p.docNum
+}
+
+func (p *UnadornedPosting) Frequency() uint64 {
+	return 0
+}
+
+func (p *UnadornedPosting) Norm() float64 {
+	return 0
+}
+
+func (p *UnadornedPosting) Locations() []segment.Location {
+	return nil
+}
+
+func (p *UnadornedPosting) Size() int {
+	return reflectStaticSizeUnadornedPosting
+}