snapshot of https://github.com/blevesearch/bleve.git (master, 2026-09-15; blobs > 131072B omitted)

docs/search_autocomplete.md

@@ -0,0 +1,358 @@
+# Edge N-gram Autocomplete in Bleve
+
+Search autocomplete is a feature which we see in search boxes when suggestions appear while we type.
+So when we type `jav`, we see suggestions like: `java` `javascript` `javascript programming` etc.
+This is helpful because it saves users time in finding what they are looking for.
+
+![Alt Text](/docs/images/search_autocomplete.png "search autocomplete suggestion")
+
+## 2. How Does It Work?
+
+Autocomplete generally works in three steps:
+
+1. **Index Time**: Breaking text into searchable pieces (tokens)
+2. **Query Time**: Matching user input against indexed tokens  
+3. **Results**: Returning relevant suggestions quickly
+
+But before we jump into the flow, let's understand different methods to achieve this and why edge n-grams are the most efficient approach.
+
+## 3. Different Tokenization Methods
+
+There are several tokenization approaches, each with its own strengths and weaknesses:
+
+### 3.1 Single Token Tokenizer
+
+```go
+// analysis/tokenizer/single/single.go
+func (t *SingleTokenTokenizer) Tokenize(input []byte) analysis.TokenStream {
+    return analysis.TokenStream{
+        &analysis.Token{
+            Term:     input,    // Here entire input as one token
+            Position: 1,
+            Start:    0,
+            End:      len(input),
+            Type:     analysis.AlphaNumeric,
+        },
+    }
+}
+```
+
+**How it works**: Treats the entire input as a single token.
+
+**Example**: "JavaScript Programming" → [`"JavaScript Programming"`]
+
+**Pros**:
+
+- Simple and fast
+- Perfect for exact phrase matching
+- Minimal index size
+
+**Cons**:
+
+- No autocomplete support (can't match partial text)
+- Not flexible for search
+
+**Use case**: Keyword fields, IDs, exact phrase matching
+
+### 3.2 Whitespace Token Tokenizer
+
+```go
+// analysis/tokenizer/whitespace/whitespace.go  
+func TokenizerConstructor(config map[string]interface{}, cache *registry.Cache) (analysis.Tokenizer, error) {
+    return character.NewCharacterTokenizer(notSpace), nil
+}
+
+func notSpace(r rune) bool {
+    return !unicode.IsSpace(r)  // Split on whitespace
+}
+```
+
+**How it works**: Splits text on whitespace characters.
+
+**Example**: "JavaScript Programming" → [`"JavaScript"`, `"Programming"`]
+
+**Pros**:
+
+- Simple word-based tokenization
+- Works well for basic prefix search
+
+**Cons**:
+
+- Only matches from word beginnings
+- No support for partial word matching
+- Limited autocomplete capabilities
+
+**Use case**: Basic search, word-level indexing
+
+### 3.3 N-gram Method
+
+```go
+// analysis/token/ngram/ngram.go
+func (s *NgramFilter) Filter(input analysis.TokenStream) analysis.TokenStream {
+    rv := make(analysis.TokenStream, 0, len(input))
+    
+    for _, token := range input {
+        runeCount := utf8.RuneCount(token.Term)
+        runes := bytes.Runes(token.Term)
+        
+        // ..generate all possible n-grams
+    }
+    return rv
+}
+```
+
+**How it works**: Creates ALL possible substrings of specified lengths.
+
+**Example**: "java" with 2-3 grams → [`"ja"`, `"av"`, `"va"`, `"jav"`, `"ava"`]
+
+**Pros**:
+
+- Supports partial matching anywhere in the text
+- Coverage of all possible substrings
+
+**Cons**:
+
+- **MASSIVE index size** (exponential growth)
+- Brings more noise as irrelevant matches (e.g., "av" matching "java")
+- Poor performance for autocomplete
+- High memory usage
+
+**Use case**: Full-text substring search (not ideal for autocomplete)
+
+### 3.4 Edge N-gram Method (preferred for autocomplete)
+
+```go
+// analysis/token/edgengram/edgengram.go
+func (s *EdgeNgramFilter) Filter(input analysis.TokenStream) analysis.TokenStream {
+    rv := make(analysis.TokenStream, 0, len(input))
+    
+    for _, token := range input {
+        runeCount := utf8.RuneCount(token.Term)
+        runes := bytes.Runes(token.Term)
+        // ..builds tokens based on either end, specified in the input
+    }
+    return rv
+}
+```
+
+**How it works**: Creates substrings only from the beginning (or end) of each word.
+
+**Example**: "javascript" with front edge n-grams (1-5) → [`"j"`, `"ja"`, `"jav"`, `"java"`, `"javas"`]
+
+**Pros**:
+
+- Perfect for autocomplete (matches prefixes naturally)
+- Efficient index size (linear growth vs exponential)  
+- Fast queries (direct term matching, no complex processing)
+- Intuitive results (matches what users expect)
+- Highly scalable for large datasets
+
+**Cons**:
+
+- Only supports prefix matching (but that's preferred for autocomplete!)
+- Slightly larger index than basic tokenization
+
+**Use case**: Search autocomplete, prefix-based search
+
+## 4. Why Edge N-grams Are Most Efficient for Autocomplete
+
+Let's see what happens when a user types "java" with edge_ngram tokenizer:
+
+```text
+Index contains: ["j", "ja", "jav", "java", "javas", "javasc", "javascr", ...]
+User types: "java"  
+Query: ExactTermQuery("java")
+Process: Direct hash table lookup for term "java"
+Result: Instant match, then retrieve documents containing this term
+```
+
+**Key advantages of Edge N-gram approach:**
+
+1. **O(1) lookup**: Direct term matching
+2. **No query-time processing**: Terms are pre-computed at index time  
+3. **Better caching**: Exact term queries cache better than prefix queries
+4. **Consistent performance**: Query time doesn't increase with index size
+
+## 5. Step-by-Step Implementation: Building Search Autocomplete
+
+Now let's see how to implement this in practice using our exact configuration. We'll build a complete autocomplete system step by step.
+
+### Step 1: Create Custom Edge N-gram Token Filter
+
+First, we need to create a custom token filter for edge n-grams. Here's how it looks in our configuration:
+
+```go
+// Create a new index mapping
+indexMapping := mapping.NewIndexMapping()
+
+// 1. Define the edgeGram token filter
+edgeGramFilter := map[string]interface{}{
+    "type": edgengram.Name,
+    "min":  2.0,
+    "max":  4.0,
+    "back": false,
+}
+
+// Register the token filter
+if err := indexMapping.AddCustomTokenFilter("Engram", edgeGramFilter); err != nil {
+    log.Fatal(err)
+}
+```
+
+**What each setting does:**
+
+- `"type": "edge_ngram"` - Tells Bleve to use the edge n-gram filter
+- `"min": 2` - Start creating tokens from 2 characters ("ja", "sc", etc.)
+- `"max": 4` - Stop at 4 characters ("java", "scri", etc.)
+- `"back": false` - Create tokens from the front (beginning) of words
+
+### Step 2: Create Custom Analyzer
+
+Next, we create an analyzer that uses our custom token filter along with other helpful filters:
+
+```go
+// 2. Define a custom analyzer that uses it
+customAnalyzer := map[string]interface{}{
+    "type":      custom.Name,
+    "tokenizer": "unicode",
+    "char_filters": []string{
+         zerowidthnonjoiner.Name,
+    },
+    "token_filters": []string{
+        "Engram", // our custom edge_ngram filter
+        "to_lower",
+        "stop_en",
+    },
+}
+
+if err := indexMapping.AddCustomAnalyzer("edgeGramAnalyzer", customAnalyzer); err != nil {
+    log.Fatal(err)
+}
+```
+
+**The pipeline works like this:**
+
+1. **Input Text**: "Schaumbergfest Event"
+
+2. **Tokenizer** (`unicode`): Splits into words
+
+   ```text
+   ["Schaumbergfest", "Event"]
+   ```
+
+3. **Character Filter** (`zero_width_spaces`): Removes invisible characters
+
+   ```text
+   ["Schaumbergfest", "Event"] (cleaned)
+   ```
+
+4. **Token Filter 1** (`Engram`): Creates edge n-grams (2-4 chars)
+
+   ```text
+   ["Sc", "Sch", "Scha", "Ev", "Eve", "Even"]
+   ```
+
+5. **Token Filter 2** (`to_lower`): Makes everything lowercase
+
+   ```text
+   ["sc", "sch", "scha", "ev", "eve", "even"]
+   ```
+
+6. **Token Filter 3** (`stop_en`): Removes common words (none in this case)
+
+   ```text
+   ["sc", "sch", "scha", "ev", "eve", "even"] (final tokens)
+   ```
+
+### Step 3: Configure Field Mapping
+
+Now we tell Bleve which fields to apply our autocomplete analyzer to:
+
+```go
+    // 3. Assign analyzer to a field mapping
+    fieldMapping := mapping.NewTextFieldMapping()
+    fieldMapping.Analyzer = "edgeGramAnalyzer"
+    
+    indexMapping.DefaultMapping.AddFieldMappingsAt("title", fieldMapping)
+    
+    indexPath := "example.bleve"
+    index, err := bleve.New(indexPath, indexMapping)
+    if err != nil {
+        log.Fatal(err)
+    }
+```
+
+### Step 4: How It Works in Real Search
+
+When someone searches for "sc", here's what happens:
+
+**Index contains these tokens:**
+
+```text
+"sc" → [document1: "Schaumbergfest", document2: "Script", ...]
+"sch" → [document1: "Schaumbergfest", ...]  
+"scha" → [document1: "Schaumbergfest", ...]
+```
+
+**User types "sc":**
+
+1. Query: `name:sc`
+2. Bleve looks up exact term "sc" in the index
+3. Finds document with "Schaumbergfest" and "Script"
+4. Returns suggestion instantly
+
+```go
+
+type Document struct {
+  ID    string `json:"id"`
+  Title string `json:"title"`
+}
+// 4. Index Documents
+documents := []Document{
+  {
+    ID:    "doc1",
+    Title: "Schaumbergfest",
+  },
+  {
+    ID:    "doc2",
+    Title: "Script",
+  },
+}
+
+batch := index.NewBatch()
+for _, doc := range documents {
+  batch.Index(doc.ID, doc)
+}
+if err := index.Batch(batch); err != nil {
+  log.Fatal(err)
+}
+
+// 5. Search the created index
+query := bleve.NewMatchQuery("sc")
+query.SetField("title")
+searchRequest := bleve.NewSearchRequest(query)
+searchRequest.Explain = true
+searchRequest.Fields = []string{"title"}
+searchResult, err := index.Search(searchRequest)
+if err != nil {
+  log.Fatal(err)
+}
+fmt.Println(searchResult)
+```
+
+Output:
+
+```bash
+
+$ go run main.go
+
+2 matches, showing 1 through 2, took 189.041µs
+    1. doc2 (0.343255)
+        title
+                Script
+    2. doc1 (0.343255)
+        title
+                Schaumbergfest
+```
+
+Note: To run code, enclose code starting from Step 1 in func main.