ZeroProximity.VectorizedContentIndexer 1.0.2

🔍 ZeroProximity.VectorizedContentIndexer

A high-performance hybrid content indexing library for .NET — combining BM25 keyword search, semantic vector search, and RRF fusion with a fully embedded ONNX model. Zero config. Works offline.

📋 Overview

ZeroProximity.VectorizedContentIndexer brings production-grade hybrid search to any .NET 9 application without requiring external services, API keys, or model downloads. It ships an embedded MiniLM-L6-v2 ONNX model (384 dimensions, 22.7M parameters) and a custom binary vector index format designed for memory-efficient, high-throughput retrieval.

Why this library?

✨ Key Features

Search Modes

• Lexical Search — Lucene.NET BM25 keyword search with inverted index; fast and precise for exact term matching
• Semantic Search — Dense vector search using ONNX-generated embeddings; finds conceptually similar content even without shared keywords
• Hybrid Search — Reciprocal Rank Fusion (RRF) combines lexical and semantic rankings into a single, balanced result set

Document Model

• ISearchable — Minimal interface: provide text and a timestamp, the library handles the rest
• IDocument — Extended interface for richer content with ID, metadata, and field control
• IHierarchicalDocument<TChild> — First-class support for parent-child structures (e.g., a session containing messages), enabling child-level indexing with parent-level context retrieval

Performance

• Parallel lexical + semantic query execution in hybrid mode
• Memory-mapped AJVI index for low-latency random reads at scale
• SIMD cosine similarity (System.Numerics intrinsics)
• Float16 storage cuts vector memory footprint by ~50% vs Float32
• DirectML GPU acceleration for batch embedding workloads

Thread Safety

• Concurrent reads fully supported
• Writes are serialized — safe for multi-threaded indexing pipelines

📦 Installation

dotnet add package ZeroProximity.VectorizedContentIndexer --prerelease

This package is currently in prerelease (1.0.0-beta.1). The --prerelease flag is required until the stable release.

🚀 Quick Start

Basic: Hybrid Search with ISearchable

using ZeroProximity.VectorizedContentIndexer.Embeddings;
using ZeroProximity.VectorizedContentIndexer.Search;
using ZeroProximity.VectorizedContentIndexer.Models;

// Define your searchable content type
public record DocumentChunk : ISearchable
{
    public required string Id { get; init; }
    public required string Content { get; init; }
    public required DateTime CreatedAt { get; init; }

    public string GetSearchableText() => Content;
    public DateTime GetTimestamp() => CreatedAt;
}

// Initialize the embedded ONNX embedding provider (no download required)
IEmbeddingProvider embeddings = await EmbeddingProviderFactory.CreateAsync();

// Compose a hybrid search engine
ISearchEngine<DocumentChunk> searchEngine = new HybridSearcher<DocumentChunk>(
    luceneEngine: new LuceneSearchEngine<DocumentChunk>("./index/lucene"),
    vectorEngine: new VectorSearchEngine<DocumentChunk>("./index/vector", embeddings),
    lexicalWeight: 0.5,
    semanticWeight: 0.5
);

// Index a document
await searchEngine.IndexAsync(new DocumentChunk
{
    Id = "doc1",
    Content = "How to optimize async performance in C#",
    CreatedAt = DateTime.UtcNow
});

// Search — hybrid mode by default
IReadOnlyList<SearchResult<DocumentChunk>> results =
    await searchEngine.SearchAsync("async optimization", maxResults: 10);

foreach (SearchResult<DocumentChunk> result in results)
{
    Console.WriteLine($"[{result.Score:F3}] {result.Document.Content}");
}

RAG Pipeline

Retrieve semantically relevant context chunks to augment an LLM prompt:

// Retrieve the top 5 most semantically relevant chunks
IReadOnlyList<SearchResult<DocumentChunk>> chunks =
    await searchEngine.SearchAsync(userQuery, maxResults: 5, mode: SearchMode.Semantic);

string context = string.Join("\n\n", chunks.Select(r => r.Document.Content));

string prompt = $"""
    Context:
    {context}

    Question: {userQuery}

    Answer:
    """;

Hierarchical Documents (IHierarchicalDocument)

Index parent documents whose children are individually searchable, and expand context at retrieval time:

public class Message
{
    public required string Id { get; init; }
    public required string Text { get; init; }
    public required DateTime SentAt { get; init; }
}

public class ConversationSession : IHierarchicalDocument<Message>
{
    public required string Id { get; init; }
    public required IReadOnlyList<Message> Messages { get; init; }

    // Each message is indexed individually
    public IReadOnlyList<Message> GetChildren() => Messages;

    // Retrieve N messages of context before a matched message
    public IReadOnlyList<Message> GetChildrenBefore(string messageId, int count) =>
        Messages
            .TakeWhile(m => m.Id != messageId)
            .TakeLast(count)
            .ToList();
}

// The engine indexes each Message as a searchable unit,
// but returns the parent ConversationSession with surrounding context
var sessionEngine = new HybridSearcher<ConversationSession>(
    luceneEngine: new LuceneSearchEngine<ConversationSession>("./index/lucene"),
    vectorEngine: new VectorSearchEngine<ConversationSession>("./index/vector", embeddings)
);

await sessionEngine.IndexAsync(session);

IReadOnlyList<SearchResult<ConversationSession>> hits =
    await sessionEngine.SearchAsync("authentication error", maxResults: 5);

🔍 Search Modes

// Explicit mode selection
var results = await searchEngine.SearchAsync(query, maxResults: 10, mode: SearchMode.Semantic);

🏗️ Architecture

┌──────────────────────────────────────────────────────────────────┐
│                        ISearchEngine<T>                          │
└───────────────────────────┬──────────────────────────────────────┘
                            │
              ┌─────────────▼─────────────┐
              │      HybridSearcher<T>    │
              │       (RRF Fusion)        │
              └──────┬────────────┬───────┘
                     │            │
        ┌────────────▼──┐   ┌─────▼──────────────┐
        │LuceneSearch   │   │VectorSearchEngine<T>│
        │Engine<T>      │   │(Semantic Search)    │
        │(BM25 Keyword) │   └─────────┬───────────┘
        └───────┬───────┘             │
                │               ┌─────▼──────────────┐
        ┌───────▼───────┐       │   AJVI Index        │
        │  Lucene.NET   │       │   (Memory-Mapped,   │
        │  Inverted     │       │    Float16 / F32)   │
        │  Index        │       └─────────┬───────────┘
        └───────────────┘                 │
                                 ┌────────▼────────────┐
                                 │  IEmbeddingProvider  │
                                 ├──────────────────────┤
                                 │  OnnxEmbeddingProvider│
                                 │  (MiniLM-L6-v2,      │
                                 │   384 dims, DirectML) │
                                 └──────────────────────┘

⚡ Performance

Benchmarks measured on typical developer hardware (CPU: modern x64; GPU: DirectML-compatible discrete GPU).

Indexing

Search

Storage

Scalability

• Under 100K vectors — Excellent performance with brute-force nearest neighbor
• 100K–1M vectors — Good performance; review performance tuning guidance
• Over 1M vectors — Approximate nearest neighbor support is a planned enhancement

⚙️ Configuration

Register and configure the search engine via IOptions<SearchEngineOptions>:

services.Configure<SearchEngineOptions>(options =>
{
    options.IndexPath         = "./index";
    options.DefaultMode       = SearchMode.Hybrid;
    options.Precision         = VectorPrecision.Float16;  // ~50% storage savings vs Float32
    options.LexicalWeight     = 0.5;
    options.SemanticWeight    = 0.5;
    options.RrfK              = 60;    // Reciprocal Rank Fusion constant
    options.DecayHalfLifeDays = 90.0;  // Exponential temporal decay half-life
});

📋 Requirements

• .NET 9.0 or later
• Platforms — Windows, Linux, macOS
• GPU (optional) — DirectML-compatible GPU for accelerated embedding generation

Dependencies

📚 Documentation

🤝 Contributing

Contributions are welcome. Please read CONTRIBUTING.md before opening a pull request.

• Open an issue to report bugs or request features
• Start a discussion for questions and ideas

📄 License

MIT License — see LICENSE for full terms.

🙏 Acknowledgments

Embedding Model

sentence-transformers/all-MiniLM-L6-v2 384 dimensions, 22.7M parameters. Apache 2.0 License. Trained by Nils Reimers.

Research

• Reciprocal Rank Fusion (RRF) — Cormack, Clarke, and Buettcher. Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods. SIGIR 2009.

Inspiration

• Lucene.NET — The foundation of the lexical search layer
• Elasticsearch — Hybrid search implementation patterns
• Vespa.ai — Multi-phase retrieval architecture