Fast by layers.

“Make vector search faster” sounds like one task. In practice it is a stack of different problems: examine fewer vectors, make each comparison cheaper, keep data close to the CPU, avoid rebuilding unchanged work, and use parallel hardware where its overhead is justified.

Each optimization also introduces a new way to be wrong. Approximate traversal can miss neighbors. Quantization changes scores. Parallel graph construction can become nondeterministic. A GPU cache can be stale while writes continue on the CPU. Even a faster query is a regression if it silently returns worse results.

I designed FocalVector around that tension. Exact search remains the correctness oracle. Benchmarks measure recall against it, durable APIs rerank with authoritative full-precision vectors, and accelerated state remains rebuildable. That gives every optimization a reference answer and a clear failure boundary.

First, avoid most comparisons.

Exact search scores every eligible vector. It is simple, deterministic, and indispensable for validation, but its work grows linearly with the collection. FocalVector’s primary approximate index is HNSW: a layered proximity graph that navigates toward promising regions instead of scanning the full corpus.

There is no single “HNSW speed.” Graph density (M), build breadth (ef_construction), and query breadth (ef_search) spend memory and computation at different times. On a deterministic one-million-vector, 128-dimensional cosine corpus, a high-recall graph built with M=32 andef_construction=400 reached 96.0% recall@10 at a mean 2.430 ms with ef_search=384. Raising query breadth to 768 reached 97.7% at 4.827 ms. The curve matters more than either endpoint.

An important lesson from that benchmark was that query effort cannot recover information the graph never captured. A faster-to-buildM=16 graph reached only 85.7% recall even atef_search=1536. Construction quality, resident memory, and query latency are one coupled decision.

Then, make graph traversal cheaper.

HNSW reduces the number of candidates, but graph traversal still performs many vector comparisons. Storing every traversal vector asf32 costs four bytes per component and moves correspondingly more data through the cache hierarchy. FocalVector’s durable indexes use symmetric scalar-int8 quantization: one signed byte per component, plus a per-vector scale and cached squared norm.

Quantization is used to navigate, not to define the final answer. A query traverses the graph with quantized cosine, dot-product, or Euclidean scores, oversamples four times the requested result count, then reranks those candidates against the authoritative full-precision vectors. Filtered exact search and the recent-write delta remain full precision as well. Returned durable-API scores therefore retain f32semantics even though the fast path used approximate arithmetic to find candidates.

SIMD turns compact data into useful throughput.

The compact representation creates an opportunity for wider arithmetic. On x86-64, FocalVector runtime-detects AVX2 and processes 16 signed-byte component pairs at a time. It widens them to 16-bit lanes, multiplies and accumulates into 32-bit lanes, then finishes any tail with the portable implementation. The dispatch is guarded at runtime, so the persisted format is not tied to one CPU instruction set.

Two details matter here. First, the accumulator path is bounded: vectors above the safe AVX2 length use the checked portable implementation. Second, tests compare the vectorized and portable results at boundary lengths and extreme values. SIMD is an implementation choice, not a new scoring definition.

The quiet wins: locality and allocation control.

Arithmetic was only part of the cost. The original graph shape could also lose time chasing allocations. FocalVector now stores vector components in contiguous arenas. After construction, neighbor lists are flattened into compact ranges over one u32 link array, and point IDs live in a single ordinal table instead of being duplicated in every graph node.

Graph traversal uses generation-marked visitation arrays rather than allocating and hashing a fresh visited set for each search. Query workspaces come from a bounded, contention-safe pool and return after use. The same generation-marked workspaces are reused by construction workers. None of these changes is as dramatic as “GPU acceleration,” but together they reduce allocator traffic and give the CPU more predictable memory access.

Parallelize construction without randomizing it.

HNSW insertion is order-sensitive, which makes naïve parallel construction both difficult to reproduce and difficult to debug. FocalVector builds a 4,096-point seed graph serially, then plans groups of 512 insertions in parallel against a stable graph snapshot. Plans are applied in input order.

Rayon supplies the worker pool, but scheduling does not decide the graph. Stable input ordering, deterministic levels, stable score tie-breaking, and ordered plan application produce the same graph across worker counts. This preserves reproducible benchmarks and storage while using available cores for the expensive planning phase.

Stop copying and rebuilding unchanged data.

Query speed can be undermined by what happens between queries. FocalVector recovers immutable segments through read-only memory maps. Persisted int8 graph codes and aligned full-precision vectors are scored directly from mapped files instead of being copied into second heap arenas. An owned public point is materialized lazily only when an ID is explicitly fetched; search and reranking stay zero-copy.

Writes use another form of work avoidance. Recent mutations remain in a small exact-search delta and merge with results from immutable graphs. Routine flushes build a new segment only for changed live IDs rather than rebuilding the entire collection. In the repository benchmark, updating 1,000 points in a 100k × 128 collection produced a 0.83 MB segment instead of 83.6 MB and reduced flush time from 8.73 seconds to 121.6 milliseconds. Search fans out across at most eight segments; the next flush compacts them into a new full graph.

A filter should reduce work, not destroy recall.

Applying metadata filters after a small ANN result set is fast but wrong: it can return fewer than k results even when enough qualifying points exist. FocalVector builds equality and numeric-range indexes for immutable snapshots, estimates selectivity, and chooses between exact candidate scoring, filtered graph traversal, and broader HNSW search.

For selective filters, an indexed exact scan can be cheaper and more reliable than graph traversal. For broad filters, HNSW remains useful as long as traversal can expand when the filtered result set is too small. The optimization is the plan choice—not a blanket assumption that ANN is always faster.

CUDA is an optional exact query engine.

FocalVector can also keep an immutable full-precision matrix on an NVIDIA GPU. For an unfiltered query, it uploads one vector and uses cuBLAS GEMV to compute dot products against the snapshot. Cosine, dot-product, and Euclidean scoring then follow the collection’s existing semantics.

This path is intentionally optional at two levels. CUDA support is a Cargo feature with dynamically loaded CUDA 12 driver and cuBLAS libraries, so a normal build has no CUDA dependency. At runtime, CUDA can be off, preferred above a configurable collection-size threshold, or required. Preferred mode falls back to CPU/HNSW when initialization fails or the collection is too small to amortize device overhead.

The device matrix is a rebuildable cache, never durable state. After a flush, FocalVector refreshes it from authoritative points. Mutations newer than the GPU snapshot remain on the exact CPU delta path; dirty IDs are excluded from device results and both candidate lists are merged. Filtered searches continue through metadata planning and the CPU engine.

The current baseline still copies one score per snapshot vector back to the host for deterministic top-k selection and full-precision reranking. Device-side top-k reduction is the obvious next optimization for larger collections. Memory is another honest constraint: one million 128-dimensional f32 vectors occupy about 488 MiB of VRAM before scores and cuBLAS workspaces. CUDA is a query option to benchmark, not a badge that automatically makes every workload faster.

HNSW avoids work. Quantization and SIMD cheapen work. Memory mapping avoids duplicate work. CUDA moves exact work. The oracle decides whether any of it was correct.

The optimization stack is the product.

FocalVector’s fastest path is not one algorithm. It is coordination among representations and execution engines: quantized graph traversal finds a candidate set, full-precision vectors settle the ranking, metadata indexes change the plan, a CPU delta preserves fresh writes, and an optional GPU snapshot accelerates the cases that fit it.

That layering is what keeps the system understandable. Each fast path has a workload where it wins, a boundary where it stops, and an exact path to compare against. Performance engineering becomes a series of measurable, reversible decisions instead of one irreversible bet.