Vector Databases for AI

Vector databases are databases and search systems designed to store high-dimensional vectors and efficiently find vectors that are similar to a query vector. They are commonly used with AI embeddings to power semantic search, retrieval-augmented generation, recommendation systems, duplicate detection, and other similarity-based applications.

The basic idea is straightforward: convert data into embeddings, store those vectors together with the original content and metadata, convert a search query into another embedding, and find nearby vectors. The engineering challenge is making this process fast, accurate, secure, and cost-effective across millions or billions of vectors.

Table of Contents

• What Is a Vector Database?
• How Vector Search Works
  • Query Embeddings
  • Similarity Metrics
• Exact vs Approximate Nearest Neighbor Search
• Vector Indexes
  • HNSW
  • IVF
• Metadata and Filtering
• Hybrid Search
• Vector Databases in RAG
• Designing a Production Vector Search Pipeline
  • Ingestion Pipeline
  • Query Pipeline
• Scaling Vector Databases
• Production Considerations
• Conclusion
  • More Articles to Read

What Is a Vector Database?

A traditional database usually retrieves records using exact values, ranges, indexes, or structured predicates. A vector database adds another important operation: find records whose vectors are closest to a supplied vector.

Suppose a knowledge base contains this document:

A read replica may fall behind the primary when it cannot replay database changes quickly enough.

A user searches:

Why is my secondary database delayed?

The query contains different words, so exact keyword matching may not recognize the relationship. An embedding model can convert both pieces of text into vectors that are close in semantic space.

A stored record might look conceptually like:

{
  "id": "chunk-1842",
  "document_id": "database-guide",
  "text": "A read replica may fall behind the primary...",
  "embedding": [0.18, -0.42, 0.71, 0.09],
  "category": "databases",
  "language": "en"
}

When a search arrives, the application creates an embedding for the query and asks the database for the nearest stored vectors.

Text Query → Embedding Model → Query Vector → Vector Database → Similar Records

The vectors themselves normally do not contain the original text. Applications typically store or reference both the embedding and the source content so that retrieved records can be displayed, reranked, or passed to another model.

The process of converting data into vectors is covered in AI Embeddings.

How Vector Search Works

Vector search is a nearest-neighbor problem. Given a query vector, the system attempts to find stored vectors that are closest according to a selected similarity or distance function.

For a small dataset, the application could compare the query against every stored vector. At large scale, specialized indexes are used to avoid examining the entire dataset.

Query Embeddings

The query and stored content must be represented in compatible vector spaces.

If documents were indexed using one embedding model, queries should generally use the same model or another explicitly compatible representation.

def search_documents(query: str, limit: int = 10):
    query_vector = embedding_model.embed(query).vector

    return vector_database.search(
        vector=query_vector,
        limit=limit,
    )

Mixing unrelated embedding models can make similarity scores meaningless. Even when two models produce vectors with the same number of dimensions, those dimensions do not necessarily represent compatible spaces.

This is why the embedding model should be treated as part of the index definition rather than an interchangeable request-time dependency.

Similarity Metrics

The vector database needs a metric for comparing vectors. Common options include cosine similarity, dot product, and Euclidean distance.

The metric should normally follow the embedding model's recommendation. Choosing a different metric simply because it performs well in another application can reduce retrieval quality.

Similarity scores also should not automatically be interpreted as probabilities. A cosine similarity of 0.82 does not necessarily mean an 82% probability that a document is relevant.

Thresholds should be determined using representative application data and retrieval evaluation.

Exact vs Approximate Nearest Neighbor Search

The simplest vector-search algorithm compares a query against every stored vector and sorts the results by similarity.

Conceptually:

def exact_search(query_vector, records, limit):
    scored = []

    for record in records:
        score = cosine_similarity(
            query_vector,
            record.embedding,
        )
        scored.append((score, record))

    scored.sort(key=lambda item: item[0], reverse=True)

    return scored[:limit]

This provides exact nearest neighbors, but the amount of work grows with the number of stored vectors.

Searching 1,000 vectors is easy. Comparing every query with 100 million high-dimensional vectors is a very different workload.

Large vector systems therefore commonly use Approximate Nearest Neighbor (ANN) algorithms. Instead of examining every vector, ANN indexes navigate a smaller candidate set that is likely to contain good matches.

This creates an important trade-off:

Less Search Work → Lower Latency → Possible Recall Loss

The index may occasionally miss a vector that an exhaustive search would have returned.

This is measured using metrics such as recall@k. If the exact top 10 contains ten ideal nearest neighbors and an approximate search returns nine of them, recall@10 is 90% for that query.

Production tuning therefore should not optimize latency alone. An extremely fast index that consistently misses relevant documents can make the entire AI application worse.

Vector Indexes

Vector indexes organize high-dimensional vectors so that nearest-neighbor queries do not need to scan the entire dataset.

Several indexing families exist. Two useful concepts to understand are graph-based indexes such as HNSW and partition-based approaches such as IVF.

HNSW

Hierarchical Navigable Small World (HNSW) organizes vectors into a graph. Each vector is connected to selected nearby vectors, and the graph contains multiple layers that support efficient navigation.

Instead of comparing a query against every record, search starts from selected graph entry points and moves toward increasingly similar vectors.

Conceptually:

Entry Point → Better Neighbor → Better Neighbor → Candidate Region

Higher graph layers contain fewer nodes and help make large jumps across the vector space. Lower layers contain more detail and refine the search around promising regions.

HNSW is popular because it can provide strong recall with low query latency. However, the graph introduces additional memory and indexing overhead.

Several parameters affect its behavior. Increasing the number of graph connections can improve search quality but consumes more memory. Exploring more candidates during queries can improve recall but increases latency.

The correct configuration depends on workload requirements rather than one universally optimal value.

IVF

Inverted File Index (IVF) approaches divide vectors into clusters. Each cluster has a representative centroid.

During search, the query is compared with cluster centroids first. Only vectors inside the most promising clusters are examined.

Conceptually:

Query → Nearest Clusters → Candidate Vectors → Best Matches

If a dataset contains 10 million vectors divided into many clusters, a query may need to inspect only a small subset of those clusters.

Searching more clusters generally improves recall while increasing query cost. Searching fewer clusters reduces latency but increases the chance of missing useful vectors.

IVF can also be combined with vector compression techniques to reduce memory and storage requirements.

The important architectural idea is not memorizing every vector-index algorithm. It is understanding that vector indexing exchanges some combination of memory, build time, precision, and query work for lower search latency.

Metadata and Filtering

Semantic similarity is usually only one condition in a real application.

Suppose an AI documentation platform stores vectors from many customers. A query from customer A might be semantically closest to a document belonging to customer B. Returning that result would be a serious security failure.

Records therefore commonly include metadata:

{
  "id": "chunk-1842",
  "tenant_id": "tenant-42",
  "document_id": "doc-91",
  "language": "en",
  "status": "published",
  "created_at": "2026-08-12",
  "embedding": [0.18, -0.42, 0.71]
}

A query can combine similarity search with deterministic filters:

results = vector_database.search(
    vector=query_vector,
    filters={
        "tenant_id": authenticated_tenant_id,
        "language": "en",
        "status": "published",
    },
    limit=10,
)

Common filters include tenant, user permissions, language, product, document type, region, time range, content status, and data classification.

Filtering introduces its own performance challenges. Consider a vector index containing 100 million records where a tenant can access only 2,000.

If the system first retrieves globally similar vectors and filters unauthorized results afterward, it may return too few usable candidates. It may also perform unnecessary search work.

Vector systems therefore use different approaches to combine ANN traversal with metadata filtering. Depending on the database, filters may be applied before search, during index traversal, or to a larger candidate set before final ranking.

Authorization must never depend on vector similarity. Access control should remain a deterministic constraint enforced by the application and storage layer.

Hybrid Search

Vector search is strong at semantic similarity, but exact lexical matching remains extremely valuable.

Consider these queries:

How do I recover a delayed database replica?

and:

SQLSTATE 40001

The first query is semantic. Several differently worded documents may answer it.

The second contains an exact technical identifier. Keyword search may locate the relevant documentation more reliably than a semantic vector alone.

Hybrid search combines vector retrieval with lexical retrieval.

                → Vector Search  ─┐
Query →─────────                  ├→ Merge / Rerank → Results
                → Keyword Search ─┘

Each retrieval method produces candidates. Their rankings can then be combined using weighted scores, rank fusion, or a separate reranking model.

For example, vector search might retrieve documents about transaction conflicts while lexical search ensures documents containing the exact error code receive strong consideration.

Hybrid retrieval is particularly useful for technical documentation, product catalogs, legal data, source code, and enterprise search where semantic concepts and exact identifiers frequently appear together.

Vector search should therefore be viewed as an additional retrieval capability rather than a universal replacement for traditional search.

Vector Databases in RAG

One of the most common uses of vector databases is the retrieval layer of Retrieval-Augmented Generation (RAG).

Documents are split into chunks and embedded before queries arrive:

Documents → Chunking → Embeddings → Vector Database

When a user asks a question:

Question
   ↓
Query Embedding
   ↓
Vector Search
   ↓
Relevant Chunks
   ↓
LLM Context
   ↓
Generated Answer

The vector database does not generate the answer. Its job is to retrieve useful evidence.

Suppose a company has 500,000 operational documents and a user asks:

How long should failed payment events remain available for replay?

The vector database might retrieve several relevant sections from event-processing and retention documentation. The application can then place those passages into the LLM context.

If retrieval returns irrelevant documents, even a strong language model may produce a poor answer. If the correct evidence is missing entirely, the model may have no reliable basis for answering.

This makes retrieval quality one of the most important components of a RAG system. RAG (Retrieval-Augmented Generation) covers the complete architecture.

Designing a Production Vector Search Pipeline

A production vector-search system has two distinct workloads: ingestion and querying. They have different performance characteristics and should usually be designed separately.

Ingestion Pipeline

The ingestion pipeline prepares source content for search.

A typical flow is:

Source
  → Parse
  → Clean
  → Chunk
  → Embed
  → Add Metadata
  → Index

Each stage can fail independently. A document may be unreadable, chunking may produce empty content, the embedding service may time out, or vector indexing may fail after embeddings have already been generated.

For large datasets, ingestion is commonly asynchronous. Work can be placed into a queue and processed by workers:

def process_chunk(job):
    chunk = load_chunk(job.chunk_id)

    vector = embedding_model.embed(chunk.text).vector

    vector_database.upsert(
        id=chunk.id,
        vector=vector,
        metadata={
            "document_id": chunk.document_id,
            "tenant_id": chunk.tenant_id,
            "version": chunk.version,
        },
    )

Jobs should be idempotent so retries do not create duplicate records.

Source versioning is also important. If a document changes, old chunks may need to be removed and new chunks embedded. Otherwise search can return information that no longer exists in the current source.

For large re-indexing operations, batching embedding requests and database writes can significantly improve throughput.

Query Pipeline

The query path is usually latency-sensitive because it sits directly between the user request and the final response.

A more complete search pipeline might look like:

Query
  → Normalize
  → Embed
  → Apply Filters
  → ANN Search
  → Keyword Search
  → Merge Candidates
  → Rerank
  → Return Top Results

Not every application needs every stage. A small semantic-search service may need only embedding and vector search.

More sophisticated systems often retrieve more candidates than they finally return. For example, the vector index might retrieve 50 candidates, a reranker evaluates them more carefully, and the final system returns the best 5.

This architecture separates fast candidate generation from more expensive ranking.

The first stage should find a high-recall candidate set quickly. The second stage can spend more computation deciding which candidates are actually most relevant.

Scaling Vector Databases

Vector databases can become memory-intensive because vectors are large compared with conventional scalar index keys.

Suppose each vector contains 1,536 float32 dimensions:

1,536 dimensions × 4 bytes ≈ 6 KB per vector

At 100 million vectors, raw vectors alone require roughly:

100,000,000 × 6 KB ≈ 600 GB

This excludes metadata, index structures, replication, graph connections, database overhead, and temporary memory used during indexing.

Several techniques can reduce the footprint. Lower-dimensional embeddings store fewer values. Reduced-precision representations use fewer bytes per dimension. Quantization compresses vectors, sometimes at the cost of retrieval accuracy.

At sufficiently large scale, vectors may be partitioned across multiple nodes. A query is routed to relevant shards or broadcast across several shards, and partial results are merged.

Sharding strategy affects both scalability and retrieval quality.

Partitioning by tenant can provide strong isolation and efficient filtering for large tenants, but a system with millions of tiny tenants may create operational complexity. Hash-based sharding balances data more evenly but may require searching multiple shards for each query.

Replication improves availability and read capacity but multiplies storage and index memory requirements.

Capacity planning should consider:

• number of vectors;
• vector dimensions and precision;
• index overhead;
• metadata size;
• replication factor;
• query rate;
• filter selectivity;
• target recall;
• index build and update rate.

A vector database storing 50 million rarely changing document chunks has different requirements from a recommendation system continuously updating hundreds of millions of item and user vectors.

Production Considerations

Vector search should be evaluated as a retrieval system rather than only as a database benchmark.

Query latency matters, but returning the correct information matters more. Production evaluation should therefore combine infrastructure metrics with retrieval-quality metrics.

Embedding model changes require special handling. Vectors generated by different incompatible models should not simply be mixed in the same similarity index.

A safer migration creates a new vector collection or index, re-embeds source content, evaluates retrieval quality, shifts query traffic, and removes the old representation after validation.

Applications should store enough information to identify how each vector was generated:

{
  "id": "chunk-1842",
  "embedding_model": "embedding-model-v3",
  "embedding_version": 3,
  "source_version": 17
}

Deletion also requires careful design. Removing a source document should eventually remove its chunks, vectors, cached retrieval results, and other derived representations where required.

Security controls must operate before retrieved text reaches an LLM. Tenant filters and document permissions should be enforced during retrieval rather than asking the model to ignore unauthorized results afterward.

Finally, not every AI application needs a dedicated vector database. If an existing relational database or search engine already supports the required vector workload at acceptable scale and latency, adding another distributed storage system may create unnecessary operational complexity.

The correct architecture is the simplest storage and retrieval system that satisfies the required scale, latency, filtering, recall, availability, and operational constraints.

Conclusion

Vector databases make embedding-based retrieval practical at scale. They store high-dimensional vectors, use similarity metrics to compare them, and employ specialized indexes such as HNSW or IVF to avoid exhaustive searches across large datasets.

Production systems require more than nearest-neighbor search. Metadata filtering provides security and domain constraints, hybrid search combines semantic and lexical retrieval, reranking improves final relevance, and carefully designed ingestion pipelines keep vectors synchronized with source data.

The central trade-off is between retrieval quality, latency, memory, and operational complexity. Faster approximate search is valuable only when it still retrieves the information needed by the application.

A vector database does not understand or answer questions. It efficiently narrows a large information space to a small set of potentially relevant candidates. The quality of everything downstream depends heavily on how well that retrieval step works.

More Articles to Read

• AI Embeddings
• RAG (Retrieval-Augmented Generation)
• AI Tokens and Context Windows
• Large Language Models (LLMs)