Search Engines Explained: How Modern Search Works

By Oleksandr Andrushchenko — Published on
0 Likes
0 Dislikes
Search Engine Explained: Elasticsearch Simple Architecture
Search Engine Explained: Elasticsearch Simple Architecture

Search looks simple from the outside: enter a query and receive a list of relevant results. Behind that interface is a specialized data-processing system designed to find useful documents across millions or billions of records within milliseconds.

A modern search engine does much more than scan text. It processes documents, analyzes language, builds specialized indexes, interprets queries, retrieves candidate documents, calculates relevance scores, ranks results, and continuously updates its indexes as source data changes.

Understanding these components is important when designing product search, document search, log search, knowledge bases, marketplaces, social platforms, and other systems where traditional database queries eventually become insufficient.

Table of Contents

Why Search Is Different from Database Lookup

A relational database is primarily designed to retrieve records using structured conditions:

SELECT *
FROM products
WHERE category_id = 42
  AND price BETWEEN 50 AND 100
ORDER BY created_at DESC;

The query describes exact constraints. Database indexes such as B-trees allow the database to efficiently locate rows matching those constraints.

Search queries are different:

"wireless noise cancelling headphones"

The system may need to return documents containing:

  • all query terms;
  • only some query terms;
  • different grammatical forms;
  • synonyms;
  • terms in different fields;
  • possibly misspelled terms.

It must then decide which matching document is more relevant.

For example:

Query:
"wireless headphones"

Document A:
"Premium wireless noise-cancelling headphones"

Document B:
"USB wireless adapter compatible with headphones"

Document C:
"Headphones with Bluetooth connectivity"

All three documents may be related to the query, but they should not necessarily receive the same score.

This is the fundamental difference between database lookup and information retrieval:

Databases primarily answer "Which records satisfy these conditions?" Search engines additionally answer "Which matching documents are most relevant?"

Search Engine Architecture

A typical search architecture separates the system responsible for authoritative application data from the infrastructure optimized for retrieval.

                  WRITE PATH

Application
    |
    v
Primary Database
    |
    | changes
    v
Indexing Pipeline
    |
    v
Search Index


                  READ PATH

User Query
    |
    v
Search API
    |
    v
Query Processing
    |
    v
Search Index
    |
    v
Ranking
    |
    v
Top Results

The primary database remains the source of truth. The search engine maintains a derived representation optimized specifically for searching.

This separation allows each storage system to serve a different access pattern.

Primary Database:
transactions
constraints
relationships
authoritative state

Search Engine:
text retrieval
relevance ranking
faceting
filtering
autocomplete
high-volume search reads

Document Indexing

Search begins before the user sends a query. Documents must first be transformed into structures that can be searched efficiently.

Consider a product:

{
    "id": 1042,
    "name": "Wireless Noise Cancelling Headphones",
    "description": "Bluetooth headphones with active noise cancellation",
    "brand": "SoundWave",
    "category": "Audio",
    "price": 199.99
}

The search engine analyzes selected fields and creates an index from their contents.

Text Analysis and Tokenization

Raw text is usually processed through an analysis pipeline.

"Wireless Noise Cancelling Headphones"
                |
                v
           Tokenization
                |
                v
["Wireless", "Noise", "Cancelling", "Headphones"]
                |
                v
           Lowercasing
                |
                v
["wireless", "noise", "cancelling", "headphones"]
                |
                v
       Optional normalization
                |
                v
["wireless", "noise", "cancel", "headphone"]

Depending on the language and search requirements, analysis may include:

  • tokenization;
  • lowercasing;
  • stemming;
  • lemmatization;
  • stop-word handling;
  • accent normalization;
  • synonym expansion.

Analysis determines what the search engine considers a searchable term. Poor analyzer configuration can therefore produce poor search quality even when the rest of the architecture is correct.

The Inverted Index

The central data structure behind traditional full-text search is the inverted index.

Suppose the system contains three documents:

Doc 1: "wireless headphones"
Doc 2: "wireless keyboard"
Doc 3: "gaming headphones"

A simplified inverted index looks like this:

Term Documents
wireless 1, 2
headphones 1, 3
keyboard 2
gaming 3

Instead of scanning every document looking for wireless, the search engine directly retrieves the posting list:

wireless -> [1, 2]

For a two-term query:

wireless   -> [1, 2]
headphones -> [1, 3]

intersection -> [1]

Real inverted indexes contain substantially more information. Posting lists may include term frequency, document frequency, field information, token positions, and other metadata used for phrase matching and relevance scoring.

The inverted index is the key reason search engines can perform full-text retrieval over very large collections without scanning every document.

Query Processing

Once the index exists, incoming search queries must be interpreted and converted into operations against it.

Query Analysis

The query usually passes through analysis similar to the document indexing process.

User query:
"Wireless Headphones"

        |
        v

["wireless", "headphones"]

        |
        v

Search inverted index

Using compatible analysis at indexing and query time is important. If documents store normalized terms while queries search completely different representations, relevant documents may never become candidates.

More sophisticated query processing can also perform:

  • spelling correction;
  • synonym expansion;
  • query rewriting;
  • language detection;
  • phrase recognition;
  • intent detection.

For example:

"laptop under 1000"

            |
            v

Text query: "laptop"
Filter: price < 1000

Separating textual relevance from structured filters can significantly improve both search quality and execution efficiency.

Candidate Retrieval

Search engines usually do not run expensive ranking logic against every indexed document.

Instead, search commonly happens in stages:

100,000,000 documents
        |
        v
Inverted Index Retrieval
        |
        v
10,000 candidates
        |
        v
Initial Scoring
        |
        v
1,000 candidates
        |
        v
Advanced Ranking
        |
        v
Top 20 results

The first stage prioritizes efficient retrieval. Later stages can use increasingly expensive ranking signals because they operate on much smaller candidate sets.

This retrieve then rank architecture is fundamental to large-scale search systems.

Relevance Scoring and Ranking

Finding documents containing the query terms is only the beginning. The search engine must determine the order in which results appear.

A basic ranking system may consider:

  • how often query terms appear;
  • how rare those terms are across the index;
  • which fields contain the terms;
  • document length;
  • phrase proximity;
  • document popularity;
  • freshness;
  • business-specific signals.

BM25 and Lexical Relevance

BM25 is a widely used ranking function for lexical search. Conceptually, it rewards documents where query terms occur meaningfully while accounting for how common those terms are and how long the document is.

Consider the query:

"distributed database"

A document containing distributed ten times is not automatically ten times more relevant than one containing it once. BM25 applies saturation so repeated occurrences provide diminishing benefits.

Rare terms can also carry more information than extremely common terms.

"the"         -> appears almost everywhere -> weak signal
"postgresql"  -> less common                -> stronger signal

Field importance can be incorporated separately. A match in a product title may be more valuable than the same match deep inside a long description.

title match       x 3.0
category match    x 2.0
description match x 1.0

Combining Search and Business Signals

Text relevance alone is rarely enough for production search.

An e-commerce system might combine lexical relevance with popularity, availability, ratings, and freshness:

final_score =
    0.60 * text_relevance +
    0.15 * popularity +
    0.10 * rating +
    0.10 * availability +
    0.05 * freshness

A marketplace may prioritize sellers with strong quality signals. A news platform may place more weight on freshness. A documentation system may prioritize authoritative pages.

Ranking is therefore both an information-retrieval problem and a product-specific optimization problem.

A single machine eventually becomes insufficient when indexes grow beyond available storage, memory, CPU capacity, or query throughput.

Distributed search engines divide an index into shards.

                 Search Index
                      |
          +-----------+-----------+
          |           |           |
          v           v           v
       Shard 1     Shard 2     Shard 3
       Docs A-H    Docs I-P    Docs Q-Z

A query can be sent to multiple shards in parallel:

                   Query
                     |
             +-------+-------+
             |       |       |
             v       v       v
           Shard   Shard   Shard
             1       2       3
             |       |       |
          Top 10  Top 10  Top 10
             \       |       /
              \      |      /
               v     v     v
                Coordinator
                     |
                     v
                Global Top 10

Each shard retrieves and scores local candidates. A coordinating node merges the shard-level results into the final ranking.

Replication can provide availability and additional read capacity:

Shard 1
  |
  +-- Primary
  +-- Replica

Shard 2
  |
  +-- Primary
  +-- Replica

Sharding enables horizontal scalability, but it also introduces distributed-system trade-offs: shard placement, rebalancing, replication, node failures, query fan-out, hot shards, and cross-shard ranking behavior.

These operational concerns become particularly important for Elasticsearch and similar distributed search engines and will be covered later in Scaling Elasticsearch Clusters.

Keeping the Search Index Updated

A search index is usually a derived copy of data rather than the authoritative source. Changes in the primary system must therefore propagate into the index.

A simple architecture might perform synchronous indexing:

Application
    |
    +----> Database
    |
    +----> Search Engine

This creates a dual-write problem. The database write may succeed while the search update fails, leaving the two systems inconsistent.

A more resilient architecture decouples indexing from the application write path:

Application
    |
    v
Database
    |
    | change event
    v
Queue / Event Stream
    |
    v
Indexing Workers
    |
    v
Search Engine

The application commits authoritative state first. Changes are then propagated asynchronously to the search infrastructure.

This means search is commonly eventually consistent.

T0  Product updated in database
T1  Change published
T2  Indexing worker receives change
T3  Search index updated
T4  New value visible in search

The delay may be milliseconds or seconds depending on architecture and load.

For most search use cases this is acceptable, but the acceptable indexing delay should be explicitly defined and monitored.

End-to-End Search Request

Putting the components together, a typical search request follows this path:

User
 |
 | "wireless headphones"
 v
Search API
 |
 v
Query Analysis
 |
 | tokens, filters, synonyms
 v
Candidate Retrieval
 |
 | inverted index
 v
Initial Ranking
 |
 v
Top Candidates
 |
 v
Business Ranking
 |
 v
Filters / Permissions
 |
 v
Top Results
 |
 v
User

The complete request may need to finish in tens or hundreds of milliseconds, even when the underlying index contains millions of documents.

Latency therefore depends on much more than raw search-engine performance. Query complexity, shard fan-out, cache effectiveness, ranking logic, network calls, authorization filters, result enrichment, and serialization all contribute to the final response time.

Production Design Example

Consider product search for a large marketplace containing 50 million products.

                         WRITE SIDE

Seller API
    |
    v
Product Service
    |
    v
Product Database
    |
    v
Change Stream
    |
    v
Indexing Workers
    |
    v
Search Cluster


                         READ SIDE

Customer
    |
    v
Search API
    |
    +--> Query Analysis
    |
    +--> Search Cluster
    |       |
    |       +--> Shard 1
    |       +--> Shard 2
    |       +--> Shard 3
    |       +--> ...
    |
    +--> Ranking
    |
    v
Search Results

The indexed document might intentionally duplicate data from several authoritative systems:

{
    "product_id": 1042,
    "title": "Wireless Noise Cancelling Headphones",
    "description": "...",
    "brand": "SoundWave",
    "category": "Audio",
    "price": 199.99,
    "rating": 4.7,
    "review_count": 8421,
    "in_stock": true,
    "popularity_score": 0.91
}

This denormalization is intentional.

Executing relational joins across multiple application databases during every search request would create latency and runtime dependencies. Search documents are instead shaped around the search read pattern.

The search engine can retrieve candidates using text fields, apply structured filters such as category and price, and use ranking fields such as rating and popularity without contacting several backend services for every candidate.

Only the small set of final results may need additional enrichment from other systems.

50,000,000 products
        |
        v
Search index
        |
        v
1,000 candidates
        |
        v
Ranking
        |
        v
20 results
        |
        v
Optional enrichment

This architecture keeps expensive cross-service operations away from the large candidate set.

Common Mistakes

SQL databases can provide useful text-search capabilities, and for small systems they may be completely sufficient.

Problems appear when applications repeatedly rely on patterns such as:

SELECT *
FROM products
WHERE LOWER(name) LIKE '%wireless headphones%';

Substring scans become expensive as datasets and query volume grow. They also provide limited relevance ranking compared with specialized information-retrieval systems.

The decision is not that SQL search is always wrong. The correct threshold depends on dataset size, query complexity, ranking requirements, and operational constraints. The trade-offs are explored in Full-Text Search vs SQL Search.

Indexing Everything

Search indexes should be designed around search requirements, not treated as complete replicas of application databases.

Indexing unnecessary fields increases:

  • index size;
  • memory pressure;
  • storage requirements;
  • indexing cost;
  • network traffic;
  • recovery time.

Only fields needed for retrieval, filtering, sorting, ranking, display, or another explicit search operation should normally be included.

Ranking Only by Text Similarity

A textually relevant result is not necessarily the best result.

Consider two products with nearly identical titles:

Product A
text score: 9.8
rating: 2.1
out of stock

Product B
text score: 9.4
rating: 4.8
in stock

A ranking system based exclusively on lexical similarity may consistently produce results that users consider poor.

Production ranking should incorporate signals that represent actual product quality and user intent. These techniques will be explored in Search Relevance and Ranking Strategies.

Making Search Indexing Part of the Write Path

Requiring both database and search writes to succeed before completing a request tightly couples application availability to the search cluster.

Request
   |
   +--> Database       OK
   |
   +--> Search Engine  FAILED
   |
   v
What should happen?

There is no simple atomic transaction across independent storage technologies.

Asynchronous indexing through a durable event or change stream usually creates a cleaner boundary. The database remains authoritative while indexing workers retry failed updates independently.

Production Checklist

  • Define which system owns the authoritative data.
  • Treat the search index as a derived read model.
  • Design indexed documents around actual search access patterns.
  • Choose analyzers according to language and product requirements.
  • Separate full-text queries from structured filters where appropriate.
  • Measure relevance using representative production queries.
  • Include business signals when lexical relevance alone is insufficient.
  • Keep expensive ranking stages limited to small candidate sets.
  • Use asynchronous indexing when immediate consistency is unnecessary.
  • Monitor indexing lag and failed indexing operations.
  • Measure p50, p95, and p99 search latency.
  • Monitor shard size, query throughput, CPU, memory, and storage growth.
  • Plan index rebuilding and schema migrations before they are required.
  • Test search behavior using realistic dataset sizes and query distributions.

Conclusion

Modern search engines achieve fast retrieval by transforming documents into specialized indexes before queries arrive. Text is analyzed into searchable terms, inverted indexes map those terms back to documents, query processing identifies candidate documents, and ranking algorithms determine which results should appear first.

At larger scale, the index is distributed across shards and replicated for availability and throughput. Application changes flow through indexing pipelines so that the search representation remains synchronized with authoritative data, usually with some degree of eventual consistency.

The result is an architecture optimized around a fundamentally different problem from ordinary database access: efficiently finding and ranking the most useful documents from a very large candidate set.

Key Takeaway: A search engine is not simply a faster implementation of LIKE '%query%'. It is a specialized retrieval system built around text analysis, inverted indexes, candidate selection, relevance scoring, distributed execution, and ranking. Good production search comes from designing all of these stages around real query patterns and relevance requirements.

Author

Oleksandr Andrushchenko

Oleksandr Andrushchenko

Software Engineer & Solution Architect
Nov 03, 2025 4 167
Enjoyed this article?

Support Oleksandr Andrushchenko

Buy me a coffee

This helps Oleksandr Andrushchenko continue creating useful content

Comments (0)