System Design Interviews retrieval-augmented-generation ai-system-design system-design-interviews

System Design Interview: How Would You Design a Secure RAG Assistant for Internal Documents?

An internal-document assistant needs more than a model that produces plausible answers. Consider an engineer asking, “Who must approve a production deployment of Atlas?” The answer must come from the right organization's current policy, respect the engineer's permissions, and identify the evidence behind the claim.

This interview walkthrough develops that request into a secure retrieval-augmented generation (RAG) system. The same deployment question connects document ingestion, authorization, retrieval, context selection, answer validation, and capacity planning, so each architectural choice follows from a problem already encountered.

Table of Contents

Start with the Answer That Must Be Defensible

Assume Atlas belongs to a hypothetical organization called Northstar. Its deployment policy says that a service owner must approve routine production releases, while a separate security-review document is restricted to the security team. An ordinary engineer may read the policy; the assistant must not retrieve restricted instructions to improve its answer.

This distinction sets the first interview boundary: version one answers questions and cites documents, but cannot approve a deployment, change a policy, or execute a command. Before choosing a search engine or model, establish what counts as a valid response to the engineer's question.

  • Answered: Accessible evidence supports the claim, and the response identifies the document version used.
  • Insufficient evidence: The available documents do not establish who can approve this deployment.
  • Temporarily unavailable: A dependency prevents a safe answer, such as an unavailable authorization service.

The last two outcomes must remain distinct. If search times out, reporting that no approval policy exists would turn an infrastructure failure into a misleading policy answer. That requirement will shape both the request path and its fallback behavior.

Make the Policy Retrievable Without Losing Its History

The assistant cannot answer from a policy document until that document becomes searchable evidence. A connector therefore fetches Northstar's policy, a worker extracts its text, and a chunker creates passages that keep an approval rule together with the section needed to interpret it.

Each passage needs more than text and a vector. For the running example, the following record identifies section 4 of policy version 17 and the access-policy revision observed during ingestion. The identifiers and revisions are illustrative; the text is an example policy clause.

{
  "tenant_id": "northstar",
  "document_id": "release-policy",
  "document_version": "v17",
  "chunk_id": "northstar:release-policy:v17:section-4",
  "observed_acl_revision": 12,
  "allowed_groups": ["engineering"],
  "text": "Routine Atlas production deployments require approval from the service owner."
}

The versioned chunk ID lets a later citation refer to the exact evidence used, even after the policy changes. The observed ACL revision serves a different purpose: it shows what the index knew at ingestion time. It cannot establish whether the engineer is still allowed to read the passage when asking a question.

Activate Complete Versions Instead of Partially Updated Documents

Suppose the policy changes to version 18. Replacing passages one at a time could let one answer combine an old approval clause with a new exception. Prepare the new lexical and vector representations separately, verify their readiness, and then move an authoritative active-version pointer.

This pointer is needed because two search indexes do not automatically share a transaction. Query-time eligibility checks reject candidates that do not match the active version; old physical entries can be removed later. The accepted content-freshness window determines whether version 17 remains usable while version 18 is being prepared.

  1. Deduplicate: Identify work by tenant, source document, and source revision before parsing or embedding again.
  2. Prepare: Write the new passages and required search representations without declaring them active.
  3. Activate: Update the authoritative version pointer only after the required representations are ready.
  4. Recover: Retry failed items independently and quarantine an unparseable revision rather than blocking other documents.

Keeping that work asynchronous protects the interactive path from slow parsers and embedding batches. More importantly, the request can now ask a precise question about every candidate: is this passage from the active version, and may this caller read it?

Follow the Question Through Authorized Retrieval

When the engineer submits the Atlas question, the service first resolves Northstar and the engineer's groups from trusted authentication and authorization state. Tenant identity and security filters are constructed by the server, so a client cannot widen the search by editing a tenant field in the request.

The search stage uses those constraints to find candidate IDs. Before passage text reaches a reranker or model, the service checks each candidate against current access policy and the active-version pointer introduced above, then hydrates only eligible content. The restricted security-review document is excluded even if it would rank first semantically.

Retrieve the Approval Rule and Its Relevant Exceptions

“Atlas” is an exact product name, while “who can approve a release?” may be a semantic paraphrase of the stored clause. Lexical and vector retrieval can therefore contribute different candidates. Combine them through an explicit ranking strategy and evaluate whether the authorized approval rule and any applicable exceptions actually survive.

Searching a global top-k and removing unauthorized hits afterward can leave the engineer with no useful evidence. Apply security constraints during retrieval, recheck eligibility before hydration, and use bounded additional batches when rejected candidates leave too few passages. Filter-aware approximate search still needs recall and latency measurements for the chosen engine.

A reranker can now compare the approval clause, its exception section, and an accessible deployment checklist. It cannot recover a passage that candidate retrieval never found. This is why increasing model quality would not fix a missing approval rule: the failure occurred before generation began.

Distinguish Access Failure from Missing Evidence

If current permissions cannot be established, the pipeline cannot safely declare a candidate eligible. Fail closed for the affected evidence and return a temporary failure when the question cannot otherwise be served. If authorization succeeds but no supporting rule is found, the correct outcome is insufficient evidence.

The distinction established in the answer contract is now enforced at a concrete boundary. Only the successful path produces the authorized, versioned passage set consumed by the context-selection step.

Turn Eligible Passages into a Bounded Answer

Retrieval can return overlapping chunks, long appendices, and several passages from the same policy. Sending all of them adds input cost and can bury the approval rule. Context selection must preserve useful evidence while leaving room for application instructions, the question, conversation history, and output.

Select Evidence with an Explicit Budget

The following Python example operates on passages that have already passed the eligibility checks. It preserves ranking order, removes repeated chunk IDs, and caps passages per document. The per-document cap is a diversity heuristic; evaluation must check that it does not discard an important exception.

from dataclasses import dataclass


@dataclass(frozen=True)
class Passage:
    document_id: str
    chunk_id: str
    token_cost: int
    text: str


def select_evidence(
    ranked: list[Passage], token_budget: int, per_document: int = 2
) -> list[Passage]:
    # Authorization and active-version checks happen before this function.
    if token_budget < 0 or per_document < 1:
        raise ValueError("Invalid evidence limits")

    selected: list[Passage] = []
    seen: set[str] = set()
    document_counts: dict[str, int] = {}
    remaining = token_budget

    for passage in ranked:
        if passage.token_cost <= 0:
            raise ValueError("Passage token cost must be positive")
        # Overlapping retrieval methods may return the same chunk twice.
        if passage.chunk_id in seen:
            continue
        if document_counts.get(passage.document_id, 0) >= per_document:
            continue
        # Skip oversized candidates; a later useful passage may still fit.
        if passage.token_cost > remaining:
            continue

        selected.append(passage)
        seen.add(passage.chunk_id)
        document_counts[passage.document_id] = (
            document_counts.get(passage.document_id, 0) + 1
        )
        remaining -= passage.token_cost

    return selected


# Costs represent illustrative serialized blocks; text is shortened here.
rule = Passage("release-policy", "northstar:release-policy:v17:section-4",
               260, "Routine Atlas production deployments require service-owner approval.")
exception = Passage("release-policy", "northstar:release-policy:v17:section-5",
                    340, "Emergency releases follow the documented exception process.")
checklist = Passage("deployment-checklist", "northstar:deployment-checklist:v6:step-2",
                    180, "Record the service owner's approval before deployment.")

# The duplicate rule consumes no extra budget. All three unique blocks fit.
chosen = select_evidence([rule, rule, exception, checklist], token_budget=800)
assert [p.chunk_id for p in chosen] == [
    rule.chunk_id, exception.chunk_id, checklist.chunk_id
]
assert sum(p.token_cost for p in chosen) == 780

The selected 780-token evidence set contains the rule, its exception, and the checklist that records approval. These are illustrative block costs, not measured token counts for the shortened strings. In the real request, calculate costs using the selected model's tokenizer and verify the complete serialized prompt because token counts at concatenation boundaries may differ.

The function does not prove that the evidence answers the question. If the approval rule was missing or an exception was wrongly filtered out, fitting the remaining passages into 800 tokens would not repair that mistake. Evidence quality therefore remains a separate check when constructing the answer.

Bind the Answer to the Selected Evidence

Assign citation labels to the selected immutable chunk IDs before invoking the model. For this request, label the approval rule c1 and the exception passage c2; keep the checklist as additional supporting context. The application resolves those labels to sources; it does not accept an invented model-produced URL.

{
  "request_id": "req-7d31",
  "status": "answered",
  "answer": "Routine Atlas production deployments require service-owner approval. [c1] Emergency releases follow a separate exception process. [c2]",
  "citations": [
    {
      "id": "c1",
      "document_id": "release-policy",
      "document_version": "v17",
      "chunk_id": "northstar:release-policy:v17:section-4"
    },
    {
      "id": "c2",
      "document_id": "release-policy",
      "document_version": "v17",
      "chunk_id": "northstar:release-policy:v17:section-5"
    }
  ]
}

This response completes the engineer's original request: the answer names the routine approver, distinguishes the exception, and identifies both source passages. A citation-ID check only establishes provenance, however. The system must also evaluate whether the cited passage supports the claim, including whether the question actually describes an emergency exception.

The exception passage does not name an emergency approver. An emergency-specific question therefore requires the actual exception procedure, further clarification, or an insufficient-evidence response. If validation must finish before any answer is released, buffer the answer or validate bounded sections; streaming everything immediately would prevent a later validator from withdrawing unsupported text.

Keep the Same Request Path Safe as State Changes

The first successful answer establishes the basic path, but the policy, engineer's permissions, and dependencies will change. Each change should alter a known eligibility or response decision rather than introduce a second, less controlled route to the model.

Revocation Changes Cache and Conversation Design

Suppose Northstar restricts the policy after the first answer. A previously generated response and its conversation history now contain evidence the engineer may no longer read. Reusing either without checking provenance would bypass the authorization boundary even if fresh search requests were filtered correctly.

For that reason, start without cross-user answer caching. Reauthorize cached passages and the sources behind reusable answers, and revalidate evidence-bearing conversation history before another model call. If affected history cannot be separated safely, discard those turns or restart the conversation.

  • Cache identity: Include the tenant and relevant security and content scope; question text alone is insufficient.
  • Restrictive policy changes: Make current authorization authoritative before waiting for asynchronous index updates.
  • In-flight answers: Define whether revocation applies at request start, before release, or through periodic checks. Cancellation cannot retract text already delivered.
  • Source deletion: Mark the document ineligible first, stop its reuse under the agreed policy, then clean search entries and cached material.

Malicious Content Does Not Get Application Authority

The policy might also contain an injected instruction claiming that the assistant should ignore access rules or execute a deployment. It remains document data, even if it appears in an otherwise eligible passage. Separating evidence from application instructions helps, but a prompt cannot enforce the authorization policy.

The read-only boundary chosen at the start now limits the damage: the model has no deployment credentials or arbitrary network tools. Citation resolution and output escaping remain application responsibilities. Adding deployment actions later would require a separately authorized tool path, not a stronger sentence in the prompt.

Dependency Failures Preserve the Answer Contract

The same principle applies when a component fails: remove or reject the unsafe path rather than quietly relaxing the contract. The following cases follow the engineer's request through the dependencies already introduced.

Failure in this requestVisible outcomeContainment and recovery
Current permission check fails.A temporary failure when the policy cannot be used safely.No stale permission grant is substituted. Track authorization errors separately from empty search results.
Search times out before finding the approval rule.A temporary retrieval failure.Bound search time and extra candidate batches. Do not invent an approval policy from model memory.
The model is throttled after evidence selection.A capacity response, or explicitly labeled authorized search results if that path is healthy.Use bounded admission and retries only while the remaining deadline and provider policy permit them.
A response stream breaks after partial text.The answer is marked incomplete.Correlate attempts and usage by request ID; do not automatically append a regenerated answer to the broken stream.

These failure choices also determine resource limits. Every retry, queued question, and long-running generation occupies capacity that another engineer might need, so the next design step is to quantify the successful path and bound its work.

Size the Path That Produces This Answer

For an illustrative interview workload, extend the same design to 10 million chunks and 50 questions per second at peak. Assume 768-dimensional float32 embeddings, 4,000 input tokens per question, and a maximum planned answer size of 400 output tokens. These are planning assumptions, not observed benchmarks.

The 800-token selection example showed one evidence budget, not the total model request. Instructions, the question, formatting, and bounded history consume the rest of the input allowance. Measure their serialized size instead of multiplying the evidence size alone when estimating model demand.

ResourceIllustrative calculationWhat it changes in the design
Raw vector payload10M × 768 × 4 bytes ≈ 30.7 GB decimal.Provision separately for metadata, search structures, replicas, lexical indexes, and headroom.
Input token demand50 questions/s × 4,000 tokens = 200,000 tokens/s.Check input-token quotas as well as request-rate limits.
Output token demand50 questions/s × 400 tokens = 20,000 tokens/s.Answer length directly affects serving capacity and cost.
Concurrent generation requests50 requests/s × 8 seconds mean generation service time ≈ 400.Bound generation concurrency and queue wait; this average omits tail-load headroom.

At an assumed decode rate of 60 tokens per second, a 400-token answer takes about 6.7 seconds to decode before accounting for retrieval and prefill. A short target for time to first token therefore cannot also be treated as a promise for the final answer.

Protect that latency budget with per-tenant admission limits, a bounded generation pool, and short queues. Propagate disconnects and deadlines where supported, but do not assume client cancellation stops provider charges. Model token cost must be added to ingestion embeddings, search, reranking, storage, and retry costs.

The estimates identify what to measure: queue wait and tenant interference before generation, retrieval and authorization latency before prompt construction, then first-token latency, final-answer latency, and tokens per completed request. Those stage measurements also explain why a correct architecture can still need a different capacity allocation.

Test the Design by Changing the Original Question

A useful evaluation set starts with the Atlas question and changes one condition at a time. This keeps a regression attributable to a particular decision instead of hiding retrieval, authorization, and generation failures inside one overall quality score.

  • Paraphrase the question: “Who can sign off an Atlas release?” checks whether retrieval still finds the same authorized rule.
  • Specify an emergency release: The exception section must affect the answer; a routine-deployment answer is no longer sufficient.
  • Remove access: The rule must disappear from new eligible evidence, reusable answers, and history under the defined request-time policy.
  • Activate version 18: The answer must not combine version 17's rule with version 18's exception.
  • Remove supporting evidence: The assistant must distinguish insufficient evidence from an infrastructure outage.
  • Add malicious document instructions: The assistant must not gain access to restricted documents or deployment actions.
  • Increase one tenant's traffic: Queueing and admission should contain the impact on other organizations.

Review candidate recall, answer support, citation accuracy, and appropriate abstention separately for these cases. Calibrate automated graders against reviewed examples, and keep raw questions and passages out of general logs; request IDs and restricted diagnostic access are enough for many operational investigations.

Extend the Design When the Interviewer Changes a Constraint

If the interviewer proposes fine-tuning on Northstar's documents, return to the revocation case: learned parameters do not provide document-level authorization or reliable fact removal. Fine-tuning may be evaluated for behavior on permitted training data, but it does not replace the evidence-access path.

If shared search causes unacceptable tenant interference, compare dedicated tenant indexes against stronger workload controls using the measured bottleneck and isolation requirements. A dedicated index can help meet specific scale or residency constraints, but adds provisioning and migration work and does not remove application authorization.

Finally, a model or embedding migration must pass the same question variants before traffic moves. Build a compatible new index for a new embedding space rather than mixing vectors merely because their dimensions match, and version chunking, ranking, prompts, and evaluation data so a rollback has a defined target.

Conclusion

The Atlas question becomes a complete design by following its dependencies: a versioned policy becomes eligible evidence, retrieval finds its rule and exceptions, a bounded prompt produces a supported answer, and state changes and load test the guarantees behind that answer. Each component exists to preserve a property established earlier.

Key Takeaway: In an AI system design interview, develop the architecture from one defensible request. Explain how each decision enables the next step, then show what happens when its assumptions change.

Comments (0)

Related articles