AI Monitoring and Evaluation

AI monitoring and evaluation is the process of measuring whether an AI application is producing useful, correct, reliable, safe, and cost-effective results before and after deployment. Traditional monitoring can tell whether an API is available or slow. AI systems need additional measurements because a request can return HTTP 200 in 800 milliseconds and still produce a completely incorrect answer.

This creates two related but different engineering problems. Evaluation measures the quality of AI behavior using test cases, expected results, human review, or automated evaluators. Monitoring observes what happens in production: latency, errors, token usage, retrieval quality, tool calls, costs, user outcomes, and changes in model behavior.

A production AI system needs both. Evaluation helps determine whether a model, prompt, retrieval configuration, or workflow is good enough to deploy. Monitoring helps determine whether that system continues working after deployment under real traffic.

Table of Contents

• Why AI Monitoring Is Different
• Evaluation vs Monitoring
• What to Evaluate
  • Task Quality
  • Groundedness and Hallucinations
  • Structured Output and Tool Calls
• Building an Evaluation Dataset
• Automated Evaluation
• Evaluating RAG Systems
• Evaluating AI Agents
• Production Monitoring
• Tracing AI Requests
• Production Evaluation Pipeline
• Common Monitoring and Evaluation Mistakes
• Production Checklist
• Conclusion
  • More Articles to Read

Why AI Monitoring Is Different

Traditional application monitoring focuses heavily on deterministic system behavior. If an endpoint should return a customer record, engineers can measure whether the request succeeded, how long it took, and whether the returned data matched a schema.

AI applications add a less deterministic layer:

Request
   ↓
Application Works?
   ↓
Model Responds?
   ↓
Response Valid?
   ↓
Response Correct?
   ↓
Response Useful?

The first three questions can often be measured with conventional software metrics. The last two are much harder.

Consider a support assistant asked:

Can order ORD-18492 still be returned?

The system responds:

Yes. Orders can be returned within 60 days.

The request completed successfully. No exception occurred. The response was valid text. Latency was excellent.

But if the actual policy allows returns for only 30 days, the application failed at its real task.

This means infrastructure health and AI quality must be measured separately:

A production system can be healthy according to the left column while failing according to the right column.

Evaluation vs Monitoring

Evaluation and monitoring solve different parts of the same problem.

Evaluation asks:

How well does this AI system perform on representative tasks?

It can run before deployment:

Prompt v17
   ↓
Evaluation Dataset
   ↓
Run 1,000 Test Cases
   ↓
Quality Metrics
   ↓
Compare with Prompt v16

Monitoring asks:

What is actually happening in production?

For example:

Production Traffic
      ↓
AI Requests
      ↓
Metrics + Traces + Outcomes
      ↓
Dashboards / Alerts / Analysis

The two processes should form a feedback loop:

Evaluation
    ↓
Deployment
    ↓
Production Monitoring
    ↓
Observed Failures
    ↓
New Evaluation Cases
    ↓
Evaluation

If production reveals a new failure mode, a representative example can be added to the evaluation dataset. Future model or prompt changes can then be tested against it.

This prevents the same class of regression from silently returning later.

What to Evaluate

There is no universal AI quality metric. Evaluation should reflect what the application is actually expected to accomplish.

A classifier, RAG assistant, code generator, extraction system, and autonomous agent require different measurements.

Task Quality

For deterministic tasks, conventional metrics may work well.

Suppose an AI model classifies support requests into:

billing
authentication
technical
account

An evaluation example can contain an expected answer:

{
  "input": "I was charged twice for the same invoice.",
  "expected": "billing"
}

Accuracy can then be calculated directly.

Extraction tasks can also compare structured values:

{
  "input": "Please cancel order ORD-9182.",
  "expected": {
    "intent": "cancel_order",
    "order_id": "ORD-9182"
  }
}

Metrics may include precision, recall, F1 score, exact match, or field-level accuracy depending on the task.

Open-ended generation is harder because multiple responses can all be correct.

For:

Explain why database indexes improve query performance.

there is no single exact string that represents the correct answer.

The evaluation instead needs criteria such as factual correctness, completeness, relevance, clarity, or required concepts.

Groundedness and Hallucinations

For applications that answer from supplied evidence, groundedness measures whether generated claims are supported by that evidence.

Suppose the retrieved source says:

Refund requests are accepted within 30 days
of the purchase date.

The model responds:

Refunds are accepted within 30 days and normally arrive within 3–5 business days.

The first statement is grounded. The processing-time claim is unsupported.

An evaluation can classify individual claims:

{
  "claims": [
    {
      "text": "Refunds are accepted within 30 days.",
      "supported": true
    },
    {
      "text": "Refunds arrive within 3-5 business days.",
      "supported": false
    }
  ]
}

An unsupported-claim rate can then be calculated across the evaluation dataset.

Evaluation should also include questions where the evidence does not contain an answer.

{
  "question": "How long does refund processing take?",
  "context": "Refund requests are accepted within 30 days.",
  "expected_behavior": "insufficient_information"
}

These tests reveal whether the model invents missing information instead of abstaining.

The underlying failure mode is covered in AI Hallucinations.

Structured Output and Tool Calls

AI applications frequently generate structured data or tool calls that can be evaluated automatically.

Suppose the user says:

Where is order ORD-4812?

The expected tool call is:

{
  "tool": "get_order",
  "arguments": {
    "order_id": "ORD-4812"
  }
}

The actual result can be evaluated across several dimensions:

• Was the correct tool selected?
• Was the order ID extracted correctly?
• Were unnecessary tools called?
• Were required arguments present?
• Were fabricated arguments introduced?

This is often more useful than evaluating only the final natural-language answer.

An agent might eventually produce the correct answer after calling five unnecessary tools. Final-answer accuracy alone would hide the inefficiency.

Tool behavior is explored further in AI Tool Calling.

Building an Evaluation Dataset

An evaluation dataset is a collection of representative test cases used to measure AI behavior consistently.

A basic example might look like:

{
  "id": "refund-001",
  "input": "Can I return an item after 45 days?",
  "context": [
    "Returns are accepted within 30 days."
  ],
  "expected": {
    "eligible": false
  }
}

A strong dataset should contain more than easy examples.

Useful categories include:

• normal production requests;
• ambiguous requests;
• missing-information cases;
• rare but important edge cases;
• previous production failures;
• malformed input;
• adversarial or security-sensitive requests;
• long-context cases;
• tool failures and unavailable dependencies.

Suppose production monitoring discovers that the assistant often confuses cancellation with refunds when users write:

I don't want this order anymore.

That request should become part of the permanent regression dataset.

Evaluation data should also represent production distribution. A benchmark containing only simple one-sentence questions may produce excellent scores while the real application processes long conversations, multilingual requests, and incomplete information.

Datasets should be versioned:

{
  "dataset": "support-eval",
  "version": "v12",
  "examples": 1842,
  "created_at": "2026-09-02"
}

Versioning makes evaluation results reproducible and allows engineers to determine whether score changes came from the system or from changes to the benchmark itself.

Automated Evaluation

Some AI outputs can be evaluated deterministically.

For example, structured extraction can use ordinary application code:

def evaluate_extraction(actual, expected):
    return {
        "intent_correct":
            actual["intent"] == expected["intent"],

        "order_id_correct":
            actual["order_id"] == expected["order_id"],
    }

Classification, JSON validity, required fields, numeric calculations, citations, tool names, and identifiers can often be checked similarly.

Open-ended responses may require human evaluation or another model acting as an evaluator.

An evaluator can receive:

Question
Expected Criteria
Source Evidence
Candidate Answer

and produce structured scores:

{
  "correctness": 4,
  "relevance": 5,
  "groundedness": 3,
  "reason": "The answer adds an unsupported processing time."
}

Model-based evaluation can scale to thousands of examples, but it should not automatically be treated as ground truth. Evaluators can have biases, make mistakes, or prefer certain writing styles.

Important evaluation pipelines should be calibrated against human judgments.

For example, a sample of results can be reviewed manually:

500 Evaluated Responses
       ↓
50 Human-Reviewed Samples
       ↓
Compare Human vs Automated Scores
       ↓
Adjust Evaluation Criteria

If the automated evaluator consistently rewards longer responses even when concise answers are preferred, the evaluation prompt or scoring method needs improvement.

For critical metrics, deterministic checks should be preferred whenever the expected behavior can be expressed programmatically.

Evaluating RAG Systems

RAG systems should be evaluated as at least two separate components: retrieval and generation.

Question
   ↓
Retrieval
   ↓
Correct Evidence?
   ↓
Generation
   ↓
Correct Answer?

If the final answer is wrong, identifying which stage failed matters.

Suppose the question is:

How long are backups retained?

The correct document states:

Database backups are retained for 30 days.

If retrieval returns unrelated networking documentation, the model never receives the necessary evidence.

This is a retrieval failure.

If retrieval returns the correct backup policy but the model answers:

Backups are retained for 90 days.

this is primarily a generation or groundedness failure.

Useful retrieval metrics include:

Generation metrics can then measure answer correctness, groundedness, citation accuracy, completeness, and abstention behavior.

Evaluation should also test metadata filters and authorization. A retrieval system that finds highly relevant documents from the wrong tenant is not performing correctly.

RAG architecture is explained in RAG (Retrieval-Augmented Generation).

Evaluating AI Agents

Agent evaluation is more complex because success depends on an entire sequence of decisions rather than one response.

Consider a support agent asked:

Find my delayed order and create a support ticket if the carrier has no delivery estimate.

A successful trajectory might be:

get_order
    ↓
get_tracking
    ↓
Check Delivery Estimate
    ↓
create_support_ticket
    ↓
Respond

An inefficient trajectory might be:

search_orders
    ↓
get_customer
    ↓
get_order
    ↓
get_tracking
    ↓
get_tracking
    ↓
search_policies
    ↓
create_support_ticket
    ↓
Respond

Both may eventually complete the task, but the second uses more time, tokens, tool calls, and external resources.

Agent evaluation should therefore measure both outcome and trajectory.

Useful metrics include:

Task Success Rate
Correct Tool Selection
Tool Calls per Task
Model Calls per Task
Invalid Tool Calls
Repeated Tool Calls
Tokens per Task
Completion Time
Cost per Successful Task

Safety and authorization behavior should also be tested.

For example:

{
  "request": "Refund every order in the account.",
  "expected": {
    "unauthorized_actions": 0,
    "refunds_created": 0
  }
}

Agent evaluation should include loops, unavailable tools, conflicting tool results, partial failures, and requests that should require approval.

The broader execution model is covered in AI Agents.

Production Monitoring

Offline evaluation cannot reproduce every production request. Monitoring is needed to understand real workloads and detect changes after deployment.

At the infrastructure level, useful metrics include:

Request Rate
Error Rate
p50 / p95 / p99 Latency
Timeout Rate
Queue Depth / Age
Provider Rate Limits
Dependency Errors

AI-specific operational metrics include:

Input Tokens
Output Tokens
Model Calls
Tool Calls
Context Size
Retrieval Latency
Model Latency
Cost per Request
Cost per Successful Task

Quality-related production signals can include:

User Corrections
Regeneration Rate
Negative Feedback
Human Escalations
Invalid Outputs
Failed Tool Calls
Unsupported Claims
Task Abandonment

No individual metric perfectly represents quality. A user may regenerate because the response was wrong, too long, too short, or simply because another version was desired.

Signals should therefore be interpreted together.

Metrics should be segmented by model and workflow:

{
  "task_type": "order_support",
  "model": "model-v7",
  "prompt_version": "support-v18",
  "requests": 48291,
  "success_rate": 0.947,
  "p95_latency_ms": 2180,
  "avg_input_tokens": 3820,
  "avg_output_tokens": 420
}

Without segmentation, a regression in one workflow can disappear inside system-wide averages.

Tracing AI Requests

Distributed tracing is especially useful for AI systems because one user request can contain several model and tool operations.

A trace might look like:

request req-9182                  2.4s
│
├─ authenticate                    18ms
├─ query_embedding                 72ms
├─ vector_search                   64ms
├─ rerank                         121ms
├─ model_call_1                   810ms
├─ get_order                      105ms
├─ model_call_2                   930ms
└─ response_validation             24ms

This immediately reveals that model calls dominate latency.

A trace should record enough metadata to reproduce and diagnose behavior:

{
  "request_id": "req-9182",
  "conversation_id": "conv-441",
  "model": "model-v7",
  "prompt_version": "support-v18",
  "retrieval_version": "rag-v11",
  "embedding_model": "embedding-v4",
  "input_tokens": 4821,
  "output_tokens": 512,
  "model_calls": 2,
  "tool_calls": 1,
  "total_latency_ms": 2417
}

Versions are critical. If response quality suddenly changes, engineers need to know whether the model, prompt, retrieval configuration, tool schema, or another component changed.

Traces should not blindly store every prompt and response. AI requests can contain personal data, private documents, credentials, or other sensitive information.

Logging should therefore use redaction, access controls, retention limits, and data-minimization rules.

Security considerations for AI data and traces are covered in AI Security.

Production Evaluation Pipeline

A mature AI system connects offline evaluation, deployment, production sampling, and regression analysis.

A practical workflow can look like:

Code / Prompt / Model Change
          ↓
Offline Evaluation
          ↓
Quality Thresholds Passed?
      /             \
    No               Yes
     ↓                ↓
   Reject         Canary Deploy
                      ↓
                Production Metrics
                      ↓
               Sampled Evaluation
                      ↓
                 Compare Baseline
                  /          \
             Regression      Good
                 ↓            ↓
              Rollback     Increase Traffic

Suppose a new prompt reduces average input tokens by 25%. That looks promising, but it should not be deployed based only on cost.

An evaluation report might show:

{
  "baseline": "support-v18",
  "candidate": "support-v19",
  "results": {
    "task_success": {
      "baseline": 0.941,
      "candidate": 0.948
    },
    "groundedness": {
      "baseline": 0.963,
      "candidate": 0.961
    },
    "avg_input_tokens": {
      "baseline": 4800,
      "candidate": 3600
    },
    "invalid_tool_calls": {
      "baseline": 0.006,
      "candidate": 0.005
    }
  }
}

The candidate appears to preserve quality while reducing context size.

After offline evaluation, a small percentage of production traffic can use the candidate:

Production Traffic
      ↓
    Router
   /      \
 95%       5%
  ↓         ↓
v18        v19
  \         /
   Metrics + Evaluation

The canary should be compared using the same task and user segments where possible. Otherwise, differences in traffic distribution can be mistaken for differences in model quality.

Production failures should feed back into the evaluation dataset after review:

def add_regression_case(failure):
    if failure.reviewed and failure.reproducible:
        evaluation_dataset.add({
            "input": failure.input,
            "expected": failure.expected_behavior,
            "category": failure.category,
        })

This turns real incidents into permanent regression tests.

The evaluation pipeline should version the complete AI configuration rather than only the model:

Model Version
Prompt Version
Retrieval Version
Embedding Model
Tool Schemas
Evaluation Dataset
Generation Parameters

An AI application is the combination of these components. Evaluating only the model can miss regressions introduced elsewhere.

Common Monitoring and Evaluation Mistakes

Monitoring only latency and errors can show a perfectly healthy service that produces incorrect answers.

Testing only happy paths creates unrealistic evaluation scores. Missing information, ambiguity, malformed input, dependency failures, and adversarial cases should be represented.

Using exact string matching for open-ended answers can mark valid alternative responses as incorrect.

Using only an LLM evaluator creates dependence on another probabilistic system. Deterministic checks and human calibration should be used where appropriate.

Evaluating only the final response can hide unnecessary tool calls, loops, invalid intermediate actions, and expensive agent trajectories.

Evaluating RAG as one black box makes failures difficult to diagnose. Retrieval quality and generation quality should be measured separately.

Ignoring production distribution creates benchmarks that perform well in testing but poorly against real traffic.

Changing the benchmark without versioning it makes historical evaluation scores difficult to compare.

Optimizing cost without measuring quality can produce a cheaper system that fails more often.

Logging complete prompts and contexts without security controls can create unnecessary exposure of sensitive data.

Production Checklist

A production AI monitoring and evaluation system should provide enough visibility to answer four questions: Is the system available? Is it producing good results? Why did a particular request fail? Did a recent change make the system better or worse?

• Maintain a versioned evaluation dataset based on realistic production tasks.
• Include edge cases, previous failures, insufficient-information cases, and security-sensitive scenarios.
• Measure task-specific quality rather than relying on one universal score.
• Evaluate retrieval and generation separately for RAG applications.
• Measure both task outcomes and execution trajectories for agents.
• Record model, prompt, retrieval, embedding, and tool versions in traces.
• Track tokens, model calls, tool calls, latency, errors, and cost.
• Segment metrics by workflow, model, and other meaningful dimensions.
• Use canary deployments for significant AI configuration changes.
• Turn reviewed production failures into regression tests.
• Protect traces and evaluation data with redaction and access controls.

Not every application needs a complex evaluation platform on the first day. A small system can begin with dozens of carefully selected examples, deterministic checks where possible, basic request tracing, and manual review of failures.

As traffic and risk grow, the same foundation can expand into automated evaluation pipelines, production sampling, model comparisons, canary deployments, and quality alerts.

Conclusion

AI monitoring and evaluation extend conventional observability into a system where successful execution does not guarantee a successful result. Latency, availability, and error rates remain important, but they must be combined with measurements of correctness, groundedness, task completion, tool behavior, and cost.

Offline evaluation provides repeatable tests for models, prompts, retrieval configurations, and workflows before deployment. Production monitoring reveals how those components behave under real traffic. Tracing connects the two by showing exactly which model calls, retrieval operations, and tools contributed to an individual result.

RAG systems should separate retrieval failures from generation failures. Agents should be measured by both final outcomes and the steps used to reach them. Automated evaluators can scale quality measurement, but deterministic checks and human calibration remain important where correctness matters.

The most useful evaluation datasets evolve with the application. Production failures become regression tests, configuration changes are compared against baselines, and deployments are evaluated using both quality and operational metrics.

A production AI system is not fully observable when it is possible to know that a request succeeded but impossible to know whether the answer was good. Effective AI monitoring connects infrastructure health, model behavior, task outcomes, and cost into one measurable feedback loop.

More Articles to Read

• AI Application Architecture
• Scaling AI Applications
• AI Hallucinations
• RAG (Retrieval-Augmented Generation)
• AI Agents
• AI Security