LLM monitoring is the ongoing measurement of a large language model (LLM) application or AI agent in production: its answer quality, groundedness, safety, fairness, escalation behavior, cost and latency. Teams sample live conversations, score them with automated graders, send the uncertain and risky ones to human reviewers, alert a named owner when a threshold is crossed, and turn each confirmed failure into a regression test.

A loan assistant passes every pre-launch test. Three weeks later, a reviewer finds it telling applicants "you're approved" before underwriting has decided. Nothing in the infrastructure broke. Latency was fine and error rates were flat. The behavior drifted, and only quality monitoring would have caught it.

This guide is for AI product owners, platform engineers and the risk and compliance reviewers who share responsibility for a customer-facing LLM or agent after launch. It covers what to watch, how to sample and grade, how to set thresholds and owners, when a change needs re-approval, and which records to keep. It builds on our LLM evaluation guide and does not repeat it.

Key takeaways
  • Monitor quality, groundedness, safety, fairness, escalations, cost and latency. Infrastructure metrics alone miss behavior drift.
  • Sample deliberately: a fixed random share for trend lines, plus every conversation that carries a risk signal such as a complaint, a thumbs-down or a sensitive topic.
  • Automated graders score at volume; human annotation queues decide the uncertain and high-impact cases and keep the graders honest.
  • Every alert needs a threshold, a window, a severity and a named owner, or it becomes noise.
  • Confirmed failures become regression tests, and a model, prompt, retrieval or tool change triggers a rerun and, when material, re-approval.

What is LLM monitoring, and how is it different from evaluation?

Evaluation tests a versioned application against a fixed set of cases before release. Monitoring measures what the released application actually does with real users. Evaluation answers "is this version safe to ship?" Monitoring answers "is the live version still behaving the way we approved, and what are users bringing that our tests missed?"

Monitoring is also different from observability in the engineering sense. Observability shows what happened inside a request: each LLM call, retrieval, tool call and error. Monitoring adds judgment: was the answer correct, grounded, safe and fair, and should someone act? For an agent, that judgment covers the trajectory (which tools it called and in what order), not only the final message. Our post on evaluating AI agents explains trajectory checks.

What should you monitor in a production LLM or AI agent?

Start from how the application can fail, then attach a measurable signal and an owner to each failure. The table below is a starting set for a customer-facing assistant in a regulated business.

SignalWhat it catchesHow it is usually measuredTypical owner
Answer quality driftAnswers that are less relevant, less complete or more confusing than at approvalGrader score trends against the approved baseline; business review of samplesBusiness owner
GroundednessClaims not supported by the retrieved policy, product terms or recordFaithfulness or hallucination grader; spot checks against source documentsBusiness owner and AI team
SafetyToxic output, jailbreaks, prompt injection, PII or PHI exposureSafety graders, PII detection, guardrail trigger ratesRisk or security
FairnessQuality or outcomes that differ across customer groupsSegment-level comparison of grader scores, refusals and escalations where permitted data or approved proxies existCompliance or fair-lending
EscalationsMissed handoffs to a human, or a sudden rise in handoffsEscalation rate; graders that flag cases that should have escalated; user frustration signalsOperations lead
Agent behaviorWrong tool, repeated tool calls, loops, empty outputsTool-call correctness grader; repeated-call and empty-output flagsAI team
Cost and latencySlow answers, runaway token use, cost outliersp50 to p99 latency, tokens and cost per conversationAI team or platform owner
Errors and data gapsFailures, silent failures, missing instrumentationError rate; spans with no model or token dataPlatform owner

Two notes. First, groundedness and correctness differ: an answer can be faithful to an outdated source and still be wrong, which is why hallucination detection needs both a source check and periodic ground-truth review. Second, fairness monitoring for LLMs is harder than for scoring models because protected attributes are rarely in the conversation. Agree with compliance on what data or proxies may be used before you build the report.

How should you sample live conversations?

Grading every conversation with an LLM judge is often affordable for low volumes and wasteful for high ones. Human review of every conversation is never affordable. A practical design uses three layers:

  1. A fixed random sample for trends. Grade a steady percentage of traffic so score trends are comparable week to week. Make the sampling deterministic, so the same conversation is always in or out and reruns give the same set.
  2. Targeted capture for risk. Always include conversations with a thumbs-down, a complaint keyword, an escalation, a guardrail trigger, a sensitive topic (hardship, disputes, clinical symptoms) or a high-value action.
  3. Filtering before human review. Remove duplicates and noise, and keep variety so reviewers do not read fifty versions of the same balance inquiry.

Mask personal data before it is stored, and decide retention up front. Sampling and masking choices are themselves evidence: a reviewer will ask how cases were selected and whether sensitive content was protected.

How do automated graders and human review work together?

Automated graders give coverage. Humans give judgment and calibration. Neither works well alone.

  • Graders score sampled traffic on defined labels, such as grounded or not grounded, escalated correctly or missed. Rubrics matter more than model choice; our LLM-as-a-judge guide covers rubric design, bias and calibration.
  • Annotation queues route conversations to the right reviewer: business experts for answer quality and policy, risk or compliance for safety and fairness, clinicians for clinical content. Reviewers see the full conversation and the grader's verdict, and record their own.
  • Calibration compares grader verdicts with human labels on a shared sample. Each disagreement is a reason to fix the rubric, the grader, or the reviewer guidance. Track agreement over time; a grader that drifts away from human labels should not be trusted for alerting.

Send to humans what graders are least sure about, what graders flag as failing, and what carries the most consequence. Everything else can rely on grader scores and trends.

How do you set alert thresholds and owners?

An alert without an owner is a log line. For each monitor, write down six things:

  1. The measure: for example, the share of sampled answers labeled not grounded.
  2. The threshold and direction: above or below what value.
  3. The window: the period the measure is computed over, short for safety, longer for slow quality drift.
  4. The persistence rule: how many consecutive breaching checks before an alert, and a minimum number of conversations so a quiet hour does not fire a false alarm.
  5. The severity: what is a page, what is a ticket, what is a weekly review item.
  6. The owner and the action: who is notified, and what they do first.

Example (fictional): a credit union's member-service assistant is approved with a groundedness rate set by the business owner and risk reviewer during evaluation. The monitor alerts the business owner if the not-grounded share stays above the agreed limit for two consecutive daily checks with at least a minimum number of graded conversations. A PII exposure monitor uses a one-hour window, a single-breach rule and critical severity, and alerts the security lead and compliance. Each alert opens a finding that records the evidence, the owner, the decision and the resolution.

Set initial thresholds from the evaluation baseline the version was approved on, then adjust them only through the same review that set them, and record the change.

How do production failures become regression tests?

Monitoring earns its cost when it improves the next release. The loop is short:

  1. A grader or reviewer flags a conversation.
  2. A qualified reviewer confirms the failure and writes the expected behavior.
  3. The conversation, with personal data removed, becomes a test case with that expected behavior in the versioned suite.
  4. The fix is tested against the full suite, including the new case, before it ships.

In the loan example above, "Has my loan been approved?" becomes a saved test with the expected behavior "state that underwriting has not decided and give the next step." Every later version must pass it. Over time, the regression suite reflects what real users actually ask, which is the gap our LLM evaluation guide warns about.

When does a change require re-approval?

Any change to the model, system prompt, retrieval sources, tools, guardrails or orchestration can change behavior. Agree in advance which changes trigger what:

  • Always rerun the saved suite after any model, prompt, retrieval or tool change, including a vendor's silent model update if you can detect it.
  • Require re-approval when a critical case fails, when scores on a risk category move beyond the agreed tolerance, when the agent gains a new tool or action, or when the use case or audience changes.
  • Allow a lighter review for changes that pass every critical case and stay within tolerance, with the results attached to the record.

The approval should point to an exact version, so anyone can see which configuration is live and which evidence approved it. Monitoring history should continue across the change, so the new version is compared against the old baseline rather than starting fresh.

What monitoring evidence do examiners expect?

Requirements differ by regulator and use case, and much generative AI guidance is still forming. Our posts on SR 26-2 and generative AI, the NAIC AI Model Bulletin and the NIST AI RMF for generative AI cover the specifics. The NIST AI Risk Management Framework includes post-deployment monitoring plans in its MANAGE function, and the Generative AI Profile applies that to generative systems. In practice, examiners and internal auditors tend to ask for:

  • A written monitoring plan: signals, sampling method, thresholds, windows and owners, approved before launch.
  • Evidence the plan ran: grader results, review records and alert history over the period.
  • Issue records: each finding with severity, evidence, owner, decision, resolution and timestamps.
  • Change records: what changed, which tests were rerun, the results, and who approved the new version.
  • Calibration records showing automated graders were checked against human judgment.
  • Data handling: how personal data was masked, who could see conversations, and how long they were kept.

None of this proves an application is safe. It shows the institution knows how the application is behaving and acts on what it finds. For the wider review, see model risk management for generative AI.

How GenGuardX handles production monitoring

GenGuardX, from Corridor Platforms, is the GenAI and AI agent governance and evaluation platform for regulated enterprises. Monitoring is one part of a record that starts with business testing and risk approval. Per the GenGuardX documentation:

  • Live conversation capture. Traces of LLM calls, tool calls and retrievals arrive through the Python SDK, framework integrations such as LangChain, LangGraph, LlamaIndex, CrewAI and the OpenAI Agents SDK, or any OpenTelemetry exporter. Each trace is tied to its deployment and agent version. Data masking hides sensitive content before storage.
  • Automated AI graders. LLM-judge evaluators score a deterministic sample of traffic, with templates for hallucination, user frustration, session coherence, safety guardrail, tool-call correctness and agent trajectory. Each verdict carries a label, an optional score and an explanation.
  • Annotation queues. Business and risk reviewers label production conversations, and their reviews become approved answers and new test cases.
  • Monitors and alerts. Monitors cover traffic, errors, latency percentiles, cost, tokens, agent flags and grader scores, with a window, consecutive-breach rule, minimum trace count and severity. Alerts go to named people in GenGuardX, by email or to Slack, and never include conversation content.
  • Findings. GenGuardX checks traces every 15 minutes for PII leaks, error spikes, silent failures, retry loops, cost outliers, evaluator regressions and monitor alerts. Each finding is tied to the deployment, agent version and model, with status, severity, evidence and a full change history, and can be added to the version's risk assessment so approvers see it.
  • Re-evaluation. When a model, prompt or tool changes, saved tests are rerun and the approval workflow reopens only what changed.

Production monitoring and alerts are part of the Scale tier; live conversation capture and automated graders start at the Approve tier. GenGuardX is SOC 2 Type II and runs on AWS, Microsoft Azure and Google Cloud. If you are comparing tools, our GenGuardX vs Fiddler AI comparison covers where an engineering observability platform fits alongside it. Industry pages show the risks monitored for banks, credit unions, insurers and payers and healthcare providers.

FAQ

What should you monitor in a production LLM?

Answer quality against the approved baseline, groundedness to sources, safety (toxicity, jailbreaks, PII), fairness across customer groups, escalation behavior, agent tool use, errors, cost and latency. Each signal needs a threshold and an owner.

How is LLM monitoring different from traditional ML model monitoring?

Traditional monitoring compares input and prediction distributions and accuracy against labels. LLM outputs are free text, so quality has to be judged by graders and people, and agents add tool calls and multi-step trajectories to check.

How many production conversations should humans review?

There is no universal number. Review every conversation a grader flags as a critical failure, a risk-targeted sample, and enough of the random sample to keep graders calibrated. Size the sample with your risk reviewer and revisit it as agreement data builds up.

Can an LLM judge replace human reviewers in monitoring?

No. Judges give coverage at volume, but they have known biases and drift. Human labels on a shared sample are how you know whether to trust the judge.

Who should own LLM monitoring alerts in a bank or credit union?

Split ownership by signal: the business owner for quality and groundedness, risk or compliance for safety and fairness, operations for escalations, and the AI or platform team for errors, cost and latency. Record the owner on each monitor.

What is the best tool for AI monitoring in regulated industries?

It depends on whether the gap is engineering visibility or governance evidence. Engineering observability tools trace and debug at depth. Governance platforms such as GenGuardX tie monitoring to business review, findings and version-tied approval. Many regulated teams use one of each.

Resources

Internal resources

External resources