Consider an appointment agent that is answering calls and responding on time. A change to the booking system starts causing updates to fail. Some callers leave the conversation believing their appointment has moved, while the original booking remains in place.
The hosting dashboard can remain healthy throughout. The operating team still needs to find the failed requests and help the affected callers.
Managed Agent Operations is the ongoing work of keeping a deployed agent reliable, useful and accountable as its models, knowledge and connected systems change. It brings monitoring, evaluation, incident handling, releases and cost management into an owned service.
The appointment workflow in this article is an illustrative operating example.
Give the live service a clear owner
Start with a service description that people can use during an ordinary working day and during an incident. Record the agent's purpose, supported channels, permitted actions, business dependencies and escalation routes. Name the person accountable for the service and the people who can change its behaviour or pause its actions.
Ownership also extends to the information it uses. Someone needs to approve changes to service policies, maintain the knowledge sources and decide how exceptions should be handled.
A managed platform provides part of the operating environment. Responsibility for configuration, data access and agent behaviour depends on the deployment model. Microsoft's agent shared responsibility guidance makes that division explicit across hosted software, managed platforms and self-managed infrastructure. [1]
Document the actual support arrangement: service hours, human escalation coverage, incident contacts and authority to approve changes. Record any contractual service-level agreement and the commitments it covers. If an agent can accept requests outside staffed hours, define where those requests wait and what the caller is told.
The service team can then route a policy question, integration failure or urgent incident to the person with authority to act.
Measure completed work as well as availability
Agree what a successful request looks like before choosing dashboard metrics. For a booking change, success should include the correct authorised record being updated and the result being communicated accurately. For an information request, it might mean a supported answer based on an approved, current source.
A service-level indicator, or SLI, measures an aspect of the service. A service-level objective, or SLO, sets the target for that measure over a defined period. Google's SRE guidance recommends measures that reflect the experience of the service's users. [2]
For the appointment workflow, a practical operating view could include:
| Area | Signal to review | Question it should answer |
|---|---|---|
| Business outcome | Booking changes verified in the business system | Did the requested work finish correctly? |
| Conversation quality | Reviewed answers, policy adherence and repeat contacts | Was the caller given accurate, useful guidance? |
| Performance | Response delay, slow requests and failed tool calls | Where are callers waiting or losing service? |
| Human handover | Escalation reasons, delivery to the receiving team and unresolved cases | Can someone take responsibility when automation stops? |
| Cost | Spend and resource use by workflow and outcome | What does delivering the service actually cost? |
Define the population behind each rate. For example, an automated-completion measure needs a documented set of eligible requests, a verification rule and a treatment for unfinished work. Establish eligibility rules in advance and retain failed or unresolved eligible requests in the denominator. Track correct refusals and planned human handovers separately so the completion measure remains interpretable.
Read the measures together. A falling handover rate needs investigation alongside resolution quality, repeat contacts and customer feedback. Set targets from business impact and pilot evidence, then review them as the service changes.
Make each request traceable
When a request fails, the team needs to follow it across the conversation, model, retrieval service and business tools. Connect those steps with a trace or interaction identifier, and record the relevant agent configuration, tool result, timing and final outcome. Label any trace sampling so that recorded traces can be distinguished from the full request population.
Microsoft's observability guidance combines evaluation, monitoring and distributed tracing, including integration with Azure Monitor Application Insights. These capabilities can help an Azure-based team investigate the path through a request, subject to the services and instrumentation actually configured. [3]
Voice adds its own investigation points. Separate time to first audio, speech recognition, model processing, tool execution and speech synthesis where the pipeline exposes them. Define the start and end of each timing measure. Microsoft's voice-agent guidance illustrates why these stages need distinct views. [4]
For the failing booking update, an operator should be able to use the trace to find the rejected operation, identify the affected version and check whether a handover was created.
Keep telemetry deliberate. Collect the metadata needed for operations, and apply appropriate access and retention controls to any captured conversation content. OpenTelemetry's sensitive-data guidance recommends minimising collection and reviewing what instrumentation libraries emit. [5]
Review a real trace before launch and after instrumentation changes. Confirm that it is useful for diagnosis and that secrets, unnecessary personal information and unrestricted tool responses are not being copied into the monitoring system.
Keep quality checks close to real use
Maintain a versioned evaluation set that reflects the work the agent handles. Include routine requests, missing information, policy exceptions, unsupported requests and failures in connected systems. For voice, add supported languages, noisy audio and ambiguous spoken dates.
Choose the check to match the task. A booking operation can be checked against the resulting business record. Answer accuracy and conversational clarity may need a written rubric and human review. Anthropic's agent-evaluation guidance distinguishes outcomes from an agent's account of what happened, and describes code-based, model-based and human grading. [6]
Use automated scoring to extend coverage, while calibrating model-based judges against human assessments. A fluent explanation or a high evaluator score still needs the relevant evidence for the business outcome.
Run the evaluation set before material changes and use an appropriate sample of production interactions for continuing review. Show the review volume and sampling method beside quality rates. Record the evaluator version and criteria so a scoring change can be distinguished from a change in the agent. Microsoft distinguishes production sampling from scheduled evaluation against test datasets. [3]
Investigate results by request type, language, channel and agent version. An overall score can hide a recurring problem in one workflow. Trace a failed answer back to the knowledge source, permission decision and tool response before deciding what to change. Add reviewed production failures to the test set, using synthetic or appropriately protected examples, and retain established regression cases when expanding coverage.
The booking-system failure should eventually become a repeatable test: the operation is rejected, the agent communicates the limitation accurately and the request follows the agreed recovery path.
Treat changes to behaviour as releases
A production agent can change even when its application code does not. A new prompt, tool definition, model deployment, permission or knowledge source can alter the result a customer receives. NCSC's operations guidance calls for testing and update processes that account for changes to data, models and prompts. [7]
Keep a release record that identifies the configuration, supporting knowledge, relevant dependencies, evaluation results and approving owner. Record model versions where the provider exposes them, and track scheduled provider changes where available. For multi-turn conversations, define whether active sessions stay on their original configuration and test any migration of stored state.
Use a controlled rollout when the architecture supports it. A canary release exposes a limited part of the service to a change and evaluates the result before wider rollout. Google's SRE guidance also stresses representative traffic and appropriate evaluation metrics. [8]
For an agent, compare business outcomes, quality, latency and cost, as well as technical errors. Agree the conditions for promotion, pausing or rollback before the rollout starts. With low traffic, allow enough representative use to gather evidence; a quiet monitoring period alone gives little assurance.
Check the recovery path as part of the release. Returning to an earlier configuration may require compatible tools, knowledge and stored state. An old prompt cannot restore a removed API. Any customer records already changed need their own reconciliation or correction process.
Make alerts actionable and incidents recoverable
An alert should identify a problem that needs a defined response. Give it an owner, severity, supporting evidence and a runbook. Keep longer-term quality trends in the review process, with urgent notifications for conditions that need immediate action.
Google's SRE alerting guidance connects alerts to significant events and the risk to service objectives. That principle helps keep the operating team focused on customer impact. [9]
For an agent, triggers might include repeated failures in a critical tool, a broken handover path, suspicious access attempts or a sustained fall in verified outcomes. Choose thresholds and response times for the actual service and risk.
During the illustrative booking incident, the first decision is whether to pause booking changes while keeping any unaffected functions available. Route affected callers to the agreed alternative, and check work already submitted. Google describes incident response as a combination of restoring service and coordinating people, communications and records. [10]
Keep a record of the affected versions, time window, actions taken and customers or transactions needing follow-up. If a write timed out, establish its actual status before replaying it: Microsoft's retry guidance explains how a completed operation with a lost response can lead to duplicate effects. [11]
Test recovery with a controlled request before restoring wider access. The follow-up should produce an assigned corrective action, an improved alert or a new regression test. Use incident exercises to check who can pause actions and whether the alternative service route really works.
Control cost at the workflow level
Bring the full operating cost into view: models, speech processing, telephony, retrieval, hosting, monitoring, evaluation and the people needed to support the service. Microsoft's AI architecture guidance asks teams to consider direct and indirect costs alongside the other requirements of the workload. [12]
A useful service measure is cost per verified outcome for a defined workflow and period. Document which costs are allocated and which outcomes count. Include the cost of failed and repeated attempts; show whether human-assisted resolutions are included, and keep the scope consistent when comparing periods.
For the appointment agent, an increase in spend might follow extra retries, growing conversation context, a change in call mix or more expensive processing. Investigate the associated traces and outcomes before choosing an optimisation. Evaluate any smaller model, shorter retrieval result or altered routing policy against the same quality and performance criteria before release.
Use billing budgets and alerts alongside runtime controls. Azure Cost Management budgets can notify teams about spending thresholds, but do not themselves stop consumption. [13]
Set appropriate limits on request size, execution steps, retries and concurrent work in the application. Define what happens when a limit is reached, including how the user receives a useful next step. Account for the delay between activity and billing data when choosing controls.
Keep a practical operating routine
Start with a review rhythm that matches the service's volume, risk and support arrangement. The following is a proposed starting point to adapt with the service owner:
| When | Review | Expected output |
|---|---|---|
| During supported operations | Service health, critical failures, handover queues and spend anomalies | A response to active issues and a clear record of unresolved work. |
| Regular quality review | Sampled interactions, failed outcomes, feedback and repeated problems | Prioritised fixes and updates to the evaluation set. |
| Before a material release | Regression results, access changes, rollout and recovery readiness | An explicit release decision and named approver. |
| Periodic service review | Objectives, costs, dependencies, knowledge freshness, access and recovery exercises | Decisions about scope, investment and service improvements. |
Bring evidence to these reviews. A useful service report explains what changed, who was affected, what was corrected and which decision is needed next. Include unresolved quality issues and limitations alongside successful outcomes.
For the appointment agent, the work is complete when the booking problem is contained, affected requests are reconciled and the next release is protected by a meaningful test. The service owner can see what happened and what has changed as a result.
Putting a voice or chat agent into everyday service? Explore Managed Agent Operations or discuss your use case with RZK.
Sources
Primary documentation consulted on 20 October 2026. The appointment example, operating measures and review routine are illustrative recommendations.
- Microsoft — AI agent shared responsibility model
- Google SRE — Implementing SLOs
- Microsoft — Observability in generative AI
- Microsoft — Voice agent tracing, monitoring, and evaluation
- OpenTelemetry — Handling sensitive data
- Anthropic — Demystifying evals for AI agents
- NCSC — Secure operation and maintenance
- Google SRE — Canarying releases
- Google SRE — Alerting on SLOs
- Google SRE — Incident response
- Microsoft — Retry pattern
- Microsoft — Design principles for AI workloads on Azure
- Microsoft — Create and manage budgets