The fact that the system responds does not mean it is still working well
A traditional application usually fails visibly. An AI system can continue delivering results while losing accuracy, using different data, increasing its cost or harming a segment. Monitoring must observe both the infrastructure and the quality of decisions.
The dashboard depends on the case, but it always needs owners, thresholds and actions. An alert without someone who can interpret and act on it is just noise.
Do not monitor only whether AI is available. Monitor whether it remains useful, safe and economically defensible.
The six layers of observability
- Service: availability, latency, errors, queues and dependencies.
- Data: volume, completeness, validity, freshness and distribution changes.
- Model or instructions: accuracy, quality, stability, version and rejection rate.
- Business: incremental result, savings, conversion, resolution or risk avoided.
- People: adoption, corrections, escalations and satisfaction.
- Risk: bias, privacy, security, harmful content and incidents.
Add cost per operation, per active user and per result achieved. For generative AI it is useful to record tokens, retries, tool use, caching and human review. The technical bill does not represent the total cost.
What drift is
Drift is a change that reduces the validity of the system. It can affect input data, the relationship between variables and outcome or user behaviour. For example, a new campaign changes the profile of leads and degrades a scoring model even though the code has not changed.
Detecting it requires comparing time windows, segments and the baseline. Not every change requires retraining: first identify the cause and check whether the business objective remains the same.
Evaluate by sample and by segment
Create a set of reference cases with expected results and run it after changes to the model, prompt, data or provider. Combine automated metrics with human review, because fluency and accuracy are not equivalent.
Break results down by channel, customer type, language, complexity and affected group. A stable average can hide deterioration in infrequent but critical cases. Also record false positives and false negatives according to their real cost.
Design actionable alerts
- Indicator and exact source.
- Warning threshold and critical threshold.
- Time window to avoid reacting to noise.
- Responsible person and backup.
- Immediate action: limit, review, roll back the version or disconnect.
- Evidence that allows the alert to be closed.
Use several severity levels. A moderate rise in latency can wait; a data leak or harmful decision requires activating the incident response plan.
Control changes and versions
Record the model, instructions, RAG sources, parameters, rules, data and deployment date. Link each result to its configuration. Without versioning it is impossible to investigate why behaviour changed or reproduce a decision.
Roll out progressively and compare the new version with the previous one. Keep a rollback path. Apparently minor changes to prompts or documents can alter results in areas you did not anticipate.
Operational review and business review
The operations team can review alerts weekly; the business should assess value, adoption and cost monthly; the governing body should review risks, providers and continuity quarterly. Adjust the frequency to the system’s impact.
Keep the documentation in the AI technical documentation: metrics, versions, incidents, evaluations and decisions. Traceability improves operations and makes it easier to demonstrate control.
When to correct, retrain or retire
Correct the integration when the problem lies in the data or flow. Retrain or change instructions when the learned relationship no longer represents reality. Retire the system when it no longer adds value, the risk cannot be mitigated or a simpler alternative exists. Keeping it out of inertia also has a cost.
Example of a minimum dashboard
- Service: availability, p95 latency and errors.
- Data: valid records, critical fields and delay.
- Quality: acceptance, correction and critical failures.
- Business: result versus baseline.
- Cost: cost per useful task and per client.
- Risk: incidents, complaints and differences by segment.
For each indicator document the definition, formula, source, frequency, threshold and owner. Do not change a definition without versioning it. If “correct answer” depends on human evaluation, create a guide and check agreement between reviewers.
Frequently asked questions about monitoring
How many indicators are needed?
Start with the minimum that detects loss of service, quality, value and control. Add metrics when a decision needs them; a huge dashboard can hide what is critical.
Does every degradation require retraining?
No. It may be caused by integration, data, instructions, user behaviour or a change in objective. Investigate the cause before modifying the model.
How long should records be kept?
It depends on the risk, obligations and need for investigation. Define retention, access and minimisation; recording everything indefinitely increases exposure and cost.
How we are monitoring the quality and value of I3OS
In I3OS we do not consider a result finished when the model responds. We are observing execution time, corrections, human interventions, quality and effect on the client to understand which capabilities can be expanded and which need adjustments. This discipline helps us avoid confusing generated activity with created value and keep autonomy proportional to the evidence available.
If you need to deploy observability and controls for your AI systems, Impulsa3 can help you define metrics, alerts and continuous-improvement processes.
