Human oversight in AI systems: what Human-in-the-Loop is and when it is mandatory

The AI Act requires human oversight for all high-risk AI systems. But putting a person in front of a screen is not oversight. We explain the three HITL models, when each applies, how to avoid “automation bias” and how to design an oversight system that really works

Human oversight in artificial intelligence systems sounds intuitive: a person reviews what the machine does and corrects it when it makes a mistake. In practice, it is one of the most poorly implemented concepts in the industry.

Many companies think complying with human oversight means putting an employee in front of an AI system dashboard. But if that employee does not understand how the model works, has no authority to stop it, cannot interpret the metrics on screen, and is processing 200 decisions per hour, the “oversight” is an illusion. The human becomes a rubber stamp: approving whatever the machine says because they have neither the time nor the ability to challenge it.

The AI Act knows this, which is why it does not simply ask for a human to be involved. It requires that person to have competence, authority, and tools to supervise effectively. And for high-risk systems (Annex III), this oversight is not optional: it is a legal obligation with direct implications for accountability.

In this article, we explain what human oversight in AI really means, the three HITL models that exist, when it is mandatory, how to avoid the most common mistakes, and how to design a system that complies with the regulations and works operationally.

The three models of human oversight

Human oversight is not always the same. There are three models with different levels of intervention:

Human-in-the-Loop (HITL)

The human is inside the decision loop. AI generates a recommendation and the human approves, modifies, or rejects it before it is executed. It is the most robust model: no decision is made without human validation.

Example: an AI system classifies CVs and generates a candidate ranking, but a human recruiter reviews the ranking and makes the final decision about whom to invite to an interview.

Human-on-the-Loop (HOTL)

The human supervises the system but does not validate every individual decision. They monitor aggregate metrics and can intervene when they detect anomalies or deviations. The system operates autonomously within defined parameters.

Example: an anti-fraud system automatically blocks transactions with a score >80, and an analyst reviews false-positive metrics daily and adjusts the thresholds when necessary.

Human-in-Command (HIC)

The human has final authority over the system but does not supervise its daily operation. They define the rules, limits, and objectives, and can stop the system at any time. This is the right model for low-risk systems with a high degree of automation.

Example: the ecommerce director defines the rules for the recommendation engine (which products to promote, minimum margins) and has a kill switch to deactivate it if it produces unsuitable results.

General rule: The greater the impact of a decision on people, the higher the required level of oversight. Systems for personnel selection, credit scoring, or medical diagnosis require HITL. Product recommendation systems or email classification can operate with HOTL or HIC.

When human oversight is mandatory under the AI Act

The AI Act establishes human oversight as a mandatory requirement for all AI systems classified as high risk (Annex III). This includes:

• Employment and personnel selection: CV screening, candidate evaluation, hiring, or dismissal decisions.
• Access to essential services: Credit scoring, insurance risk assessment, and allocation of public benefits.
• Law enforcement: Surveillance systems, criminal risk assessment, and fraud detection with an impact on rights.
• Education: Systems that assess or classify students.
• Critical infrastructure: Management of electricity, water, gas, and transport networks.

For these systems, the AI Act requires human oversight to include:

1. Understanding the system: The supervisor must understand the system’s capabilities, limitations, and risks. It is not enough to know how to use it: they must understand how it generates its decisions.
2. Ability to interpret outputs: They must be able to interpret the system’s scores, rankings, or recommendations and evaluate them critically.
3. Authority to discard or modify: They must have real authority (not just theoretical authority) to override the system’s decisions.
4. Ability to stop the system: Access to a stop mechanism (kill switch) that can deactivate the system if anomalous behavior is detected.

Oversight without authority is theatre. Without veto power, the human is decorative.

The invisible enemy: automation bias

Automation bias is the tendency to place excessive trust in the recommendations of an automated system, even when they contradict one’s own judgment. It is the phenomenon that turns human oversight into an empty formality.

Automation bias becomes worse with:

• Volume: When the supervisor has to review 200 decisions per hour, they lose the ability to critically assess each one. They start approving by default.
• Complexity: If the supervisor does not understand how the model reaches its conclusions, they tend to trust the system because they have no basis on which to question it.
• Track record of successes: If the system has been right the last 50 times, the supervisor lets their guard down. But AI errors are sporadic and systematic: when it fails, it fails consistently for a specific type of case.
• Time pressure: If there is pressure to process quickly, human review becomes a box-ticking exercise. The human clicks “approve” without reading.

How to mitigate automation bias:

1. Volume limit. Define the maximum number of decisions a supervisor can review per hour. If the system generates more, you need more supervisors or a preliminary filter to reduce the volume.
2. Test cases. Periodically introduce deliberately incorrect cases to verify that the supervisor detects them. If they do not, the oversight system is not working.
3. Supervisor rotation. Do not assign the same person to the same type of review indefinitely. Familiarity breeds complacency.
4. Explainability. Show the supervisor the reasons behind the system’s output, not just the result. If the supervisor sees that the AI recommends rejecting a candidate because their university correlates with a postcode, they can identify a proxy bias.
5. Intervention metrics. Monitor the supervisor’s intervention rate. If they approve 100% of the recommendations, they are not supervising. An intervention rate of 5–15% is healthy.

Traffic light: Green: intervention rate 5–15% and detection of >80% of test cases. Amber: intervention <5% or detection 60–80%. Red: intervention <2% or detection <60% → oversight is not working.

How to design an HITL system that works

An effective human oversight system has five components:

1. Scope definition. Which decisions does the human review? All of them, or only those in the grey zone (intermediate score)? For high-volume systems, selective HITL (only borderline cases) is more viable than full HITL.
2. Oversight interface. The supervisor needs an interface that shows the system’s decision, the reasons (explainability), the key case data, and the available actions (approve/modify/reject/escalate). If the interface is an email with a PDF attachment, the system will fail.
3. Escalation protocol. What happens when the supervisor disagrees with the system? Can they override it directly, or do they need a superior’s approval? Define a clear protocol with escalation levels.
4. Continuous training. The supervisor needs initial training on the system and ongoing training whenever the model changes. Every model update is an opportunity to recalibrate the supervisor’s understanding.
5. Intervention log. Every supervisor action must be recorded: what they decided, when, why, and what the system recommended. This record feeds model improvement and provides evidence of compliance with the AI Act.

HITL by use case: a quick guide

Not all systems need the same level of oversight. Here is a practical guide by use case:

• Personnel selection: HITL mandatory. No rejection decision without human review. Especially strict in borderline cases.
• Credit scoring: HITL mandatory. The customer has the right to obtain a human explanation for a rejection.
• Customer service chatbot: HOTL is sufficient. Monitor aggregate metrics (CSAT, escalation rate, hallucinations). Human intervention at the customer’s request.
• Recommendation engine: HIC is sufficient. Control business rules and the kill switch. Review results periodically.
• Fraud detection: HOTL for most cases. HITL for high-impact blocks (>€1,000 or recurring customers). Review false positives weekly.
• Demand forecasting: HOTL. The purchasing team reviews forecasts weekly and can adjust them manually.

The real cost of human oversight (and the cost of not having it)

Human oversight has a real operating cost that many companies underestimate when calculating the TCO of an AI project:

• Direct cost: Supervisor hours based on the volume of decisions. If a supervisor can review 20 qualified decisions per hour and your system generates 100 borderline cases per day, you need ~5 hours of oversight every day. At €25/hour, that is €125/day or ~€2,500/month.
• Training cost: Initial supervisor training (understanding the model, interpreting outputs, and using the interface) requires between 8 and 16 hours, plus 2–4 hours of ongoing training per quarter when the model is updated.
• Interface cost: If your AI tool does not include a native oversight interface, you need to develop one or use a workflow management platform. Typical cost: €2,000–€10,000 depending on complexity.

Now, the cost of not having human oversight:

• Regulatory fines: Up to 3% of global turnover for non-compliance with the AI Act in high-risk systems.
• Discrimination lawsuits: Without documented human oversight, the company cannot demonstrate that it took reasonable steps to prevent biased decisions. The burden of proof is reversed.
• Errors at scale: An undetected biased model can systematically reject a group of people for months before anyone discovers it. With human oversight, that bias is detected within weeks.
• Loss of trust: When an AI incident becomes public, the first question is always: “Was a human supervising?” If the answer is no, the reputational damage is multiplied.

Oversight ROI: HITL typically represents between 5% and 15% of the project’s total TCO. The risk it mitigates (fines + lawsuits + errors at scale) can be 10–50 times that cost. It is the quality-control investment with the best risk-to-return ratio in any AI project.

GDPR and human oversight: Article 22

In addition to the AI Act, the GDPR already establishes human oversight obligations in Article 22: people have the right not to be subject to decisions based solely on automated processing that produce legal or similarly significant effects.

This means that if your AI system makes decisions that affect people (rejecting a candidate, denying credit, blocking an account), the affected person has the right to have a human review that decision. And that review must be real, not a bot that repeats the algorithm’s decision.

The combination of the GDPR and the AI Act creates a dual framework of obligations: the GDPR protects the affected individual (the right to human review), while the AI Act protects the system as a whole (governance, traceability, and continuous oversight). They complement each other, and both are enforceable with significant penalties.

Human oversight is an operational control, not a box-ticking exercise

Well-designed human oversight does not slow down AI. It makes it more reliable, fairer, and easier to defend before regulators, customers, and employees. HITL is not a cost: it is a quality-control mechanism that protects your investment in AI.

The difference between a company that implements HITL as a checkbox and one that implements it as an operating system comes down to five factors: the supervisor’s understanding of the model, real authority to intervene, a suitable interface, manageable volume, and a record of every decision.

If you want to design the human oversight system for your AI projects, define HITL/HOTL/HIC protocols, or train your team of supervisors, Impulsa3 designs custom HITL systems.

impulsa3.com · Digital Transformation and AI for SMEs and ecommerce · servicios@impulsa3.com

Human oversight must be integrated with FRIA and fundamental rights.