Metacognitive Readiness as a Missing Variable in AI Governance

Why knowing your model is not enough

Keywords

AI governance · XAI · metacognition · human oversight · EU AI Act · human-AI interaction

1. THE SILENT ASSUMPTION

Contemporary AI governance frameworks rest on a foundational premise that is rarely made explicit: when a human operator receives the output of an AI system, they can meaningfully evaluate it. The EU AI Act’s high-risk system provisions require transparency, explainability, and human oversight. The United States AI Action Plan calls for maintaining meaningful human control over consequential AI decisions. The OECD AI Principles embed human-centredness at their core.

These commitments are necessary — but they are insufficient. They address what AI systems must provide to human operators. They leave entirely unexamined what human operators must be capable of in order to act on what they receive. This gap is not incidental. It is a structural blind spot in the current architecture of AI governance.

Governance frameworks tell AI systems to explain themselves. They say nothing about whether humans can understand the explanation.

The implicit model of the human supervisor in most regulatory texts is that of a competent, attentive professional who, given accurate and legible system outputs, will make sound judgments. This is a plausible assumption in stable, low-stakes, well-understood environments. It is a fragile one in high-stakes, time-pressured, and technically opaque contexts — precisely the contexts in which AI systems are increasingly being deployed.

2. WHAT COGNITIVE SCIENCE TELLS US

Metacognition - the ability to monitor and regulate one’s own cognitive processes— is the psychological basis for effective supervision. It is neither a personality trait nor an indicator of value. It is a situational skill, sensitive to context, training, cognitive load, and working conditions.

Metacognition — the capacity to monitor and regulate one’s own cognitive processes — is the psychological substrate of effective oversight. A metacognitively aware operator knows when they understand something, when they are guessing, and when their judgment is being shaped by factors they have not consciously registered. This capacity cannot be assumed; it varies significantly across individuals, training backgrounds, and situational pressures.

Research in human-automation interaction has documented a cluster of failure modes that emerge specifically when humans supervise automated or AI-powered systems. Automation bias — the tendency to over-weight machine-generated recommendations relative to one’s own judgment — is among the most robust findings in the field. In a widely cited study, Parasuraman and Riley (1997) demonstrated that humans routinely misuse automation by applying it in domains where it is unreliable, and disuse it by ignoring it in domains where it would be beneficial — both errors reflecting a failure of metacognitive calibration rather than a failure of information provision.¹

The challenge is compounded by the specific properties of contemporary AI systems. Kahneman’s framework of dual-process cognition provides a useful lens: outputs produced by fluent, confident-sounding generative AI systems engage the fast, associative processing of System 1, bypassing the slower, more deliberate evaluation of System 2.² An explanation that looks plausible activates different cognitive responses than one that is plausible. When AI explanations are syntactically fluent, numerically precise, and visually formatted as authoritative outputs, they suppress the very scepticism that meaningful oversight requires^3,4 .

Perhaps most counterintuitively, providing more explanation does not reliably improve human judgment about AI outputs. Bansal et al. (2021), in a rigorous experimental study, found that AI explanations did not consistently improve human-AI team performance and, in several conditions, degraded it — particularly when explanations were technically accurate but cognitively incompatible with how operators formed their own judgments.⁵ Explainability is a necessary but insufficient condition for effective oversight. What mediates the gap between the two is metacognitive readiness.

Providing more explanation does not reliably improve human judgment. What mediates the gap is metacognitive readiness.

3. THREE GOVERNANCE IMPLICATIONS FROM INTER-DISCIPLINARY RESEARCH

If metacognitive readiness is a real and variable property of human operators — and the evidence suggests it is — then governance frameworks that mandate explainability without addressing operator metacognition are building on incomplete foundations. Three implications follow. They differ deliberately in their epistemic status: the first is grounded in established empirical findings; the second identifies a structural gap in the research and regulatory literature; the third draws on evidence from a neighbouring discipline to formulate a hypothesis that, to our knowledge, has not yet been systematically investigated in the context of AI governance.

First, documentation-centred transparency is insufficient — a finding from the empirical literature. This implication is not speculative. Parasuraman and Riley’s foundational work on human-automation interaction (1997) documents that providing humans with accurate information does not reliably produce accurate judgments. Bansal et al.’s controlled experiment (2021) extends this finding specifically to AI-generated explanations: technically correct explanations degraded human-AI team performance in several experimental conditions, particularly when the explanation format was cognitively incompatible with the operator’s reasoning style. The EU AI Act’s technical documentation requirements define explainability in terms of what system providers must produce. They do not define it in terms of what operators must be able to do with what they receive. A governance standard centred on the recipient of explanations — rather than solely their formal provision — would require that high-risk AI system deployment be accompanied by validated evidence that target user populations can meaningfully interpret system outputs in realistic operational conditions. This is a logical consequence of findings that are already in the literature; the regulatory architecture has not yet caught up.

Second, operator metacognitive qualification as a governance variable — a gap the research literature has named but not yet filled. Existing scholarship on human-AI collaboration increasingly identifies operator metacognition — specifically the capacity to detect one’s own over-reliance, recognise distribution shift, and resist automation bias under time pressure — as a key determinant of oversight quality. Recent work in Information Systems Research (2025) explicitly characterises this as an open research gap: whether and how technical features of AI systems influence the metacognitive processes of human operators remains empirically unresolved. More directly, legal analyses of Article 14 of the EU AI Act note that while the text requires deployers to assign oversight to persons with ‘necessary competence, training and authority’, it provides no criteria for what these competencies consist of, and imposes no research requirement on the organisational and human conditions at the user’s end. The implication we draw is not that such a qualification framework already exists — it does not — but that constructing one is both technically feasible and governance-relevant. Domain-specific, role-specific assessment criteria for metacognitive readiness in AI supervision are researchable and operationalisable. They have simply not been included in any major regulatory framework. This is the gap.

Third, the unequal distribution of metacognitive readiness as a structural safety risk — a hypothesis drawn from educational psychology that has not yet been studied in the AI governance context, and that may define the next research frontier. This third implication differs from the preceding two in its epistemic standing: it is not a conclusion from the AI literature, but a transposition from an adjacent field that warrants systematic investigation. Research in educational psychology has consistently demonstrated that metacognitive skills — the capacity to monitor one’s own understanding, detect errors in reasoning, and regulate cognitive strategies — are not uniformly distributed across populations. They vary as a function of educational exposure, institutional culture, professional training environment, and access to formative feedback. These findings are robust within their original domain. Their implications for AI governance have not, to our knowledge, been empirically examined. We propose the following hypothesis: governance frameworks designed around the operator capabilities of well-resourced institutions risk producing a structural safety asymmetry — organisations with substantial training infrastructure can operationalise oversight requirements; those without cannot. The result is compliance on paper, and potential failure on the ground. If this hypothesis holds — and the cross-disciplinary logic is strong — then inclusivity and safety are not separate governance objectives. They are the same objective viewed from different angles. We submit this as a research question that governments and funding agencies should actively prioritise, precisely because the window for incorporating such findings into regulatory design is closing as the AI Act’s implementation deadlines approach. The strength of this brief’s contribution lies partly in reading across disciplinary boundaries: what educational psychology has demonstrated about the unequal distribution of metacognitive capacity may be the missing variable in global AI governance.

An important clarification is needed here regarding the last point. Talking about operators’ metacognitive skills does not mean questioning the value or intelligence of the people who supervise AI systems. It is not a matter of ranking humans according to their ability to “think well.” Metacognition is not a personality trait or an indicator of value. It is a situational skill, sensitive to context, training, cognitive load, and working conditions. An experienced surgeon may have excellent metacognitive calibration in their field and be just as vulnerable to automation bias as a novice faced with an opaque AI system in a context for which they have received no specific training. What is proposed here is to make visible and assessable a variable that already determines, in practice, the quality of supervision, and affects it without anyone measuring it.

POLICY RECOMMENDATIONS

1. Expand explainability standards to include recipient-centred criteria: require that high-risk AI deployments demonstrate, through validated user studies, that target operator populations can meaningfully interpret system outputs under realistic conditions — not only that technically accurate explanations have been provided.

2. Develop operator metacognitive qualification frameworks: working through the EU AI Act’s implementation guidelines and equivalent US executive guidance, introduce domain-specific competency criteria for AI supervisory roles that address cognitive calibration, automation bias resistance, and error detection capacity.

3. Design compliance pathways for less-resourced contexts: ensure that international AI governance standards — including the AI Action Plan’s global outreach components — include tiered implementation mechanisms that account for variation in institutional training capacity, so that oversight requirements do not inadvertently entrench a two-tier safety architecture.

CONCLUSION

The central problem this brief identifies is not a failure of ambition in existing AI governance frameworks. The EU AI Act, the OECD AI Principles, and the US AI Action Plan all express genuine commitments to human oversight and meaningful control. The problem is a structural gap between what these frameworks require of AI systems and what they require of the humans who supervise them.

Closing this gap requires treating metacognitive readiness not as a soft human-factors concern peripheral to governance, but as a constitutive variable of effective AI oversight — one that is empirically researchable, operationally relevant, and unevenly distributed across the institutions that AI governance is meant to protect. Until it is incorporated into the formal architecture of AI regulation, ‘human oversight’ will remain a declaration of intent rather than a guarantee of safety.

REFERENCES

¹ Parasuraman, R., & Riley, V. (1997). Humans and automation : Use, misuse, disuse, abuse. Human Factors, 39(2), 230–253. https://doi.org/10.1518/001872097778543886

² Kahneman, D. (2011). Thinking, Fast and Slow. Farrar, Straus and Giroux. Le cadre de la cognition à double processus distingue le traitement rapide et associatif (Système 1) du traitement lent et délibéré (Système 2), avec des implications importantes pour la manière dont les humains répondent aux outputs automatisés à haute confiance apparente.

³ Tankelevitch, L., Kewenig, V., Simkute, A., Scott, E.A., Sarkar, A., Sellen, A., &
Rintel, S. (2024). The Metacognitive Demands and Opportunities of Generative AI. Proceedings of the CHI Conference on Human Factors in Computing Systems^{. https://doi.org/10.1145/3613904.3642902}

⁴ Alter, A.L., & Oppenheimer, D.M. (2009). Uniting the tribes of fluency to form a
metacognitive nation. Personality and Social Psychology Review, 13(3), 219–235.

⁵ Bansal, G., Vaughan Ngyuen, T., Vaughan, J. W., Weld, D., & Lasecki, W. S. (2021). Does the Whole Exceed Its Parts? The Effect of AI Explanations on Complementary Team Performance. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3411764.3445717

⁶ Parlement européen. (2024). Règlement (UE) 2024/1689 — Loi sur l’intelligence artificielle (AI Act). Articles 13 (Transparence), 14 (Supervision humaine), Annexe III (Systèmes d’IA à haut risque).

⁷ OCDE. (2024). Principes de l’OCDE sur l’IA. Principe 1.4 : Centrage sur l’humain et
supervision humaine. https://oecd.ai/en/ai-principles

About the Author

Ikram Chraibi Kaadoud holds a PhD in computer science and cognitive science (University of Bordeaux/INRIA, FRANCE, 2018). She has contributed to several projects at the intersection of industry and academia, including a private research laboratory and the management of a European project on the responsible deployment of trustworthy AI with 13 institutional partners (EDIH) and more than 100 manufacturing companies. Her research focuses on the intersection of interpretability and explainability of AI systems, human cognitive factors in AI adoption, and responsible governance frameworks. Her scientific work has been published in Neural Networks, Knowledge-Based Systems, and Scientific Reports. She is a regular contributor to the Blog Binaire (La Recherche) on AI and societal issues, and is a member of the WomenTechMaker community.