As AI agents move from answering questions to taking actions, the technology leader’s mandate has fundamentally shifted. Deploying AI was the first challenge. Being accountable for what AI agents decide and do across banking and financial services, healthcare, and manufacturing is the next, harder one. The technology is advancing. Governance required to deploy it responsibly is not keeping pace.
AI agents can inform a decision but cannot be held accountable for one. BCG’s 2026 research highlights the leadership responsibility: 72% of CEOs say they are now their organisation’s main AI decision maker. That gap is no longer theoretical.
The evidence is instructive. In controlled simulations published in 2025, Anthropic found that models from several developers sometimes chose blackmail or corporate espionage to pursue assigned goals. These were deliberately constructed tests, not reported production incidents. Separately, Anthropic’s September 2026 threat report documented biological research uses of Claude that could support weapons development, while explicitly stating that it was not asserting harmful intent. The industry’s own leaders have brought these broader concerns onto the international stage. On 23 September 2026, the CEOs of Anthropic and OpenAI briefed the UN Security Council on the benefits and risks of advanced AI. When builders bring questions of oversight and control before the Council, the case for governance that evolves with AI capability becomes harder to ignore.
Traditional governance was designed for static systems. An AI agent evaluated in January does not behave identically in September, having encountered months of new production data. Risk assessments, one-time audits and policy sign-offs assumed agents would stay within the bounds of their validation; they do not.
Trust must be engineered into the architecture of AI deployment from the start, not applied afterwards as a compliance layer. The harness surrounding an AI agent governing how it behaves, what it refuses to do, how it sequences its reasoning, and when it defers to human judgment is the first line of that architecture. Equally critical is what agents reason over: the structured, contextual intelligence that grounds their outputs. An agent reasoning over poorly assembled context does not just under-perform; it produces outputs that look authoritative but carry no traceable lineage. Continuous evaluation disciplines close the loop, agents checking their own outputs against defined criteria, with traces and logs making every reasoning step attributable and auditable.
Explainability is the outcome: every output traceable to what the agent knew, what it was instructed to prioritize, and why it concluded what it did. Continuous monitoring detects capability drift in production, not after an incident forces a review. These are not static measures, they are disciplines that must evolve as AI capability does, and the foundation that determines whether AI can be trusted at scale.
The regulatory direction is clear, even where legislation is still developing. The EU AI Act applies in phases, with major provisions applying from August 2026 and high-risk requirements following in 2027 and 2028. Its requirements for high-risk AI systems include human oversight, logging, and post-market monitoring. The United States NIST AI Risk Management Framework provides a voluntary approach to managing AI risks, including accountability and traceability. India’s AI Governance Guidelines, released in November 2025, set out an approach to safe and responsible AI adoption. Together, these developments reinforce the case for continuous accountability, not periodic validation.
The technology leaders moving fastest with AI, built trust into the architecture early. Explainability, traceability, and human oversight are not what slow AI agents down. They are what make AI agents safe enough to deploy at speed. The mandate for technology leaders is no longer to adopt AI, it is to be accountable for what AI agents do after adoption as they move from answering to acting and to engineer that accountability in, before the first decision is made at scale.
About the Author
As the Vice President of the PULSEEAI platform and Assurance service line at Maveric Systems, Sriram Sivaraman is responsible for leading the design, build, and commercialization of AI-native platforms for banking transformation. He is also entrusted with advancing Maveric’s AI-first strategy to help global banks modernize technology delivery, strengthen governance, and scale operations across regulated environments.
Bringing over two decades of cross-functional leadership in IT consulting and digital transformation, his expertise spans large-scale program governance, core banking modernization, and enterprise delivery architectures. Prior to his current mandate, he led high-impact practice groups and client delivery portfolios, consistently steering complex, multi-million-dollar technology engagements from vision to execution across global BFSI markets.
Originally published in ETEdge Insights
