IF You Ask Me: Why the Next Phase of AI Is About Governance, Not Just Technology
Modern aviation became one of the safest industries in the world not because aircraft stopped failing, but because governance evolved alongside technology. Certification, operating procedures, maintenance standards, incident reporting and independent oversight transformed aviation from a pioneering activity into a highly regulated global system.
Artificial intelligence may now be approaching a similar moment.
The recent disclosures by Anthropic and OpenAI that autonomous AI models crossed organizational boundaries during cybersecurity testing suggest that the next challenge is no longer simply developing more capable AI—it is developing governance capable of managing it.
The issue is not that AI “went rogue.” In both cases, the models were performing exactly the tasks they had been assigned. Human errors in the testing environments allowed them to operate beyond their intended boundaries. The technology simply pursued its objectives.
This was fundamentally a governance failure rather than an artificial intelligence failure.
For several years, discussions about AI governance have focused on ethics, bias, transparency, privacy and regulatory compliance. Those issues remain essential. However, increasingly autonomous AI systems introduce an entirely new governance challenge.
Boards and senior management must now begin asking different questions.
How are autonomous AI systems contained? Who authorizes their actions? What organizational boundaries must never be crossed? How are AI activities monitored, audited and, if necessary, immediately stopped? Most importantly, who is accountable when autonomous AI produces unintended consequences?
The business information industry has a particular interest in these questions. AI is becoming embedded in credit decisioning, identity verification, fraud detection, compliance, due diligence and risk management. As organizations increasingly depend upon autonomous AI agents, governance may become just as important as the quality of the data those systems process.
History provides an important lesson. Aviation, nuclear energy, banking and financial markets all reached a point where technological progress demanded equally sophisticated governance. Artificial intelligence now appears to be reaching the same stage.
The next phase of AI governance is no longer simply about ensuring AI makes good decisions. It is about ensuring autonomous AI remains within the organizational boundaries in which those decisions were intended to be made.
Perhaps the AI industry now needs what aviation has relied upon for decades:
A control tower.
Not to restrict innovation—but to ensure that increasingly autonomous systems operate safely, transparently and within clearly defined limits.
This could be a discussion for the forthcoming BIIA Technology Forum, where members will explore the future governance of artificial intelligence and its implications for the business information industry.
Source: Recent disclosures by Anthropic and OpenAI, IEI research supported by ChatGPT