The Reflex
When a new category of technology risk emerges in a large, regulated organisation, the institutional response is predictable. A working group is formed. A framework is drafted. Policies are written. Review boards are established. Approval workflows are designed. Training is mandated. And then, gradually, the governance apparatus becomes so heavy that the thing it was designed to govern never actually gets deployed.
This has happened with cloud computing, with open source software, and with data sharing. Each time, the governance pendulum swung too far toward control, slowed adoption to a crawl, and eventually had to be recalibrated once the organisation realised it had governed itself into irrelevance. The cycle took years each time.
It’s happening again with AI. But the stakes are higher, because the organisations that get the governance balance wrong won’t just be slow. They’ll be left behind by competitors who figured it out faster.
The Paradox
The AI governance paradox is this: the organisations most capable of governing AI are also the ones most likely to over-govern it into irrelevance, while the organisations that move fastest with AI are the ones least likely to govern it responsibly.
Heavily regulated industries (financial services, healthcare, government, energy) have decades of experience managing operational risk. They have the institutional muscle to build governance frameworks. The problem is that their governance instincts are calibrated for a different era of technology. Traditional IT governance assumes deterministic systems: given the same input, you get the same output. You can test exhaustively, document completely, and validate before deployment.
AI, particularly generative AI, doesn’t work that way. Outputs are probabilistic. The same input can produce different outputs. Full explainability isn’t always possible. And the rate of change in model capability means that governance frameworks designed with twelve-month review cycles are outdated before they’re approved.
The result in most regulated organisations is one of two failure modes:
Governance that prevents deployment. Approval processes so thorough that by the time a model is approved for production, the business case has evaporated, the sponsoring executive has moved on, or the technology has been superseded. I’ve seen organisations with eighteen-month model approval cycles trying to govern LLM-based applications that didn’t exist eighteen months ago.
Governance theatre. Impressive frameworks that look good in board presentations and satisfy auditors but don’t actually catch the risks that matter. The policies exist, the committees meet, the documents are produced, but nobody is asking the hard questions about what happens when the model gets it wrong in ways that affect real people.
Neither of these is governance. The first is prohibition dressed up as prudence. The second is compliance dressed up as risk management.
The following quadrant maps the two failure modes and the target state. The horizontal axis is governance rigour. The vertical axis is AI deployment velocity. Most regulated enterprises sit in the bottom-right: heavy governance, almost no deployed AI. Most tech companies sit in the top-left: moving fast, governing later (or never).
quadrantChart
title The AI Governance Balance
x-axis Low Governance Rigour --> High Governance Rigour
y-axis Slow AI Deployment --> Fast AI Deployment
quadrant-1 "Governed and Delivering"
quadrant-2 "Moving Fast, Governing Later"
quadrant-3 "Not Trying"
quadrant-4 "Governed into Irrelevance"
"Most Regulated Enterprises": [0.82, 0.18]
"Most Tech Companies": [0.20, 0.82]
"Target State": [0.72, 0.68]The target state is top-right: high governance rigour and fast deployment. Getting there requires abandoning the assumption that rigour and speed are inversely related. They’re not, if you design governance for the technology you’re actually governing rather than retrofitting frameworks built for something else.
What Makes AI Governance Different
AI governance isn’t IT governance with a new label. There are structural differences that require a fundamentally different approach:
Non-determinism. Traditional systems produce the same output for the same input. AI models, especially generative ones, don’t. This breaks the testing and validation paradigms that most governance frameworks depend on. You can’t test every possible output, because the output space is effectively infinite.
Emergent behaviour. Large language models exhibit capabilities that weren’t explicitly trained for and weren’t predicted by their creators. Governing emergent behaviour with frameworks designed for predictable systems is like using a building code to regulate weather.
Continuous drift. Models degrade over time as the data they were trained on becomes less representative of the current world. Governance that treats model approval as a point-in-time event misses the reality that a model’s risk profile changes continuously after deployment.
Supply chain opacity. Most enterprise AI now depends on foundation models built by external providers. The organisation using GPT-4 or Claude or Gemini in a customer-facing application doesn’t control, and often can’t inspect, the model’s training data, architecture, or update cycle. Governing a system you can’t fully see inside requires different tools than governing one you built yourself.
These differences don’t make AI ungovernable. They make it differently governable. And that distinction matters, because most organisations are trying to force AI into governance frameworks that were designed for a completely different category of technology.
A Tiered Governance Model
The approach I’ve found effective borrows from the defensive and offensive data strategy framework: match the intensity of governance to the actual risk, not to the institutional anxiety level.
Not all AI applications carry the same risk. A model that recommends blog articles to employees carries fundamentally different risk than one that influences clinical decisions for patients with cochlear implants. Governing them with the same framework is wasteful for the first and potentially inadequate for the second.
flowchart LR
subgraph Risk Tier
T1["Tier 1: Exploration\nSandboxed, no real\ndecisions affected"]
T2["Tier 2: Operational Support\nAI-assisted processes,\nhuman in the loop"]
T3["Tier 3: Decision Influence\nAI directly informs\nbusiness decisions"]
T4["Tier 4: Autonomous Decision\nAI decides with minimal\nhuman intervention"]
end
subgraph Governance Response
G1["Light Touch\nUsage guidelines,\nbasic monitoring"]
G2["Structured\nBias testing, audit trails,\nperformance monitoring"]
G3["Rigorous\nModel validation,\nexplainability, impact assessment"]
G4["Intensive\nRegulatory review, continuous\nmonitoring, ethics board"]
end
T1 --> G1
T2 --> G2
T3 --> G3
T4 --> G4Tier 1: Exploration
Internal experimentation, sandbox environments, no real decisions affected. Think: data scientists exploring a new modelling technique, teams testing whether an LLM can usefully summarise internal documents, hackathon projects.
Governance requirement: Minimal. Usage guidelines, data handling rules (don’t put customer PII into external LLM APIs), basic monitoring of what tools and services are being used. The goal is to enable learning, not to control it.
Common mistake: Applying Tier 3 or Tier 4 governance to Tier 1 activities, which kills experimentation before it starts and drives usage underground. If your people are using ChatGPT on personal devices because the corporate approval process takes three months, your governance hasn’t succeeded. It’s failed in the most fundamental way possible: it’s pushed activity outside your visibility entirely.
Tier 2: Operational Support
AI that assists human-led processes where a human reviews and acts on the output. Think: AI-generated first drafts of reports, automated data quality flagging, copilot-style coding assistance, meeting summarisation.
Governance requirement: Structured but not heavy. Bias testing where relevant, audit trails for outputs that feed into business records, performance monitoring to ensure the model continues to be useful, clear disclosure when AI-generated content is used externally.
Common mistake: Treating Tier 2 applications as if they carry Tier 4 risk. A coding assistant might introduce a bug. That’s what code review is for. A summarisation tool might miss a nuance. That’s what the human reading it is for. The human in the loop isn’t decoration. They’re the primary control.
Tier 3: Decision Influence
AI output that directly informs significant business or clinical decisions. Think: credit risk scoring, treatment pathway recommendations, fraud detection alerts, customer churn predictions used to allocate retention resources.
Governance requirement: Rigorous. Formal model validation before deployment. Explainability requirements appropriate to the decision context. Impact assessments that consider what happens when the model is wrong. Regular revalidation as data and conditions change. Clear escalation paths for edge cases the model wasn’t designed to handle.
Common mistake: Assuming that because a human technically signs off on the decision, the governance can be lighter. If a human consistently follows the model’s recommendation without independent assessment (and they will, because automation bias is well documented), the practical risk profile is closer to Tier 4 than Tier 2.
Tier 4: Autonomous Decision
AI that makes or executes decisions with minimal human intervention. Think: automated trading systems, real-time fraud blocking, dynamic pricing, clinical decision support systems that interact with implantable medical devices.
Governance requirement: The full apparatus. Regulatory review where applicable. Continuous monitoring with automated alerting. Human override capability that is tested regularly, not just documented. Ethics board review. Extensive testing including adversarial scenarios. Ongoing bias monitoring. Periodic external audit.
Common mistake: Not recognising when a system has drifted from Tier 3 to Tier 4. This happens gradually: the human review step becomes a rubber stamp, the override process becomes so cumbersome that nobody uses it, and the system becomes de facto autonomous without anyone explicitly deciding that it should be. This is one of the most dangerous failure modes in AI governance, because nobody made a conscious decision to operate without oversight. It just happened.
The Governance Operating Model
Getting the tiers right is necessary but not sufficient. The operating model matters just as much as the framework itself. Three principles from experience:
Governance Must Be Embedded, Not Bolted On
This is the same principle from my defensive and offensive data strategy work: governance that exists as a separate process from delivery will always be treated as overhead. The most effective AI governance I’ve seen is embedded directly into the development lifecycle. Model risk assessment happens during design, not after deployment. Bias testing is part of the build pipeline, not a quarterly review. Monitoring is automated, not dependent on someone remembering to check a dashboard.
Speed Is a Governance Requirement
If your governance process takes longer than the useful life of the model it’s assessing, you don’t have a governance framework. You have a prohibition framework with better branding. The time from model development to governance clearance is itself a metric that the governance function should be measured on and held accountable for. This doesn’t mean cutting corners. It means designing governance processes that are efficient enough to operate at the speed the technology moves. If the answer to “how long does approval take?” is measured in quarters, something is broken.
Governance Should Learn
AI models learn from data. AI governance should learn from outcomes. Every model failure, every near-miss, every instance where governance caught a real problem or failed to catch one should feed back into the governance framework. Static governance applied to a technology that changes monthly is a contradiction that resolves itself in the worst possible way: the governance becomes irrelevant and people route around it.
The GenAI Complication
Generative AI introduces governance challenges that most existing model risk frameworks weren’t designed for:
Prompt injection. Users, or attackers, can manipulate LLM behaviour through carefully crafted inputs. This is a novel attack surface that traditional application security doesn’t address because traditional applications don’t accept freeform natural language instructions from users.
Hallucination. LLMs generate plausible but false information with high confidence. In a Tier 3 or Tier 4 context, this isn’t a minor inconvenience. It’s a material risk that needs specific monitoring and mitigation strategies.
Training data provenance. When you use a foundation model from an external provider, you inherit whatever biases, errors, and legal risks exist in their training data. Your governance framework needs to account for risk you can’t directly inspect or control.
Output variability. The same prompt can produce meaningfully different outputs on different occasions. This makes traditional testing approaches (define expected output, compare actual output) insufficient. Governance for generative AI needs to think in terms of acceptable ranges of behaviour, not deterministic correctness.
These challenges are real, but they’re solvable with the right approach. What’s not acceptable is using their difficulty as a reason to either block GenAI deployment entirely or to deploy it without adequate controls. Both responses represent a failure of governance thinking: the first prioritises control over value, the second prioritises speed over responsibility. The organisations that will succeed are the ones that refuse to accept that as a binary choice.
This framework is shaped by direct experience governing AI in two very different regulated environments: financial services at Westpac (where model risk governance is mature but was designed for classical statistical models, not LLMs) and healthcare technology at Cochlear (where the regulatory bar is set by the real-world consequence of getting it wrong, which is a patient relying on an implanted device). The tiered approach emerged from the practical necessity of enabling AI innovation while maintaining the governance rigour that both environments demand, and from the observation that trying to apply one level of governance to all AI applications satisfies nobody and protects nothing.