Devansh Kamlani
When I look back at the AI programmes I have helped enterprises stand up, the difference between the ones that scaled and the ones that stalled almost always came down to governance. Not glamorous governance, not boardroom governance, but the practical, daily mechanics of knowing which models are in production, who approved them, what data they were trained on and what to do when something goes wrong.
AI Model Governance for Enterprise Architectures
In this article I want to share the AI model governance framework I now use when I advise large organisations. I will walk through the model registry, approval workflows, change control, audit trails, model cards, vendor model risk, policy and standards, integration with risk and compliance functions, the NIST AI Risk Management Framework, ISO/IEC 42001 and the tooling landscape. My intention is not to give you a perfect template, because no template survives contact with a real organisation. My intention is to give you the architectural building blocks that you can adapt to your own context.
If you are an architect being asked to build the governance backbone for an AI programme, this is the article I wish someone had handed me five years ago.
Why AI model governance is now an architectural concern
For a long time, model governance lived in the data science function. A team would track models in a spreadsheet, get sign-off from a risk manager and move on. That model has broken down for three reasons that any architect needs to understand.
First, the surface area has exploded. A modern enterprise might have dozens of fine-tuned models, hundreds of prompts in production, multiple vendor model endpoints and a sprawling RAG corpus. A spreadsheet cannot describe that estate, let alone govern it.
Second, the regulatory environment has hardened. The EU AI Act, the NIST AI RMF, ISO/IEC 42001, sector-specific regulators and state-level laws like the Colorado AI Act all expect documented governance. Auditors now ask for evidence.
Third, the failure modes have shifted. AI systems fail in ways that look more like reputational or behavioural failures than traditional software outages. Governance is the only mechanism that catches these failures before they reach production.
Governance is not paperwork. It is the architectural layer that makes AI systems explainable, reversible and accountable.
The architect’s job is to design that layer so that it accelerates delivery rather than blocking it.
The model registry as the central source of truth
The single most important component of an enterprise AI governance architecture is the model registry. It is the system of record for every model, prompt, retrieval index and evaluation artefact in the organisation.
A registry is not just a list. It is a queryable, versioned, access-controlled service that records the lifecycle of every model from training to retirement. Every entry has an owner, a status, a version history and links to its training data, evaluation results, approval records and deployment locations.
In my experience, the registry must cover at minimum:
- Foundation models in use (vendor and self-hosted)
- Fine-tuned model versions, including LoRA adapters
- Embedding and reranking models
- Prompt templates and their versions
- Retrieval indexes and their refresh schedules
- Agents and tool definitions
- Evaluation datasets and benchmarks
The registry should integrate with deployment systems so that nothing reaches production without a registry entry. The simplest enforcement is a CI gate that fails if the artefact reference is missing or unapproved.
I have seen organisations build their own registry on top of MLflow, use the registries inside SageMaker or Vertex AI, or adopt purpose-built products like Weights and Biases, Comet or Domino. The choice matters less than the discipline of using it for every artefact.
Approval workflows and gating
A registry without an approval workflow is just a catalogue. The governance value comes from gating: nothing moves to production without the right approvals, and every approval is recorded.
I usually design a tiered approval workflow that matches the risk of the use case. Low-risk internal tools might require only a peer review and a product owner sign-off. Customer-facing systems require risk, legal, security and compliance reviews. Systems in regulated functions require additional review by the relevant control owners.
| Tier | Typical use cases | Required approvals |
|---|---|---|
| 1 (low) | Internal productivity, draft generation | Engineering peer review, product owner |
| 2 (medium) | Customer-facing chat, internal decision support | Add: security, privacy, model risk |
| 3 (high) | Underwriting, hiring, credit, clinical | Add: legal, compliance, business unit risk officer |
| 4 (critical) | Anything in scope of EU AI Act high-risk | Add: external assurance, board-level sign-off |
The approval workflow lives inside the registry and is automated wherever possible. Reviewers receive notifications, see the evaluation results, can request changes and digitally sign off. Each approval is timestamped, attributed to a named individual and stored as an immutable record.
I have seen approval workflows go wrong in two ways. Either they are too heavy, so engineers route around them, or they are too light, so risk slips through. The right answer is risk-tiered, automated and integrated with the developer workflow.
Change control for models and prompts
Once a model is live, every change needs to be controlled. This is where many AI programmes break down because prompts and models change far more frequently than traditional software.
Change control for AI artefacts has to cover at least four kinds of change:
- New model versions (foundation model updates, fine-tune retrains)
- Prompt template changes
- Retrieval corpus updates
- Tool and agent configuration changes
Each change type needs its own process, but all of them need versioning, testing, approval and the ability to roll back. I treat prompts as code and store them in version control alongside the application that uses them. Retrieval corpora live in a versioned index with documented refresh procedures. Fine-tuned models live in the registry with semantic version numbers.
The most important architectural pattern here is the evaluation gate. Every change must pass an evaluation suite before it can be promoted. The suite includes regression tests, quality benchmarks, safety checks and cost metrics. If the suite fails, the promotion is blocked.
If you cannot tell me what evaluation suite blocks a bad prompt from reaching production, your AI system is not under change control.
The same gating logic applies to vendor model updates. When OpenAI or Anthropic or Google ships a new model version, your evaluation suite must run against it before you switch traffic.
Audit trails and lineage
Audit trails turn governance from a promise into evidence. Every approval, change, deployment, incident and retirement must be recorded with enough detail that an external auditor can reconstruct what happened.
The lineage view is particularly important. For any prediction or generation, you should be able to trace back to the model version, prompt version, retrieval results, input data and user identity. This sounds expensive, and at scale it is, but it is what regulators and incident responders will ask for.
I recommend an architecture where each AI request emits a structured event that includes:
- Request identifier and timestamp
- Model and prompt versions
- Retrieved context identifiers (not the raw text)
- Tool calls made
- Output identifiers
- User and tenant identifiers
- Cost and latency metrics
These events flow into an immutable log store, typically a write-once bucket or a regulated archive. Personal data is handled according to your privacy policy, often by hashing or referencing rather than storing raw values.
The lineage data also feeds drift detection, evaluation regression and cost analysis. Investing in this telemetry pays back across many governance functions, not just audit.
Model cards and data sheets
Model cards and data sheets are the human-readable documentation that accompanies every model in the registry. They are the artefact that auditors, risk officers and downstream teams will actually read.
A good model card answers the questions a thoughtful reviewer would ask. What is this model for? What is it not for? What data was it trained or fine-tuned on? How was it evaluated? What are its known limitations? What populations or use cases were tested? Who owns it and how do you contact them?
I structure model cards around the following sections:
- Intended use and out-of-scope use
- Training and fine-tuning data summary
- Evaluation results and benchmarks
- Fairness and bias evaluation
- Safety evaluation and red team results
- Operational metrics (latency, cost, throughput)
- Known limitations and failure modes
- Owner, reviewers and approval history
- Lifecycle status and review cadence
Data sheets do the same job for datasets. They describe provenance, collection methodology, consent basis, sensitive attributes, known biases and recommended uses.
These artefacts are not optional. They are required by the EU AI Act for high-risk systems, by ISO/IEC 42001 as part of an AI management system and by NIST AI RMF as part of the Map function. Building the discipline early is far cheaper than retrofitting it under audit pressure.
Vendor and third-party model risk
Most enterprise AI today is built on vendor foundation models. That means vendor model risk is a first-class governance concern, not a procurement footnote.
The architect’s job is to ensure that the organisation understands and manages the risks of every external model in use. That includes the obvious large vendors but also embedded AI in SaaS products, open-source models pulled from public hubs and any model accessed through a marketplace.
A vendor model risk programme covers:
- Contractual terms, including data usage, retention and indemnity
- Security posture, including penetration testing and certifications
- Privacy posture, including data residency and processing locations
- Model behaviour, including safety evaluations and red team results
- Operational reliability, including SLAs and incident history
- Lifecycle, including deprecation policy and notice periods
I have seen organisations caught out when a vendor deprecated a model with limited notice, breaking production systems. I have also seen privacy incidents triggered by SaaS vendors quietly adding AI features that processed customer data through third-party models. The mitigation is a vendor model inventory that sits inside the same registry as your own models.
Treat every vendor model as if you owned it. The regulator will.
Policy and standards architecture
Governance does not work without a policy layer that defines what is allowed and what is not. The architect’s role is to translate organisational policy into enforceable standards that fit into the engineering workflow.
I usually advocate a three-tier policy structure. The top tier is the AI policy, owned by the board or executive committee, that sets principles and risk appetite. The middle tier is standards, owned by risk and security, that define mandatory controls. The bottom tier is technical guidance, owned by the architecture function, that translates standards into patterns and tooling.
The standards I see most often include:
- Acceptable use of AI systems
- Approved model and vendor list
- Data handling and personal data restrictions
- Evaluation and testing requirements
- Human oversight and escalation requirements
- Logging and audit requirements
- Incident reporting requirements
The technical guidance layer is where architects spend most of their time. It is where the standards become reference architectures, code libraries, default settings and CI checks. When policy is expressed only as a document, engineers ignore it. When it is expressed as a library call or a CI gate, it is followed automatically.
The policy layer must also be versioned, reviewed regularly and tied to the regulatory environment. A policy that does not change when the EU AI Act becomes effective is a policy that is failing its job.
Integration with enterprise risk and compliance
AI governance cannot live in isolation. It must integrate with the existing enterprise risk and compliance functions, or it will be perceived as a parallel bureaucracy and quietly ignored.
The integration points I always design for include:
- The enterprise risk register, where AI risks are recorded and tracked
- The model risk management function, often in finance or insurance, which extends to AI models
- The privacy office, which reviews data processing for AI
- The security function, which assesses AI-specific threats
- Internal audit, which tests the effectiveness of AI governance controls
- The technology risk committee, which oversees the overall portfolio
In regulated industries, model risk management already has a mature framework, often based on SR 11-7 in the US or the equivalent local guidance. AI governance extends that framework rather than replacing it. The same lifecycle of identification, validation, monitoring and retirement applies, but the techniques are different.
The trick is to extend, not duplicate. If your bank has a model risk function, the AI governance framework should plug into it. If your insurer has an actuarial review process, the AI governance framework should extend it. Architects who build parallel structures lose, because they end up fighting the established functions for authority.
NIST AI Risk Management Framework
The NIST AI Risk Management Framework, published in early 2023 and refined since, has become the de facto reference for AI governance in the US and increasingly elsewhere. Architects need to understand it because boards, auditors and regulators are using it as a benchmark.
The framework has four core functions: Govern, Map, Measure and Manage.
- Govern establishes the organisational culture, policies and accountability for AI risk
- Map identifies the context, intended use, stakeholders and risks of each AI system
- Measure evaluates AI systems against the identified risks, using both quantitative and qualitative methods
- Manage prioritises and treats the risks, including monitoring, response and retirement
The framework is voluntary, but in practice it is becoming the expected baseline. Federal contracts, sector regulators and enterprise customers increasingly ask for evidence that an organisation has adopted it.
For architects, the most useful aspect is the focus on context and trustworthiness characteristics: valid and reliable, safe, secure and resilient, accountable and transparent, explainable and interpretable, privacy-enhanced, and fair with harmful bias managed.
I use these characteristics as a checklist when reviewing AI architectures. Each system should have a documented position on each characteristic, with evidence behind it. Where the system falls short, there should be a documented mitigation or accepted risk.
ISO/IEC 42001 and management systems
ISO/IEC 42001, published in late 2023, is the first international standard for an AI management system. It is the AI equivalent of ISO 27001 for information security, and it follows the same management-system structure.
The standard requires organisations to establish an AI management system that covers leadership, planning, support, operation, performance evaluation and improvement. It expects documented policies, defined roles, risk assessments, controls and continual improvement.
For enterprises that already operate under ISO 27001 or ISO 9001, adopting ISO/IEC 42001 is largely a matter of extending existing management systems with AI-specific controls. For enterprises that do not have a management-system culture, it is a larger lift.
The standard is becoming a procurement requirement, particularly in Europe. Customers and partners increasingly ask whether their vendors are certified or working toward certification. Architects should expect to see this requirement land in tenders and contracts within the next two years.
ISO/IEC 42001 is the standard your procurement team will start asking about. NIST AI RMF is the framework your board will ask about. They are complementary, not competitive.
I recommend organisations adopt the NIST framework as their working model and align with ISO/IEC 42001 if they need a certifiable management system.
Tooling landscape and build-vs-buy
The tooling for AI governance is evolving fast. In 2026 the landscape includes several categories that an architect needs to evaluate.
| Category | Examples | Build-vs-buy default |
|---|---|---|
| Model registry | MLflow, SageMaker, Vertex, W&B, Comet, Domino | Buy or use cloud-native |
| Prompt management | LangSmith, PromptLayer, Humanloop, Langfuse | Buy |
| Evaluation | Promptfoo, OpenAI Evals, DeepEval, Patronus | Buy and extend |
| Observability | Arize, Fiddler, Whylabs, Langfuse, Datadog AI | Buy |
| Policy enforcement | OPA, in-house gateways | Build on top of buy |
| Governance platforms | Credo AI, Fairly AI, Holistic AI | Buy if mature programme |
| Vendor risk | Existing GRC platforms | Extend existing |
My default advice is to buy or use cloud-native services for the commodity layers, build only the integration glue and policy enforcement that is specific to your organisation. Custom-built registries and observability platforms become a maintenance burden that distracts from the actual AI work.
The integration story matters as much as the individual products. The registry must talk to the evaluation system, which must talk to the observability platform, which must talk to the incident management system. Architects should design the data model and event schema once and then choose tools that fit it.
Operating model and roles
Governance only works if there are named humans accountable for it. The operating model I recommend has three layers.
The first layer is the AI governance committee, chaired by a senior executive, with representation from technology, risk, legal, compliance, privacy, security and the business. It owns the policy, approves high-risk use cases and reviews the portfolio.
The second layer is the AI risk function, often led by a head of AI risk or chief AI officer, with model risk specialists, AI ethicists and red team specialists. This function operates the framework day to day, performs validations and reviews and reports to the committee.
The third layer is the AI engineering and architecture function, where the controls are implemented in code and operations. This is where the registry runs, where evaluations execute, where models are deployed and monitored. Architects here are the bridge between the policy world and the engineering world.
Each layer needs clear accountabilities. I have seen too many programmes where everyone assumes someone else is approving, validating or monitoring. A simple RACI matrix, kept up to date and visible to the committee, prevents most of these gaps.
The roles also need investment. AI governance is not a side-of-desk activity. It needs dedicated people with the skills to challenge engineering decisions on their merits.
Common pitfalls and how to avoid them
I have seen organisations make the same mistakes often enough to catalogue them.
- Governance theatre. Heavy documentation, light enforcement. Avoid by making controls code-enforceable wherever possible.
- Spreadsheet registries. Shadow inventories that no one trusts. Avoid by investing in a real registry and making it the only path to production.
- Approval bottlenecks. Single reviewers who become a critical path. Avoid by tiering approvals and resourcing reviewers properly.
- Vendor blind spots. No inventory of vendor models embedded in SaaS. Avoid by extending procurement processes to flag AI features.
- Disconnected functions. AI governance running in parallel to existing risk and compliance. Avoid by integrating from day one.
- Policy without standards. Vague principles with no enforceable rules. Avoid by writing standards that an engineer can implement.
- Static lifecycle. Models that never get re-evaluated after launch. Avoid by scheduling periodic reviews and tying them to operational events.
The thread that runs through all of these is the gap between intent and implementation. Architects who close that gap by making governance code-enforceable, integrated and resourced will build programmes that scale. Architects who treat governance as a deliverable rather than a system will build programmes that audit poorly.
Good governance is invisible to the engineer who is doing the right thing and immediate to the engineer who is not.
