AI Security Architecture: Threats, Defenses & Frameworks

The first AI security incident I worked on was almost comical in hindsight. A junior engineer had built a customer-facing assistant, plugged in a vendor LLM, given it access to an internal API and shipped it. Within a week, a curious user had convinced the assistant to dump configuration that included an internal endpoint. No exotic attack, no zero-day, just a polite request the model could not refuse. That incident changed how I think about AI security.

In this article I want to share the AI security architecture framework I now use when I review production AI systems. I will walk through the OWASP Top 10 for LLM Applications, the defence-in-depth patterns I rely on, the red team approach that helps me find real issues, and the secure deployment patterns I now consider table stakes. My focus is on what an architect actually builds, not on a theoretical threat catalogue.

If you are responsible for AI systems that touch real users, real data or real money, the cost of getting security wrong has gone up dramatically. This article is the framework I wish I had when I shipped my first production assistant.

AI security shares principles with traditional application security but introduces new failure modes. The model is not just code that processes input. It is a probabilistic component that follows instructions, including instructions that are smuggled in alongside data.

The implications are wide ranging. The trust boundary between data and instructions is blurred. The output cannot be fully predicted. The model can be coerced into behaviours that the developer never intended. The tools the model can call become attack surfaces. The retrieval corpus becomes an instruction injection vector.

Traditional security frameworks like OWASP Top 10 for web applications still apply for the surrounding system. But the model itself introduces a new category that those frameworks do not cover well. That is why OWASP published the Top 10 for LLM Applications and why the NIST AI RMF includes security as a core trustworthiness characteristic.

The AI model is the only component in your system that is willing to follow instructions from anyone. Design accordingly.

The architect’s job is to draw the trust boundaries clearly, apply defence in depth at each boundary and assume that any individual control will eventually fail.

The OWASP Top 10 for LLM Applications has become the de facto reference for AI security risks. The 2025 revision covers the categories that any architect should design against.

I use the list as a review checklist for every AI architecture. For each risk, the design must show a primary control and at least one secondary control. The mantra is defence in depth, because each individual control has a non-trivial failure rate against motivated attackers.

The OWASP list is not exhaustive. There are emerging risks around multi-modal models, agent autonomy and multi-tenant data leakage that are not yet in the list but are worth considering. Architects should treat the list as a baseline, not a ceiling.

Prompt injection is the AI security problem that keeps me up at night, because there is no clean solution. The model cannot reliably distinguish between trusted instructions and untrusted content. Any text that reaches the model can be interpreted as an instruction.

Direct prompt injection happens when a user types instructions that override the system prompt. Indirect prompt injection is more dangerous: it happens when instructions are smuggled in through retrieved content, tool output, file uploads or even image content for multi-modal models.

The defences are a layered set, not a single fix:

• Separate trusted instructions from untrusted data using structured prompts and clear delimiters
• Strip or escape control sequences in untrusted content
• Use fine-tuned models that are more resistant to instruction overrides
• Apply output filtering and behavioural checks before downstream actions
• Reduce the blast radius of model actions through least privilege
• Implement human-in-the-loop for high-impact actions

The defence I have come to trust most is the principle that the model should never have authority over an action that the model alone can authorise. Whatever the model wants to do, a deterministic system must check it against rules and quotas before it happens.

Treat the model like an intern with administrator access. Helpful, eager, occasionally manipulated. Design the controls around that reality.

Prompt injection will not be solved by a single model improvement. It will be managed by architecture.

The second OWASP risk is sensitive information disclosure, and it shows up in three patterns I see often.

The first pattern is system prompt leakage. The system prompt often contains business logic, API keys, customer-specific configuration or behavioural rules that should not be visible. Users find ways to extract it. The defence is to keep secrets out of the system prompt and to treat the system prompt itself as semi-public.

The second pattern is training data leakage. Models can memorise training examples and reproduce them on prompting. For fine-tuned models, this is a serious concern if the training data contains personal data or proprietary information. The defence is data minimisation, differential privacy techniques where appropriate and explicit testing for memorisation.

The third pattern is retrieval-based leakage. RAG systems retrieve content based on the query, but the retrieval layer often does not enforce access control. A user can phrase a query in a way that retrieves content they should not be able to see. The defence is to enforce access control at retrieval time, filtering the index based on user identity before the model ever sees the content.

The control I always recommend is authorisation at retrieval, not at presentation. Filter what the user is allowed to see before retrieval, not after generation. By the time the model has generated a response, the data has already crossed the trust boundary.

The supply chain risk is growing as more organisations adopt open-source models, community fine-tunes and public datasets. The risks include compromised model weights, malicious code embedded in model loading scripts, poisoned training data and dependency vulnerabilities in the AI stack.

The defences are largely the same as for traditional software supply chain security:

• Use trusted sources for models and datasets
• Verify checksums and signatures where available
• Scan model files for known threats
• Pin versions and avoid implicit updates
• Maintain an inventory of every model, dataset and library
• Apply the same SBOM and SCA practices you use for traditional code

For fine-tuning, data provenance becomes a serious concern. Where did the training data come from? Who labelled it? Are there poisoning vectors in user-contributed data? The architect should design a curation pipeline that includes provenance metadata, quality checks and adversarial filtering.

For vendor models, the supply chain extends to the vendor’s own security posture. Contracts should require evidence of penetration testing, secure development practices and incident notification. This is increasingly standard in enterprise procurement.

The supply chain risk that catches teams out most often is the silent introduction of AI into existing SaaS products. Architects should extend procurement processes to flag new AI features and bring them into the inventory before they reach production.

Insecure output handling is the risk that downstream systems trust the model’s output too much. It is the AI version of injection attacks against traditional applications.

If the model can generate SQL and the application runs it, you have a SQL injection vector. If the model can generate shell commands and the application executes them, you have a remote code execution vector. If the model can generate URLs and the application fetches them, you have a server-side request forgery vector. If the model can generate HTML and the application renders it, you have a cross-site scripting vector.

The defence is to never trust the model’s output for security-sensitive actions. Treat the model as untrusted input to the rest of the system. Apply the same sanitisation, parameterisation and validation you would apply to any user input.

For structured outputs like JSON, use schema validation before any downstream system consumes the output. For free text rendered to users, apply the same XSS protections you would for any user-generated content. For actions that the model proposes, route them through a deterministic authorisation layer.

I have seen elegant architectures undone by a single point where the model’s output was treated as trusted code. Every output should be considered untrusted by default. The trust should be earned by validation, not assumed by source.

Model denial of service is a real and growing problem. Attackers can craft inputs that trigger expensive computation, exhaust context windows, force tool loops or simply rack up token costs that the victim has to pay.

The attack patterns include:

• Long-context inputs that maximise per-call cost
• Recursive tool calls that never converge
• Adversarial inputs that cause the model to retry repeatedly
• Bulk traffic from compromised accounts

The defences are familiar to anyone who has run a public API but need to be adapted to the AI context:

• Per-user rate limits based on requests, tokens and cost
• Per-tenant cost caps with circuit breakers
• Maximum context window enforcement
• Maximum tool call depth and breadth limits
• Timeout policies at every layer
• Cost-based throttling that adapts as load grows

Cost-based throttling is the AI-specific control that I now insist on. Track the token cost of every request, attribute it to a user or tenant, and enforce limits in real time. A free-tier user should not be able to spend more than a few dollars of token cost per day, regardless of how clever their prompts are.

If you do not have a per-user cost limit, you have an unbounded liability.

The same controls protect against accidental overspend by your own users, which is often a larger problem than malicious attacks.

Input validation and output filtering form the perimeter of an AI system. They do not solve every problem, but they catch many of the easiest attacks.

For input validation, I implement at least three layers:

• Format and length validation, rejecting inputs that exceed reasonable limits
• Content classification, detecting prompt injection patterns and known jailbreak attempts
• Domain-specific validation, ensuring the input matches the use case

For output filtering, I implement at least three layers:

• Schema validation for structured outputs
• Content safety classification, detecting harmful or out-of-scope content
• Domain-specific validation, ensuring the output is plausible for the use case

These layers can be built with a combination of rule-based checks, classifier models and the LLM itself. A common pattern is to run a small classifier model alongside the main model, with the classifier acting as a gate for both input and output.

The vendor landscape includes options like Lakera, Protect AI, Microsoft Prompt Shields, AWS Bedrock Guardrails and various open-source projects. Each has different strengths. I usually combine a vendor product with custom rules that encode the specific risks of the use case.

Input validation and output filtering are necessary but not sufficient. They catch the bulk of attacks but are bypassed by sophisticated ones. The deeper controls of least privilege, sandboxing and human oversight matter more for high-risk systems.

When the model can call tools, the tool layer becomes the most important security boundary. Every tool the model can invoke is a capability the attacker can potentially leverage.

The design principles I now insist on:

• Each tool has a clearly defined scope, with no implicit access to other resources
• Tools authenticate as the model with limited privileges, not as the user with full privileges
• Tools enforce authorisation based on the user identity carried through the request
• Tools have explicit allow lists for arguments rather than implicit permissions
• Tools are sandboxed at the network and filesystem level
• Tools have their own rate limits and cost limits

The principle of least privilege is more important here than in any other part of the system. The model will request capabilities it should not have. The tool must refuse those requests.

I think of tools in three tiers. Read-only tools that return public information are the lowest risk. Read-only tools that access user-specific data are medium risk. Tools that take actions on behalf of the user are high risk and should require additional validation, often including a human confirmation step for irreversible actions.

For agent systems, the tool design is the dominant security concern. An agent with broad tool access is effectively a user, and any compromise of the agent is a compromise of those tools. I push hard for narrow, well-scoped tools rather than general-purpose interfaces.

Beyond cost-based throttling, AI systems need the same abuse prevention measures as any public-facing API, plus AI-specific extensions.

The baseline includes per-IP, per-user and per-tenant rate limits, CAPTCHA or similar friction for anonymous traffic, anomaly detection on usage patterns and reputation-based throttling. These are well understood for traditional APIs and the same patterns apply.

The AI-specific extensions include:

• Token cost limits in addition to request limits
• Tool call depth and breadth limits
• Context length limits per request
• Cumulative cost limits per session
• Behavioural quotas that limit certain kinds of action regardless of cost

Behavioural quotas are an underused control. For example, an agent should not be allowed to send more than five emails in a session, even if its rate limits permit more. This kind of action-level quota catches abuse that resource-level quotas miss.

Abuse detection should also include AI-specific signals. Rapid sequences of prompt injection attempts. Unusual tool call patterns. Outputs flagged repeatedly as unsafe. These signals feed into the same abuse pipelines as traditional API abuse.

The architect should also design for graceful degradation. When the system is under attack, it should reject suspicious traffic without taking down legitimate users. Circuit breakers, priority queues and bot detection layers are all part of the toolkit.

Red teaming for AI systems is different from traditional security testing. The attack surface is the model’s behaviour, not just its interfaces.

I structure AI red teaming around three workstreams.

The first workstream is policy red teaming. The team tries to make the system violate its stated policies. Generate prohibited content, give regulated advice it should refuse, leak information it should protect. The output is a list of policy bypasses and recommended mitigations.

The second workstream is technical red teaming. The team tries to exploit the system as a piece of software. Prompt injection through every input vector, indirect injection through retrieved content, tool abuse, output handling attacks. The output is a list of technical vulnerabilities.

The third workstream is operational red teaming. The team tries to disrupt or exploit the system in production. Cost attacks, denial of service, abuse pattern probing. The output is a list of operational weaknesses.

The team should include diverse expertise. Application security testers for the technical workstream. Domain experts for the policy workstream. Reliability engineers for the operational workstream. The model itself can be used as an attacker, generating adversarial prompts at scale.

An AI red team that has not made your system misbehave is not trying hard enough.

The findings should feed back into the design, the controls and the evaluation suite. Every confirmed attack becomes a permanent test case that the system must pass on every release.

Beyond the application layer, AI systems need secure deployment patterns. These follow well-established security principles, adapted to the AI context.

The deployment patterns I now consider standard:

• Dedicated infrastructure tier for AI workloads, separate from general applications
• Network segmentation between the model inference layer and other systems
• Egress filtering to prevent the model from reaching arbitrary external systems
• Secrets management that keeps credentials out of prompts and out of model context
• Encryption in transit and at rest for all training and inference data
• Tenant isolation for multi-tenant systems, with hard boundaries between tenants
• Logging and monitoring that captures security-relevant events

For self-hosted models, the inference servers should be hardened, patched and isolated. For vendor models, the contract should specify data handling, residency, retention and incident notification. For hybrid deployments, the boundary between self-hosted and vendor components needs explicit security review.

The deployment should also be designed for safe rollback. Every model version, prompt version and tool configuration should be feature-flagged so it can be turned off quickly. The blast radius of a misbehaving deployment is much larger for AI systems than for traditional ones, because the bad behaviour scales with traffic.

I also recommend a kill switch at the platform level, accessible to security and risk teams without requiring engineering involvement. When something is going wrong, you want the ability to stop traffic immediately.

Identity is the foundation of AI security. The system must know who is making each request, what they are authorised to do and how their data should be handled.

For single-tenant systems, identity flows through standard authentication and authorisation patterns. The model receives the user identity as part of the request context, and tools enforce permissions based on that identity. Logging captures who did what.

For multi-tenant systems, the design is more involved. Each tenant’s data must be isolated at every layer: retrieval indexes, fine-tuned models, prompts, logs and evaluations. The model itself does not enforce tenant boundaries, so the surrounding architecture must.

I have seen multi-tenant systems where retrieval indexes were shared across tenants, with tenant filtering applied only at query time. That is a fragile pattern. A bug in the filter, a misconfigured query or a clever prompt can expose other tenants’ data. I now insist on separate indexes per tenant, even if it is more expensive, because the security guarantee is much stronger.

The same principle applies to fine-tuned models. A model fine-tuned on one tenant’s data should not be served to another tenant, even if the data was de-identified. Memorisation effects mean that the boundary cannot be fully relied upon.

For prompts and configurations, tenant-specific values should never appear in shared system prompts. Use tenant context that is loaded dynamically based on the authenticated identity.

Security monitoring for AI systems extends the standard observability stack with AI-specific signals.

The events I always capture include:

• Every request and response, with model and prompt versions
• Every tool call, with arguments and results
• Every input that was flagged by validation
• Every output that was flagged by filtering
• Every authentication and authorisation event
• Every rate limit and quota breach

These events feed into the same SIEM and incident management systems as traditional security events. AI-specific detection rules look for prompt injection patterns, abnormal tool call sequences, cost anomalies and behavioural shifts.

Incident response for AI security incidents requires playbooks that are slightly different from traditional ones. Common AI incident types include data leakage through model outputs, prompt injection that triggered an unintended action, vendor model behaviour change that bypassed safety controls and abuse of generative capabilities to produce harmful content.

Each playbook covers detection, containment, eradication and recovery. Containment often involves a kill switch on the affected use case. Eradication often involves a prompt update or model rollback. Recovery includes communication to affected users and update of the evaluation suite to prevent recurrence.

Every AI security incident should result in a permanent test case. The evaluation suite is your collective memory of what attackers have tried.

Post-incident, the findings should feed into the red team programme, the design standards and the governance documentation. AI security maturity is built through this feedback loop.