AITM01: AI Threat Modeling Framework
Securing an AI/LLM deployment means thinking end to end, from the moment a user submits a prompt to how models and data are stored, served, and retired. This interactive threat model is built around four foundational pillars, each targeting a different vantage point on your system
Use Case:
Deploying a publicly accessible AI-powered Chatbot (LLM-based).
Step-by-Step Walkthrough:
Step 1: Select AI System Components
- Identify all AI System components involved in the deployment.
Example:
-
Application Layer
- (Frontend and API Gateway) – Public entry points.
-
Model Layer
- (Serving Runtime) – Generates chatbot responses.
-
Infrastructure
- (Network & Host) – Supports secure communication and hosting.
Clearly defines your scope of analysis, ensuring no critical areas are missed.
Step 2: Identify Threats
- For each chosen component, pick potential threats from your framework.
Example:
| Component | Threat | Explanation of threat |
|---|---|---|
| Application/API Gateway | Spoofing | Attackers impersonating legitimate users via stolen tokens or cookies. |
| Application/API Gateway | Tampering | Attackers altering requests to trigger unintended model behavior. |
| Model Layer | Information Disclosure | Sensitive data being leaked in responses or logs. |
| Infrastructure (Network) | Denial of Service (DoS) | Attackers flooding the system to disrupt legitimate services. |
One must clearly understand threats relevant to your scenario before applying controls.
Step 3: Identify Relevant Controls
- Use your Controls Catalog to mitigate each threat explicitly.
Example:
| Threat | Controls Applied | Why this Control? |
|---|---|---|
| Spoofing | JWT Validation (LLM01), Rate Limiting (LLM01) | Ensures user authenticity; limits spoof attempts. |
| Tampering (Prompt Injection) | Input Sanitization, Context Separation (LLM01) | Prevents malicious input altering model behavior. |
| Information Disclosure | Output Sanitization, Data Masking (LLM02, LLM06) | Ensures sensitive data isn't exposed to users. |
| Denial of Service | Throttling, Autoscaling (LLM04) | Ensures service remains available during traffic spikes. |
Applying the correct controls minimizes the likelihood and impact of each identified threat.
Step 4: Map and Analyze Data Flows
- Identify which Data Flows relate directly to these components.
Example:
- F8 (Inference Request): Client → Prediction API
- F9 (Inference Response): Prediction API → Client
Analysis:
- Ensure all data inputs (F8) are strictly validated.
- Confirm responses (F9) undergo sanitization to avoid leaks.
Explicitly understanding data flow helps ensure each step handles data securely, reducing exploitation opportunities.
Step 5: Clarify Trust Boundaries
- Define relevant Trust Boundaries clearly and secure transitions between boundaries.
Example:
-
TB1 (Internet → API Gateway):
- HTTPS encryption; authentication enforced.
-
TB8 (Client → Prediction API):
- API keys or JWT tokens validated to authenticate legitimate users.
Explicitly marking trust boundaries helps manage risks associated with external interactions clearly.
Step 6: Detailed Threat-Control Mapping
- Clearly map each threat explicitly to the control measures.
Example of Detailed Threat-Control Mapping:
| Component | Threat | Control Applied | How this mitigates the threat |
|---|---|---|---|
| Application/API Gateway | Prompt Injection | Input Sanitization (LLM01) | Filters inputs, preventing harmful manipulation. |
| Application/API Gateway | Spoofing | JWT Validation, Context Separation (LLM01) | Ensures requests are from genuine authenticated users. |
| Model Layer | Information Disclosure | Data Masking, Output Filtering (LLM02, LLM06) | Removes sensitive data before sending to users. |
| Infrastructure | Denial of Service | Rate Limiting, Circuit Breaker (LLM04) | Keeps service available during overload attempts. |
This explicit mapping clarifies your defensive strategy clearly, ensuring stakeholders understand the importance of each security measure.
Step 7: Document Clearly
- Summarize your entire threat modeling activity clearly for future reference and audits.
Example Documentation Summary:
Chatbot Deployment Security Summary
Components analyzed: Application (API Gateway), Model Layer (Serving Runtime), Infrastructure (Network).Identified and mitigated threats explicitly include:
- Prompt Injection: Prevented by sanitizing and validating inputs.
- Spoofing: Mitigated via JWT authentication and strict request validation.
- Information Disclosure: Addressed via output sanitization and masking sensitive data.
- Denial of Service: Managed by throttling requests, autoscaling infrastructure, and implementing circuit breakers.
Trust boundaries (Internet → API Gateway, Client → Prediction API) have been secured with TLS and authentication mechanisms.
Good documentation simplifies auditing, training, and continuous improvement of your security posture.
Why is this structured approach effective?
- Clearly links components → threats → controls → data flows → trust boundaries explicitly.
- Ensures threats are not overlooked.
- Provides reasoning behind each security measure clearly.
- Simplifies the audit and management processes, ensuring your security posture remains strong and documented.