AI Security for Security Teams培訓

Module 1 — AI Systems for Security Engineers

Lab: Lab 01 — 01-Introduction

Understanding the architecture.

Topics:

• LLMs vs normal apps
• AI inference pipelines
• Prompt flow
• RAG architecture
• embeddings/vector databases
• agentic workflows
• tool calling
• AI gateways
• copilots
• MCP and agent protocols
• where WAF visibility exists
• where WAF visibility disappears

Key insight: Traditional WAFs often lose visibility after the prompt reaches the model.

Module 2 — OWASP GenAI Top 10

Lab: none — interactive recap/discussion

Core AI attack categories.

Topics: 

• Prompt Injection
• Insecure Output Handling
• Training Data Poisoning
• Model DoS
• Supply Chain Vulnerabilities
• Sensitive Information Disclosure
• Excessive Agency
• Vector/Embedding Weaknesses
• Misinformation
• Unbounded Consumption

Include:

• Differences from classic OWASP
• Mapping to defensive controls (WAF, gateway, app-layer)
• Where each control helps
• Where each control fails

Module 3 — Prompt Injection Detection

Lab: Lab 02 — 02-Prompt-Injection

The “SQL injection moment” for AI.

Topics:

• Direct prompt injection
• Indirect prompt injection
• Hidden instructions
• Document-based attacks
• HTML/Markdown injection
• Jailbreak patterns
• Context override attacks
• Role confusion attacks

Detection strategies:

• keyword heuristics
• semantic classification
• prompt linting
• instruction boundary enforcement
• allow/deny policies
• AI-aware regex patterns

Hands-on labs:

• Attack a chatbot
• Bypass naïve filters
• Build layered detection

Module 4 — AI-Aware WAF Rules

Lab: Lab 03 — 03-WAF-Basics

How WAF rules evolve for AI systems.

• Topics:
• protecting LLM endpoints
• inference API protection
• token-aware rate limiting
• prompt size inspection
• AI-specific signatures
• conversation anomaly detection
• multi-turn abuse patterns
• model enumeration attempts
• inference scraping
• denial-of-wallet protection

Examples:

• protecting /v1/chat/completions
• defending streaming APIs
• blocking recursive agent calls

Module 5 — Securing RAG Pipelines

Lab: Lab 04 — 04-RAG-Security

One of the biggest new attack surfaces.

Topics:

• vector DB threats
• embedding poisoning
• malicious PDFs/docs
• retrieval manipulation
• semantic poisoning
• hidden instructions in documents
• cross-document contamination
• data exfiltration via retrieval

Defenses:

• ingestion sanitization
• trust scoring
• metadata isolation
• document provenance
• retrieval policies
• segmentation

Case study: “Upload a poisoned PDF and take over the AI assistant.”

Module 6 — Agentic AI Security

Lab: Lab 05 — 05-Agent-Security

Where things become dangerous.

Topics:

• excessive agency
• tool abuse
• API chaining
• autonomous loops
• permission escalation
• memory poisoning
• indirect tool execution
• agent impersonation
• credential leakage
• multi-agent attacks

Defenses:

• least privilege for agents
• approval gates
• runtime policy engines
• sandboxing
• scoped credentials
• tool whitelisting
• human-in-the-loop

This is the section managers usually care about most because the risk becomes operational and business-impacting.

Module 7 — API Security for AI

Lab: Lab 06 — 06-Denial-of-Wallet

AI systems are API-heavy.

Topics:

• API gateways
• GraphQL AI risks
• MCP/API abuse
• JWT protection
• AI plugin security
• agent authentication
• delegated authorization
• secret management
• signed prompts
• API inventory for AI

Tie into: OWASP API Security Top 10

Module 8 — Detection Engineering & SOC Integration

Lab: Lab 07 — 07-Detection

Operational defense.

Topics:

• AI telemetry
• prompt logging
• token analytics
• anomaly detection
• semantic SIEM pipelines
• AI attack indicators
• threat hunting for LLM abuse
• AI runtime observability

Examples:

• detecting jailbreak campaigns
• spotting automated agent abuse
• identifying model scraping

Module 9 — Cloud WAFs and AI Security

Lab: none — interactive recap/discussion

Vendor-specific implementations.

Topics:

• AWS WAF for AI APIs
• Azure WAF
• Cloudflare AI Gateway
• API gateways
• Envoy AI filtering
• Kong AI Gateway
• NGINX AI security patterns

Comparison:

• traditional WAF vs AI gateway vs app-layer guardrail
• proxy-based vs semantic filtering

Module 10 — Building a Layered AI Defense

Lab: Lab 08 — 08-Layered-Defense

Important philosophical conclusion:

No single layer can secure AI (a WAF least of all, on its own).

Students build a layered model:

1. WAF
2. API gateway
3. AI gateway
4. Guardrails
5. Runtime monitoring
6. Identity/authorization
7. Sandbox
8. Human approval
9. Observability
10. Incident response

This aligns strongly with the “multi-layer security” model.

Module ↔ Lab map

Labs run in lab order, which follows module order.

The course has 10 modules but 8 labs: Modules 2 and 9 are interactive recap/discussion and have no lab.

Each lab is tagged with its module throughout this outline.

• Lab 01 (Module 1)
  • Folder: 01-Introduction
  • Title: Explore an AI system — what's on the wire
• Lab 02 (Module 3)
  • Folder: 02-Prompt-Injection
  • Title: Attack a chatbot & bypass naïve filtering
• Lab 03 (Module 4)
  • Folder: 03-WAF-Basics
  • Title: Build AI-aware WAF rules
• Lab 04 (Module 5)
  • Folder: 04-RAG-Security
  • Title: Poison a RAG pipeline
• Lab 05 (Module 6)
  • Folder: 05-Agent-Security
  • Title: Secure an autonomous agent
• Lab 06 (Module 7)
  • Folder: 06-Denial-of-Wallet
  • Title: Detect denial-of-wallet attacks
• Lab 07 (Module 8)
  • Folder: 07-Detection
  • Title: Monitor AI abuse patterns in logs
• Lab 08 (Module 10)
  • Folder: 08-Layered-Defense
  • Title: Build a layered AI defense architecture

Capstone

Students defend a simulated enterprise AI assistant.

Attackers attempt: 

1. prompt injection
2. tool abuse
3. credential theft
4. retrieval poisoning
5. excessive API consumption
6. agent escalation

Teams build:

• WAF rules
• AI gateway policies
• runtime detection
• guardrails
• incident response