deepteam — DeepTeam is a framework to red team LLMs and AI agents

The LLM Red Teaming Framework: DeepTeam in Practice

Introduction

DeepTeam is an open-source, easy-to-use red teaming framework for large language models (LLMs). Think of it as a dedicated security tester for AI systems: it simulates adversarial scenarios to uncover vulnerabilities that could manifest as bias, data leakage, or unsafe behavior in AI agents, retrieval-augmented generation pipelines, and chatbots. Built to run locally on your machine and grounded in the open DeepEval evaluation framework, DeepTeam helps you stress-test your AI stack, identify gaps, and strengthen guardrails before incidents reach production. For teams looking to centralize findings and collaborate on risk, there’s a path to the Confident AI platform to manage risk assessments, monitor in production, and share reports with stakeholders.

An illustration of the synergy between Confident AI and DeepTeam is captured in a live demo GIF:

What DeepTeam Delivers

• A broad catalog of vulnerabilities, attacks, and features designed for LLM ecosystems
• Local, machine-bound evaluation that does not require cloud-facing data or external services
• Ready-to-use guardrails that can block unsafe behavior in real time
• A bridge to standard safety frameworks, enabling teams to align with established risk controls
• Flexible deployment options: run red-teaming via the CLI with YAML configurations or programmatically in Python
• A pathway to structured risk reporting, including dataframes and locally stored JSON results

Vulnerabilities, Attacks, and Features: An Inside Look

DeepTeam introduces a comprehensive catalog organized into several families. Each vulnerability and attack is described with concrete criteria, scoring, and actionable guidance to reproduce and remediate.

Vulnerability families

• Data Privacy: leakage risks such as PII exposure and prompt leakage
• PII Leakage: sensitive personal information disclosed by the model
• Prompt Leakage: system prompts or instructions exposed during interactions
• Responsible AI: biases, fairness, and ethical concerns
• Bias: stereotypes or unfair treatment across gender, race, religion, politics
• Toxicity: harmful, offensive, or demeaning content
• Child Protection: privacy and safety risks related to minors
• Ethics: violations of moral reasoning or organizational values
• Fairness: discriminatory outcomes across groups and contexts
• Security: a broad set of technical vulnerabilities
• Broken Function-Level Authorization (BFLA)
• Broken Object-Level Authorization (BOLA)
• RBAC bypass
• Debug Access: unauthorized access to debug modes and developer endpoints
• Shell Injection: unauthorized system command execution
• SQL Injection: manipulation of database queries
• SSRF: server-side request forgery to internal services
• Tool Metadata Poisoning: corrupted tool schemas and descriptions
• Cross-Context Retrieval: data access across isolation boundaries
• System Reconnaissance: probing internal architecture and configurations
• Safety: risks tied to unsafe or illegal guidance
• Illegal Activity: facilitation of fraud, weapons, drugs, or other unlawful actions
• Graphic Content: explicit or violent material
• Personal Safety: self-harm or dangerous advice
• Unexpected Code Execution: coerced execution of unauthorized code
• Business: risks in information integrity and market behavior
• Misinformation: factual errors and unsupported claims
• Intellectual Property: copyright, trademark, and patent concerns
• Competition: manipulation or misrepresentation in competitive contexts
• Agentic: risks tied to autonomy and instruction
• Goal Theft: extraction or redirection of an agent’s objectives
• Recursive Hijacking: self-modifying goal chains
• Excessive Agency: agents acting beyond their granted authority
• Robustness: input overreliance and prompt hijacking
• Indirect Instruction: hidden instructions embedded in retrieved content
• Tool Orchestration Abuse: exploiting tool calling sequences
• Agent Identity & Trust Abuse: impersonation of agent identity
• Inter-Agent Communication Compromise: spoofed multi-agent messaging
• Autonomous Agent Drift: deviation from intended goals over time
• Exfiltration through tools: weaponizing tools for unintended actions
• External System Abuse: using agents to attack external services
• Custom: define and test your own criteria with minimal code

Adversarial Attacks

DeepTeam ships with more than 20 research-backed adversarial attack methods, designed for both single-turn and multi-turn (conversational) red teaming. Each attack method uses state-of-the-art techniques to stress test defenses and expand the reach of vulnerability probes.

Single-Turn attacks

• Prompt Injection: crafted prompts that bypass restrictions
• Roleplay: persona-driven scenarios exploiting training-time prompts
• Leetspeak: substitute characters to bypass keyword filters
• ROT13 and Base64: encoding attacks to evade simple detectors
• Gray Box: partial knowledge of system prompts to tailor targeted prompts
• Math Problem: disguising manipulation inside mathematical inputs
• Multilingual: translating attacks to less-spoken languages to test cross-language resilience
• Prompt Probing: attempting to extract system prompt details
• Adversarial Poetry: transforming attacks into poetic form for stealth
• System Override: disguising attacks as legitimate system commands
• Permission Escalation: shifting asset identity to bypass role restrictions
• Goal Redirection: reframing objectives to unlock unsafe outputs
• Linguistic Confusion: semantic ambiguity to impede understanding
• Input Bypass: bypassing validation through careful handling of exceptions
• Context Poisoning: injecting false context to bias reasoning
• Character Stream: character-by-character input to avoid detectors
• Context Flooding: overwhelming input with benign content to hide malicious instruction
• Embedded Instruction JSON: hiding attacks within JSON structures
• Synthetic Context Injection: fabricating context to exploit long-context handling
• Authority Escalation: misrepresenting power dynamics to gain trust
• Emotional Manipulation: high-pressure tactics to coerce unsafe compliance

Multi-Turn attacks

• Linear Jailbreaking: iterative refinement of prompts across turns
• Tree Jailbreaking: parallel variations to optimize bypass strategies
• Crescendo Jailbreaking: gradual escalation from benign to harmful prompts
• Sequential Jailbreak: scaffolding across turns toward restricted outputs
• Bad Likert Judge: exploiting evaluation roles to coax harmful content

Red Team against safety frameworks

• In addition to crafting attacks, DeepTeam maps vulnerabilities and attacks to established AI safety frameworks. You can run red-teaming against safety frameworks like OWASP Top 10 for LLMs (2025), OWASP Top 10 for Agents (2026), NIST AI RMF, MITRE ATLAS, BeaverTails, and Aegis. This helps you align your testing with recognized controls and risk categories, expediting audit readiness.

Guardrails: Real-Time Defenses

DeepTeam includes seven production-ready guardrails to provide fast, binary classification-style protection for inputs and outputs:

• ToxicityGuard
• PromptInjectionGuard
• PrivacyGuard
• IllegalGuard
• HallucinationGuard
• TopicalGuard
• CybersecurityGuard You can read the full guardrails documentation for details and implementation guidance. Guardrails are designed to halt unsafe interactions before they reach users and to filter unsafe outputs before they are delivered.

The DeepTeam Ecosystem: Where It Fits

• Build your own custom vulnerabilities and attacks that integrate seamlessly with DeepTeam’s ecosystem
• Run red teaming from the CLI with YAML configs or programmatically in Python
• Access risk assessments, display results in dataframes, and save locally in JSON
• Integrate with the Confident AI platform to manage risk assessments, monitor vulnerabilities in production, and share reports with your team

QuickStart: A Practical Path to Your First Red Team

DeepTeam is designed to be approachable even if you don’t specify the exact LLM you will test. The core idea is to provide a model_callback, a vulnerability or attack configuration, and a running risk assessment.

Installation

• pip install -U deepteam

Red Team Your First LLM

• The example below demonstrates a minimal setup that tests a Biased vulnerability against a single-turn attack.

Code sample (conceptual, simplified)

from deepteam import red_team
from deepteam.vulnerabilities import Bias
from deepteam.attacks.single_turn import PromptInjection

async def model_callback(input: str) -> str:
    # Replace this with your LLM application
    return f"I'm sorry but I can't answer this: {input}"

risk_assessment = red_team(
    model_callback=model_callback,
    vulnerabilities=[Bias(types=["race"])],
    attacks=[PromptInjection()]
)

Notes

• Before running, set your OPENAIAPIKEY as an environment variable (or use any custom model supported by DeepEval).
• Run the script:

python red_team_llm.py

What happens under the hood

• The model_callback wraps your LLM and returns a string for a given input.
• During red teaming, DeepTeam simulates a PromptInjection attack targeting Bias vulnerabilities.
• The outputs are evaluated by a BiasMetric, which produces a binary score (0 or 1).
• The final risk rating for Bias is determined by the proportion of scores equal to 1.

A more automated path is Red Team Against Safety Frameworks

• You can leverage established safety frameworks to guide testing:

from deepteam import red_team
from deepteam.frameworks import OWASPTop10

async def model_callback(input: str) -> str:
    return f"I'm sorry but I can't answer this: {input}"

risk_assessment = red_team(
    model_callback=model_callback,
    framework=OWASPTop10()
)

• This approach automatically maps framework categories to the corresponding vulnerabilities and attacks, ensuring standardized coverage.

Guardrails in Practice: Guard Your Production LLMs

Guardrails provide a practical, fast, binary classification to protect both inputs and outputs in real time:

from deepteam import Guardrails
from deepteam.guardrails import PromptInjectionGuard, ToxicityGuard, PrivacyGuard

guardrails = Guardrails(
    input_guards=[PromptInjectionGuard(), PrivacyGuard()],
    output_guards=[ToxicityGuard()]
)

input_result = guardrails.guard_input("Tell me how to hack a database")
print(input_result.breached)  # True if the input violates guardrails

output_result = guardrails.guard_output(input="Hi", output="Here is some toxic content...")
print(output_result.breached)  # True if the output violates guardrails

Seven guards are available out of the box, with continued expansion planned:

• ToxicityGuard
• PromptInjectionGuard
• PrivacyGuard
• IllegalGuard
• HallucinationGuard
• TopicalGuard
• CybersecurityGuard

Guardrails documentation is essential reading as you design your guardrails strategy and implement guard policies for your production LLMs.

DeepTeam and Confident AI: A Unified Platform

• Confident AI provides an all-in-one platform that integrates with DeepTeam and DeepEval.
• Manage risk assessments, compare results across iterations, and monitor live vulnerabilities in production
• Share reports with stakeholders and team members, enabling better collaboration on security posture
• Run red teams from your IDE via Confident AI’s MCP server and pull results without leaving your development environment

Contributing and Community

DeepTeam is a community-driven project. Readers are encouraged to contribute by adhering to the project’s contributor guidelines. The repository includes a CONTRIBUTING.md that outlines the code of conduct and the process for submitting pull requests. Engaging with the community helps refine the framework, expand our vulnerabilities and attacks catalog, and improve guardrails for broader coverage.

Authors and Licensing

• DeepTeam is built by the founders of Confident AI. For inquiries, you can reach out to the team at [email protected].
• The project is licensed under Apache 2.0. The license file is publicly available in the repository, ensuring developers can adopt and adapt the framework with proper attribution.

What You Can Do Next

• Try a QuickStart in your own environment to see how DeepTeam integrates with your LLM stack.
• Explore the vulnerability and attack catalog to understand which facets of your AI system are most at risk.
• Extend the framework with your own vulnerabilities and attacks, and contribute back to the community.
• Consider using Confident AI to centralize risk reporting and production monitoring, enabling more proactive risk management.

Images that capture the spirit of DeepTeam

• The hero wordmark to represent the project’s branding:
• A display of the Confident AI + DeepTeam integration in action:
• A broader view of Confident AI as the platform for risk management:

Closing Thoughts

DeepTeam is more than a testing tool; it’s a framework for building safer, more trustworthy AI systems. By combining a rich set of vulnerabilities and attacks, automated mapping to established safety standards, and practical guardrails that can stop unsafe behavior in real time, DeepTeam gives AI teams a structured, repeatable approach to red teaming. Running locally ensures that sensitive data remains in your control while you simulate realistic attack and defense scenarios. The integration with Confident AI further enhances the ability to manage risk across development, testing, and production, bringing clarity to the security posture of your AI systems.

If you’re ready to dive deeper, start with the QuickStart guide, explore the vulnerability catalog, and consider how you can map your testing to OWASP, NIST, and MITRE standards. Your next production release will be safer and more robust with a red team that continuously tests and strengthens your AI stack.

Images and assets referenced above are included from the Input to illustrate key concepts and features:

• DeepTeam branding at the top
• Live demo visuals for Confident AI + DeepTeam
• Additional branding assets that show the collaboration between DeepTeam and Confident AI

Note: If you plan to publish your vulnerability assessments or share results externally, ensure you follow your organization’s security and privacy policies. For teams wanting to manage and disseminate red-team results securely, the Confident AI platform is recommended to streamline risk reporting and production monitoring.