🔴 LLM Red Teaming

A modular, extensible toolkit for adversarial testing of large language models and NLP systems.

Adversarial text attacks · prompt injection · jailbreaking · fairness probing · pluggable model targets

---

📖 Overview

Modern AI systems are increasingly deployed in sensitive contexts — yet their robustness to adversarial inputs remains poorly understood. LLM Red Teaming provides a structured, reproducible framework to:

• Attack language models at multiple levels: character, word, sentence, semantic, and prompt
• Jailbreak instruction-tuned LLMs using standardised benchmarks and custom templates
• Evaluate robustness metrics: accuracy drop, attack success rate (ASR), stealth score, risk score
• Flag high-risk adversarial examples for human review with priority queuing
• Align every evaluation to industry standards: MITRE ATLAS, NIST AI RMF, NIST AI 600-1, OWASP LLM Top 10, EU AI Act

Where this toolkit fits

Adversarial ML attacks span the whole pipeline — training data, the model, its inputs, and its outputs. This toolkit now covers most of that surface: input attacks (evasion, jailbreak, prompt injection, reasoning robustness), output/data attacks (sensitive-data disclosure, PII/memorization extraction, RAG exfiltration), and agentic tool hijacking — with data poisoning, membership inference, and model extraction remaining on the roadmap:

Scope — model-level by default. The notebooks probe the model (plus the vendor's platform content filter), which is the right unit for model assurance and a conservative upper bound on risk. To test a deployed application — with its own system prompt, guardrails, retrieval, and tools — point the harness at the app via ApplicationTarget and measure what the guardrails catch (the delta). See model-level vs application-level testing.

📐 Methodology: Industry alignment · Dataset strategy · Roadmap — future attacks, datasets & testing strategies

---

🧭 Workstreams

Each workstream is a code-light demo notebook backed by reusable modules. Status, notebook, and full write-up (results, methodology, regulatory mapping) for each:

#	Workstream	Status	Notebook	Full Results & Design
01	🧬 Adversarial NLP	✅ Complete	notebook	docs/01
02	🔓 Jailbreaking	✅ Complete	notebook	docs/02
03	💉 Prompt Injection	✅ Complete	notebook	docs/03
04	⚖️ Bias & Fairness	✅ Complete	notebook	docs/04
05	🧩 NLI Robustness	✅ Complete	notebook	docs/05
06	🔐 Data Red-Teaming	✅ Complete	notebook	docs/06
07	🤖 Agentic Tool Attacks	🛠️ Built — run pending	notebook	docs/07

Notebooks are intentionally code-light — they import from the modules below and focus on results, visualisations, and interpretation.

---

🗂️ Repository Structure

llm_red_teaming/
│
├── attacks/                    # All attack implementations
│   ├── character/ word/ sentence/ semantic/ structural/   # NB01 perturbations   [✅]
│   ├── jailbreak/              # JailbreakBench + PAIR artifact runners, HarmBench [✅]
│   ├── prompt/                 # Prompt injection (direct + indirect)             [✅]
│   ├── fairness/                # BBQ + counterfactual fairness probes            [✅]
│   ├── robustness/             # NLI runner + MultiNLI/ANLI/AdvGLUE               [✅]
│   ├── data/                   # Disclosure, memorization (+Enron), exfiltration  [✅]
│   └── agent/                  # Tool-using agent sandbox + attacks               [🛠️]
│
├── judges/                     # Response evaluation — rule-based + BART-MNLI + LLM-as-judge
├── targets/                    # Pluggable model connectors — OpenAI-compatible, Azure, ApplicationTarget
├── evaluate/                   # Metrics & reporting — ASR, risk score, regulatory mapping, executive reports
├── eval_datasets/               # Cached evaluation datasets (SST-2, JailbreakBench, HarmBench, BBQ, NLI, …)
├── notebooks/                  # The 7 demo notebooks (see Workstreams table above)
├── docs/                       # Per-workstream results & deep dives (see table above) + roadmap/methodology
│
├── configs/                    # Experiment configuration files
├── results/                    # Output files (gitignored)
├── .env.example                # API key template
├── requirements.txt
└── LICENSE

---

🚀 Quick Start

git clone https://github.com/minw0607/llm_red_teaming.git
cd llm_red_teaming
pip install -r requirements.txt
cp .env.example .env          # fill in your Azure OpenAI credentials

Open any notebook in notebooks/ and set the parameters in its config cell — everything else runs end-to-end.

# Programmatic usage
from attacks.character import TextBugger
from attacks.word import TextFooler
from targets.azure_openai import AzureOpenAITarget
from evaluate import run_all_attacks, compute_attack_summary

attacks = {"TextBugger": TextBugger(), "TextFooler": TextFooler()}
target  = AzureOpenAITarget()
results = run_all_attacks(attacks, target, eval_df, n_samples=50)
summary = compute_attack_summary(results)

---

🧬 Adversarial NLP (Notebook 01)

✅ Complete

10 black-box attacks across 5 perturbation levels (character → structural) test how much text perturbation degrades a model's accuracy on SST-2 sentiment classification.

Headline (GPT-5-4 via Azure, n=872): NegationInjection dominates — a 17.5% accuracy drop at 0.941 stealth, 5× the next attack, and undetectable by perplexity monitors. Character-level attacks are effectively neutralised at frontier scale.

📄 Full results, risk matrix, executive report →

---

🔓 Jailbreaking (Notebook 02)

✅ Complete

Tests whether harmful-intent prompts bypass safety alignment, using JailbreakBench (100 behaviors + PAIR artifacts) and HarmBench (400 behaviors) across direct goals, artifact templates, and PAIR transfer — scored by a BART-MNLI classifier or LLM-as-judge, with StrongREJECT graded scoring.

Headline (GPT-5-4 via Azure, 172 prompts): the model held firm — 0% ASR on direct and template-wrapped attacks, a single borderline PAIR-transfer case, and 0 violations on a HarmBench cross-check (130 harder CBRN/illegal/misinformation prompts).

📄 Full results, both datasets, StrongREJECT, regulatory mapping →

---

💉 Prompt Injection (Notebook 03)

✅ Complete

Tests whether adversarial instructions override the system prompt or hijack behaviour — directly (user input) and indirectly (content the model retrieves) — using the Open-Prompt-Injection taxonomy (5 strategies) and real-world payloads from deepset/prompt-injections. Success is measured deterministically via canary detection.

Headline (GPT-5-4 via Azure, 280 attempts): 0% indirect override (held across every strategy); a 4.6% overall override rate, with the meaningful signal being ~4% partial compliance on real-world payloads.

📄 Full results, attack vectors, canary methodology, regulatory mapping →

---

⚖️ Bias & Fairness (Notebook 04)

✅ Complete

Unlike NB01–03, bias is a harm, not an attack — there's no adversary; the model exhibits disparate behaviour on its own. Two methods: BBQ (does the model fall back on stereotypes when underdetermined?) and counterfactual probes (does swapping a protected attribute flip a hiring/lending/housing decision?).

Headline (GPT-5-4 via Azure, 440 BBQ items + 64 counterfactual checks): 99.5% ambiguous accuracy, 0% decision-flip rate, but 2/440 BBQ answers were wrong and stereotype-aligned (incl. a pregnancy-discrimination concern) — low but non-zero.

📄 Full results, worked examples, regulatory mapping (strongest of any workstream) →

---

🧩 NLI Robustness (Notebook 05)

✅ Complete

Tests whether the model still reasons correctly under adversarial pressure. Natural Language Inference asks whether a hypothesis is entailed by, neutral to, or contradicts a premise — unlike NB01, the dataset is the adversary (ANLI items are human-crafted to fool strong models).

Headline (GPT-5-4 via Azure, 13,298 items): clean accuracy 85.5% vs. a +20.8% robustness gap on the hardest ANLI round — the model degrades gracefully, not catastrophically, but clean accuracy overstates reliability on hard reasoning. Dominant failure mode: hedging to "neutral."

📄 Full results, robustness gap, ANLI difficulty curve →

---

🔐 Data Red-Teaming (Notebook 06)

✅ Complete

Targets confidentiality — the model as a data-leak vector — across three tracks: system-prompt/secret disclosure, memorization/PII regurgitation (incl. real Enron PII extraction), and RAG context exfiltration.

Headline (GPT-5-4 via Azure, 61 probes): clean across all three tracks — 0% sensitive-leak rate, including 0/20 real Enron PII reproduced. The only flags are benign public-domain recall, correctly excluded from the headline leak rate.

📄 Full results, three tracks, industry alignment →

---

🤖 Agentic Tool Attacks (Notebook 07)

🛠️ Built — run pending

The frontier — validated by the OpenAI/Google/IEEE Kaggle competition on multi-step tool attacks. Tests whether untrusted input can move a tool-using agent to an unsafe action (send email, delete files, make a payment) via a ReAct loop over a safe mock sandbox, in the style of AgentDojo.

Harness, sandbox, 5 scenarios, metrics, and executive report are built and validated end-to-end; results are published here after a run against the assessed model.

📄 Design & methodology →

---

📚 References & Standards

Frameworks: MITRE ATLAS · NIST AI RMF · NIST AI 600-1 · OWASP LLM Top 10 · EU AI Act

Attacks & Benchmarks: TextFooler · TextBugger · DeepWordBug · BERT-Attack · PAIR · GCG · StrongREJECT · JailbreakBench · AdvBench · HarmBench

Datasets: SST-2 · AdvGLUE · ANLI · ToxiGen · HateXplain

Related tools: Microsoft PyRIT · NVIDIA Garak · TextAttack

Forward-looking attacks, datasets, and metrics are cited in the Roadmap; per-workstream methodology and regulatory mapping are in each doc page above.

---

🤝 Contributing

Contributions are welcome. To add a new attack, dataset, or testing strategy:

1. Fork the repo and create a feature branch
2. Follow the existing module structure — attacks inherit from the base class in attacks/base.py
3. Add an entry to the relevant roadmap table (with standards mapping)
4. Open a PR with a short description of the attack and at least one worked example

---

📄 License

MIT License — see LICENSE for details.

---

⚠️ Responsible Use

This toolkit is intended for security research, model evaluation, and AI safety work. All jailbreak goals used in testing are sourced from published academic benchmarks. Do not use this toolkit to generate or distribute harmful content.

---

_{Built for AI safety practitioners, ML engineers, and red team researchers.}