AI Red Teaming & Defensive Techniques

The Big Picture First#

The previous section walked through the various ways an AI system can actually be attacked. This section flips that same picture around, covering how you actually go about deliberately finding those weaknesses yourself, before a genuine real attacker ever does, and how you actually go about building real, concrete defenses against the specific threats we already covered. There’s a genuinely useful mental model worth holding onto throughout this whole explanation: red teaming is fundamentally about finding the problems, while the various specific defensive techniques covered afterward are all fundamentally about fixing or actually preventing them. Together, these two halves form a genuinely continuous, ongoing cycle — you test, you find real weaknesses, you fix them, and then you genuinely test again, since new weaknesses can always keep emerging over time, whether through actual model updates, new features being added, or genuinely new attack techniques being actively discovered elsewhere out in the world. Let’s work through it all.

1. AI Red Teaming Fundamentals#

Red teaming is a term genuinely borrowed from a much older military and cybersecurity tradition — a “red team” is a group specifically tasked with actually attacking their own organization’s own systems, deliberately playing the role of a genuine, realistic adversary, specifically to find real, genuine weaknesses before any actual, real, genuinely hostile outside attacker ever manages to find and exploit them first. AI red teaming applies this exact same well-established underlying idea specifically to AI systems — deliberately, systematically trying to actually break, manipulate, or otherwise genuinely misuse your own AI application, specifically to discover its real weaknesses ahead of time, under genuinely safe, fully controlled conditions.

The core underlying mindset shift here matters quite a lot, and it’s genuinely worth sitting with for a moment. Most of the actual work of building an AI application, throughout this whole series, has naturally been focused on the question “does this genuinely work well for a normal, well-intentioned user asking normal, reasonable questions?” Red teaming instead asks a meaningfully different, distinct question: “what happens if someone actually, deliberately tries to make this system genuinely misbehave?” This requires actually, genuinely thinking somewhat like an attacker would — creatively trying out the various specific attack techniques we covered in the previous explanation (prompt injection, jailbreaking, and so on), and more generally trying to actually probe for any and all sorts of genuinely unexpected, unintended edge-case behavior a normal, well-intentioned everyday user would likely never actually happen to stumble onto entirely on their own.

AI red teaming exercises can genuinely be conducted by dedicated internal security specialists, by external, outside firms that specifically specialize in exactly this kind of work, or increasingly, through automated tools specifically designed to help systematically generate and actually test a genuinely large number of different potential attack attempts (which connects directly to the security testing automation topic we’ll cover later below). Whichever specific approach a given organization actually chooses, the genuinely core underlying goal remains exactly the same: proactively, deliberately discovering real weaknesses under safe, fully controlled conditions, rather than only ever discovering them reactively, after the fact, once real, actual, genuine harm has already actually occurred out in the real world.

2. Adversarial Prompt Testing#

Adversarial prompt testing refers to the actual, concrete, hands-on practice of systematically crafting and actually trying out prompts that are specifically, deliberately designed to try to make an AI system genuinely misbehave — this is really the specific, concrete, hands-on activity that red teaming, as a broader overall concept, actually practically, concretely involves doing, day to day.

This genuinely goes well beyond simply just trying a handful of obviously, blatantly malicious-sounding requests. A genuinely thorough adversarial testing process actually, deliberately tries a genuinely wide, varied range of different approaches — directly, bluntly asking for clearly problematic content outright; trying various different jailbreak techniques (like the specific role-play and hypothetical framings we already discussed in the previous explanation); trying various genuinely subtle, indirect approaches (like gradually, incrementally working a conversation forward, step by step, toward some genuinely problematic eventual outcome, rather than directly, bluntly asking for it all at once, in one single obvious step); and trying various forms of genuinely disguised or obfuscated language specifically designed to try to help slip past straightforward, simple keyword-based filtering (for example, deliberately misspelling certain particular sensitive words, or using coded, indirect language specifically instead of more direct, obvious terminology).

A genuinely well-organized adversarial testing effort generally maintains an actual, ongoing, evolving library of these kinds of specific known test prompts, organized by exactly which specific category of vulnerability each individual one is actually specifically testing for, and genuinely, regularly runs a given system against this whole accumulated library, not merely just once, right before an actual initial launch, but on a genuinely regular, ongoing, continuing basis over real time — since, as we already touched on in the previous explanation, both the specific underlying models themselves and the broader, wider array of known attack techniques out there both continue to genuinely, actively keep evolving and changing over time.

3. Prompt Injection Testing#

Prompt injection testing is a specific, focused subset of the broader adversarial prompt testing we just discussed, specifically and deliberately focused on the particular prompt injection vulnerability we covered in real, careful detail in the previous explanation — deliberately trying to actually get a given AI system to genuinely follow some kind of injected instruction, rather than genuinely, properly sticking to and following its own actual, legitimate, originally intended instructions.

Good prompt injection testing generally needs to genuinely, carefully cover both of the specific major variants we already discussed earlier. Direct injection testing genuinely means actually trying various different ways of directly typing in instructions specifically designed to try to override a given system’s own actual original instructions — variations on things like “ignore your previous instructions,” or more genuinely subtle, less obvious attempts specifically designed to try to more quietly, gradually redirect a given system’s actual overall behavior. Indirect injection testing genuinely means actually specifically testing what happens when a given malicious instruction is instead deliberately planted inside some piece of external content the system might later actually end up reading or retrieving — for example, actually deliberately planting a hidden instruction inside a test document specifically designed to be later fed into a genuine RAG pipeline, and then genuinely, carefully checking whether the given system actually, genuinely ends up following that particular hidden instruction, or whether it correctly, properly treats it purely as inert content instead, to simply read but never actually obey.

Testing specifically for this particular vulnerability matters so much precisely because, as we already discussed at real length in the previous explanation, prompt injection is currently genuinely ranked as the single most significant, top overall risk specifically facing today’s LLM applications — and because this particular vulnerability is genuinely so fundamentally, deeply rooted in exactly how language models actually inherently process natural language instructions in the first place, it genuinely can’t ever be considered fully, completely, permanently solved once and for all — meaning genuinely ongoing, continuing testing specifically remains a genuinely necessary, essential, recurring practice, rather than something that could ever reasonably be treated as some kind of one-time, single, isolated checkbox to simply tick off once and then never actually need to revisit again.

4. Jailbreak Detection#

Jailbreak detection refers specifically to actually identifying when a given user is genuinely, actively attempting to use one of the known jailbreak techniques we already discussed in the previous explanation — meaning specifically catching the actual, genuine attempt itself, ideally, before it can actually even genuinely, successfully manage to work at all.

This can genuinely be approached from a couple of meaningfully different, complementary angles. One approach specifically involves actually looking directly for known, recognizable jailbreak patterns in the incoming user input itself — certain particular characteristic phrasings and specific structures that tend to genuinely, commonly show up quite repeatedly and consistently across many different known jailbreak attempts (like certain particular common role-play framings, or certain particular commonly recurring hypothetical-scenario setups), and specifically flagging or blocking input that genuinely, closely matches these kinds of known, recognizable patterns. Another genuinely complementary approach instead specifically involves actually monitoring a given model’s own resulting output, specifically looking for telltale signs that a genuine, successful jailbreak may have actually genuinely, successfully occurred — for example, actually noticing when a model suddenly, unexpectedly starts producing genuinely, noticeably different content than it would otherwise typically, normally produce, given similar related past questions and past requests.

A meaningfully important, genuinely realistic point worth being upfront and honest about here: jailbreak detection is genuinely, inherently a continuing arms race, rather than ever being some kind of fully, permanently solved, once-and-for-all static problem. As various specific defenders actually get progressively better and better over time at reliably detecting and catching various specific known jailbreak techniques, various specific attackers correspondingly, in turn, actively keep developing and discovering various new, novel techniques specifically designed to try to actually get around those exact same particular defenses. This is precisely why jailbreak detection genuinely needs to be treated as an ongoing, continuing, evolving practice, rather than as some kind of single, fixed, static, permanently completed piece of work that a given team could reasonably consider themselves fully, completely, permanently finished with at any single particular given point in time.

5. Output Validation#

Output validation refers to actually, carefully checking a given model’s own generated response before it’s actually ever allowed to genuinely reach a real, actual end user — connecting directly back to the “output guardrails” concept we already specifically introduced back in the earlier AI Guardrails explanation from the previous Week 10 material.

The genuinely core underlying reasoning here is fairly straightforward and intuitive: even with genuinely excellent, well-designed, careful input validation firmly in place, a given model can still, sometimes, potentially end up producing a genuinely problematic response — whether that’s specifically due to some genuine model mistake, some genuinely successful jailbreak attempt that actually managed to slip past a given system’s own defenses, or some other unexpected, genuinely unforeseen edge case that nobody had actually, specifically anticipated or planned for in advance. Output validation is specifically the genuinely necessary, important second line of overall defense, specifically checking the actual, real output itself, rather than exclusively relying only on carefully, thoroughly controlling and validating the original input alone.

Good output validation generally checks for several genuinely different, distinct things together. It can check for the specific various content-related issues we already discussed at real length back in the earlier Week 10 material — toxicity, genuinely sensitive PII, and various other content-moderation-related concerns. It can also specifically check for genuine format-related, structural correctness — making sure a given response genuinely, actually conforms properly to whatever specific expected structure it was actually, originally supposed to have (connecting directly back to the structured outputs concept we already discussed in the earlier tool calling explanation). And it can specifically check for genuine consistency with a given system’s own actual intended, defined purpose and scope — for example, actually, specifically flagging a response from a supposedly narrowly-scoped customer support bot that’s somehow unexpectedly, suddenly started freely discussing completely unrelated topics entirely outside its own actual, originally intended scope, which could genuinely, plausibly indicate that some kind of successful, genuine prompt injection or jailbreak attempt may have actually, genuinely occurred somewhere along the way.

6. Input Validation#

Input validation is the genuine mirror image and natural complement of output validation, specifically focused instead on actually, carefully checking what’s actually coming into a given system, before it’s ever actually allowed to reach the underlying model at all — connecting directly back to the “input guardrails” concept we already specifically introduced back in that same earlier AI Guardrails explanation.

Good input validation generally involves checking for several genuinely different, distinct kinds of potential problems together. It can specifically check for known attack patterns, like the specific prompt injection and jailbreak signatures we’ve already discussed at real length throughout this whole explanation. It can specifically check that a given input actually, genuinely stays properly within its own reasonably expected format and reasonably expected length — an input that’s genuinely, unusually, suspiciously excessively long, for example, might potentially, plausibly represent some kind of genuine attempt to actually try to overwhelm or otherwise meaningfully manipulate a given system in some way. It can specifically check for genuinely known malicious content more generally, connecting directly back to the broader content moderation concepts we already covered back in the earlier Week 10 material.

An important, genuinely worthwhile nuance worth being upfront and clear about here: input validation alone, entirely on its own, genuinely can never be considered fully, completely, perfectly sufficient by itself, precisely because, as we already discussed at real length in the previous explanation, indirect prompt injection specifically means malicious content can potentially still genuinely, effectively enter a given system through channels other than the actual, direct original user input alone — like a retrieved document, or a given tool’s own returned result. This is exactly why input validation genuinely needs to actually work together, closely, in real tandem with output validation, and with the various other specific defensive techniques we’re covering throughout this whole overall explanation, rather than ever being reasonably relied upon as some kind of single, sole, complete, standalone defense entirely, all on its own.

7. AI Firewall Concepts#

An AI firewall applies a genuinely familiar, well-established general concept from more traditional network security — a firewall, in that original, more traditional sense, sits directly between a given network and the broader outside world, specifically inspecting all the actual traffic passing through it, and specifically blocking anything that genuinely, clearly looks suspicious or otherwise genuinely dangerous. An AI firewall applies that exact same well-established general underlying concept specifically to AI traffic — sitting directly between a given user (or a given external system) and the actual underlying language model itself, specifically inspecting both the incoming requests and the resulting outgoing responses, and specifically blocking or otherwise appropriately, actively modifying anything that genuinely, clearly appears problematic.

This idea genuinely, naturally connects several of the specific individual concepts we’ve already covered throughout this whole explanation into one single, unified, cohesive overall system. Rather than each individual specific check (input validation, output validation, jailbreak detection, and so on) being implemented as its own separate, individual, disconnected piece scattered independently throughout a given application’s own underlying codebase, an AI firewall specifically centralizes all of these various different checks together into one single, unified, consistently-applied layer — connecting quite directly and closely back to the AI gateway concept we already covered at real length back in the earlier Week 11 enterprise architecture material, just now specifically, particularly emphasizing the security-related, protective aspect of exactly that same shared, centralized infrastructure, rather than emphasizing routing or cost-related aspects specifically.

The genuinely core practical advantage of centralizing things this way, once again, comes back to consistency — rather than every single individual application across an entire organization needing to separately, redundantly implement its own entirely separate version of each of these individual specific security checks, all completely on its own, a single, unified, shared AI firewall can genuinely, consistently apply exactly the same well-thought-out, carefully considered set of protections uniformly across an entire organization’s overall traffic, while also being considerably easier to reliably keep properly, consistently up to date as genuinely new specific threats continue to actively emerge and evolve over real time.

8. Prompt Shielding#

Prompt shielding refers to a genuinely more specific, focused, targeted technique specifically designed to help protect a given system’s own particular system prompt — the actual private, confidential set of foundational instructions a given developer originally, carefully wrote to actually define exactly how their own particular AI application is supposed to genuinely behave — from both being successfully overridden through prompt injection, and, separately, from actually being inadvertently leaked or otherwise inappropriately revealed to a given end user.

This directly connects back to two genuinely distinct, separate specific risks we’ve already touched on throughout this whole explanation. The first genuine risk is that a given user might successfully manage to actually override a given system’s own carefully-written original instructions through prompt injection, exactly as we already discussed at real length earlier. The second, separate genuine risk is “system prompt leakage” — which we actually already specifically mentioned back when we covered the current OWASP Top 10 list in the previous explanation — where a given user manages to actually successfully trick a given model into directly revealing its own actual private, confidential system prompt, verbatim, word for word. This second particular risk matters more than it might genuinely, initially seem to, since a given system prompt often genuinely contains real, meaningful proprietary business logic, and knowing its exact specific contents can also genuinely, separately help a given attacker actually craft considerably more effective, more targeted prompt injection attempts specifically going forward.

Practical prompt shielding techniques generally include clearly, explicitly instructing a given model, directly within its own system prompt itself, to genuinely never actually reveal its own specific instructions no matter how directly it’s actually asked to; structuring a given overall prompt in ways specifically designed to help make it genuinely harder for later user input to actually be successfully mistaken for a legitimate, genuine new instruction (for example, using clear, explicit, consistent formatting specifically designed to help clearly, reliably separate actual instructions from genuine user content); and specifically, actively testing a given system, on an ongoing, regular basis, against known, common leakage attempts, connecting directly back to the adversarial testing practices we already covered earlier above in this same explanation.

9. Response Filtering#

Response filtering is closely related to output validation, which we already discussed above, but it’s genuinely worth understanding as its own particular, distinct piece specifically focused on actually, actively modifying a given response, rather than simply just outright blocking it entirely — the genuinely key distinction worth holding onto here is between outright rejecting an entire problematic response completely, versus instead actually cleaning it up or otherwise meaningfully adjusting it in some way before it’s actually finally delivered.

Think about the genuinely meaningful practical difference here. If a given generated response happens to contain one small, isolated piece of genuinely sensitive PII buried somewhere in the middle of an otherwise entirely fine, perfectly acceptable response, entirely blocking and rejecting that whole response completely might reasonably feel like a fairly unnecessarily heavy-handed, overly blunt overall approach — response filtering instead specifically means actually, carefully redacting or otherwise appropriately masking just that one specific particular problematic piece, while still actually, genuinely delivering the remainder of that same otherwise perfectly fine response through to the actual user as normal. Similarly, if a given response happens to contain some kind of minor, relatively small formatting issue, response filtering might instead specifically mean actually, automatically correcting that particular issue directly, rather than needing to completely regenerate the entire response completely from scratch, all over again.

This particular technique genuinely represents a thoughtful, sensible middle ground between two other more extreme overall approaches — never actually checking a given response’s own content at all (which is genuinely far too permissive and risky), versus completely, entirely blocking absolutely any response at all that happens to trip even one single specific individual guardrail check, no matter how genuinely minor that particular issue might actually be (which, in turn, risks becoming considerably too restrictive, and can genuinely end up meaningfully hurting a given system’s overall usefulness and its overall practical usability, entirely unnecessarily). Well-designed response filtering specifically tries to actually, thoughtfully preserve as much of a given response’s own genuine usefulness and inherent overall value as reasonably possible, while still genuinely, reliably addressing and properly resolving whatever specific particular problem was actually, genuinely identified within it.

10. Human-in-the-Loop Validation#

We already covered the broader, general human-in-the-loop concept at real length back in the earlier Agent Patterns explanation, but it’s genuinely worth specifically revisiting here, in this particular current security context, since it plays a genuinely distinct, particular role specifically as a defensive security technique, rather than purely serving as a general reliability or quality mechanism alone.

In this particular specific security context, human-in-the-loop validation genuinely means actually routing certain particular kinds of requests or responses to an actual real human reviewer for their own explicit approval, specifically before those particular requests or responses are actually, finally allowed to fully, completely proceed — and specifically doing so based on genuine, real security-related risk criteria, rather than purely based only on general functional or business-related considerations alone. For example, a given system might reasonably be specifically configured to actually flag any request that happens to trip a given jailbreak detector, or that happens to genuinely involve some kind of unusually high-risk specific tool call, for direct, explicit human review, rather than the given system simply, automatically proceeding entirely on its own, fully autonomously, without any actual human oversight whatsoever.

The genuinely core underlying value that human review specifically brings to this particular security context is a kind of judgment and contextual understanding that automated systems, even genuinely quite good, well-designed ones, still often, quite reasonably, actually genuinely lack — an actual human reviewer can typically pick up on genuinely subtle, unusual context, or a genuinely unusual, novel-looking combination of specific factors that a purely automated, rules-based detection system might well end up completely missing entirely. This makes human-in-the-loop validation a genuinely particularly valuable, important additional safeguard specifically for a given system’s own highest-stakes, most genuinely consequential decisions — even while it’s genuinely, correctly recognized as being simultaneously far too slow, and far too directly, fundamentally dependent on genuinely limited human attention and human availability, to actually be reasonably, sensibly applied to absolutely every single individual request a given system happens to actually process, across the board, indiscriminately.

11. Security Testing Automation#

Security testing automation refers to actually building tooling specifically designed to help systematically run the kinds of adversarial tests we already discussed above — automatically, and at genuine real scale, rather than needing to entirely, exclusively rely on human security testers manually crafting and then manually, individually trying out each single individual test prompt themselves, entirely by hand, one at a time.

This matters quite a lot precisely because thorough, genuinely comprehensive red teaming actually, genuinely involves an awful lot of individual, distinct test cases — potentially many hundreds or even many thousands of different specific variations across all the different specific attack categories we’ve already discussed throughout this whole overall explanation — and manually testing all of that, entirely by hand, would genuinely be an extremely slow, extremely labor-intensive, and generally quite impractical undertaking to actually carry out consistently, especially on any kind of genuinely regular, recurring, ongoing basis over real time. Automated security testing tools can specifically help systematically, automatically generate genuinely large numbers of different variations of known attack patterns, automatically actually run a given target system against all of them in one single, unified, efficient batch, and then automatically, systematically flag any specific individual cases where the given system’s actual real response genuinely, clearly appears to indicate some kind of real, genuine vulnerability having actually, successfully been exploited.

A genuinely particularly powerful, increasingly common specific approach here specifically involves actually using one dedicated AI model specifically to help automatically generate genuinely creative, novel new adversarial test cases specifically designed to try to actually attack a completely separate, different target AI system — essentially, effectively using AI itself to help actually, meaningfully red-team other AI systems. This connects quite directly and closely back to the broader debate pattern we already discussed at real length back in the earlier Agent Patterns explanation, just now specifically applied in a genuinely more explicitly adversarial context — one system specifically, deliberately trying its best to actually attack another, while the given target system correspondingly, in turn, actually genuinely tries its own best to properly, effectively defend itself against those exact same specific, ongoing attempts. Genuinely well-integrated security testing automation, in a mature, well-run organization, generally becomes an actual, ongoing, integrated part of the broader AI CI/CD pipeline we already discussed at real length back in the earlier Week 10 material — meaning any given new prompt or given new model change specifically, automatically gets genuinely tested against this whole accumulated, ever-growing security test suite, alongside all its various other more standard, regular quality checks, before that particular given change is ever actually finally, fully allowed to genuinely go live, out into real, actual production.

12. AI Incident Response#

AI incident response refers to what a given organization actually, genuinely does after something has actually already, genuinely gone wrong — a real, genuine security incident of some kind has actually, genuinely occurred, despite whatever various specific defenses a given organization might have already had actually, genuinely in place at the actual time it happened. No matter how genuinely good and how genuinely thorough a given organization’s overall defenses might actually happen to be, it’s still always genuinely wise and sensible to actually have a real, clear, well-thought-out plan specifically in place for exactly what to actually do once something has already, genuinely gone wrong regardless, since no given set of defenses can ever be reasonably, realistically expected to be perfectly, completely, permanently sufficient, entirely on their own, forever.

A genuinely well-prepared AI incident response plan generally, typically covers several genuinely distinct, important stages together. Detection means actually, reliably having the genuinely necessary means in place to actually notice that some kind of genuine incident has actually, really occurred in the first place — connecting quite directly and closely back to the whole production monitoring and observability material we already covered at real length back in the earlier Week 10 explanation. Containment means actually taking genuinely swift, immediate action specifically designed to help limit and properly contain the overall further, ongoing damage — this might reasonably mean things like actually temporarily disabling some particular specific tool or specific capability that’s actually, genuinely currently, actively being exploited or abused, or actually temporarily, entirely taking a given affected specific feature fully offline while the underlying genuine problem is actually being properly, thoroughly investigated. Investigation means actually, carefully, thoroughly understanding exactly what actually, genuinely happened — connecting quite directly and closely back to the failure analysis concepts we already covered at real length back in the earlier Week 10 explanation, just now specifically applied here through more of a genuinely security-focused overall lens. And remediation means actually, properly fixing the underlying genuine root cause of the problem, and specifically making sure genuinely similar future incidents actually, genuinely can’t quite so easily happen again, going forward, in exactly the same particular way.

A genuinely important, worthwhile point to close on here: incident response, red teaming, and every single other one of the various specific defensive techniques we’ve actually covered throughout this whole entire explanation are all really, genuinely meant to work together, in real, ongoing, continuous combination, as one single, unified, cohesive, overall cycle, rather than any single one of them ever really being reasonably considered genuinely sufficient entirely, completely on its own, standing fully alone. Red teaming genuinely helps proactively find real weaknesses before they’re actually, genuinely exploited by any real, genuine attacker; the various different specific defensive techniques we’ve covered throughout this whole explanation genuinely help meaningfully reduce the actual overall likelihood of any given real, genuine successful attack actually occurring at all in the first place; and a genuinely well-prepared incident response plan specifically ensures that when something inevitably, eventually still does manage to genuinely slip through regardless, despite everything else already actually, genuinely in place, the actual resulting real-world overall damage actually, genuinely stays as fully contained, and the given underlying problem itself actually, genuinely gets properly fixed, as quickly, and as reliably, and as thoroughly as reasonably, genuinely possible.