Red-Teaming Text-to-Image Models via In-Context Experience Replay and Semantic-Preserving Prompt Rewriting

ICER treats red-teaming as a learning problem: accumulating successful jailbreaks into a playbook that transfers across defenses and to commercial systems like DALL·E 3 and Midjourney.

Motivation & Intuition

Almost every automated red-teaming method treats prompt attempts as independent: generate a candidate, query the model, discard the experience, repeat. That’s a strong assumption which says yesterday’s successful jailbreak has nothing useful to tell us about tomorrow’s. Look across concepts, defenses, and even model families, and that assumption breaks: successful attacks share exploitable structure.

ICER’s answer is the playbook. Every successful or near-successful jailbreak goes into a shared database, and at each step we replay a handful as in-context exemplars for the LLM that writes the next prompt. The rewriter never starts from scratch — it always sees what has worked before. The framework below shows how we pick which past experiences to lean on at each step.

The ICER Framework

ICER has three components: a Candidate Sampler that asks an LLM to rewrite the current prompt using k=3 in-context exemplars, a Prior Sampler that uses Thompson Sampling over a Beta posterior to pick those exemplars from the playbook \( \mathcal{D} \), and an Evaluator that gates each candidate by semantic similarity to the original intent and scores its harmfulness. The results write back into \( \mathcal{D} \).

Figure Algorithm

Key Findings

Finding 02: Replay learns patterns, not just longer prompts.

A natural worry is that ICER simply jailbreaks by making prompts longer. But experience replay lifts FR just as much when the LLM rewrites a prompt at its original length as when it extends it, so what carries over from past attempts is a transferable attack strategy, not extra prompt budget. That is the whole premise of the playbook: each success teaches the next attempt how to attack, not just how to write more.

Finding 04: Semantic constraint separates targeted attacks from arbitrary.

Once we require a jailbreak to still match the original concept not just produce any unsafe image, token-level methods drop sharply, because much of their success came from unrelated harmful content rather than the intended concept. ICER's LLM rewrites stay on-concept by construction, so it degrades only modestly, capturing the kind of targeted misuse that reflects realistic threats.

Discussion & Implications

1. The playbook is a diagnostic, not just a weapon. The accumulated jailbreaks are themselves a dataset of which prompt structures defeat which defenses. If analyzed rather than merely executed, this playbook could reveal the weaknesses different safety mechanisms share, turning red-teaming from finding holes into understanding why defenses fail.
2. Defenses should be evaluated against an adversary that remembers. If attacks compound across attempts, concepts, and model families, then defenses should also pointing toward continual, online defense against adaptive attackers rather than robustness measured on a fixed prompt set.
3. Realistic threat models need living benchmarks. High scores on a static prompt set do not imply prompts that survive a deployed system’s moderation; our fluency and transfer results suggest evaluation should be continuously updated rather than measured once.
4. Double-edged sword. The same playbook that makes safety auditing cheap for developers also makes accumulated misuse cheap for malicious actors, which motivates shared, continuously-updated red-teaming infrastructure.

BibTeX

@misc{chin2024icer,
    title={Red-Teaming Text-to-Image Models via In-Context Experience Replay and Semantic-Preserving Prompt Rewriting}, 
    author={Zhi-Yi Chin and Pin-Yu Chen and Wei-Chen Chiu and Mario Fritz},
    year={2024},
    eprint={2411.16769},
    archivePrefix={arXiv},
    primaryClass={cs.LG},
    url={https://arxiv.org/abs/2411.16769}
}