🔓 Automatic Jailbreaking of the
Text-to-Image Generative AI Systems

Recent AI systems have shown extremely powerful performance, even surpassing human performance, on various tasks such as information retrieval, language generation, and image generation based on large language models (LLMs). At the same time, there are diverse safety risks that can cause the generation of malicious contents by circumventing the alignment in LLMs, which are often referred to as jailbreaking. However, most of the previous works only focused on the text-based jailbreaking in LLMs, and the jailbreaking of the text-to-image (T2I) generation system has been relatively overlooked. In this paper, we first evaluate the safety of the commercial T2I generation systems, such as ChatGPT, Copilot, and Gemini, on copyright infringement with naive prompts. From this empirical study, we find that Copilot and Gemini block only 12% and 17% of the attacks with naive prompts, respectively, while ChatGPT blocks 84% of them. Then, we further propose a stronger automated jailbreaking pipeline for T2I generation systems, which produces prompts that bypass their safety guards. Our automated jailbreaking framework leverages an LLM optimizer to generate prompts to maximize degree of violation from the generated images without any weight updates or gradient computation. Surprisingly, our simple yet effective approach successfully jailbreaks the ChatGPT with 11.0% block rate, making it generate copyrighted contents in 76% of the time. Finally, we explore various defense strategies, such as post-generation filtering and machine unlearning techniques, but found that they were inadequate, which suggests the necessity of stronger defense mechanisms.

To prevent such potential copyright violations, commercial T2I systems (ChatGPT, Copilot) censor user requests by blocking generation of copyrighted materials or rephrase the users' prompts, to prevent them. However, are they really secure against unauthorized reproduction of copyrighted materials? To the best of our knowledge, there is no work on quantitative evaluation of the copyright violation by the commercial T2I systems, making it difficult for the service providers to red-team their systems. Furthermore, for intellectual property (IP) owners, it requires a large amount of effort to verify the usage of contents in those systems via manual trial-and-error processes

we further propose a simple yet effective Automated Prompt Generation Pipeline (APGP) which automatically generates jailbreaking prompts by optimizing a large language model (LLM) using the self-generated QA score and keyword penalty. To bypass the word-based detection, we give a penalty when prompts contain specific keywords, such as "Mickey Mouse," when describing the copyrighted content. Simultaneously, to prevent overly generic descriptions without these keywords, we introduce a self-generated QA score. This score assesses how well the answers that are generated solely from the prompt match the questions, where questions are derived from the target image. Our scoring function effectively optimizes LLM to refine prompts that are at high risk of inducing copyright infringement in T2I systems.

1) Optimizing the instruction with LLM for vision-language models to generate a seed prompt that describes the given target image.

2) Revision optimization step uses the LLM to refine the prompt to accurately depict the image that achieves a higher score according to the proposed scoring function.

3) Post processing suffix prompts step, e.g., keyword-suppressing suffix, and intention added suffix, append the suffix that can bypass simple word-based block systems.

The overall pipeline does not require any weight updates or gradient computations; it only needs inference with LLMs and T2I models, which is fast and computationally inexpensive.

To quantify the violations, we conducted a human evaluation on 63 participants to determine the copyright violation based on the reference image. The copyright violation is highly occurring in the product and logo category where 96.24% and 82.71% of participants examine the images as copyright infringement. Upon examining the images classified as identical violations, it was found that over 50% were deemed to be cases of copyright infringement in product and logo. Furthermore, 30% of characters are also considered as similar violations which are determined as severe similarity. When we employ a consensus vote to determine violations, there are 10 images that all participants determine as violations.

Our approach involves searching for prompts that may lead to copyright infringement in commercial text-to-image (T2I) systems. There is a concern that our work could enable adversaries to exploit these systems. Additionally, we are worried about the potential misuse and abuse of our approach, as we have identified instances of not only copyright infringement but also violations of publicity rights. However, these risks have been inherent in commercial T2I systems from the beginning. We believe it is crucial to acknowledge these issues and investigate ways to enhance the safety of real-world AI applications in the future.