# Emoji Attack: Enhancing Jailbreak Attacks Against Judge LLM Detection

**Authors:** Zhipeng Wei, Yuqi Liu, N. Benjamin Erichson  
**arXiv:** [2411.01077](https://arxiv.org/abs/2411.01077)  
**Date:** 2024-11-01  
**Summary page:** https://www.summarizepaper.com/en/paper/2411.01077v5/

## Key points

- Jailbreaking techniques pose a threat to Large Language Models (LLMs), manipulating them into producing restricted or harmful output.
- Defense mechanism involves using another LLM as a Judge to evaluate text for harmfulness, but Judge LLMs are susceptible to token segmentation bias.
- Emoji Attack technique strategically inserts emojis into text before evaluation by a Judge LLM, reducing detection accuracy and allowing unsafe content to slip past moderation filters.
- Emojis introduce semantic ambiguity, making them effective in evading detection compared to traditional delimiters.
- Research question focuses on whether seemingly harmless constructs like emojis can alter decision boundaries of Judge LLMs and enable harmful content to bypass moderation filters.

## Summary

, , , , 

In recent years, jailbreaking techniques have emerged as a threat to Large Language Models (LLMs), manipulating them into producing restricted or harmful output. To counter this, one defense mechanism involves using another LLM as a Judge to evaluate the generated text for harmfulness. However, it has been discovered that these Judge LLMs are susceptible to token segmentation bias, where delimiters in the text alter the tokenization process and split words into smaller sub-tokens. This manipulation of embeddings reduces detection accuracy, allowing harmful content to be misclassified as safe. In response to this vulnerability, a novel strategy known as Emoji Attack has been introduced. This technique leverages in-context learning to strategically insert emojis into text before evaluation by a Judge LLM. By inducing embedding distortions through the use of emojis, the likelihood of detecting unsafe content is significantly lowered. Unlike traditional delimiters, emojis introduce semantic ambiguity, making them particularly effective in evading detection. Through experiments on cutting-edge Judge LLMs, it has been demonstrated that Emoji Attack successfully reduces the rate of unsafe predictions and bypasses existing safeguards. The research question posed in this study revolves around whether seemingly harmless linguistic constructs like emojis can alter the decision boundaries of Judge LLMs and enable harmful content to slip past moderation filters. Furthermore, it has been highlighted that many Judge LLMs assign numerical scores to indicate the severity of content violations on ethical, legal, or safety grounds. If a response surpasses a predefined threshold score, it is flagged as unsafe. While automated moderation mechanisms show promise in addressing such issues, they remain susceptible to specific exploits like token segmentation bias. This study sheds light on the critical weakness in Judge LLMs caused by token segmentation bias and underscores the importance of understanding how minor input modifications can impact contextual understanding and ultimately influence the effectiveness of moderation mechanisms in safeguarding against harmful content generated by LLMs.