The New Yorker Caption Contest is known for its sophisticated and nuanced humor, which often involves unexpected relationships between images and captions as well as allusions to a wide range of human experiences. In an effort to challenge AI models to "demonstrate understanding" of this type of humor, a team of researchers has developed three carefully circumscribed tasks that require grasping these complex relationships and allusions. The team investigates both vision-and-language models that take in cartoon pixels and captions directly as well as language-only models that use textual descriptions of the image. To aid future research in this area, the team publicly releases their corpora along with rich annotations for the cartoon images including descriptions of locations/entities, unusual aspects of the scene and explanations of the jokes. Despite providing such resources however there are still performance gaps between high-quality machine learning models (such as a fine-tuned 175B parameter language model) and humans. The study's authors include Jack Hessel, Ana Marasović, Jena D. Hwang, Lillian Lee, Jeff Da, Rowan Zellers, Robert Mankoff and Yejin Choi. Their paper is titled "Do Androids Laugh at Electric Sheep? ".
The New Yorker Caption Contest is a funny competition where people make jokes about pictures. Sometimes the jokes are tricky and use things that you might not expect. The jokes can be about lots of different things that people go through in life. Some smart people made some tests for computers to understand these jokes too. They had to think really hard about how the pictures and words go together. These smart people shared their work so other people can learn from it too. Even though computers are getting better at understanding jokes, they still aren't as good as humans yet. The paper with all this information was written by a group of authors including Jack Hessel, Ana Marasović, Jena D. Hwang, Lillian Lee, Jeff Da, Rowan Zellers, Robert Mankoff and Yejin Choi."
Definitions
- Caption: A sentence or phrase that goes with a picture to explain what's happening or make it funny.
- Humor: Something that makes you laugh or smile.
- Allusions: References to something else that people might know about.
- AI models: Computers that can learn and do things on their own without being told exactly what to do.
- Corpora: A collection of information used for research.
- Annotations: Notes or explanations added to something to help understand it better.
- Machine learning models: Computers that can learn and improve on their own over time.
- Authors: People who wrote the paper with all this information in it.
Exploring the Complexities of Humor with AI: A Study on The New Yorker Caption Contest
Humor is a complex and nuanced form of communication that often involves unexpected relationships between images and captions as well as allusions to a wide range of human experiences. To challenge AI models to "demonstrate understanding" of this type of humor, a team of researchers led by Jack Hessel, Ana Marasović, Jena D. Hwang, Lillian Lee, Jeff Da, Rowan Zellers, Robert Mankoff and Yejin Choi developed three carefully circumscribed tasks for their paper titled "Do Androids Laugh at Electric Sheep?".
The Tasks
The tasks developed by the research team involve both vision-and-language models that take in cartoon pixels and captions directly as well as language-only models that use textual descriptions of the image. In order to aid future research in this area, the team publicly released their corpora along with rich annotations for the cartoon images including descriptions of locations/entities, unusual aspects of the scene and explanations of the jokes.
Performance Gaps Between Humans & Machines
Despite providing such resources however there are still performance gaps between high-quality machine learning models (such as a fine-tuned 175B parameter language model) and humans when it comes to recognizing humor. This suggests that there is still much work to be done before machines can truly understand human humor on an intuitive level.
Conclusion
The study conducted by Hessel et al provides valuable insight into how AI can be used to recognize complex relationships between images and captions in order to better understand human humor. Despite some progress being made in this area however there is still much room for improvement before machines can truly grasp these nuances on par with humans.
We challenge AI models to "demonstrate understanding" of the sophisticated multimodal humor of The New Yorker Caption Contest. Concretely, we develop three carefully circumscribed tasks for which it suffices (but is not necessary) to grasp potentially complex and unexpected relationships between image and caption, and similarly complex and unexpected allusions to the wide varieties of human experience; these are the hallmarks of a New Yorker-caliber cartoon. We investigate vision-and-language models that take as input the cartoon pixels and caption directly, as well as language-only models for which we circumvent image-processing by providing textual descriptions of the image. Even with the rich multifaceted annotations we provide for the cartoon images, we identify performance gaps between high-quality machine learning models (e.g., a fine-tuned, 175B parameter language model) and humans. We publicly release our corpora including annotations describing the image's locations/entities, what's unusual about the scene, and an explanation of the joke.
