In their paper titled "A Pipeline for Creative Visual Storytelling," authors Stephanie M. Lukin, Reginald Hobbs, and Clare R. Voss explore the realm of computational visual storytelling. They investigate how advancements in natural language processing, generation techniques, and computer vision have made it possible to generate textual descriptions that interpret events depicted in a series of images. The authors introduce the concept of computational creative visual storytelling and discuss its three key aspects: discussing different environments, creating variations based on narrative objectives, and tailoring the narrative to suit the audience. To construct a creative visual storyteller, the authors present a pipeline consisting of task-modules such as Object Identification, Single-Image Inferencing, and Multi-Image Narration as a foundational framework. This pipeline was tested in an annotation task involving a sequence of images and analyzed to create a corpus. The authors also outline plans for automating this process in the future. Originally presented at the First Workshop on Storytelling (StoryNLP) in 2018 at the North American Association for Computational Linguistics (NAACL), this research sheds light on unexplored aspects of creative storytelling within visual narratives. By delving into how these aspects influence the overall narrative structure, Lukin et al. 's work contributes to advancing computational methods for generating engaging and adaptable visual stories. Through their innovative approach and proposed pipeline, they pave the way for further exploration and development in this emerging field of study.
SummaryAuthors Stephanie M. Lukin, Reginald Hobbs, and Clare R. Voss explore using computers to tell stories with pictures. They use new technology to help computers understand images and write descriptions about them. They focus on creating different scenes, changing stories based on goals, and adjusting stories for different audiences. A storyteller's process includes tasks like identifying objects in images and telling a story with multiple pictures. The researchers tested their method by annotating images and plan to automate the process in the future.
Definitions
- Computational: Relating to using computers or technology.
- Visual storytelling: Telling a story using images or visuals.
- Advancements: Improvements or progress made in a particular field.
- Narrative: The story being told or written.
- Automation: Using machines or technology to perform tasks automatically without human intervention.
Introduction
The art of storytelling has been an integral part of human culture since the beginning of time. From cave paintings to modern-day films, humans have always used visual narratives to convey their stories and experiences. However, with advancements in technology, there has been a shift towards computational methods for generating visual stories.
In their paper titled "A Pipeline for Creative Visual Storytelling," authors Stephanie M. Lukin, Reginald Hobbs, and Clare R. Voss delve into the realm of computational visual storytelling and explore how it can be used to generate textual descriptions that interpret events depicted in a series of images. This research was originally presented at the First Workshop on Storytelling (StoryNLP) in 2018 at the North American Association for Computational Linguistics (NAACL).
The Concept of Computational Creative Visual Storytelling
The authors introduce the concept of computational creative visual storytelling as a way to use natural language processing, generation techniques, and computer vision to create engaging and adaptable visual narratives. They highlight three key aspects that are crucial in this process: discussing different environments, creating variations based on narrative objectives, and tailoring the narrative to suit the audience.
Discussing Different Environments: The first aspect involves considering different environments or settings within which a story may take place. This could include indoor or outdoor spaces, specific locations such as cities or forests, or even virtual worlds.
Creating Variations Based on Narrative Objectives: The second aspect focuses on creating variations within the story based on its narrative objectives. This could involve changing elements such as characters' emotions or actions to evoke different responses from the audience.
Tailoring the Narrative to Suit the Audience: The third aspect emphasizes adapting the narrative according to the intended audience's preferences and interests. This could involve using different language styles or cultural references depending on who will be reading or viewing the story.
The Pipeline for Creative Visual Storytelling
To construct a creative visual storyteller, the authors present a pipeline consisting of task-modules as a foundational framework. These task-modules include Object Identification, Single-Image Inferencing, and Multi-Image Narration.
Object Identification: This module involves using computer vision techniques to identify objects within an image. This is crucial in understanding the visual elements that will be incorporated into the narrative.
Single-Image Inferencing: The next module focuses on generating textual descriptions for individual images based on their identified objects. This involves natural language processing techniques to interpret and describe the events depicted in each image.
Multi-Image Narration: The final module combines the single-image descriptions to create a cohesive narrative that spans across multiple images. This requires considering the overall story arc and how each individual image contributes to it.
Testing and Analysis
To test their proposed pipeline, Lukin et al. conducted an annotation task involving a sequence of images and analyzed the results to create a corpus. Through this process, they were able to identify patterns and commonalities in how people interpret visual narratives.
The analysis also revealed potential areas for improvement in their pipeline, such as incorporating more advanced natural language generation techniques or expanding the range of environments considered in storytelling.
Future Directions
In addition to testing their pipeline with human annotators, Lukin et al. also outline plans for automating this process in the future. They suggest using machine learning algorithms trained on large datasets of annotated visual narratives to generate more accurate and diverse descriptions automatically.
They also propose exploring other aspects of creative storytelling within visual narratives that could be incorporated into their pipeline, such as character development or plot twists.
Conclusion
Overall, Lukin et al.'s paper sheds light on unexplored aspects of computational creative visual storytelling and its potential applications in generating engaging and adaptable visual narratives. Through their innovative approach and proposed pipeline, they pave the way for further exploration and development in this emerging field of study.
As technology continues to advance, computational methods for storytelling will only become more sophisticated and prevalent. With Lukin et al.'s research as a foundation, we can expect to see even more creative and immersive visual stories being generated by machines in the future.
Computational visual storytelling produces a textual description of events and interpretations depicted in a sequence of images. These texts are made possible by advances and cross-disciplinary approaches in natural language processing, generation, and computer vision. We define a computational creative visual storytelling as one with the ability to alter the telling of a story along three aspects: to speak about different environments, to produce variations based on narrative goals, and to adapt the narrative to the audience. These aspects of creative storytelling and their effect on the narrative have yet to be explored in visual storytelling. This paper presents a pipeline of task-modules, Object Identification, Single-Image Inferencing, and Multi-Image Narration, that serve as a preliminary design for building a creative visual storyteller. We have piloted this design for a sequence of images in an annotation task. We present and analyze the collected corpus and describe plans towards automation.
