AI summary · arXiv: 2204.13988v2 · cs.MM

A Taxonomy of Prompt Modifiers for Text-To-Image Generation

Jonas Oppenlaender
Original paper · DOI 10.1080/0144929x.2023.2286532 Submitted to arXiv on 20 Apr. 2022 Published in Behaviour & Information Technology (28 Nov. 2023) Accepted on 16 Nov. 2023 15 pages License: NONEXCLUSIVE-DISTRIB 1.0

How to read this summary? Illustrated guide

The essentials

  • Text-to-image generation has gained significant attention in recent years
  • Deep generative models can synthesize beautiful and intriguing digital images and artworks from textual inputs
  • Online communities focused on text-to-image generation and AI-generated art have emerged
  • Jonas Oppenlaender presents a novel taxonomy of prompt modifiers used by practitioners in the online community based on a 3-month ethnographic study
  • The taxonomy identifies six types of prompt modifiers that provide researchers with a conceptual starting point for investigating the practice of text-to-image generation
  • The paper discusses research opportunities presented by this novel creative practice in the field of Human-Computer Interaction (HCI)
  • The paper concludes with a discussion of broader implications of prompt engineering from the perspective of Human-AI Interaction (HAI) beyond just text-to-image generation and AI generated art
  • The taxonomy provides insight into how prompts can be modified to generate more diverse and creative outputs while highlighting potential ethical concerns related to HAI.

Key points written by AI: they may contain errors, check them against the paper.

AI-generated keywords: Text-to-Image Generation Taxonomy Prompt Modifiers Human-Computer Interaction Human-AI Interaction

Read further

  • Comprehensive Summary
  • Layman's Summary
  • Blog article
  • Original abstract
The field of text-to-image generation has gained significant attention in recent years, with the ability to synthesize beautiful and intriguing digital images and artworks from textual inputs using deep generative models. As a result, online communities focused on text-to-image generation and AI-generated art have emerged. In this paper titled "A Taxonomy of Prompt Modifiers for Text-To-Image Generation," Jonas Oppenlaender presents a novel taxonomy of prompt modifiers used by practitioners in the online community based on a 3-month ethnographic study. The taxonomy identifies six types of prompt modifiers that provide researchers with a conceptual starting point for investigating the practice of text-to-image generation. The paper also discusses research opportunities presented by this novel creative practice in the field of Human-Computer Interaction (HCI). Furthermore, the paper concludes with a discussion of broader implications of prompt engineering from the perspective of Human-AI Interaction (HAI) beyond just text-to-image generation and AI generated art. The taxonomy provides insight into how prompts can be modified to generate more diverse and creative outputs while highlighting potential ethical concerns related to HAI. Overall, Oppenlaender's work offers valuable insights into the emerging field of text-to image generation and its impact on HAI. It provides researchers and practitioners with a framework for exploring new ways to create compelling visual content using deep generative models while considering ethical implications.

Was this synthesis useful?

Score: 0

Why do we need votes?

Votes are used to determine whether we need to re-run our summarizing tools. If the count reaches -10, our tools can be restarted.

Figures and plots

Ask about a figure by number or use its question button to request visual analysis. Up to two figures and four image panels per question. Values read from plots are approximate; check the original data.

AI assistant instructions?

Tools for this paper

Explain it differently 1 free rewrite left today
Save to a collection Group papers in collections to compare them, chat with them and export them. Continue with Google Create a free account Log in

Advanced tools Premium

For researchers who follow a field closely: citation alerts, journal-club slides, version comparison and Zotero export. Discover Premium

Zotero Zotero guide
Citation alerts 204 papers already cite it: be emailed each new one. Citation alerts guide
Who works on it?

Among the 31 papers around this one, the name that comes back most: Vivian Liu (3 papers)

2 more people sign several of them. The full map adds their institutions and approaches, from the papers of the last years.

Continue with Google Create a free account: your first map is free
Weekly letter The latest letter of its field, "Computing beyond AI this week" of 16 November 2026: Computing beyond AI this week: The costs hidden in our maps, networks and algorithms · Receive one by email for free

Papers around this one

See them on a map

Ranked by how close their title and abstract are to this paper’s. Select some to read them together.

Closest papers

  1. Very close
    The Prompt Artists
    Minsuk Chang, Stefania Druga, A. Fiannaca et al. · 2023 · Creativity & Cognition · cited 127 times · Not summarized yet
    Abstract

    This paper examines the art practices, artwork, and motivations of prolific users of the latest generation of text-to-image models. Through interviews, observations, and a user survey, we present a sampling of the artistic styles and describe the developed community of practice around generative AI. We find that: 1) artists hold the text prompt and the resulting image can be considered collectively as a form of artistic expression (prompts as art), and 2) prompt templates (prompts with “slots” for others to fill in with their own words) are developed to create generative art styles. We discover that the value placed by this community on unique outputs leads to artists seeking specialized voc…

  2. Very close Cites this paper
    PrompTHis: Visualizing the Process and Influence of Prompt Editing During Text-to-Image Creation
    Yuhan Guo, Hanning Shao, Can Liu et al. · 2024 · IEEE Transactions on Visualization and Computer Graphics · cited 38 times · Not summarized yet
    Abstract

    Generative text-to-image models, which allow users to create appealing images through a text prompt, have seen a dramatic increase in popularity in recent years. However, most users have a limited understanding of how such models work and often rely on trial and error strategies to achieve satisfactory results. The prompt history contains a wealth of information that could provide users with insights into what has been explored and how the prompt changes impact the output image, yet little research attention has been paid to the visual analysis of such process to support users. We propose the Image Variant Graph, a novel visual representation designed to support comparing prompt-image pairs …

  3. Very close Cites this paper Uses this paper in depth
    Examining the Text-to-Image Community of Practice: Why and How do People Prompt Generative AIs?
    Téo Sanchez · 2023 · Creativity & Cognition · cited 64 times · Not summarized yet
    Abstract

    Image generation gained popularity with machine learning (ML) models generating images from text, fuelling new online communities of practices. This work explores the sociology, motivations, and usages of AI art hobbyists. We analyzed an online questionnaire answered by 64 practitioners and a dataset of user prompts sent to the Stable Diffusion generative model. Our findings suggest that TTI generation is a recreational activity mainly conducted by narrow socio-demographic groups who use auxiliary techniques across platforms and beyond request-response interactions. Inherent model limitations and finding suitable prompt formulation are the main obstacles practitioners face. A taxonomy and a …

  4. Very close Cites this paper
    PromptMagician: Interactive Prompt Engineering for Text-to-Image Creation
    Yingchaojie Feng, Xingbo Wang, Kam-Kwai Wong et al. · 2023 · IEEE Transactions on Visualization and Computer Graphics · cited 183 times · Not summarized yet
    Abstract

    Generative text-to-image models have gained great popularity among the public for their powerful capability to generate high-quality images based on natural language prompts. However, developing effective prompts for desired images can be challenging due to the complexity and ambiguity of natural language. This research proposes PromptMagician, a visual analysis system that helps users explore the image results and refine the input prompts. The backbone of our system is a prompt recommendation model that takes user prompts as input, retrieves similar prompt-image pairs from DiffusionDB, and identifies special (important and relevant) prompt keywords. To facilitate interactive prompt refineme…

  5. Very close
    Beyond Text-to-Image: Multimodal Prompts to Explore Generative AI
    Vivian Liu · 2023 · CHI Extended Abstracts · cited 44 times · Not summarized yet
    Abstract

    Text-to-image AI systems have proven to have extraordinary generative capacities that have facilitated widespread adoption. However, these systems are primarily text-based, which is a fundamental inversion of what many artists are traditionally used to: having full control over the composition of their work. Prior work has shown that there is great utility in using text prompts and that AI augmented workflows can increase momentum on creative tasks for end users. However, multimodal interactions beyond text need to be further defined, so end users can have rich points of interaction that allow them to truly co-pilot AI-generated content creation. To this end, the goal of my research is to eq…

  6. Very close Cited by this paper
    Design Guidelines for Prompt Engineering Text-to-Image Generative Models
    Vivian Liu, Lydia B. Chilton · 2021 · International Conference on Human Factors in Computing Systems · cited 796 times · Not summarized yet
    Abstract

    Text-to-image generative models are a new and powerful way to generate visual artwork. However, the open-ended nature of text as interaction is double-edged; while users can input anything and have access to an infinite range of generations, they also must engage in brute-force trial and error with the text prompt when the result quality is poor. We conduct a study exploring what prompt keywords and model hyperparameters can help produce coherent outputs. In particular, we study prompts structured to include subject and style keywords and investigate success and failure modes of these prompts. Our evaluation of 5493 generations over the course of five experiments spans 51 abstract and concre…

  7. Very close
    PromptCharm: Text-to-Image Generation through Multi-modal Prompting and Refinement
    Zhi-Jie Wang, Yu-Heng Huang, Da Song et al. · 2024 · International Conference on Human Factors in Computing Systems · cited 205 times · Not summarized yet
    Abstract

    The recent advancements in Generative AI have significantly advanced the field of text-to-image generation. The state-of-the-art text-to-image model, Stable Diffusion, is now capable of synthesizing high-quality images with a strong sense of aesthetics. Crafting text prompts that align with the model’s interpretation and the user’s intent thus becomes crucial. However, prompting remains challenging for novice users due to the complexity of the stable diffusion model and the non-trivial efforts required for iteratively editing and refining the text prompts. To address these challenges, we propose PromptCharm, a mixed-initiative system that facilitates text-to-image creation through multi-moda…

  8. Very close Cited by this paper Cites this paper Used in depth by this paper
    Prompting AI Art: An Investigation into the Creative Skill of Prompt Engineering
    Jonas Oppenlaender, Rhema Linder, Johanna Silvennoinen · 2023 · Human-Computer Interaction · cited 282 times · AI summary available
    Abstract

    We are witnessing a novel era of creativity where anyone can create digital content via prompt-based learning (known as prompt engineering). This paper delves into prompt engineering as a novel creative skill for creating AI art with text-to-image generation. In a pilot study, we find that many crowdsourced participants have knowledge about art which could be used for writing effective prompts. In three subsequent studies, we explore whether crowdsourced participants can put this knowledge into practice. We examine if participants can 1) discern prompt quality, 2) write prompts, and 3) refine prompts. We find that participants could evaluate prompt quality and crafted descriptive prompts, bu…

Prior work it builds on

  1. Close Cites this paper Used in depth by this paper
    The Creativity of Text-to-Image Generation
    J. Oppenlaender · 2022 · International Conference on Entertainment and Media in the Ubiquitous Era · cited 328 times · Not summarized yet
    Abstract

    Text-guided synthesis of images has made a giant leap towards becoming a mainstream phenomenon. With text-to-image generation systems, anybody can create digital images and artworks. This provokes the question of whether text-to-image generation is creative. This paper expounds on the nature of human creativity involved in text-to-image art (so-called “AI art”) with a specific focus on the practice of prompt engineering. The paper argues that the current product-centered view of creativity falls short in the context of text-to-image generation. A case exemplifying this shortcoming is provided and the importance of online communities for the creative ecosystem of text-to-image art is highligh…

  2. Very close
    Is Writing Prompts Really Making Art?
    J. Mccormack, Camilo Cruz Gambardella, N. Rajcic et al. · 2023 · EvoMUSART@EvoStar · cited 46 times · Not summarized yet
    Abstract

    In recent years Generative Machine Learning systems have advanced significantly. A current wave of generative systems use text prompts to create complex imagery, video, even 3D datasets. The creators of these systems claim a revolution in bringing creativity and art to anyone who can type a prompt. In this position paper, we question the basis for these claims, dividing our analysis into three areas: the limitations of linguistic descriptions, implications of the dataset, and lastly, matters of materiality and embodiment. We conclude with an analysis of the creative possibilities enabled by prompt-based systems, asking if they can be considered a new artistic medium.

  3. Very close
    Initial Images: Using Image Prompts to Improve Subject Representation in Multimodal AI Generated Art
    Han Qiao, Vivian Liu, Lydia B. Chilton · 2022 · Creativity & Cognition · cited 50 times · Not summarized yet
    Abstract

    Advances in text-to-image generative models have made it easier for people to create art by just prompting models with text. However, creating through text leaves users with limited control over the final composition or the way the subject is represented. A potential solution is to use image prompts alongside text prompts to condition the model. To better understand how and when image prompts can improve subject representation in generations, we conduct an annotation experiment to quantify their effect on generations of abstract, concrete plural, and concrete singular subjects. We find that initial images improved subject representation across all subject types, with the most noticeable impr…

  4. Close Used in depth by this paper
    PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts
    S. Bach, Victor Sanh, Zheng-Xin Yong et al. · 2022 · Annual Meeting of the Association for Computational Linguistics · cited 436 times · Not summarized yet
    Abstract

    PromptSource is a system for creating, sharing, and using natural language prompts. Prompts are functions that map an example from a dataset to a natural language input and target output. Using prompts to train and query language models is an emerging area in NLP that requires new tools that let users develop and refine these prompts collaboratively. PromptSource addresses the emergent challenges in this new setting with (1) a templating language for defining data-linked prompts, (2) an interface that lets users quickly iterate on prompt development by observing outputs of their prompts on many examples, and (3) a community-driven set of guidelines for contributing new prompts to a common po…

  5. Related* Used in depth by this paper
    The Creativity of Text-based Generative Art
    J. Oppenlaender · 2022 · arXiv.org · preprint, not peer reviewed · cited 17 times · Not summarized yet

Papers that cite it

  1. Very close Uses this paper in depth
    Promptify: Text-to-Image Generation through Interactive Prompt Exploration with Large Language Models
    Stephen Brade, Bryan Wang, Maurício Sousa et al. · 2023 · ACM Symposium on User Interface Software and Technology · cited 230 times · Not summarized yet
    Abstract

    Text-to-image generative models have demonstrated remarkable capabilities in generating high-quality images based on textual prompts. However, crafting prompts that accurately capture the user’s creative intent remains challenging. It often involves laborious trial-and-error procedures to ensure that the model interprets the prompts in alignment with the user’s intention. To address these challenges, we present Promptify, an interactive system that supports prompt exploration and refinement for text-to-image generative models. Promptify utilizes a suggestion engine powered by large language models to help users quickly explore and craft diverse prompts. Our interface allows users to organize…

  2. Close Uses this paper in depth
    Prompt Stealing Attacks Against Text-to-Image Generation Models
    Xinyue Shen, Yi-Ting Qu, M. Backes et al. · 2023 · USENIX Security Symposium · cited 84 times · Not summarized yet
    Abstract

    Text-to-Image generation models have revolutionized the artwork design process and enabled anyone to create high-quality images by entering text descriptions called prompts. Creating a high-quality prompt that consists of a subject and several modifiers can be time-consuming and costly. In consequence, a trend of trading high-quality prompts on specialized marketplaces has emerged. In this paper, we perform the first study on understanding the threat of a novel attack, namely prompt stealing attack, which aims to steal prompts from generated images by text-to-image generation models. Successful prompt stealing attacks directly violate the intellectual property of prompt engineers and jeopard…

  3. Close Uses this paper in depth
    Can (A)I Have a Word with You? A Taxonomy on the Design Dimensions of AI Prompts
    Marvin Braun, Maike Greve, Felix Kegel et al. · 2024 · Hawaii International Conference on System Sciences · cited 11 times · Not summarized yet
    Abstract

    Emerging generative artificial intelligence (AI) is revolutionizing human collaboration with machines through their remarkable processing of human language. Consequently, the role of AI is transitioning as it becomes adaptable to specific tasks through human language, without the need for additional training data. These "prompts" enable end-users to fully harness the potential of AI. However, the process of designing prompts is currently unstructured, relying mostly on trial-and-error processes. Studies emphasize the significant impact of prompt design on the AI's generated results. Therefore, there is a clear need to establish a structured approach to prompt design. To address this need, we…

  4. Very close
    Optimizing Prompts for Text-to-Image Generation
    Y. Hao, Zewen Chi, Li Dong et al. · 2022 · Neural Information Processing Systems · cited 321 times · Not summarized yet
    Abstract

    Well-designed prompts can guide text-to-image models to generate amazing images. However, the performant prompts are often model-specific and misaligned with user input. Instead of laborious human engineering, we propose prompt adaptation, a general framework that automatically adapts original user input to model-preferred prompts. Specifically, we first perform supervised fine-tuning with a pretrained language model on a small collection of manually engineered prompts. Then we use reinforcement learning to explore better prompts. We define a reward function that encourages the policy to generate more aesthetically pleasing images while preserving the original user intentions. Experimental r…

  5. Close
    Dynamic Prompt Optimizing for Text-to-Image Generation
    Wenyi Mo, Tianyu Zhang, Ya-Long Bai et al. · 2024 · Computer Vision and Pattern Recognition · cited 61 times · Not summarized yet
    Abstract

    Text-to-image generative models, specifically those based on diffusion models like Imagen and Stable Diffusion, have made substantial advancements. Recently, there has been a surge of interest in the delicate refinement of text prompts. Users assign weights or alter the injection time steps of certain words in the text prompts to improve the quality of generated images. However, the success of fine-control prompts depends on the accuracy of the text prompts and the careful selection of weights and time steps, which requires significant manual intervention. To address this, we introduce the Prompt Auto-Editing (PAE) method. Besides refining the original prompts for image generation, we furthe…

How are these papers found?

We compare the title and abstract of this paper with those of every paper summarized on this site, and with the papers Semantic Scholar recommends, those it cites and those citing it (for a much-cited paper, its most recent citations and, from OpenAlex, the most cited ones). Among the prior work and the follow-ups, the papers that use each other in depth, a method it uses and the much-cited follow-ups come a little higher. The levels (very close, close, related) come from the similarity of the texts, not from a judgment of quality; * marks a paper without an abstract, compared by its title. The sentence under a paper is written by an AI from the two abstracts: check it in the papers. A paper marked not relevant by three readers leaves the list.

Updated on 6 November 2026.

The illustrated guide to the papers around a paper

The previous summary was created more than a year ago and can be re-run (if necessary) by clicking on the Run button below.

Disclaimer: The AI-based summarization tool and virtual assistant provided on this website may not always provide accurate and complete summaries or responses. We encourage you to carefully review and evaluate the generated content to ensure its quality and relevance to your needs.