A Taxonomy of Prompt Modifiers for Text-To-Image Generation
Auteurs : Jonas Oppenlaender
Résumé scientifique
Text-to-image generation has seen an explosion of interest since 2021. Today, beautiful and intriguing digital images and artworks can be synthesized from textual inputs ("prompts") with deep generative models. Online communities around text-to-image generation and AI generated art have quickly emerged. This paper identifies six types of prompt modifiers used by practitioners in the online community based on a 3-month ethnographic study. The novel taxonomy of prompt modifiers provides researchers a conceptual starting point for investigating the practice of text-to-image generation, but may also help practitioners of AI generated art improve their images. We further outline how prompt modifiers are applied in the practice of "prompt engineering." We discuss research opportunities of this novel creative practice in the field of Human-Computer Interaction (HCI). The paper concludes with a discussion of broader implications of prompt engineering from the perspective of Human-AI Interaction (HAI) in future applications beyond the use case of text-to-image generation and AI generated art.
Les articles autour de lui, par date et par proximité
Plus haut, c’est plus proche de cet article ; à gauche, c’est plus ancien. Les traits relient les articles proches entre eux : les groupes de traits sont des sous-thèmes. Sélectionnez un point pour voir pourquoi il est là. Retour au résumé de cet article
Les articles autour de celui-ci
Classés selon la proximité de leur titre et de leur résumé avec ceux de cet article. Sélectionnez-en pour les lire ensemble.
Les articles les plus proches
-
The Prompt ArtistsMinsuk Chang, Stefania Druga, A. Fiannaca et al. · 2023 · Creativity & Cognition · cité 127 fois · Pas encore résumé
Résumé scientifique
This paper examines the art practices, artwork, and motivations of prolific users of the latest generation of text-to-image models. Through interviews, observations, and a user survey, we present a sampling of the artistic styles and describe the developed community of practice around generative AI. We find that: 1) artists hold the text prompt and the resulting image can be considered collectively as a form of artistic expression (prompts as art), and 2) prompt templates (prompts with “slots” for others to fill in with their own words) are developed to create generative art styles. We discover that the value placed by this community on unique outputs leads to artists seeking specialized voc…
-
PrompTHis: Visualizing the Process and Influence of Prompt Editing During Text-to-Image CreationYuhan Guo, Hanning Shao, Can Liu et al. · 2024 · IEEE Transactions on Visualization and Computer Graphics · cité 38 fois · Pas encore résumé
Résumé scientifique
Generative text-to-image models, which allow users to create appealing images through a text prompt, have seen a dramatic increase in popularity in recent years. However, most users have a limited understanding of how such models work and often rely on trial and error strategies to achieve satisfactory results. The prompt history contains a wealth of information that could provide users with insights into what has been explored and how the prompt changes impact the output image, yet little research attention has been paid to the visual analysis of such process to support users. We propose the Image Variant Graph, a novel visual representation designed to support comparing prompt-image pairs …
-
Examining the Text-to-Image Community of Practice: Why and How do People Prompt Generative AIs?Téo Sanchez · 2023 · Creativity & Cognition · cité 64 fois · Pas encore résumé
Résumé scientifique
Image generation gained popularity with machine learning (ML) models generating images from text, fuelling new online communities of practices. This work explores the sociology, motivations, and usages of AI art hobbyists. We analyzed an online questionnaire answered by 64 practitioners and a dataset of user prompts sent to the Stable Diffusion generative model. Our findings suggest that TTI generation is a recreational activity mainly conducted by narrow socio-demographic groups who use auxiliary techniques across platforms and beyond request-response interactions. Inherent model limitations and finding suitable prompt formulation are the main obstacles practitioners face. A taxonomy and a …
-
PromptMagician: Interactive Prompt Engineering for Text-to-Image CreationYingchaojie Feng, Xingbo Wang, Kam-Kwai Wong et al. · 2023 · IEEE Transactions on Visualization and Computer Graphics · cité 183 fois · Pas encore résumé
Résumé scientifique
Generative text-to-image models have gained great popularity among the public for their powerful capability to generate high-quality images based on natural language prompts. However, developing effective prompts for desired images can be challenging due to the complexity and ambiguity of natural language. This research proposes PromptMagician, a visual analysis system that helps users explore the image results and refine the input prompts. The backbone of our system is a prompt recommendation model that takes user prompts as input, retrieves similar prompt-image pairs from DiffusionDB, and identifies special (important and relevant) prompt keywords. To facilitate interactive prompt refineme…
-
Beyond Text-to-Image: Multimodal Prompts to Explore Generative AIVivian Liu · 2023 · CHI Extended Abstracts · cité 44 fois · Pas encore résumé
Résumé scientifique
Text-to-image AI systems have proven to have extraordinary generative capacities that have facilitated widespread adoption. However, these systems are primarily text-based, which is a fundamental inversion of what many artists are traditionally used to: having full control over the composition of their work. Prior work has shown that there is great utility in using text prompts and that AI augmented workflows can increase momentum on creative tasks for end users. However, multimodal interactions beyond text need to be further defined, so end users can have rich points of interaction that allow them to truly co-pilot AI-generated content creation. To this end, the goal of my research is to eq…
-
Design Guidelines for Prompt Engineering Text-to-Image Generative ModelsVivian Liu, Lydia B. Chilton · 2021 · International Conference on Human Factors in Computing Systems · cité 796 fois · Pas encore résumé
Résumé scientifique
Text-to-image generative models are a new and powerful way to generate visual artwork. However, the open-ended nature of text as interaction is double-edged; while users can input anything and have access to an infinite range of generations, they also must engage in brute-force trial and error with the text prompt when the result quality is poor. We conduct a study exploring what prompt keywords and model hyperparameters can help produce coherent outputs. In particular, we study prompts structured to include subject and style keywords and investigate success and failure modes of these prompts. Our evaluation of 5493 generations over the course of five experiments spans 51 abstract and concre…
-
PromptCharm: Text-to-Image Generation through Multi-modal Prompting and RefinementZhi-Jie Wang, Yu-Heng Huang, Da Song et al. · 2024 · International Conference on Human Factors in Computing Systems · cité 205 fois · Pas encore résumé
Résumé scientifique
The recent advancements in Generative AI have significantly advanced the field of text-to-image generation. The state-of-the-art text-to-image model, Stable Diffusion, is now capable of synthesizing high-quality images with a strong sense of aesthetics. Crafting text prompts that align with the model’s interpretation and the user’s intent thus becomes crucial. However, prompting remains challenging for novice users due to the complexity of the stable diffusion model and the non-trivial efforts required for iteratively editing and refining the text prompts. To address these challenges, we propose PromptCharm, a mixed-initiative system that facilitates text-to-image creation through multi-moda…
-
Prompting AI Art: An Investigation into the Creative Skill of Prompt EngineeringJonas Oppenlaender, Rhema Linder, Johanna Silvennoinen · 2023 · Human-Computer Interaction · cité 282 fois · Résumé IA disponible
Résumé scientifique
We are witnessing a novel era of creativity where anyone can create digital content via prompt-based learning (known as prompt engineering). This paper delves into prompt engineering as a novel creative skill for creating AI art with text-to-image generation. In a pilot study, we find that many crowdsourced participants have knowledge about art which could be used for writing effective prompts. In three subsequent studies, we explore whether crowdsourced participants can put this knowledge into practice. We examine if participants can 1) discern prompt quality, 2) write prompts, and 3) refine prompts. We find that participants could evaluate prompt quality and crafted descriptive prompts, bu…
Les travaux antérieurs sur lesquels il s’appuie
-
The Creativity of Text-to-Image GenerationJ. Oppenlaender · 2022 · International Conference on Entertainment and Media in the Ubiquitous Era · cité 328 fois · Pas encore résumé
Résumé scientifique
Text-guided synthesis of images has made a giant leap towards becoming a mainstream phenomenon. With text-to-image generation systems, anybody can create digital images and artworks. This provokes the question of whether text-to-image generation is creative. This paper expounds on the nature of human creativity involved in text-to-image art (so-called “AI art”) with a specific focus on the practice of prompt engineering. The paper argues that the current product-centered view of creativity falls short in the context of text-to-image generation. A case exemplifying this shortcoming is provided and the importance of online communities for the creative ecosystem of text-to-image art is highligh…
-
Is Writing Prompts Really Making Art?J. Mccormack, Camilo Cruz Gambardella, N. Rajcic et al. · 2023 · EvoMUSART@EvoStar · cité 46 fois · Pas encore résumé
Résumé scientifique
In recent years Generative Machine Learning systems have advanced significantly. A current wave of generative systems use text prompts to create complex imagery, video, even 3D datasets. The creators of these systems claim a revolution in bringing creativity and art to anyone who can type a prompt. In this position paper, we question the basis for these claims, dividing our analysis into three areas: the limitations of linguistic descriptions, implications of the dataset, and lastly, matters of materiality and embodiment. We conclude with an analysis of the creative possibilities enabled by prompt-based systems, asking if they can be considered a new artistic medium.
-
Initial Images: Using Image Prompts to Improve Subject Representation in Multimodal AI Generated ArtHan Qiao, Vivian Liu, Lydia B. Chilton · 2022 · Creativity & Cognition · cité 50 fois · Pas encore résumé
Résumé scientifique
Advances in text-to-image generative models have made it easier for people to create art by just prompting models with text. However, creating through text leaves users with limited control over the final composition or the way the subject is represented. A potential solution is to use image prompts alongside text prompts to condition the model. To better understand how and when image prompts can improve subject representation in generations, we conduct an annotation experiment to quantify their effect on generations of abstract, concrete plural, and concrete singular subjects. We find that initial images improved subject representation across all subject types, with the most noticeable impr…
-
PromptSource: An Integrated Development Environment and Repository for Natural Language PromptsS. Bach, Victor Sanh, Zheng-Xin Yong et al. · 2022 · Annual Meeting of the Association for Computational Linguistics · cité 436 fois · Pas encore résumé
Résumé scientifique
PromptSource is a system for creating, sharing, and using natural language prompts. Prompts are functions that map an example from a dataset to a natural language input and target output. Using prompts to train and query language models is an emerging area in NLP that requires new tools that let users develop and refine these prompts collaboratively. PromptSource addresses the emergent challenges in this new setting with (1) a templating language for defining data-linked prompts, (2) an interface that lets users quickly iterate on prompt development by observing outputs of their prompts on many examples, and (3) a community-driven set of guidelines for contributing new prompts to a common po…
-
The Creativity of Text-based Generative ArtJ. Oppenlaender · 2022 · arXiv.org · preprint, non relu par les pairs · cité 17 fois · Pas encore résumé
Les articles qui le citent
-
Promptify: Text-to-Image Generation through Interactive Prompt Exploration with Large Language ModelsStephen Brade, Bryan Wang, Maurício Sousa et al. · 2023 · ACM Symposium on User Interface Software and Technology · cité 230 fois · Pas encore résumé
Résumé scientifique
Text-to-image generative models have demonstrated remarkable capabilities in generating high-quality images based on textual prompts. However, crafting prompts that accurately capture the user’s creative intent remains challenging. It often involves laborious trial-and-error procedures to ensure that the model interprets the prompts in alignment with the user’s intention. To address these challenges, we present Promptify, an interactive system that supports prompt exploration and refinement for text-to-image generative models. Promptify utilizes a suggestion engine powered by large language models to help users quickly explore and craft diverse prompts. Our interface allows users to organize…
-
Prompt Stealing Attacks Against Text-to-Image Generation ModelsXinyue Shen, Yi-Ting Qu, M. Backes et al. · 2023 · USENIX Security Symposium · cité 84 fois · Pas encore résumé
Résumé scientifique
Text-to-Image generation models have revolutionized the artwork design process and enabled anyone to create high-quality images by entering text descriptions called prompts. Creating a high-quality prompt that consists of a subject and several modifiers can be time-consuming and costly. In consequence, a trend of trading high-quality prompts on specialized marketplaces has emerged. In this paper, we perform the first study on understanding the threat of a novel attack, namely prompt stealing attack, which aims to steal prompts from generated images by text-to-image generation models. Successful prompt stealing attacks directly violate the intellectual property of prompt engineers and jeopard…
-
Can (A)I Have a Word with You? A Taxonomy on the Design Dimensions of AI PromptsMarvin Braun, Maike Greve, Felix Kegel et al. · 2024 · Hawaii International Conference on System Sciences · cité 11 fois · Pas encore résumé
Résumé scientifique
Emerging generative artificial intelligence (AI) is revolutionizing human collaboration with machines through their remarkable processing of human language. Consequently, the role of AI is transitioning as it becomes adaptable to specific tasks through human language, without the need for additional training data. These "prompts" enable end-users to fully harness the potential of AI. However, the process of designing prompts is currently unstructured, relying mostly on trial-and-error processes. Studies emphasize the significant impact of prompt design on the AI's generated results. Therefore, there is a clear need to establish a structured approach to prompt design. To address this need, we…
-
Optimizing Prompts for Text-to-Image GenerationY. Hao, Zewen Chi, Li Dong et al. · 2022 · Neural Information Processing Systems · cité 321 fois · Pas encore résumé
Résumé scientifique
Well-designed prompts can guide text-to-image models to generate amazing images. However, the performant prompts are often model-specific and misaligned with user input. Instead of laborious human engineering, we propose prompt adaptation, a general framework that automatically adapts original user input to model-preferred prompts. Specifically, we first perform supervised fine-tuning with a pretrained language model on a small collection of manually engineered prompts. Then we use reinforcement learning to explore better prompts. We define a reward function that encourages the policy to generate more aesthetically pleasing images while preserving the original user intentions. Experimental r…
-
Dynamic Prompt Optimizing for Text-to-Image GenerationWenyi Mo, Tianyu Zhang, Ya-Long Bai et al. · 2024 · Computer Vision and Pattern Recognition · cité 61 fois · Pas encore résumé
Résumé scientifique
Text-to-image generative models, specifically those based on diffusion models like Imagen and Stable Diffusion, have made substantial advancements. Recently, there has been a surge of interest in the delicate refinement of text prompts. Users assign weights or alter the injection time steps of certain words in the text prompts to improve the quality of generated images. However, the success of fine-control prompts depends on the accuracy of the text prompts and the careful selection of weights and time steps, which requires significant manual intervention. To address this, we introduce the Prompt Auto-Editing (PAE) method. Besides refining the original prompts for image generation, we furthe…
Aller plus loin
Comment ces articles sont-ils trouvés ?
Nous comparons le titre et le résumé de cet article avec ceux de tous les articles résumés sur ce site, et avec les articles que Semantic Scholar recommande, ceux qu’il cite et ceux qui le citent (pour un article très cité, ses citations les plus récentes et, depuis OpenAlex, les plus citées). Parmi les travaux antérieurs et les suites, les articles qui s’utilisent en profondeur, une méthode qu’il utilise et les suites très citées remontent un peu. Les niveaux (très proche, proche, apparenté) viennent de la similarité des textes, pas d’un jugement de qualité ; * signale un article sans résumé, comparé par son titre. La phrase sous un article est écrite par une IA à partir des deux résumés : vérifiez-la dans les articles. Un article jugé hors sujet par trois lecteurs quitte la liste.
Mis à jour le 6 novembre 2026.