A Taxonomy of Prompt Modifiers for Text-To-Image Generation

Auteurs : Jonas Oppenlaender

Soumis à arXiv le 20 Avr. 2022
Résumé scientifique

Text-to-image generation has seen an explosion of interest since 2021. Today, beautiful and intriguing digital images and artworks can be synthesized from textual inputs ("prompts") with deep generative models. Online communities around text-to-image generation and AI generated art have quickly emerged. This paper identifies six types of prompt modifiers used by practitioners in the online community based on a 3-month ethnographic study. The novel taxonomy of prompt modifiers provides researchers a conceptual starting point for investigating the practice of text-to-image generation, but may also help practitioners of AI generated art improve their images. We further outline how prompt modifiers are applied in the practice of "prompt engineering." We discuss research opportunities of this novel creative practice in the field of Human-Computer Interaction (HCI). The paper concludes with a discussion of broader implications of prompt engineering from the perspective of Human-AI Interaction (HAI) in future applications beyond the use case of text-to-image generation and AI generated art.

Les articles autour de lui, par date et par proximité

Plus haut, c’est plus proche de cet article ; à gauche, c’est plus ancien. Les traits relient les articles proches entre eux : les groupes de traits sont des sous-thèmes. Sélectionnez un point pour voir pourquoi il est là. Retour au résumé de cet article

Comment lire cette carte ? Guide illustré

Les articles les plus proches Les travaux antérieurs sur lesquels il s’appuie Les articles qui le citent

Les articles autour de celui-ci

Classés selon la proximité de leur titre et de leur résumé avec ceux de cet article. Sélectionnez-en pour les lire ensemble.

Les articles les plus proches

  1. Très proche
    The Prompt Artists
    Minsuk Chang, Stefania Druga, A. Fiannaca et al. · 2023 · Creativity & Cognition · cité 127 fois · Pas encore résumé
    Résumé scientifique

    This paper examines the art practices, artwork, and motivations of prolific users of the latest generation of text-to-image models. Through interviews, observations, and a user survey, we present a sampling of the artistic styles and describe the developed community of practice around generative AI. We find that: 1) artists hold the text prompt and the resulting image can be considered collectively as a form of artistic expression (prompts as art), and 2) prompt templates (prompts with “slots” for others to fill in with their own words) are developed to create generative art styles. We discover that the value placed by this community on unique outputs leads to artists seeking specialized voc…

  2. Très proche Cite cet article
    PrompTHis: Visualizing the Process and Influence of Prompt Editing During Text-to-Image Creation
    Yuhan Guo, Hanning Shao, Can Liu et al. · 2024 · IEEE Transactions on Visualization and Computer Graphics · cité 38 fois · Pas encore résumé
    Résumé scientifique

    Generative text-to-image models, which allow users to create appealing images through a text prompt, have seen a dramatic increase in popularity in recent years. However, most users have a limited understanding of how such models work and often rely on trial and error strategies to achieve satisfactory results. The prompt history contains a wealth of information that could provide users with insights into what has been explored and how the prompt changes impact the output image, yet little research attention has been paid to the visual analysis of such process to support users. We propose the Image Variant Graph, a novel visual representation designed to support comparing prompt-image pairs …

  3. Très proche Cite cet article Utilise cet article en profondeur
    Examining the Text-to-Image Community of Practice: Why and How do People Prompt Generative AIs?
    Téo Sanchez · 2023 · Creativity & Cognition · cité 64 fois · Pas encore résumé
    Résumé scientifique

    Image generation gained popularity with machine learning (ML) models generating images from text, fuelling new online communities of practices. This work explores the sociology, motivations, and usages of AI art hobbyists. We analyzed an online questionnaire answered by 64 practitioners and a dataset of user prompts sent to the Stable Diffusion generative model. Our findings suggest that TTI generation is a recreational activity mainly conducted by narrow socio-demographic groups who use auxiliary techniques across platforms and beyond request-response interactions. Inherent model limitations and finding suitable prompt formulation are the main obstacles practitioners face. A taxonomy and a …

  4. Très proche Cite cet article
    PromptMagician: Interactive Prompt Engineering for Text-to-Image Creation
    Yingchaojie Feng, Xingbo Wang, Kam-Kwai Wong et al. · 2023 · IEEE Transactions on Visualization and Computer Graphics · cité 183 fois · Pas encore résumé
    Résumé scientifique

    Generative text-to-image models have gained great popularity among the public for their powerful capability to generate high-quality images based on natural language prompts. However, developing effective prompts for desired images can be challenging due to the complexity and ambiguity of natural language. This research proposes PromptMagician, a visual analysis system that helps users explore the image results and refine the input prompts. The backbone of our system is a prompt recommendation model that takes user prompts as input, retrieves similar prompt-image pairs from DiffusionDB, and identifies special (important and relevant) prompt keywords. To facilitate interactive prompt refineme…

  5. Très proche
    Beyond Text-to-Image: Multimodal Prompts to Explore Generative AI
    Vivian Liu · 2023 · CHI Extended Abstracts · cité 44 fois · Pas encore résumé
    Résumé scientifique

    Text-to-image AI systems have proven to have extraordinary generative capacities that have facilitated widespread adoption. However, these systems are primarily text-based, which is a fundamental inversion of what many artists are traditionally used to: having full control over the composition of their work. Prior work has shown that there is great utility in using text prompts and that AI augmented workflows can increase momentum on creative tasks for end users. However, multimodal interactions beyond text need to be further defined, so end users can have rich points of interaction that allow them to truly co-pilot AI-generated content creation. To this end, the goal of my research is to eq…

  6. Très proche Cité par cet article
    Design Guidelines for Prompt Engineering Text-to-Image Generative Models
    Vivian Liu, Lydia B. Chilton · 2021 · International Conference on Human Factors in Computing Systems · cité 796 fois · Pas encore résumé
    Résumé scientifique

    Text-to-image generative models are a new and powerful way to generate visual artwork. However, the open-ended nature of text as interaction is double-edged; while users can input anything and have access to an infinite range of generations, they also must engage in brute-force trial and error with the text prompt when the result quality is poor. We conduct a study exploring what prompt keywords and model hyperparameters can help produce coherent outputs. In particular, we study prompts structured to include subject and style keywords and investigate success and failure modes of these prompts. Our evaluation of 5493 generations over the course of five experiments spans 51 abstract and concre…

  7. Très proche
    PromptCharm: Text-to-Image Generation through Multi-modal Prompting and Refinement
    Zhi-Jie Wang, Yu-Heng Huang, Da Song et al. · 2024 · International Conference on Human Factors in Computing Systems · cité 205 fois · Pas encore résumé
    Résumé scientifique

    The recent advancements in Generative AI have significantly advanced the field of text-to-image generation. The state-of-the-art text-to-image model, Stable Diffusion, is now capable of synthesizing high-quality images with a strong sense of aesthetics. Crafting text prompts that align with the model’s interpretation and the user’s intent thus becomes crucial. However, prompting remains challenging for novice users due to the complexity of the stable diffusion model and the non-trivial efforts required for iteratively editing and refining the text prompts. To address these challenges, we propose PromptCharm, a mixed-initiative system that facilitates text-to-image creation through multi-moda…

  8. Très proche Cité par cet article Cite cet article Utilisé en profondeur par cet article
    Prompting AI Art: An Investigation into the Creative Skill of Prompt Engineering
    Jonas Oppenlaender, Rhema Linder, Johanna Silvennoinen · 2023 · Human-Computer Interaction · cité 282 fois · Résumé IA disponible
    Résumé scientifique

    We are witnessing a novel era of creativity where anyone can create digital content via prompt-based learning (known as prompt engineering). This paper delves into prompt engineering as a novel creative skill for creating AI art with text-to-image generation. In a pilot study, we find that many crowdsourced participants have knowledge about art which could be used for writing effective prompts. In three subsequent studies, we explore whether crowdsourced participants can put this knowledge into practice. We examine if participants can 1) discern prompt quality, 2) write prompts, and 3) refine prompts. We find that participants could evaluate prompt quality and crafted descriptive prompts, bu…

Les travaux antérieurs sur lesquels il s’appuie

  1. Proche Cite cet article Utilisé en profondeur par cet article
    The Creativity of Text-to-Image Generation
    J. Oppenlaender · 2022 · International Conference on Entertainment and Media in the Ubiquitous Era · cité 328 fois · Pas encore résumé
    Résumé scientifique

    Text-guided synthesis of images has made a giant leap towards becoming a mainstream phenomenon. With text-to-image generation systems, anybody can create digital images and artworks. This provokes the question of whether text-to-image generation is creative. This paper expounds on the nature of human creativity involved in text-to-image art (so-called “AI art”) with a specific focus on the practice of prompt engineering. The paper argues that the current product-centered view of creativity falls short in the context of text-to-image generation. A case exemplifying this shortcoming is provided and the importance of online communities for the creative ecosystem of text-to-image art is highligh…

  2. Très proche
    Is Writing Prompts Really Making Art?
    J. Mccormack, Camilo Cruz Gambardella, N. Rajcic et al. · 2023 · EvoMUSART@EvoStar · cité 46 fois · Pas encore résumé
    Résumé scientifique

    In recent years Generative Machine Learning systems have advanced significantly. A current wave of generative systems use text prompts to create complex imagery, video, even 3D datasets. The creators of these systems claim a revolution in bringing creativity and art to anyone who can type a prompt. In this position paper, we question the basis for these claims, dividing our analysis into three areas: the limitations of linguistic descriptions, implications of the dataset, and lastly, matters of materiality and embodiment. We conclude with an analysis of the creative possibilities enabled by prompt-based systems, asking if they can be considered a new artistic medium.

  3. Très proche
    Initial Images: Using Image Prompts to Improve Subject Representation in Multimodal AI Generated Art
    Han Qiao, Vivian Liu, Lydia B. Chilton · 2022 · Creativity & Cognition · cité 50 fois · Pas encore résumé
    Résumé scientifique

    Advances in text-to-image generative models have made it easier for people to create art by just prompting models with text. However, creating through text leaves users with limited control over the final composition or the way the subject is represented. A potential solution is to use image prompts alongside text prompts to condition the model. To better understand how and when image prompts can improve subject representation in generations, we conduct an annotation experiment to quantify their effect on generations of abstract, concrete plural, and concrete singular subjects. We find that initial images improved subject representation across all subject types, with the most noticeable impr…

  4. Proche Utilisé en profondeur par cet article
    PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts
    S. Bach, Victor Sanh, Zheng-Xin Yong et al. · 2022 · Annual Meeting of the Association for Computational Linguistics · cité 436 fois · Pas encore résumé
    Résumé scientifique

    PromptSource is a system for creating, sharing, and using natural language prompts. Prompts are functions that map an example from a dataset to a natural language input and target output. Using prompts to train and query language models is an emerging area in NLP that requires new tools that let users develop and refine these prompts collaboratively. PromptSource addresses the emergent challenges in this new setting with (1) a templating language for defining data-linked prompts, (2) an interface that lets users quickly iterate on prompt development by observing outputs of their prompts on many examples, and (3) a community-driven set of guidelines for contributing new prompts to a common po…

  5. Apparenté* Utilisé en profondeur par cet article
    The Creativity of Text-based Generative Art
    J. Oppenlaender · 2022 · arXiv.org · preprint, non relu par les pairs · cité 17 fois · Pas encore résumé

Les articles qui le citent

  1. Très proche Utilise cet article en profondeur
    Promptify: Text-to-Image Generation through Interactive Prompt Exploration with Large Language Models
    Stephen Brade, Bryan Wang, Maurício Sousa et al. · 2023 · ACM Symposium on User Interface Software and Technology · cité 230 fois · Pas encore résumé
    Résumé scientifique

    Text-to-image generative models have demonstrated remarkable capabilities in generating high-quality images based on textual prompts. However, crafting prompts that accurately capture the user’s creative intent remains challenging. It often involves laborious trial-and-error procedures to ensure that the model interprets the prompts in alignment with the user’s intention. To address these challenges, we present Promptify, an interactive system that supports prompt exploration and refinement for text-to-image generative models. Promptify utilizes a suggestion engine powered by large language models to help users quickly explore and craft diverse prompts. Our interface allows users to organize…

  2. Proche Utilise cet article en profondeur
    Prompt Stealing Attacks Against Text-to-Image Generation Models
    Xinyue Shen, Yi-Ting Qu, M. Backes et al. · 2023 · USENIX Security Symposium · cité 84 fois · Pas encore résumé
    Résumé scientifique

    Text-to-Image generation models have revolutionized the artwork design process and enabled anyone to create high-quality images by entering text descriptions called prompts. Creating a high-quality prompt that consists of a subject and several modifiers can be time-consuming and costly. In consequence, a trend of trading high-quality prompts on specialized marketplaces has emerged. In this paper, we perform the first study on understanding the threat of a novel attack, namely prompt stealing attack, which aims to steal prompts from generated images by text-to-image generation models. Successful prompt stealing attacks directly violate the intellectual property of prompt engineers and jeopard…

  3. Proche Utilise cet article en profondeur
    Can (A)I Have a Word with You? A Taxonomy on the Design Dimensions of AI Prompts
    Marvin Braun, Maike Greve, Felix Kegel et al. · 2024 · Hawaii International Conference on System Sciences · cité 11 fois · Pas encore résumé
    Résumé scientifique

    Emerging generative artificial intelligence (AI) is revolutionizing human collaboration with machines through their remarkable processing of human language. Consequently, the role of AI is transitioning as it becomes adaptable to specific tasks through human language, without the need for additional training data. These "prompts" enable end-users to fully harness the potential of AI. However, the process of designing prompts is currently unstructured, relying mostly on trial-and-error processes. Studies emphasize the significant impact of prompt design on the AI's generated results. Therefore, there is a clear need to establish a structured approach to prompt design. To address this need, we…

  4. Très proche
    Optimizing Prompts for Text-to-Image Generation
    Y. Hao, Zewen Chi, Li Dong et al. · 2022 · Neural Information Processing Systems · cité 321 fois · Pas encore résumé
    Résumé scientifique

    Well-designed prompts can guide text-to-image models to generate amazing images. However, the performant prompts are often model-specific and misaligned with user input. Instead of laborious human engineering, we propose prompt adaptation, a general framework that automatically adapts original user input to model-preferred prompts. Specifically, we first perform supervised fine-tuning with a pretrained language model on a small collection of manually engineered prompts. Then we use reinforcement learning to explore better prompts. We define a reward function that encourages the policy to generate more aesthetically pleasing images while preserving the original user intentions. Experimental r…

  5. Proche
    Dynamic Prompt Optimizing for Text-to-Image Generation
    Wenyi Mo, Tianyu Zhang, Ya-Long Bai et al. · 2024 · Computer Vision and Pattern Recognition · cité 61 fois · Pas encore résumé
    Résumé scientifique

    Text-to-image generative models, specifically those based on diffusion models like Imagen and Stable Diffusion, have made substantial advancements. Recently, there has been a surge of interest in the delicate refinement of text prompts. Users assign weights or alter the injection time steps of certain words in the text prompts to improve the quality of generated images. However, the success of fine-control prompts depends on the accuracy of the text prompts and the careful selection of weights and time steps, which requires significant manual intervention. To address this, we introduce the Prompt Auto-Editing (PAE) method. Besides refining the original prompts for image generation, we furthe…

Comment ces articles sont-ils trouvés ?

Nous comparons le titre et le résumé de cet article avec ceux de tous les articles résumés sur ce site, et avec les articles que Semantic Scholar recommande, ceux qu’il cite et ceux qui le citent (pour un article très cité, ses citations les plus récentes et, depuis OpenAlex, les plus citées). Parmi les travaux antérieurs et les suites, les articles qui s’utilisent en profondeur, une méthode qu’il utilise et les suites très citées remontent un peu. Les niveaux (très proche, proche, apparenté) viennent de la similarité des textes, pas d’un jugement de qualité ; * signale un article sans résumé, comparé par son titre. La phrase sous un article est écrite par une IA à partir des deux résumés : vérifiez-la dans les articles. Un article jugé hors sujet par trois lecteurs quitte la liste.

Mis à jour le 6 novembre 2026.

Le guide illustré des articles autour d’un article