Prompting Is Programming: A Query Language For Large Language Models

Authors: Luca Beurer-Kellner, Marc Fischer, Martin Vechev

Submitted to arXiv on 12 Dec. 2022
License: CC BY 4.0
Abstract

Large language models have demonstrated outstanding performance on a wide range of tasks such as question answering and code generation. On a high level, given an input, a language model can be used to automatically complete the sequence in a statistically-likely way. Based on this, users prompt these models with language instructions or examples, to implement a variety of downstream tasks. Advanced prompting methods can even imply interaction between the language model, a user, and external tools such as calculators. However, to obtain state-of-the-art performance or adapt language models for specific tasks, complex task- and model-specific programs have to be implemented, which may still require ad-hoc interaction. Based on this, we present the novel idea of Language Model Programming (LMP). LMP generalizes language model prompting from pure text prompts to an intuitive combination of text prompting and scripting. Additionally, LMP allows constraints to be specified over the language model output. This enables easy adaption to many tasks, while abstracting language model internals and providing high-level semantics. To enable LMP, we implement LMQL (short for Language Model Query Language), which leverages the constraints and control flow from an LMP prompt to generate an efficient inference procedure that minimizes the number of expensive calls to the underlying language model. We show that LMQL can capture a wide range of state-of-the-art prompting methods in an intuitive way, especially facilitating interactive flows that are challenging to implement with existing high-level APIs. Our evaluation shows that we retain or increase the accuracy on several downstream tasks, while also significantly reducing the required amount of computation or cost in the case of pay-to-use APIs (13-85% cost savings).

The papers around it, by date and closeness

Higher is closer to this paper; left is older. Lines join papers close to each other: groups of lines are sub-topics. Select a point to see why it is there. Back to the summary of this paper

How to read this map? Illustrated guide

Closest papers Prior work it builds on Papers that cite it

Papers around this one

Ranked by how close their title and abstract are to this paper’s. Select some to read them together.

Closest papers

  1. Very close
    Automatic Prompt Selection for Large Language Models
    Viet-Tung Do, Van-Khanh Hoang, Duy-Hung Nguyen et al. · 2024 · Computation and Language · preprint, not peer reviewed · cited 16 times · AI summary available
    Abstract

    Large Language Models (LLMs) can perform various natural language processing tasks with suitable instruction prompts. However, designing effective prompts manually is challenging and time-consuming. Existing methods for automatic prompt optimization either lack flexibility or efficiency. In this paper, we propose an effective approach to automatically select the optimal prompt for a given input from a finite set of synthetic candidate prompts. Our approach consists of three steps: (1) clustering the training data and generating candidate prompts for each cluster using an LLM-based prompt generator; (2) synthesizing a dataset of input-prompt-output tuples for training a prompt evaluator to ra…

  2. Very close Cited by this paper A method it uses
    Large Language Models Are Human-Level Prompt Engineers
    Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han et al. · 2022 · Machine Learning · cited 1,708 times · AI summary available
    Abstract

    By conditioning on natural language instructions, large language models (LLMs) have displayed impressive capabilities as general-purpose computers. However, task performance depends significantly on the quality of the prompt used to steer the model, and most effective prompts have been handcrafted by humans. Inspired by classical program synthesis and the human approach to prompt engineering, we propose Automatic Prompt Engineer (APE) for automatic instruction generation and selection. In our method, we treat the instruction as the "program," optimized by searching over a pool of instruction candidates proposed by an LLM in order to maximize a chosen score function. To evaluate the quality o…

  3. Very close
    Prompt Engineering and Optimization in Large Language Models
    Ranjith Gundam, Kalyan Mandadhi · 2026 · International Scientific Journal of Engineering and Management · Not summarized yet
    Abstract

    Abstract: Large Language Models (LLMs) have emerged as transformative technologies, enabling sophisticated natural language understanding and generation across diverse applications. However, their effectiveness is heavily dependent on how instructions and queries are formulated. Prompt engineering—the art and science of crafting optimal input instructions—has become critical to unlocking the full potential of these models. This systematic review investigates contemporary approaches to prompt engineering and optimization in LLMs, examining techniques such as few-shot learning, chain-of-thought prompting, retrieval augmented generation, and dynamic prompt optimization. The study analyzes vario…

  4. Very close
    Reflective Prompt Tuning through Language Model Function-Calling
    Farima Fatahi Bayat, Moin Aminnaseri, Pouya Pezeshkpour et al. · 2026 · arXiv.org · preprint, not peer reviewed · cited 2 times · Not summarized yet
    Abstract

    Large language models (LLMs) have become increasingly capable of following instructions and complex reasoning, making prompting a flexible interface for adapting models without parameter updates. Yet prompt design remains labor-intensive and highly sensitive to formatting, phrasing, and instruction order, motivating automated prompt optimization methods that reduce manual effort while preserving inference-time flexibility. However, existing methods often search over prompt candidates or use fixed critique-refine pipelines driven by individual examples or small batches, limiting their ability to capture systematic error patterns and make targeted edits grounded in failure history. We propose …

  5. Very close Cites this paper
    Imprompt: A Language Framework for Prompt Programming
    Chentian Wu, Shengyuan Yang, Adithya Murali · 2026 · arXiv.org · preprint, not peer reviewed · Not summarized yet
    Abstract

    With the unprecedented success of Language Models (LMs), the science of Prompt Engineering has evolved the powerful idea of Prompt Programming, where prompts are treated as a programmable control surface for describing complex tasks and leveraging LM capabilities. However, existing prompt programming frameworks suffer from various complexities and inelegances, which make them hard to utilize in practice for effectively describing tasks. We propose Imprompt, a new language framework for the study and practice of prompt programming. We undertake a foundational investigation of prompt programming, and contend that prompt programs must contain only the task descriptions and must be decoupled fro…

  6. Very close
    Enhancing LLM Efficiency: A Literature Review of Emerging Prompt Optimization Strategies
    Asyafa Ditra Al Hauna, Andi Prademon Yunus, Masanori Fukui et al. · 2025 · International Journal on Robotics Automation and Sciences · cited 3 times · Not summarized yet
    Abstract

    This study focuses on enhancing the performance of Large Language Models (LLMs) through innovative prompt engineering techniques aimed at optimizing outputs without the high computational costs of model fine-tuning or retraining. The primary objective is to investigate efficient alternatives, such as black-box prompt optimization and ontology-based prompt refinement, which improve LLM performance by refining prompts externally while maintaining the model's internal parameters. The study explores various prompt optimization techniques, including instruction-based, role-based, question-answering, and contextual prompting, alongside advanced methods like CoT and ToT prompting. Methodologically,…

  7. Very close
    Progressive-Hint Prompting Improves Reasoning in Large Language Models
    Chuanyang Zheng, Zheng-Ying Liu, En-Ze Xie et al. · 2023 · arXiv.org · preprint, not peer reviewed · cited 165 times · Not summarized yet
    Abstract

    The performance of Large Language Models (LLMs) in reasoning tasks depends heavily on prompt design, with Chain-of-Thought (CoT) and self-consistency being critical methods that enhance this ability. However, these methods do not fully exploit the answers generated by the LLM to guide subsequent responses. This paper proposes a new prompting method, named Progressive-Hint Prompting (PHP), that enables automatic multiple interactions between users and LLMs by using previously generated answers as hints to progressively guide toward the correct answers. PHP is orthogonal to CoT and self-consistency, making it easy to combine with state-of-the-art techniques to further improve performance. We c…

  8. Very close
    Exploring Techniques and Applications of Prompt Engineering for Large Language Models
    Princy Medona, L. A. Felicien, N. Mani · 2025 · International Journal of Scientific Research in Engineering and Management · Not summarized yet
    Abstract

    Abstract - Prompt engineering has become an effective approach for improving the performance of large language models (LLMs) and vision–language models (VLMs) without modifying their internal architecture. By designing precise instructions, structured formats, and illustrative examples, prompts guide models to interpret tasks more accurately and generate more reliable outputs. Despite its rapid growth and significant impact on areas such as question answering, reasoning, and multimodal tasks, the field still lacks a well-organized overview of its diverse techniques. This paper presents a structured survey of recent advances in prompt engineering, explaining how each technique functions, the …

Prior work it builds on

  1. Very close Used in depth by this paper A method it uses
    Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm
    Laria Reynolds, Kyle McDonell · 2021 · Computation and Language · cited 1,462 times · AI summary available
    Abstract

    Prevailing methods for mapping large generative language models to supervised tasks may fail to sufficiently probe models' novel capabilities. Using GPT-3 as a case study, we show that 0-shot prompts can significantly outperform few-shot prompts. We suggest that the function of few-shot examples in these cases is better described as locating an already learned task rather than meta-learning. This analysis motivates rethinking the role of prompts in controlling and evaluating powerful language models. In this work, we discuss methods of prompt programming, emphasizing the usefulness of considering prompts through the lens of natural language. We explore techniques for exploiting the capacity …

  2. Close Used in depth by this paper A method it uses
    Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
    Jason Wei, Xuezhi Wang, Dale Schuurmans et al. · 2022 · Computation and Language · cited 23,723 times · AI summary available
    Abstract

    We explore how generating a chain of thought -- a series of intermediate reasoning steps -- significantly improves the ability of large language models to perform complex reasoning. In particular, we show how such reasoning abilities emerge naturally in sufficiently large language models via a simple method called chain of thought prompting, where a few chain of thought demonstrations are provided as exemplars in prompting. Experiments on three large language models show that chain of thought prompting improves performance on a range of arithmetic, commonsense, and symbolic reasoning tasks. The empirical gains can be striking. For instance, prompting a 540B-parameter language model with just…

  3. Very close
    PAL: Program-aided Language Models
    Luyu Gao, Aman Madaan, Shuyan Zhou et al. · 2022 · Computation and Language · cited 910 times · AI summary available
    Abstract

    Large language models (LLMs) have recently demonstrated an impressive ability to perform arithmetic and symbolic reasoning tasks when provided with a few examples at test time (few-shot prompting). Much of this success can be attributed to prompting methods for reasoning, such as chain-of-thought, that employ LLMs for both understanding the problem description by decomposing it into steps, as well as solving each step of the problem. While LLMs seem to be adept at this sort of step-by-step decomposition, LLMs often make logical and arithmetic mistakes in the solution part, even when the problem is correctly decomposed. We present Program-Aided Language models (PaL): a new method that uses th…

  4. Close Used in depth by this paper A method it uses
    ReAct: Synergizing Reasoning and Acting in Language Models
    Shunyu Yao, Jeffrey Zhao, Dian Yu et al. · 2022 · International Conference on Learning Representations · cited 13,626 times · Not summarized yet
    Abstract

    While large language models (LLMs) have demonstrated impressive capabilities across tasks in language understanding and interactive decision making, their abilities for reasoning (e.g. chain-of-thought prompting) and acting (e.g. action plan generation) have primarily been studied as separate topics. In this paper, we explore the use of LLMs to generate both reasoning traces and task-specific actions in an interleaved manner, allowing for greater synergy between the two: reasoning traces help the model induce, track, and update action plans as well as handle exceptions, while actions allow it to interface with external sources, such as knowledge bases or environments, to gather additional in…

  5. Related Used in depth by this paper A method it uses
    Language Models are Few-Shot Learners
    Tom B. Brown, Benjamin Mann, Nick Ryder et al. · 2020 · Neural Information Processing Systems · cited 66,241 times · AI summary available
    Abstract

    Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task. While typically task-agnostic in architecture, this method still requires task-specific fine-tuning datasets of thousands or tens of thousands of examples. By contrast, humans can generally perform a new language task from only a few examples or from simple instructions - something which current NLP systems still largely struggle to do. Here we show that scaling up language models greatly improves task-agnostic, few-shot performance, sometimes even reaching competitiveness with prior state-of-the-art fine-tuning approaches. Spec…

Papers that cite it

  1. Very close
    DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines
    Omar Khattab, Arnav Singhvi, Paridhi Maheshwari et al. · 2023 · Computation and Language · preprint, not peer reviewed · cited 1,163 times · AI summary available
    Abstract

    The ML community is rapidly exploring techniques for prompting language models (LMs) and for stacking them into pipelines that solve complex tasks. Unfortunately, existing LM pipelines are typically implemented using hard-coded "prompt templates", i.e. lengthy strings discovered via trial and error. Toward a more systematic approach for developing and optimizing LM pipelines, we introduce DSPy, a programming model that abstracts LM pipelines as text transformation graphs, i.e. imperative computational graphs where LMs are invoked through declarative modules. DSPy modules are parameterized, meaning they can learn (by creating and collecting demonstrations) how to apply compositions of prompti…

  2. Close Uses this paper in depth
    ART: Automatic multi-step reasoning and tool-use for large language models
    Bhargavi Paranjape, Scott Lundberg, Sameer Singh et al. · 2023 · Computation and Language · preprint, not peer reviewed · cited 239 times · AI summary available
    Abstract

    Large language models (LLMs) can perform complex reasoning in few- and zero-shot settings by generating intermediate chain of thought (CoT) reasoning steps. Further, each reasoning step can rely on external tools to support computation beyond the core LLM capabilities (e.g. search/running code). Prior work on CoT prompting and tool use typically requires hand-crafting task-specific demonstrations and carefully scripted interleaving of model generations with tool use. We introduce Automatic Reasoning and Tool-use (ART), a framework that uses frozen LLMs to automatically generate intermediate reasoning steps as a program. Given a new task to solve, ART selects demonstrations of multi-step reas…

  3. Close Uses this paper in depth
    SGLang: Efficient Execution of Structured Language Model Programs
    Lian-Min Zheng, Liangsheng Yin, Zhi-Qiang Xie et al. · 2023 · Neural Information Processing Systems · cited 1,843 times · AI summary available
    Abstract

    Large language models (LLMs) are increasingly used for complex tasks that require multiple generation calls, advanced prompting techniques, control flow, and structured inputs/outputs. However, efficient systems are lacking for programming and executing these applications. We introduce SGLang, a system for efficient execution of complex language model programs. SGLang consists of a frontend language and a runtime. The frontend simplifies programming with primitives for generation and parallelism control. The runtime accelerates execution with novel optimizations like RadixAttention for KV cache reuse and compressed finite state machines for faster structured output decoding. Experiments show…

  4. Very close
    APPL: A Prompt Programming Language for Harmonious Integration of Programs and Large Language Model Prompts
    Honghua Dong, Qidong Su, Yu-Bo Gao et al. · 2024 · Annual Meeting of the Association for Computational Linguistics · cited 12 times · Not summarized yet
    Abstract

    Large Language Models (LLMs) have become increasingly capable of handling diverse tasks with the aid of well-crafted prompts and integration of external tools, but as task complexity rises, the workflow involving LLMs can be complicated and thus challenging to implement and maintain. To address this challenge, we propose APPL, A Prompt Programming Language that acts as a bridge between computer programs and LLMs, allowing seamless embedding of prompts into Python functions, and vice versa. APPL provides an intuitive and Python-native syntax, an efficient parallelized runtime with asynchronous semantics, and a tracing module supporting effective failure diagnosis and replaying without extra c…

  5. Very close
    AskIt: Unified Programming Interface for Programming with Large Language Models
    Katsumi Okuda, Saman P. Amarasinghe · 2023 · IEEE/ACM International Symposium on Code Generation and Optimization · cited 10 times · Not summarized yet
    Abstract

    Large Language Models (LLMs) exhibit a unique phenomenon known as emergent abilities, demonstrating adeptness across numerous tasks, from text summarization to code generation. While these abilities open up novel avenues in software design and crafting, their incorporation presents substantial challenges. Developers face decisions regarding the use of LLMs for directly performing tasks within applications as well as for generating and executing code to accomplish these tasks. Moreover, effective prompt design becomes a critical concern, given the necessity of extracting data from natural language outputs. To address these complexities, this paper introduces AskIt, a domain-specific language …

How are these papers found?

We compare the title and abstract of this paper with those of every paper summarized on this site, and with the papers Semantic Scholar recommends, those it cites and those citing it (for a much-cited paper, its most recent citations and, from OpenAlex, the most cited ones). Among the prior work and the follow-ups, the papers that use each other in depth, a method it uses and the much-cited follow-ups come a little higher. The levels (very close, close, related) come from the similarity of the texts, not from a judgment of quality; * marks a paper without an abstract, compared by its title. The sentence under a paper is written by an AI from the two abstracts: check it in the papers. A paper marked not relevant by three readers leaves the list.

Updated on 6 November 2026.

The illustrated guide to the papers around a paper