InstructExcel: A Benchmark for Natural Language Instruction in Excel

AI-generated keywords: Large Language Models Natural Language Processing Excel InstructExcel Benchmark GPT-4

AI-generated Key Points

The license of the paper does not allow us to build upon its content and the key points are generated using the paper metadata rather than the full article.

  • The paper explores the capabilities of Large Language Models (LLMs) in solving complex Natural Language Processing (NLP) tasks within the domain of spreadsheets.
  • Focuses on whether LLMs can generate code using Excel OfficeScripts to address specific Excel operations based on natural language user instructions.
  • Introduces a large-scale benchmark called InstructExcel comprising over 10,000 samples encompassing 170+ Excel operations extracted from 2,000 publicly available Excel spreadsheets.
  • Experimental results show that InstructExcel poses a significant challenge for state-of-the-art models like GPT-4.
  • Findings suggest that utilizing GPT-4 and providing more contextual examples can enhance model performance on this benchmark.
  • Employing dynamic prompting strategies can improve model performance on challenging benchmarks like InstructExcel.
Also access our AI generated: Comprehensive summary, Lay summary, Blog-like article; or ask questions about this paper to our AI assistant.

Authors: Justin Payan, Swaroop Mishra, Mukul Singh, Carina Negreanu, Christian Poelitz, Chitta Baral, Subhro Roy, Rasika Chakravarthy, Benjamin Van Durme, Elnaz Nouri

Findings of EMNLP 2023, 18 pages

Abstract: With the evolution of Large Language Models (LLMs) we can solve increasingly more complex NLP tasks across various domains, including spreadsheets. This work investigates whether LLMs can generate code (Excel OfficeScripts, a TypeScript API for executing many tasks in Excel) that solves Excel specific tasks provided via natural language user instructions. To do so we introduce a new large-scale benchmark, InstructExcel, created by leveraging the 'Automate' feature in Excel to automatically generate OfficeScripts from users' actions. Our benchmark includes over 10k samples covering 170+ Excel operations across 2,000 publicly available Excel spreadsheets. Experiments across various zero-shot and few-shot settings show that InstructExcel is a hard benchmark for state of the art models like GPT-4. We observe that (1) using GPT-4 over GPT-3.5, (2) providing more in-context examples, and (3) dynamic prompting can help improve performance on this benchmark.

Submitted to arXiv on 23 Oct. 2023

Ask questions about this paper to our AI assistant

You can also chat with multiple papers at once here.

The license of the paper does not allow us to build upon its content and the AI assistant only knows about the paper metadata rather than the full article.

AI assistant instructions?

Results of the summarizing process for the arXiv paper: 2310.14495v1

This paper's license doesn't allow us to build upon its content and the summarizing process is here made with the paper's metadata rather than the article.

The paper "InstructExcel: A Benchmark for Natural Language Instruction in Excel" by Justin Payan, Swaroop Mishra, Mukul Singh, Carina Negreanu, Christian Poelitz, Chitta Baral, Subhro Roy, Rasika Chakravarthy, Benjamin Van Durme and Elnaz Nouri explores the capabilities of Large Language Models (LLMs) in solving complex Natural Language Processing (NLP) tasks within the domain of spreadsheets. The study focuses on whether LLMs can generate code using Excel OfficeScripts (a TypeScript API for executing tasks in Excel) to address specific Excel operations based on natural language user instructions. To facilitate this investigation, the authors introduce a novel large-scale benchmark called InstructExcel. This benchmark comprises over 10,000 samples encompassing 170+ Excel operations extracted from 2,000 publicly available Excel spreadsheets. The dataset was created by leveraging the 'Automate' feature in Excel to automatically generate OfficeScripts from users' actions. Experimental results across various zero-shot and few-shot settings demonstrate that InstructExcel poses a significant challenge for state-of-the-art models like GPT-4. The findings suggest that utilizing GPT-4 instead of GPT-3.5 and providing more contextual examples can enhance model performance on this benchmark. Additionally, employing dynamic prompting strategies can also improve model performance on challenging benchmarks like InstructExcel. Overall,the study highlights the potential of LLMs in generating code for Excel tasks based on natural language instructions and emphasizes the importance of tailored approaches to improve model performance on challenging benchmarks like InstructExcel.
Created on 09 Oct. 2024

Assess the quality of the AI-generated content by voting

Score: 0

Why do we need votes?

Votes are used to determine whether we need to re-run our summarizing tools. If the count reaches -10, our tools can be restarted.

Similar papers summarized with our AI tools

Navigate through even more similar papers through a

tree representation

Look for similar papers (in beta version)

By clicking on the button above, our algorithm will scan all papers in our database to find the closest based on the contents of the full papers and not just on metadata. Please note that it only works for papers that we have generated summaries for and you can rerun it from time to time to get a more accurate result while our database grows.

Disclaimer: The AI-based summarization tool and virtual assistant provided on this website may not always provide accurate and complete summaries or responses. We encourage you to carefully review and evaluate the generated content to ensure its quality and relevance to your needs.