# Large Language Models as Tax Attorneys: A Case Study in Legal Capabilities Emergence

**Authors:** John J. Nay, David Karamardian, Sarah B. Lawsky, Wenting Tao, Meghana Bhat, Raghav Jain, Aaron Travis Lee, Jonathan H. Choi, Jungo Kasai  
**arXiv:** [2306.07075](https://arxiv.org/abs/2306.07075)  
**Date:** 2023-06-12  
**Summary page:** https://www.summarizepaper.com/en/paper/2306.07075v1/

## Key points

- Large Language Models (LLMs) can contribute to improving legal services, AI governance, and identifying inconsistencies in tax law.
- Tax law was chosen as the focus for experiments due to its relevance and the ability to set up automated validation pipelines.
- LLMs have emerging legal understanding capabilities, with improved performance in each subsequent model release by OpenAI.
- Providing additional legal context and using few-shot prompting techniques enhances the performance of LLMs.
- LLMs can perform at high levels of accuracy when provided with correct legal texts but are not yet at expert tax lawyer levels.
- Advancements in LLMs could have significant implications for the legal profession and AI governance.
- Other studies highlight various applications of LLMs, such as generating code, improving models with external knowledge, and open-domain question answering techniques.

## Summary

This paper explores the capabilities of Large Language Models (LLMs) in applying tax law and how it can contribute to improving legal services, AI governance, and identifying inconsistencies in the law. The authors chose tax law as their focus because it allows them to set up automated validation pipelines, requires logical reasoning and math skills, and is relevant to the real-world economic lives of citizens and companies. Through experiments, they demonstrate that LLMs have emerging legal understanding capabilities, with improved performance in each subsequent model release by OpenAI. They also find that providing additional legal context to LLMs enhances their performance, particularly when combined with few-shot prompting techniques. However, while LLMs can perform at high levels of accuracy when provided with the correct legal texts, they are not yet at expert tax lawyer levels. The advancement of LLMs could have significant implications for the legal profession and AI governance as they continue to improve their ability to reason about law autonomously. Several related studies have been conducted on language models including research on discovering distributional differences through language descriptions, improving science question-answering through supervised reasoning processes, active prompting with chain-of-thought for large language models, scalable prompt generation for semi-supervised learning with language models, using language models for computer tasks, measuring and narrowing the compositionality gap in language models, synergizing reasoning and acting in language models (ReAct), exploring language models as accounts of human moral judgment, bounding the capabilities of large language models in open text generation with prompt constraints. Other studies focus on augmented language models through surveys or specific applications such as generating code by retrieving documentation (DocPrompting), improving large language models with external knowledge and automated feedback, learning to play Atari games with the help of instruction manuals (Read and Reap the Rewards), open-domain question answering techniques. There are also studies on recitation-augmented language models (Recitation-Augmented Language Models), composing retrieval and language models for knowledge-intensive NLP (Demonstrate-Search-Predict), compositional exemplars for in-context learning, and in-context retrieval-augmented language models. Overall these studies highlight the potential of LLMs in various applications and demonstrate a need to further explore their capabilities and limitations. The advancements made so far show promise towards revolutionizing the legal profession and AI governance but there is still progress needed before they can match human professionals' expertise.