# Can LLMs Generate Architectural Design Decisions? -An Exploratory Empirical study

**Authors:** Rudra Dhar, Karthik Vaidhyanathan, Vasudeva Varma  
**arXiv:** [2403.01709](https://arxiv.org/abs/2403.01709)  
**Date:** 2024-03-04  
**Summary page:** https://www.summarizepaper.com/en/paper/2403.01709v1/

## Key points

- Study focuses on Architectural Knowledge Management (AKM) and use of Large Language Models (LLMs) for Architecture Decision Records (ADRs)
- Utilizes Developer-Intent Driven Code Comment Generation and Automatic Identification of Decisions from developer mailing lists
- Tools like ADeX used for automatic curation of design decision knowledge
- Evaluation metrics include ROUGE, BLEU, METEOR, and BERTScore
- Experiment involves gathering 95 ADRs from repositories like archane-framework, winery, joelparkerhenderson's repository, cardano, and island
- LLM models explored include GPT-2, GPT-3, GPT-3.5, GPT-4, T5 in different sizes (small to XL), T0 models like ada and davinci along with Flan-T5 variants
- Results show that state-of-the-art models like GPT-4 can generate relevant Design Decisions in a 0-shot setting but fall short of human-level performance
- More cost-effective models such as GPT-3.5 show promise in few-shot settings while smaller models like Flan-T5 can yield comparable results after fine-tuning

## Summary

, , , , 


In this expanded study on Architectural Knowledge Management (AKM) and the use of Large Language Models (LLMs) for Architecture Decision Records (ADRs), we delve into the realm of Developer-Intent Driven Code Comment Generation, Automatic Identification of Decisions from developer mailing lists, and tools like ADeX for automatic curation of design decision knowledge. Drawing from foundational works such as the Goal Question Metric Approach by Basili et al. and metrics like ROUGE, BLEU, METEOR, and BERTScore for evaluation, we aim to enhance our understanding of how LLMs can revolutionize ADR generation. Our experimental subject involves gathering ADR data from various repositories like archane-framework, winery, joelparkerhenderson's repository, cardano, and island. Through web crawling and manual extraction processes, we obtained 95 ADRs that adhere to a standard format. Focusing on extracting the Context and Decision components from these ADRs, we aim to leverage LLMs for generating Design Decisions based on given contexts. We explore a range of LLM models including GPT-2, GPT-3, GPT-3.5, GPT-4, T5 in different sizes (small to XL), T0 models like ada and davinci along with Flan-T5 variants. By experimenting with 0-shot, few-shot, and fine-tuning approaches using these models on our extracted ADR data samples as shown in Figure 2 - where Python is chosen as the primary programming language - we assess their effectiveness in generating accurate Design Decisions. Our results indicate that state-of-the-art models like GPT-4 can generate relevant Design Decisions in a 0-shot setting but fall short of human-level performance. However, more cost-effective models such as GPT-3.5 show promise in few-shot settings while smaller models like Flan-T5 can yield comparable results after fine-tuning. This exploratory study suggests that LLMs have potential for generating Design Decisions but further research is needed to achieve human-level generation and establish standardized widespread adoption in AKM practices.