# An Empirical Study on Using Large Language Models to Analyze Software Supply Chain Security Failures

**Authors:** Tanmay Singla, Dharun Anandayuvaraj, Kelechi G. Kalu, Taylor R. Schorlemmer, James C. Davis  
**arXiv:** [2308.04898](https://arxiv.org/abs/2308.04898)  
**Date:** 2023-08-09  
**Summary page:** https://www.summarizepaper.com/en/paper/2308.04898v1/

## Key points

- Breaches in the software supply chain have become more detrimental
- Recent high-profile cyber attacks (SolarWinds, ShadowHammer) highlight the need for stronger cybersecurity measures
- Traditional methods of analyzing failures are time-consuming and costly
- Natural Language Processing (NLP) techniques, specifically Large Language Models (LLMs), can assist in analyzing software supply chain breaches
- Empirical study assessed LLMs' effectiveness in analyzing historical failures
- GPT 3.5s achieved an average accuracy rate of 68% in categorizing dimensions, Bard achieved 58%
- LLMs can effectively characterize failures with sufficient detail available
- LLMs cannot replace human analysts entirely
- NLP techniques like LLMs can automate analysis process and enhance cybersecurity efforts
- Further research should focus on improving LLM performance and expanding scope

## Summary

In the face of increasing reliance on software systems, breaches in the software supply chain have become more detrimental. Recent high-profile cyber attacks, such as those on SolarWinds and ShadowHammer, have led to significant financial losses and compromised data, highlighting the urgent need for stronger cybersecurity measures. To prevent future breaches, it is crucial to study past failures. However, traditional methods of analyzing these failures involve manual reading and summarizing of reports, which can be time-consuming and costly. To address this challenge, researchers have explored the use of Natural Language Processing (NLP) techniques, specifically Large Language Models (LLMs), to assist in the analysis of software supply chain breaches. In a recent empirical study conducted by Tanmay Singla et al., they assessed the effectiveness of LLMs in analyzing historical software supply chain security failures. The researchers replicated the manual analysis performed by members of the Cloud Native Computing Foundation (CNCF) on 69 such failures using LLMs. They developed prompts for LLMs to categorize these failures based on four dimensions: type of compromise, intent, nature, and impact. The findings revealed that GPT 3.5s had an average accuracy rate of 68% in categorizing these dimensions while Bard achieved an accuracy rate of 58%. This suggests that LLMs can effectively characterize software supply chain failures when there is sufficient detail available in source articles for consensus among human analysts. However, LLMs cannot yet replace human analysts entirely. The study highlights the potential benefits of leveraging NLP techniques like LLMs to automate the analysis process for software supply chain breaches. By reducing costs and enabling analysis of a larger number of failures compared to traditional methods, automated support can significantly enhance cybersecurity efforts. Moving forward, further research can focus on improving the performance of LLMs in this context and expanding the scope to include a broader range of articles and failures. This will contribute to a more comprehensive understanding of software supply chain security failures and aid in the development of more robust cybersecurity strategies.