# Evaluating and Explaining Large Language Models for Code Using Syntactic Structures

**Authors:** David N Palacio, Alejandro Velasco, Daniel Rodriguez-Cardenas, Kevin Moran, Denys Poshyvanyk  
**arXiv:** [2308.03873](https://arxiv.org/abs/2308.03873)  
**Date:** 2023-08-07  
**Summary page:** https://www.summarizepaper.com/en/paper/2308.03873v1/

## Key points

- Large Language Models (LLMs) are evaluated and explained for their effectiveness in code
- LLMs are transformer-based neural networks pre-trained on large datasets of natural and programming languages
- ASTxplainer is introduced as an explainability method specific to LLMs for code, aligning token predictions with Abstract Syntax Tree (AST) nodes
- ASTxplainer extracts and aggregates normalized model logits within AST structures to provide insights into LLM effectiveness
- Empirical evaluation conducted on 12 popular LLMs using a curated dataset of popular GitHub projects
- Demonstrates the practical benefit of ASTxplainer in improving LLM evaluation
- User study conducted to examine the usefulness of ASTxplainer-derived visualizations in aiding end-users' understanding of model predictions
- Results highlight the potential for ASTxplainer to improve LLM evaluation and aid in interpreting model predictions
- Contributes to the field of automated Software Engineering tools and provides valuable insights into the performance and interpretability of LLMs for code.

## Summary

This paper evaluates and explains the effectiveness of Large Language Models (LLMs) for code. LLMs are transformer-based neural networks pre-trained on large datasets of both natural and programming languages. To this end, the authors introduce ASTxplainer, an explainability method specific to LLMs for code that aligns token predictions with Abstract Syntax Tree (AST) nodes. This automated method extracts and aggregates normalized model logits within AST structures to provide insights into LLM effectiveness. An empirical evaluation is conducted on 12 popular LLMs using a curated dataset of popular GitHub projects to demonstrate the practical benefit of ASTxplainer. Additionally, a user study is conducted to examine the usefulness of ASTxplainer-derived visualizations in aiding end-users in understanding model predictions. The results highlight the potential for ASTxplainer to improve LLM evaluation and aid in interpreting model predictions. Overall, this research contributes to the field of automated Software Engineering tools and provides valuable insights into the performance and interpretability of LLMs for code.