# From Large to Small: Transferring CUDA Optimization Expertise via Reasoning Graph

**Authors:** Junfeng Gong, Zhiyi Wei, Junying Chen, Cheng Liu, Huawei Li  
**arXiv:** [2510.19873](https://arxiv.org/abs/2510.19873)  
**Date:** 2025-10-22  
**Summary page:** https://www.summarizepaper.com/en/paper/2510.19873v1/

## Key points

- Challenges persist in effectively harnessing massively parallel engines in CUDA programming despite advancements in programming and domain-specific libraries.
- Large language models (LLMs) show promise in generating optimized CUDA code, but face hurdles such as privacy risks with cloud-based APIs and high computational costs with local deployment.
- Small language models (SLMs) offer a more lightweight and privacy-friendly alternative to LLMs, achieving comparable performance on specific tasks but limited reasoning abilities for complex CUDA generation.
- ReGraphT is a novel training-free, retrieval-augmented generation framework that enhances the reasoning capabilities of SLMs by transferring LLM-level reasoning through structured reasoning graphs and Monte Carlo Graph Search (MCGS).
- Experimental results demonstrate that ReGraphT outperforms HPC-specific fine-tuned models and other retrieval-augmented approaches, achieving an average 2.33X speedup on CUDAEval and ParEval tasks.
- Combining ReGraphT with specific instructive models enables SLMs to approach LLM-level performance without compromising privacy or requiring excessive computing resources, offering a promising solution for optimizing CUDA code generation efficiently while maintaining privacy standards.

## Summary

, , , , 

In the realm of CUDA programming and GPU utilization, challenges persist in effectively harnessing massively parallel engines despite advancements in CUDA programming and domain-specific libraries. Large language models (LLMs) have shown promise in generating optimized CUDA code from sequential code, but practical implementation faces hurdles such as privacy risks with cloud-based APIs and high computational costs with local deployment. This has sparked interest in small language models (SLMs), which offer a more lightweight and privacy-friendly alternative. Recent studies have demonstrated that SLMs can achieve comparable performance to LLMs on specific tasks. However, their limited reasoning abilities result in suboptimal performance when it comes to complex CUDA generation. To address this gap, a novel approach called ReGraphT has been proposed. ReGraphT is a training-free, retrieval-augmented generation framework that transfers LLM-level reasoning to smaller models. By organizing CUDA optimization trajectories into a structured reasoning graph and leveraging Monte Carlo Graph Search (MCGS) for efficient exploration, ReGraphT aims to enhance the reasoning capabilities of SLMs. Furthermore, a CUDA-specific benchmark with difficulty tiers based on reasoning complexity has been introduced to evaluate models comprehensively. Experimental results show that ReGraphT outperforms HPC-specific fine-tuned models and other retrieval-augmented approaches, achieving an average 2.33X speedup on CUDAEval and ParEval tasks. When combined with DeepSeek-Coder-V2-Lite-Instruct and Qwen2.5-Coder-7B-Instruct, ReGraphT enables SLMs to approach LLM-level performance without compromising privacy or requiring excessive computing resources. Despite the resource-intensive nature of deploying LLMs locally or the potential risks associated with cloud-based APIs, the emergence of SLMs presents a promising solution for optimizing CUDA code generation while maintaining efficiency and privacy standards. Ongoing research in this direction aims to further enhance the capabilities of SLMs through innovative frameworks like ReGraphT, paving the way for more effective GPU utilization in parallel processing environments.