# HELM: Hyperbolic Large Language Models via Mixture-of-Curvature Experts

**Authors:** Neil He, Rishabh Anand, Hiren Madhu, Ali Maatouk, Smita Krishnaswamy, Leandros Tassiulas, Menglin Yang, Rex Ying  
**arXiv:** [2505.24722](https://arxiv.org/abs/2505.24722)  
**Date:** 2025-05-30  
**Summary page:** https://www.summarizepaper.com/en/paper/2505.24722v1/

## Key points

- HELM is a family of Hyperbolic Large Language Models (LLMs) that operate fully in hyperbolic space
- HELM models leverage the properties of hyperbolic space to capture semantic hierarchies and geometric structures in natural language
- HELM-MICE is a Mixture-of-Curvature Experts model where each expert operates in a distinct curvature space to encode fine-grained geometric structures from text
- HELM-D is a dense model that enhances the representational flexibility and scalability of existing hyperbolic LMs
- The researchers developed hyperbolic Multi-Head Latent Attention (HMLA) for HELM-MICE to improve training efficiency and inference accuracy
- Fully hyperbolic LLMs trained at billion-parameter scale showed consistent performance gains of up to 4% over popular Euclidean architectures like LLaMA and DeepSeek on benchmark datasets such as MMLU and ARC
- HELM models offer enhanced reasoning capabilities by embracing non-Euclidean geometries like hyperbolic space for large-scale LM pretraining

## Summary

Researchers have introduced HELM, a family of Hyperbolic Large Language Models (LLMs) that operate fully in hyperbolic space. This approach aims to address the limitations of current LLMs and their reliance on Euclidean operations, which fail to capture the inherent semantic hierarchies and geometric structure of natural language. By leveraging the expansive, scale-free, and low-distortion properties of hyperbolic space, HELM models offer a novel geometric rethinking of Transformer-based LLMs. The researchers also propose HELM-MICE, a Mixture-of-Curvature Experts model where each expert operates in a distinct curvature space to encode fine-grained geometric structures from text. Additionally, they introduce HELM-D, a dense model that enhances the representational flexibility and scalability of existing hyperbolic LMs. To improve training efficiency and inference accuracy, they develop hyperbolic Multi-Head Latent Attention (HMLA) for HELM-MICE. Furthermore, the researchers pioneer the training of fully hyperbolic LLMs at billion-parameter scale and evaluate them on benchmark datasets such as MMLU and ARC. Their results demonstrate consistent performance gains of up to 4% over popular Euclidean architectures like LLaMA and DeepSeek. This highlights the efficacy and enhanced reasoning capabilities afforded by hyperbolic geometry in large-scale LM pretraining. In conclusion, this study showcases the potential of HELM models in capturing complex semantic relationships and geometric structures within text data. By embracing non-Euclidean geometries like hyperbolic space, these models offer a promising avenue for advancing natural language processing tasks across various domains.