Results of the summarizing process for the arXiv paper: 2604.22520v1

, , , , In the field of Machine Translation (MT), Large Language Models (LLMs) have demonstrated impressive performance, but their widespread deployment is hindered by high costs. To address this challenge, a common approach is the hybrid system paradigm, which involves using a small model for most translation requests and selectively directing some to a larger model to optimize cost-effectiveness and quality. However, existing routing strategies often lack precision in determining whether the large model truly offers significant improvements over the small one. In response to this issue, this paper introduces RouteLMT (Learned Sample Routing for Hybrid LLM Translation Deployment), a novel in-model router that leverages marginal gain as the key signal for making budgeted decisions in routing. By analyzing the expected improvement of the large model over the small one through probing prompt-token representations, RouteLMT eliminates the need for external models or hypothesis decoding. Extensive experiments showcase RouteLMT's superiority over heuristic approaches and quality/difficulty estimation baselines, achieving a superior quality-budget Pareto frontier. Additionally, regression risks are examined, highlighting how a guarded variant can effectively mitigate potential significant quality losses. Authored by Yingfeng Luo, Hongyu Liu, Dingyang Lin, Kaiyan Chang, Chenglong Wang, Bei Li, Quan Du, Tong Xiao, and Jingbo Zhu; this research has been accepted for presentation at ACL 2026 Industry Track.

• - Large Language Models (LLMs) in Machine Translation have shown impressive performance but face deployment challenges due to high costs.
• - Hybrid system paradigm is a common approach, using a small model for most translation requests and selectively directing some to a larger model for cost-effectiveness and quality optimization.
• - Existing routing strategies often lack precision in determining if the large model offers significant improvements over the small one.
• - RouteLMT (Learned Sample Routing for Hybrid LLM Translation Deployment) is introduced as an in-model router that uses marginal gain to make budgeted decisions in routing, eliminating the need for external models or hypothesis decoding.
• - RouteLMT outperforms heuristic approaches and quality/difficulty estimation baselines, achieving a superior quality-budget Pareto frontier.

SummaryLarge Language Models (LLMs) are very good at translating languages, but it can be expensive to use them all the time. A hybrid system uses a mix of small and large models to save money and make translations better. Sometimes it's hard to know when to use the big model for the best results. RouteLMT is a new way to decide which model to use based on how much it helps without needing extra tools. RouteLMT works better than other methods and finds the best balance between quality and cost. Definitions- Large Language Models (LLMs): Advanced computer programs that are really good at translating languages. - Hybrid system paradigm: A way of combining different models or approaches for better results. - Routing strategies: Plans for deciding how data should be directed through a system. - RouteLMT: A new method that helps choose the best translation model based on its benefits. - Pareto frontier: The best possible trade-off between two factors, like quality and cost.

Introduction

Machine Translation (MT) has made significant strides in recent years with the development of Large Language Models (LLMs). These models have shown impressive performance in translating text from one language to another, but their widespread deployment is hindered by high costs. To address this challenge, a common approach is the hybrid system paradigm, which involves using a small model for most translation requests and selectively directing some to a larger model to optimize cost-effectiveness and quality. However, existing routing strategies often lack precision in determining whether the large model truly offers significant improvements over the small one. This can lead to inefficient use of resources and ultimately impact the overall quality of translations. In response to this issue, a team of researchers from various institutions including Alibaba Group and Tsinghua University has developed RouteLMT (Learned Sample Routing for Hybrid LLM Translation Deployment), a novel in-model router that leverages marginal gain as the key signal for making budgeted decisions in routing.

The Problem

The main challenge faced by MT systems is finding an optimal balance between cost and quality. While LLMs have shown great potential in improving translation quality, they come at a high cost due to their large size and computational requirements. As such, it is not feasible or practical to use them for every translation request. To overcome this challenge, hybrid systems were introduced where smaller models are used for most translation requests while larger models are reserved for more complex or difficult translations. However, existing routing strategies often rely on heuristics or external models which may not accurately determine if using the larger model will significantly improve translation quality.

The Solution: RouteLMT

RouteLMT addresses these issues by utilizing marginal gain as the key signal for making budgeted decisions in routing. This means that instead of relying on external models or heuristics, RouteLMT analyzes the expected improvement of the large model over the small one through probing prompt-token representations. This eliminates the need for additional models or hypothesis decoding, making it a more efficient and cost-effective solution.

Experimental Results

The researchers conducted extensive experiments to compare RouteLMT with existing routing strategies and quality/difficulty estimation baselines. The results showed that RouteLMT outperformed these approaches, achieving a superior quality-budget Pareto frontier. This means that RouteLMT was able to achieve better translation quality while using fewer resources compared to other methods. Additionally, the researchers also examined regression risks, which refers to potential significant quality losses when using a smaller model instead of a larger one. They found that by using a guarded variant of RouteLMT, these risks can be effectively mitigated without sacrificing too much in terms of translation quality.

Conclusion

In conclusion, this research paper introduces an innovative solution for optimizing cost-effectiveness and improving translation quality in hybrid LLM systems. By leveraging marginal gain as the key signal for routing decisions, RouteLMT eliminates the need for external models or heuristics and achieves superior performance compared to existing approaches. With its successful experimental results and potential applications in industry settings, RouteLMT is set to make significant contributions in advancing MT technology.