AI summary · arXiv: 2611.13058v1 · cs.CL

The Indic Token Tax in 2026: Measuring Tokenizer. Disparity and Its Cost for the Languages of India

Johin Johny Arimbur, Ankur Ashit Ghosh, Shervin Sam
Original paper Submitted to arXiv on 09 Nov. 2026 License: CC BY 4.0 Also in: fr

How to read this summary? Illustrated guide

The essentials

Across 25 Indian language-script pairs, tokenizers impose substantial and uneven token overhead versus English; Indic-focused tokenizers reduce but do not eliminate it.

Question

How much extra tokenization cost do public model tokenizers impose on Indian languages and romanized text compared with English, and how well do their token boundaries align with Hindi and Marathi morphemes? The paper also illustrates the possible API cost implications of these differences.

Method

The authors measured nine public tokenizers on 2,009 English-aligned FLORES-200 sentences across 25 Indian language-script pairs, 13,732 Hinglish-English PHINC pairs, and additional romanized text. They also compared tokenizer boundaries with MorphTok segmentations, retaining 59,676 Hindi and 42,292 Marathi words, and calculated an illustrative cost scenario using assumed monthly volumes of 100,000 chats.

Findings

  • Result On GPT-4o, token counts ranged from 1.57 times English for Hindi to 13.72 times English for Santali in Ol Chiki.
    See the passage in the paper
    on GPT-4o’s tokenizer no language of India reaches parity: the lowest ratio is Hindi at 1.57, eight of the 25 are at 2.0 or above, and three scripts are much worse: Odia (5.01), Meitei in Bengali script (3.07) and Santali in Ol Chiki (13.72, higher than GPT-4’s 12.76).
  • Result Sarvam-30B/105B had the most even results across the tested languages, with ratios from 1.21 to 2.50 relative to English.
    See the passage in the paper
    Third, the Sarvam tokenizers are the most even: Sarvam-30B/105B stays between 1.21 and 2.50 on every language, the only third-party tokenizer under 2.6 everywhere.
  • Result For most Indian languages in the Dakshina comparison, romanized text used fewer tokens than native-script text with GPT-4 and Qwen2.5, while Sarvam-30B/105B generally showed the opposite pattern.
    See the passage in the paper
    On the older or non-Indic-focused tokenizers, romanizing cuts token cost sharply: on cl100k_base the romanized text needs only 0.25 to 0.53 of the native-script tokens, and on Qwen2.5 0.28 to 0.75, so typing in Latin letters avoids most of the tax these tokenizers impose on native scripts.
  • Result No tested tokenizer exceeded a morpheme-boundary F1 score of 0.48 on Hindi and Marathi.
    See the passage in the paper
    The Indic-focused Sarvam tokenizers place 29 to 34% of their word-internal boundaries on morpheme boundaries, about twice the reference, and no tokenizer exceeds an F1 of 0.48.
  • Result The illustrative GPT-4o support-bot estimate was INR 56,640 per month for English and INR 102,829 for Marathi, based on stated assumptions rather than measured production costs.
    See the passage in the paper
    On GPT-4o, the bot costs INR 56,640 a month in English, INR 89,145 in Hindi, INR 102,829 in Marathi and INR 283,601 in Odia, so the same service in Marathi costs INR 46,190 a month more than in English.

Limitations

  • The rupee estimates rely on assumed chat lengths and parity assumptions, and the authors did not verify which tokenizer Sarvam's API uses.
    See the passage in the paper
    The rupee example rests on assumed chat lengths, list prices that change often, the assumption that output parity equals input parity, and, for Sarvam, an unverified assumption about which tokenizer its API uses
  • The tested FLORES text is formal, and the study could not use the gated conversational benchmark that might show how tokenization behaves on conversational text.
    See the passage in the paper
    FLORES is formal, Wikipedia-style text, and conversational text may behave differently; IN22-Conv, which would test this, is gated and was not used.
  • MorphTok is a curated rather than exhaustive gold standard, and the analysis excludes sandhi entries and evaluates words individually.
    See the passage in the paper
    MorphTok segmentations are partly generated from stem and affix lists and partly post-edited model output, so they are a curated reference rather than an exhaustive gold standard; we score single words out of context and drop entries with sandhi changes.

Why it matters

The authors argue that tokenizer choice can materially affect the cost and context capacity available to Indian-language users, and that the disparity should be checked language by language when selecting an API. Indic-focused tokenizers reduce the gap, but the measured token boundaries remain imperfectly aligned with morphology.

Written by AI from the paper. Passages are checked word for word against the text we analysed; the rest may contain errors.

AI-generated keywords: Tokenization Indian languages Indic scripts Token efficiency API costs

Read further

  • Comprehensive Summary
  • Layman's Summary
  • Blog article
  • Original abstract
The paper examines how much more costly Indian-language text is to process than English because of tokenization differences, and whether newer or Indic-focused tokenizers reduce this gap. The authors evaluate nine public tokenizers: GPT-4, GPT-4o, Qwen2.5, DeepSeek-V3, Mistral-Nemo, and four Sarvam tokenizers. The comparisons use 2,009 aligned sentences from FLORES-200, covering 25 Indian language-and-script combinations, and 13,732 Hinglish social-media posts paired with English translations. Additional analyses examine romanized Hindi and Marathi social-media text, matched native-script and romanized strings from Dakshina, and Hindi and Marathi word segmentations to assess alignment with morpheme boundaries. The primary measure is the ratio of tokens used for a language to those used for aligned English content. The authors also report a common-baseline comparison that accounts for differences in how efficiently tokenizers encode English. They encode sentences as plain text, without chat templates or special tokens, and use bootstrap resampling to estimate uncertainty. To estimate costs, the authors apply published API prices to a hypothetical support-bot workload. These figures are estimates based on stated assumptions, not observed production costs. GPT-4o substantially reduces the disparity compared with older tokenizers, but does not eliminate it. Its token counts are 1.57 times the English count for Hindi and 1.82 times for Marathi, rising to 5.01 times for Odia and 13.72 times for Santali in Ol Chiki. Eight of the 25 language-and-script combinations have GPT-4o ratios of at least 2.0. GPT-4 and Qwen2.5 are especially inefficient for many Indic scripts: their ratios reach 12.48 and 9.73, respectively, for Odia, and 12.76 and 8.93 for Santali. Sarvam-30B and Sarvam-105B produce the most even results across the languages measured, with ratios ranging from 1.21 to 2.50. Sarvam-1 performs well for many Indic scripts but has much higher ratios for Urdu, Sindhi, Kashmiri in Arabic script, and Santali. Romanized text changes the comparison. On the parallel PHINC set, Hinglish has relatively low parity ratios, partly because Latin-script text reuses English subword units. The authors also note that Hinglish posts contain 8% more words than their English translations. In Dakshina, romanization sharply reduces token counts relative to native script for older tokenizers such as GPT-4, but has little effect for GPT-4o in most languages. For Sarvam-30B, Sarvam-105B, and Sarvam-1, romanized text generally requires more tokens than native-script text in the languages examined. The illustrative cost calculation estimates that 100,000 monthly chats on GPT-4o would cost about INR 56,640 in English, INR 102,829 in Marathi, and INR 283,601 in Odia, given the paper’s assumed chat lengths and pricing. These results show how tokenizer choice can increase per-token costs and reduce the amount of text that fits in a fixed context window, but they do not measure model quality. In a separate morphology analysis, no tokenizer achieves a boundary F1 score above 0.48. Sarvam-30B and Sarvam-105B perform best on Hindi and Marathi, but their alignment with morpheme boundaries remains limited. The findings are constrained by FLORES’s formal, Wikipedia-style text, the lack of comparable parallel romanized data beyond Hinglish, and the absence of gated conversational benchmarks. Some Indian languages are missing, and the morphological reference is curated rather than exhaustive.

Was this synthesis useful?

Score: 0

Why do we need votes?

Votes are used to determine whether we need to re-run our summarizing tools. If the count reaches -10, our tools can be restarted.

Figures and plots

Ask about a figure by number or use its question button to request visual analysis. Up to two figures and four image panels per question. Values read from plots are approximate; check the original data.

AI assistant instructions?

Tools for this paper

Explain it differently 1 free rewrite left today
Save to a collection Group papers in collections to compare them, chat with them and export them. Continue with Google Create a free account Log in

Advanced tools Premium

For researchers who follow a field closely: citation alerts, journal-club slides, version comparison and Zotero export. Discover Premium

Zotero Zotero guide
Citation alerts 1 paper already cites it: be emailed each new one. Citation alerts guide
Who works on it?

Among the 23 papers around this one, the name that comes back most: Maharaj Brahma (2 papers)

1 more person signs several of them. The full map adds their institutions and approaches, from the papers of the last years.

Continue with Google Create a free account: your first map is free
Weekly letter Presented in "Artificial intelligence this week" of 16 November 2026. Artificial intelligence this week: What impressive scores leave out · Receive one by email for free

Papers around this one

See them on a map

Ranked by how close their title and abstract are to this paper’s. Select some to read them together.

Closest papers

  1. Very close Same question
    The Tokenizer Tax: Quantifying and Explaining the Cross-Lingual Cost of Subword Tokenization for Indian Languages
    Priyanshu Srivastava · 2026 · arXiv.org · preprint, not peer reviewed · cited 2 times · Not summarized yet

    The Tokenizer Tax measures cross-lingual tokenization costs for Indian languages on FLORES-200, as this paper does, using six tokenizers and fourteen languages.

    Abstract

    Large language models (LLMs) process text through subword tokenizers rather than directly reading characters or words. Because these tokenizers are trained predominantly on English-centric corpora, they introduce a systematic and often overlooked disadvantage for many non-English languages. In this work, we quantify this tokenizer tax for Indian languages using the FLORES-200 parallel corpus, measuring tokenization fertility across six widely used tokenizers and fourteen languages. Under cl100k_base (used by GPT-3.5 and GPT-4), Indian languages experience an average 8.0x tokenization tax relative to English, reaching 13.0x for Malayalam, reducing the effective context window to as little as …

  2. Very close Builds on this paper Cites this paper
    A Multi-Word Tokenizer for 204 Languages That Needs Fewer Tokens Than GPT-4o's, Including on English
    Johin Johny, A. Ghosh, Shervin Sam · 2026 · Not summarized yet

    Havon v3 and v4 target the token-count disparities measured here, training multilingual tokenizers that reduce counts on Indian languages relative to GPT-4o.

    Abstract

    Languages of India need 1.6 to 14 times as many tokens as English on GPT-4o's tokenizer for the same content, which makes them more expensive and slower to serve. We train tokenizers from scratch on human-written text in about 200 languages. Havon v3 is a byte-level BPE tokenizer with 262,144 tokens and a pre-tokenizer that keeps combining marks with their letters and chunks scripts written without spaces. On FLORES-200 devtest it needs fewer tokens than GPT-4o's o200k_base on all 25 languages of India (median 0.74 of GPT-4o's count) and on 172 of 204 languages, but 7% more on English. Havon v4 adds a second, multi-word training stage (SuperBPE) to v3. At exactly GPT-4o's vocabulary size of …

  3. Close Same question Cited by this paper
    Evaluating Tokenizer Performance of Large Language Models Across Official Indian Languages
    Sagar Tamang, D. Bora · 2024 · arXiv.org · preprint, not peer reviewed · cited 16 times · Not summarized yet

    Evaluates tokenization efficiency across Indian languages, comparing language-model tokenizers with normalized sequence length.

    Abstract

    Large Language Models (LLMs) based on transformer architectures have revolutionized a variety of domains, with tokenization playing a pivotal role in their pre-processing and fine-tuning stages. In multilingual models, particularly those tailored for Indic languages, effective tokenization is crucial for optimizing performance. This paper presents a comprehensive evaluation of tokenizers used by 12 LLMs across all 22 official languages of India, with a focus on comparing the efficiency of their tokenization processes. We employed the Normalized Sequence Length (NSL) as a key metric in our analysis. Our findings reveal that the SUTRA tokenizer outperforms all other models, including several I…

  4. Close Broader context
    Is Sanskrit the most token-efficient language? A quantitative study using GPT, Gemini, and SentencePiece
    Anshul Kumar · 2026 · arXiv.org · preprint, not peer reviewed · cited 1 time · Not summarized yet

    Measures token counts for Sanskrit, English, Hindi, and transliterated Sanskrit on parallel Bhagavad Gita verses using several tokenizers.

    Abstract

    Tokens are the basic units of Large Language Models (LLMs). LLMs rely on tokenizers to segment text into these tokens, and tokenization is the primary determinant of computational and inference cost. Sanskrit, one of the oldest languages, is hypothesized to express more meaning per token due to its morphology and grammar rules; however, no prior work has quantified this. We use a dataset of 701 parallel verses of the Bhagavad Gita, which comprises three languages-Sanskrit, English, and Hindi along with transliteration of Sanskrit into English. We test tokenizers including SentencePiece (SPM), older GPT models, and the latest generation tokenizers from Gemini and GPT. We use metrics of token …

  5. Close Same question
    Multilingual Tokenization through the Lens of Indian Languages: Challenges and Insights
    Maharaj Brahma, N J Karthika, Rajat Verma et al. · 2025 · Annual Meeting of the Association for Computational Linguistics · cited 5 times · Not summarized yet

    Empirically studies multilingual tokenization across Indic languages, including the effects of algorithms, script normalization, vocabulary size, and vocabulary construction.

    Abstract

    Tokenization plays a pivotal role in NLP and is fundamental to training language models. However, existing tokenizers are often skewed towards high-resource languages, limiting their effectiveness for linguistically diverse and morphologically rich languages such as those in the Indian subcontinent. In this work, we present a comprehensive empirical study of multilingual tokenization across 17 Indic languages spanning 11 scripts and two language families. We systematically evaluate the effects of (i) widely used subword algorithms: BPE and Unigram LM, (ii) script and orthography-aware normalization, (iii) vocabulary size, and (iv) multilingual vocabulary construction strategies. We use a com…

  6. Close Same question
    The Tokenizer Tax Across 25 European Languages: Domain Invariance, Cross-Lingual Few-Shot Effects, and the Ukrainian Penalty
    Volodymyr Ovcharov · 2026 · arXiv.org · preprint, not peer reviewed · cited 5 times · Not summarized yet

    Maps tokenization disparities across parallel text in European languages, measuring how tokenizer fertility varies by language and domain.

    Abstract

    Tokenizer fertility the number of tokens per word imposes a hidden cost on non-English NLP. We measure fertility for ten foundation models across 25 European languages on parallel text, producing the first controlled tokenizer tax map for the continent. The tax spans 2.5x from English (1.2 tokens/word) to Greek/Maltese (~3.1), following a clear hierarchy: Romance (1.5-1.7), Germanic (1.7-1.9), Slavic (2.2-2.5), Uralic/Baltic (2.7-3.0). Ukrainian (2.7) pays 15-18% more than cognate Slavic languages, reflecting underrepresentation in pre-training data. Fertility rankings are domain-invariant across three text registers (rho>0.97). A subword analysis reveals that high-fertility tokenizers fragm…

  7. Close Broader context
    Krutrim LLM: A Novel Tokenization Strategy for Multilingual Indic Languages with Petabyte-Scale Data Processing
    Rahul Kumar, Shubham Kakde, Divyansh Rajput et al. · 2024 · cited 1 time · Not summarized yet

    Describes data preparation and tokenization optimization for a multilingual Indic language model, including language-specific preprocessing.

    Abstract

    We present a novel approach to data preparation for developing multilingual Indic large language model. Our meticulous data acquisition spans open-source and proprietary sources, including Common Crawl, Indic books, news articles, and Wikipedia, ensuring a diverse and rich linguistic representation. For each Indic language, we design a custom preprocessing pipeline to effectively eliminate redundant and low-quality text content. Additionally, we perform deduplication on Common Crawl data to address the redundancy present in 70% of the crawled web pages. This study focuses on developing high-quality data, optimizing tokenization for our multilingual dataset for Indic large language models wit…

  8. Close Groundwork for this paper Cited by this paper
    Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models
    Orevaoghene Ahia, Sachin Kumar, Hila Gonen et al. · 2023 · Conference on Empirical Methods in Natural Language Processing · cited 298 times · Not summarized yet

    Analyzes how token-count differences across languages affect commercial language-model API pricing.

    Abstract

    Language models have graduated from being research prototypes to commercialized products offered as web APIs, and recent works have highlighted the multilingual capabilities of these products. The API vendors charge their users based on usage, more specifically on the number of ``tokens'' processed or generated by the underlying language models. What constitutes a token, however, is training data and model dependent with a large variance in the number of tokens required to convey the same information in different languages. In this work, we analyze the effect of this non-uniformity on the fairness of an API's pricing policy across languages. We conduct a systematic analysis of the cost and u…

Prior work it builds on

  1. Related Groundwork for this paper
    Language Model Tokenizers Introduce Unfairness Between Languages
    Aleksandar Petrov, Emanuele La Malfa, P. Torr et al. · 2023 · Neural Information Processing Systems · cited 344 times · Not summarized yet

    Shows that tokenization creates large cross-language differences in the token length of translated text, including for multilingual tokenizers.

    Abstract

    Recent language models have shown impressive multilingual performance, even when not explicitly trained for it. Despite this, there are concerns about the quality of their outputs across different languages. In this paper, we show how disparity in the treatment of different languages arises at the tokenization stage, well before a model is even invoked. The same text translated into different languages can have drastically different tokenization lengths, with differences up to 15 times in some cases. These disparities persist even for tokenizers that are intentionally trained for multilingual support. Character-level and byte-level models also exhibit over 4 times the difference in the encod…

  2. Related Shared data, tool or method
    IndicTrans2: Towards High-Quality and Accessible Machine Translation Models for all 22 Scheduled Indian Languages
    Jay Gala, Pranjal A. Chitale, AK Raghavan et al. · 2023 · arXiv.org · preprint, not peer reviewed · cited 350 times · Not summarized yet

    Introduces parallel data and benchmarks for machine translation across India's 22 scheduled languages.

    Abstract

    India has a rich linguistic landscape with languages from 4 major language families spoken by over a billion people. 22 of these languages are listed in the Constitution of India (referred to as scheduled languages) are the focus of this work. Given the linguistic diversity, high-quality and accessible Machine Translation (MT) systems are essential in a country like India. Prior to this work, there was (i) no parallel training data spanning all 22 languages, (ii) no robust benchmarks covering all these languages and containing content relevant to India, and (iii) no existing translation models which support all the 22 scheduled languages of India. In this work, we aim to address this gap by …

  3. Related Shared data, tool or method
    IndicLLMSuite: A Blueprint for Creating Pre-training and Fine-Tuning Datasets for Indian Languages
    Mohammed Safi Ur Rahman Khan, Priyam Mehta, Ananth Sankar et al. · 2024 · Annual Meeting of the Association for Computational Linguistics · cited 84 times · Not summarized yet

    Provides large-scale pretraining and fine-tuning datasets for Indic language models across 22 languages.

    Abstract

    Despite the considerable advancements in English LLMs, the progress in building comparable models for other languages has been hindered due to the scarcity of tailored resources. Our work aims to bridge this divide by introducing an expansive suite of resources specifically designed for the development of Indic LLMs, covering 22 languages, containing a total of 251B tokens and 74.8M instruction-response pairs. Recognizing the importance of both data quality and quantity, our approach combines highly curated manually verified data, unverified yet valuable data, and synthetic data. We build a clean, open-source pipeline for curating pre-training data from diverse sources, including websites, P…

  4. Related Broader context
    How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models
    Phillip Rust, Jonas Pfeiffer, Ivan Vulic et al. · 2020 · Annual Meeting of the Association for Computational Linguistics · cited 525 times · Not summarized yet

    Examines multilingual tokenizers in relation to monolingual model performance across nine languages and five downstream tasks.

    Abstract

    In this work, we provide a systematic and comprehensive empirical comparison of pretrained multilingual language models versus their monolingual counterparts with regard to their monolingual task performance. We study a set of nine typologically diverse languages with readily available pretrained monolingual models on a set of five diverse monolingual downstream tasks. We first aim to establish, via fair and controlled comparisons, if a gap between the multilingual and the corresponding monolingual representation of that language exists, and subsequently investigate the reason for any performance difference. To disentangle conflating factors, we train new monolingual models on the same data,…

  5. Related Shared data, tool or method
    L3Cube-HingCorpus and HingBERT: A Code Mixed Hindi-English Dataset and BERT Language Models
    Ravindra Nayak, Raviraj Joshi · 2022 · WILDRE · cited 76 times · Not summarized yet

    Presents a large Roman-script Hindi-English code-mixed corpus and language models, relevant to the paper's Hinglish tokenization analysis.

    Abstract

    Code-switching occurs when more than one language is mixed in a given sentence or a conversation. This phenomenon is more prominent on social media platforms and its adoption is increasing over time. Therefore code-mixed NLP has been extensively studied in the literature. As pre-trained transformer-based architectures are gaining popularity, we observe that real code-mixing data are scarce to pre-train large language models. We present L3Cube-HingCorpus, the first large-scale real Hindi-English code mixed data in a Roman script. It consists of 52.93M sentences and 1.04B tokens, scraped from Twitter. We further present HingBERT, HingMBERT, HingRoBERTa, and HingGPT. The BERT models have been p…

How are these papers found?

We compare the title and abstract of this paper with those of every paper summarized on this site, and with the papers Semantic Scholar recommends, those it cites and those citing it (for a much-cited paper, its most recent citations and, from OpenAlex, the most cited ones). Among the prior work and the follow-ups, the papers that use each other in depth, a method it uses and the much-cited follow-ups come a little higher. The levels (very close, close, related) come from the similarity of the texts, not from a judgment of quality; * marks a paper without an abstract, compared by its title. The sentence under a paper is written by an AI from the two abstracts: check it in the papers. A paper marked not relevant by three readers leaves the list.

Updated on 16 November 2026.

The illustrated guide to the papers around a paper

Disclaimer: The AI-based summarization tool and virtual assistant provided on this website may not always provide accurate and complete summaries or responses. We encourage you to carefully review and evaluate the generated content to ensure its quality and relevance to your needs.