From the weekly letter · 16 November 2026

Why cover this story this week?

The hidden language surcharge

Language-model services charge for tokens, the fragments into which text is divided. Those fragments are not equally economical across languages. Researchers compared nine public tokenizers using professionally translated, English-aligned sentences in the languages of India, alongside human-translated Hinglish-English social media pairs. With GPT-4o's tokenizer, equivalent content required 1.57 times as many tokens in Hindi as in English, 5.01 times in Odia, and 13.7 times in Santali written in Ol Chiki script.

This affects more than a bill: extra tokens consume context space and can increase processing time. The study also found weak alignment between token boundaries and meaningful word parts in Hindi and Marathi. Its cost calculations are explicitly illustrative rather than measurements of production services. Nevertheless, the comparison makes an infrastructure choice visible: the way a model divides writing into pieces can distribute its practical benefits unevenly across language communities.

Editorial context written by AI from abstracts, not independent scientific validation. The kit and its verified quotes rely on the source paper, not on this letter.

Read the full letter →

Talk about a paper in public without overstating it: yours, one you are asked to comment on, or one you write about as a journalist.

  • Your paper: a draft release for your press office, the questions journalists will ask with what the paper lets you answer, the likely misreadings, a lay summary and a short thread.
  • Asked to comment: what the paper shows and does not, its strengths and weaknesses with their passages, and what would settle its main claim.
  • Writing about it as a journalist: reporting angles, the wording to check before publication, and the questions to ask the authors.
  • For everyone: findings with their passages checked word for word, the novelty set against the closest earlier papers, a hype check, figures with their credit, and researchers for an outside view who have not signed with the authors.

Discover Premium+ See an example kit