A piece of token appreciation

Last weekend, this column argued why India must establish tech sovereignty

Last weekend, this column argued why India must establish tech sovereignty. . There was despondency when pointing out that as things are in tech, India is a Tier II country as opposed to China and the US—now counted as Tier I countries. How did things get to be this way? Policy wonk Venky Hariharan (Venky) says fundamentally it’s language.

Getting AI to generate a sentence in Malayalam or Tamil or any other regional language is dearer than generating one in English. He shared a sentence as an example: “People should benefit from AI, regardless of the language they speak.” This appears in a paper by CLEAR, a coalition working on language equity in AI. It points out that “Each of these sentences convey the same meaning; the content is equivalent and they contain nearly the same number of characters.” However, to make sense of it, an AI model needs tokens. Bluntly put, you pay a bill for a model to get to work on it.

So, with GPT-5, “the English sentence requires 13 tokens while the Malayalam one requires 26.” Priyansh Srivastava, an AI Engineer, has done the research. He ran Indian languages behind older models such as GPT-3.5 and GPT-4 and concluded that Indian users pay eight times more than an English language user does.

This tax is not a law. Srivastava finds it comes from how tokenizers were designed. No Indian scripts were used and that’s what brings us to BHASHINI. The government launched this in 2022 as a translation and voice platform for Indian languages. And on November 3, markets regulator SEBI signed an agreement with it to use this as a tool to bring investor information to people in their own languages. What powers the engine that runs it?

Earlier this year, BHASHINI moved its entire stack from a global cloud onto an Indian one. Many of its models are “open”, from AI4Bharat at IIT Madras among others. Anyone can read them. It is Open Source technology.

In that it’s akin to Wikipedia: volunteers write it and anyone can read it, copy it and build on it, including businesses. Nobody owns it. Selena Deckelmann, chief product and technology officer of The Wikimedia Foundation pointed out, “To date, every LLM is trained on Wikipedia content”. Now imagine that for AI. Not just an Indian one. A global one: open models, open data, open tests for every language.

Wikipedia also shows what happens when the supply of information is thin. English has more than six million articles. Tamil, among the largest languages, tied to a single state, had about 1.5 lakh in 2023. Malayalam trails it. Srivastava’s paper finds that how well a model reads a language owes directly to how much material that exists in it.

In the overall scheme, cheaper pieces make Malayalam affordable. When entities such as SEBI agreed to add securities-market terms to BHASHINI’s models, the language gets more data. Now comes the hard part to which the answers are fuzzy, says Venky. While the Wikimedia Commons evolved, who takes the responsibility to run a global AI commons so that neither Washington nor Beijing captures it? Who pays for it? A page of text is cheap to keep. But an AI model is not.

There is an opportunity here for India but also some concern. If India shares what it builds, does it lose control? Venky says, no. “An AI commons is essential for the digital sovereignty of Tier 2 powers.” Countries in that tier cannot build all of this alone. But together, they just might be able to do it. And the pooling must begin with the data.

Most of the talk in AI circles until now has been about models but the bottleneck is data. To place that in perspective consider this: it takes about three trillion tokens to train a model in Tamil. What exists now is around 250 billion, and much of it is in Tanglish (Tamil typed in English letters). That is a twelfth of what is needed, and that too in the wrong alphabet. The machine can learn Tamil but there is not enough data for it to learn from. Money cannot fill this gap. Only people can because AI can only learn from what people bother to write down.