Newsletters
Ask a chatbot the same question in English and in Turkish, and the answers may look alike. What happens behind the screen does not. Before an artificial intelligence (AI) model can understand any text, it breaks it into tiny fragments, and languages are not all split the same way.
Consider one sentence: "Apparently they will not be able to get out of our houses." In Turkish, it is just two words: "Evlerimizden cikamayacaklarmis." When Türkiye Today tested five tokenizers, the English sentence was split into 13 pieces, while the two Turkish words were split into as many as 15.
These fragments are known as tokens. AI systems tally them, charge for them and sometimes run out of them. In four test sentences, every tokenizer needed more tokens for Turkish than for English. Whether this means Turkish users are quietly paying extra remains uncertain. The size of the gap depends on the tokenizer's designer.
Tokens set the price, shape reply speed and decide how much of a conversation a model can hold onto. The cost difference for the same meaning in different languages now has a name: the "tokenizer tax."
The term comes from a September analysis by the Coalition for Language Equity in AI Research (CLEAR), as reported by Rest of World in late October. Among the 20 most popular internet languages, the most expensive used 1.5 to 5 times as many tokens as the cheapest.
The study also found that script matters: languages written in scripts the tokenizers rarely saw, such as Burmese, Tibetan and Khmer, did the worst.
The report does not include Turkish. The language uses the Latin script, the better-served side of that divide.
With no Turkish data available, Türkiye Today ran its own test. The team wrote four pairs of English and Turkish sentences and counted tokens across five tokenizers from OpenAI, Google, Meta, Alibaba and DeepSeek.
| Tokenizer | English tokens | Turkish tokens | Turkish/English |
|---|---|---|---|
| OpenAI (GPT-5 generation) | 42 | 55 | 1.31 |
| Gemma-7b | 42 | 56 | 1.33 |
| Llama-3-8B | 42 | 59 | 1.40 |
| Qwen2.5-72B | 42 | 66 | 1.57 |
| DeepSeek R1 | 42 | 77 | 1.83 |
Gemma's counts exclude the special start token its tool adds.
Across all five tokenizers, Turkish consistently required more tokens. The gap ran from 31% with the tokenizer behind OpenAI's GPT-5 generation to 83% with DeepSeek's. Only the two-word sentence broke the pattern, costing less in Turkish than in English in three of the five.
Testing just four sentences is more of a snapshot than a benchmark. Yet the direction was the same for every tokenizer.
The reason is grammar. Turkish builds meaning by adding suffixes to a root, so one word can express what English needs a whole phrase for. Tokenizers trained mostly on English often split these Turkish words into many pieces.
Researchers from four universities, three of them in Istanbul, tested four large models on 6,200 Turkish exam questions. The models performed better when the tokenizer's pieces matched real Turkish words. Model size alone did not predict the results.
Some say the cost will eventually balance out. Dheemanth Reddy, CEO of Maya Research, told Rest of World that lower token prices should help, but closing the gap will still require labs to design tokenizers specifically for these languages.
Pratyush Choudhury, co-founder of the venture fund Activate AI, told the outlet he wants every major lab to publish a scorecard showing the cost of similar tasks for each language.
No such scorecard exists for Turkish. Ugur Sergin, chief technology officer at Türkiye Today, says the disadvantage shows up in daily work. His team, which built its own content management system, puts every new model through newsroom tasks such as editing, correcting copy and finding errors in Turkish text.
He says Turkish is at a disadvantage in token counts because it is an agglutinative language. As a result, token limits can be reached more quickly, which makes working through an application programming interface (API) more expensive in Turkish.
There is a workaround, and it is telling. Even when he writes to a model in Turkish, he keeps technical terms in English because the models handle them better.
Of the models his team has tested, he says Google's Gemma understands Turkish best. He attributes this to a larger data pool.
He does not expect the gap to close soon, because he sees it as rooted in the language itself. Efforts to build large language models in Türkiye continue, he says, but none has caught up with the big companies. Two years of work on a model can fade fast, he adds, if a new one arrives just three months later.
Sergin sees the problem as rooted in the language. Research also points to the people behind the tokenizers. A tokenizer is a choice made before a single word is written, and its cost lands on users who never had a voice in the decision.
The wide variation in that test supports this view. If five tokenizers diverge so much on the same sentences, grammar alone cannot explain the penalty. Someone made these choices, and someone else could choose differently. Researchers in Türkiye have already built a tokenizer tailored for Turkish.
Until that changes, users must adapt.
A Turkish speaker who slips in an English word because the machine handles it better has already made a quiet compromise.
The models will keep evolving, but no tokenizer can tally how much of a language its speakers will have left behind by then.
Set as preferred source
Set as preferred source
4 min read
