Nzingha Tokenizer

Tokens needed for the same text by the Nzingha tokenizer and leading global tokenizers, across 26 African languages. Fewer tokens means lower cost, more room in the context window and faster responses.

Early prototype · · Text is tokenized in your browser and never sent anywhere. Pieces outlined in a box are single characters split across several tokens.
Sets the sample text and the benchmark column.
Assumption you can edit. Default is OpenAI's published GPT-4o input price.

This text

Cost is for sending this text one million times at the price above.

TokenizerTokensvs fewestCost per 1M messagesHow the text is split

Benchmark:

Token premium vs English on 1,997 held-out NTREX-128 news sentences. 1.00x means as efficient as English.

TokenizerPremiumMean, 26 languagesEnglish tokens

Method and sources

Benchmark: NTREX-128 (Federmann, Kocmi and Xin, 2022), never used in training. Training text: FineWeb-2 web sample (ODC-By) and MAFAND-MT (Masakhane), filtered to remove any document sharing a 13-word sequence with NTREX. Global tokenizers: public tokenizer files from the @lenml/tokenizer-* packages. The Nzingha tokenizer is Gemma 3's tokenizer with 32,000 African-language tokens appended; base token IDs are unchanged and English and French tokenize as before. Aya Expanse is licensed for non-commercial use only, so it appears in the benchmark but its tokenizer is not loaded on this page.

Notices

The Nzingha tokenizer is derived from the Gemma 3 tokenizer. Gemma is provided under and subject to the Gemma Terms of Use found at ai.google.dev/gemma/terms.
Llama 3.1 + Nzingha pack: Built with Llama. Llama 3.1 is licensed under the Llama 3.1 Community License, Copyright © Meta Platforms, Inc. All Rights Reserved.
Other tokenizer files on this page are published by OpenAI, Alibaba (Qwen), DeepSeek, Mistral AI and Google, and are used under their publishers' licences for comparison only.