Nine small utilities, in TypeScript and Go, for text in the 22 languages of India's Eighth Schedule: Hindi, Bengali, Tamil, Telugu, Marathi, Urdu, Gujarati, Kannada, Malayalam, Odia, Punjabi and the rest. Each one fixes one place where ordinary string code goes wrong on Indian text: Unicode normalization, search folding, stemming, letter (grapheme) segmentation, phonetic name matching and name search, transliteration to and from Latin letters, and a check that a word can be drawn whole (wellformed, not shown here; search-key joins normalize, stem and fold in one call). Every result on this page is computed in your browser by indickit 0.14.0, with examples from real government press releases.
normalize One encoding for text that looks the samenormalize(text, lang)
Gives text that looks the same one encoding, so equal text has equal bytes. It never changes a letter a reader can see.
Press offices type ড়, য়, ज़ and ਸ਼ as one code point or as two, and Malayalam chillu letters in an old way. To a computer, the two forms are different words.
The live demo needs JavaScript.
In the 50 releases (rules 2026-10-12): Bengali 51 of 51, Manipuri 50 of 50, Punjabi 48 of 50 documents change; Marathi 0 of 50. Held out: 88.1% of look-alike groups made byte-equal, 0 words changed in look (docs/normalize.md). New in 0.12.0: every font indickit holds decides what looks the same, and three faults are fixed (khanda ta, old Malayalam chillus before a vowel sign, a ZWJ inside a Malayalam letter).
fold One search key for spellings readers treat as one wordfold(text, lang)
Makes a search key: two spellings a reader treats as one word get one key. Use it for an index, not for storage.
हिंदी and हिन्दी are both correct, and so are यहाँ and यहां. A search for one spelling misses documents that use the other.
The live demo needs JavaScript.
In the 50 releases (rules 2026-10-12), fold changes 58% of Assamese words, 22.5% of Malayalam, 3.8% of Hindi. Fold rules were chosen against Lucene and Indic NLP on three answer keys; rules that merged different frequent words were dropped (माल / मॉल).
stem One index key for the forms of a wordstem(word, lang)
Cuts the case and number endings off a word, so किताब, किताबें and किताबों share one index key.
Indian languages put grammar on the end of the word. A search for योजना (scheme) misses योजनाओं (schemes) unless the index stores one stem for both.
The live demo needs JavaScript.
In the 50 releases (rules 2026-10-12): Hindi 4,438 word forms → 3,462 stems, Gujarati 6,216 → 3,568 (3,784 before Gujarati ran its endings twice, 0.11.0). Held out: stems find 3.7–15 points more of the right sentences than exact search, in all 22 languages since 0.11.0; 0.12.0 adds 6 Kannada and 10 Marathi endings. A stem can be too short: Tamil செய்தி (news) and செய்து (did) share one.
segment Count and cut letters as a reader sees themsegment(text)
Splits text into the letters a reader sees, so a cursor, a letter count or a cut at a length never splits one letter in two.
ಕ್ಷ್ಮಿ is one letter made of six code points. String.slice counts code units, and Intl.Segmenter splits some conjuncts in Kannada and Gurmukhi.
The live demo needs JavaScript.
Snippets cut at 120 units from the 50 releases (rules 2026-10-11): slice split a letter in 20–48% of them in Indian scripts; Intl.Segmenter in 22% of Kannada and 2% of Punjabi snippets. segment's own held-out error: 0.05% of Kannada words.
phonetic Match a name across 22 scripts and Latin lettersmatch(a, b) · keys(word)
Gives a name the same short key in any of 22 scripts or in Latin letters, so a name typed one way finds it written another way.
राजनाथ, ರಾಜನಾಥ್, راجناتھ and Rajnath are one name. No two of them share a letter.
The live demo needs JavaScript.
Held out: 89% of OpenStreetMap place-name pairs across scripts (romanize + Soundex 71%); 84% of Wikidata people in 21 languages (rules 2026-10-05, a lower bound for the rules that ship); 88% of a roster as written, titles and initials included (Rajya Sabha). New in 0.11.0: Bengali সিং = Singh, Tamil லட்சுமி = Lakshmi, Urdu عمران = Imran. New in 0.12.0 (rules 2026-10-11.8): Sindhi ھ is read as h (سنھا = Sinha). See all 22 scripts in the language table below.
phonetic-search Find and rank a name among many wordsloadSearch(lang).index(words, "text").search(q)
Finds a name among many words and scores each candidate from 0 to 100. The key finds candidates; a scorer learned from data ranks them.
A key says only yes or no, and a common key says yes to too much. A list of 5,000 word forms needs a ranking and a threshold.
The live demo needs JavaScript.
Held out: 93% of OpenStreetMap places found across scripts (ICU + fuzzy match 74%, at the same wrong hits). A name in running text (PIB July 2019, 10 languages; search 2026-10-09): 66% with at most 10% of hits wrong, 78% with at most 20%; romanize + fuzzy match 31% and 46%. Lok Sabha list 86%, villages 85%. A score ranks the candidates of one search; do not compare scores across searches.
romanize Transliterate Indian text into Latin letters, as people spell itload(lang, mode).text(s) · .word(w, n)
Writes Indian text in Latin letters the way people spell it (लक्ष्मी → lakshmi, not lakṣmī), with a ranked list of spellings for each word.
A transliteration scheme gives one exact spelling that nobody types. People write Laxmi, Lakshmi, Luxmi: the useful answer is the likely spellings, in order.
The live demo needs JavaScript.
Held out: running text spelled as its writers spell it 48.9% of words (IndicXlit 37.9%; ahead in all 11 languages); names 71.3% (IndicXlit 49.0%). New in 0.12.0: names mode reranks its spellings with a letter 5-gram; on names held out to the end, the first spelling is right 41.0% of the time, was 37.9%. A places mode since 0.11.0: village names 57.6% (IndicXlit 52.8%), but behind IndicXlit on OpenStreetMap places and on rare single words. Tables load on demand: 0.2–1.8 MB a language.
deromanize Transliterate Latin typing into an Indian scriptload(lang, mode).text(s) · .word(w, n)
Writes Latin typing in an Indian script (namaste → नमस्ते), with a ranked list of spellings for each word. Since 0.8.0; 0.11.0 adds textMixed for Hindi typed with English words in it.
Most people type Indian languages on an English keyboard. A search box, a form or a chat must turn "yojana" into योजना before any other step can run.
The live demo needs JavaScript.
Held out: Hindi news 87.1% of words right (IndicXlit 86.1%); names 69.7% (66.4%). Behind IndicXlit on unseen names (−2.8 points) and rare isolated words (−10.1). A names-mode fault this page found (a cut-short first spelling) is fixed in rules 2026-10-09; see “Where it fails”.
One name, typed in Latin letters, through every package in every language: deromanize writes it in the language's script, segment counts its letters, phonetic checks that the written name still matches what was typed, and romanize writes it back in Latin letters. A red cross marks a step that broke the name: the native spelling, or the Latin one written back, no longer matches what was typed.
| Language | deromanize the name | segment letters | phonetic key, match | romanize back | back = typed? | stem | phonetic-search |
|---|---|---|---|---|---|---|---|
| Assameseas · Bengali-Assamese | ✓ | ✓ | |||||
| Bengalibn · Bengali-Assamese | ✓ | ✓ | |||||
| Bodobrx · Devanagari | ✓ | ✓ | |||||
| Dogridoi · Devanagari | ✓ | ✓ | |||||
| Gujaratigu · Gujarati | ✓ | ✓ | |||||
| Hindihi · Devanagari | ✓ | ✓ | |||||
| Kannadakn · Kannada | ✓ | ✓ | |||||
| Kashmiriks · Perso-Arabic | ✓ | ✓ | |||||
| Konkanigom · Devanagari | ✓ | ✓ | |||||
| Maithilimai · Devanagari | ✓ | ✓ | |||||
| Malayalamml · Malayalam | ✓ | ✓ | |||||
| Manipurimni · Meetei Mayek | ✓ | ✓ | |||||
| Marathimr · Devanagari | ✓ | ✓ | |||||
| Nepaline · Devanagari | ✓ | ✓ | |||||
| Odiaor · Odia | ✓ | ✓ | |||||
| Punjabipa · Gurmukhi | ✓ | ✓ | |||||
| Sanskritsa · Devanagari | ✓ | ✓ | |||||
| Santalisat · Ol Chiki | ✓ | keys only | |||||
| Sindhisd · Perso-Arabic | ✓ | ✓ | |||||
| Tamilta · Tamil | ✓ | ✓ | |||||
| Telugute · Telugu | ✓ | ✓ | |||||
| Urduur · Perso-Arabic | ✓ | ✓ |
deromanize fault this page found, fixed in rules 2026-10-09. In names mode the first spelling was sometimes only the start of the name (Malayalam “murmu” gave മു, Kashmiri “Rajnath” راج, Urdu “Chandrayaan” ان): the tables let a consonant write nothing, and a short common word then won the frequency re-rank. A spelling that writes nothing for a consonant is now ranked last in names mode (on names no rule was tuned on, the first spelling is right 69.2% instead of 68.1%). Words mode was right already and is unchanged.romanize faults this page found, fixed in rules 2026-10-08. Manipuri written in Bengali script lost syllables (রামেন became “en”): its tables were grouped with Meetei Mayek alone; it now shares the Brahmic tables, and names mode writes “ramen”. Words mode still cuts some Bengali-script Manipuri short. An invisible joiner typed after a virama (प्राप्त) left only the end of the word (“the”) in 212 Hindi words of the 50 releases: joiners are now deleted before a word is romanized (“praapt”).romanize gives some frequent words an English look-alike: एक becomes “ac”, देकर becomes “decker”.phonetic-search, Hindi “singh” still ranks संघ (union) above सिंह.stem gives Tamil செய்தி (news) and செய்து (did) one stem, so a word search for one finds the other.How this was measured: the 50 releases are a held-out slice of the August 2023 month of pib-parallel, chosen by a hash of the release ID before anyone read them. No indickit rule was chosen on this text. The counts on this page were fixed before the text was read; the text was read once on 2026-10-08 and counted again with 0.12.0 on 2026-10-13 (both reads logged; nothing was chosen on either). The language table's preset names were written ahead of time by deromanize and romanize, since their tables for 22 languages weigh about 60 MB. Everything else runs live in this page. Text: Press Information Bureau, Government of India.