Hinglish pre-training data

An open, cleaned corpus of Hindi written in English letters, proved by training small models on it.

Not started · lead itLLM Engineering

0 of 11 tasks done

1 · Purpose

Hundreds of millions of people type Hindi in English letters, often mixed with English, which is called Hinglish. Today's open web datasets hold very little of it: one large open dataset has about 98,000 such documents, against more than 22 million in Hindi script, and a rough sample suggests a third of them are junk. We would harvest, clean, deduplicate and quality-score real Hinglish from the public web crawl, then show it helps by training identical small models on different versions of the data and comparing their scores. Finally we would fine-tune and shrink a small Hinglish model that can run on an Android tablet.

For: Language-model researchers, Indian AI builders, and data engineers who want to learn how pre-training data is made.

2 · Business benefits

Fills a measured gapGives builders far more clean Hinglish text than any open dataset has today.
Evidence, not claimsFixed-compute comparisons show whether the data actually improves models.
Better Indian productsChat and voice products can understand how Indians really type.
Runs on small devicesA small model that fits a tablet makes Hinglish AI usable offline and cheaply.
Open to everyoneData, pipeline and results are published so others can build on them.

3 · Status and how to join

Not started · lead it. Not started yet: join the first group and shape it from day one. Lead: Basant Choudhary.

Good first tasks this week:

  • Read 100 sampled Hinglish documents and label them real, junk or mislabelled.
  • Write 50 Hinglish multiple-choice questions for the test set.
  • Compare two language detectors on a small labelled sample.

The code is not public yet; joiners get access.

4 · Requirements

What it must do

How well it must do it

5 · Architecture, very high level

Crawl harvesterPulls text from public webcrawl dumps and keeps likelyLanguage identifierDecides which documents arereally romanized Hindi.Cleaner and deduplicatorRemoves near-copies, adultcontent, spam and personalQuality scorerGives each document a qualityscore from a small trainedDataset versionerFreezes each data version withits counts and a record of howAblation trainerTrains identical small modelson each data version with theEvaluation suiteScores models on Hinglish andHindi tests and tracks seeds.Small-model packagerFine-tunes and compresses thewinning model for a tablet.candidate pageromanized Hindclean unique dscored documentsfrozen data vemodel checkpoithe winning se
Components and what each is responsible for, with no tool names: the tools come in the next section.

6 · Tech stack, per platform

ComponentOpen stackAzureAWSGCPAndroid
Quality scorerProposal: fastText or XLM-R classifier————
PipelineProposal: datatrove on spot CPU machines————
Language IDProposal: IndicLID romanized model or GlotLID————
DeduplicationProposal: MinHash————
TrainingProposal: nanotron or a small Llama-style trainerProposal: Azure Machine Learning with H100 VMsProposal: SageMaker training jobs or EC2 GPU instancesProposal: Vertex AI custom training on GPUs—
PackagingProposal: llama.cpp with 4-bit GGUF————
ReleaseProposal: Hugging Face datasets and models————
Processing—Proposal: Azure Batch on spot VMsProposal: EMR or Batch on spot instances next to Common Crawl in S3Proposal: Dataproc on spot VMs—
Storage—Proposal: Blob StorageProposal: S3——
On-device model————Proposal: llama.cpp or MediaPipe LLM inference running the compressed model

Bold with ✓ = built and running today. Everything else is the planned or possible binding: the components stay the same, only the tools change.

7 · What needs to be done

Pilot one crawl dump 0 of 3 done

TaskStatusSizeSkill
Run the pre-filter and language ID on one crawl dump and measure the yieldtodoMdata engineering
Hand-check 500 sampled documents for real Hinglish and junktodoSHindi reading
Go or no-go decision on whether the yield justifies scalingtodoSjudgement

Build the corpus 0 of 3 done

TaskStatusSizeSkill
Clean, deduplicate and filter adult content and personal datatodoLdata engineering
Label about 50,000 documents with an open model and train the quality scorertodoMmachine learning
Freeze and document the data versionstodoSdata engineering

Build the tests 0 of 1 done

TaskStatusSizeSkill
Build the Hinglish held-out set and the hand-checked multiple-choice settodoMHindi, evaluation

Run the comparisons 0 of 2 done

TaskStatusSizeSkill
Train 150M models on each data version with two seedstodoMmodel training
Confirm at 400M and run one 1B headline comparisontodoLmodel training

Small model and release 0 of 2 done

TaskStatusSizeSkill
Fine-tune and compress a 1B Hinglish model for a tablettodoMmachine learning
Publish the data, pipeline, results and a write-uptodoMwriting

Size: S = an evening or two, M = about a week part-time, L = a few weeks.

8 · Roles

RoleHow manyHours / weekWhat they do
Data pipeline engineer24–6 hours/weekBuilds harvesting, cleaning, deduplication and versioning.
ML engineer14–6 hours/weekRuns the quality scorer, the small-model training and the comparisons.
Hindi and Hinglish reviewer22 hours/weekHand-checks samples and the test set.
On-device engineer12–4 hours/weekGets the compressed model running on an Android tablet.

9 · What you'll learn

Level-up questions

10 · Links