Hinglish pre-training data
An open, cleaned corpus of Hindi written in English letters, proved by training small models on it.
1 · Purpose
Hundreds of millions of people type Hindi in English letters, often mixed with English, which is called Hinglish. Today's open web datasets hold very little of it: one large open dataset has about 98,000 such documents, against more than 22 million in Hindi script, and a rough sample suggests a third of them are junk. We would harvest, clean, deduplicate and quality-score real Hinglish from the public web crawl, then show it helps by training identical small models on different versions of the data and comparing their scores. Finally we would fine-tune and shrink a small Hinglish model that can run on an Android tablet.
2 · Business benefits
3 · Status and how to join
Good first tasks this week:
- Read 100 sampled Hinglish documents and label them real, junk or mislabelled.
- Write 50 Hinglish multiple-choice questions for the test set.
- Compare two language detectors on a small labelled sample.
4 · Requirements
What it must do
- Pre-filter public web crawl text for likely Hinglish using common Hindi function words and Indian sites.
- Identify romanized Hindi with a language detector, telling it apart from English and Urdu.
- Remove near-duplicates, adult content, spam and personal details.
- Score each document's quality with a small classifier trained on examples labelled by an open model.
- Build a Hinglish test set: held-out text for perplexity and a hand-checked multiple-choice set.
- Train identical small models (about 150M and 400M parameters, then one 1B run) on each data version and compare.
- Fine-tune and compress the best small model so it runs on an Android tablet.
How well it must do it
- Success criteria are written down before any training run, so results cannot be cherry-picked.
- Every model traces back to the exact data version it was trained on.
- Each comparison uses at least two random seeds so noise is not mistaken for a gain.
- The pipeline is reproducible from public inputs with one documented command per stage.
- Licences of every source are recorded and respected in what is released.
5 · Architecture, very high level
6 · Tech stack, per platform
| Component | Open stack | Azure | AWS | GCP | Android |
|---|---|---|---|---|---|
| Quality scorer | Proposal: fastText or XLM-R classifier | — | — | — | — |
| Pipeline | Proposal: datatrove on spot CPU machines | — | — | — | — |
| Language ID | Proposal: IndicLID romanized model or GlotLID | — | — | — | — |
| Deduplication | Proposal: MinHash | — | — | — | — |
| Training | Proposal: nanotron or a small Llama-style trainer | Proposal: Azure Machine Learning with H100 VMs | Proposal: SageMaker training jobs or EC2 GPU instances | Proposal: Vertex AI custom training on GPUs | — |
| Packaging | Proposal: llama.cpp with 4-bit GGUF | — | — | — | — |
| Release | Proposal: Hugging Face datasets and models | — | — | — | — |
| Processing | — | Proposal: Azure Batch on spot VMs | Proposal: EMR or Batch on spot instances next to Common Crawl in S3 | Proposal: Dataproc on spot VMs | — |
| Storage | — | Proposal: Blob Storage | Proposal: S3 | — | — |
| On-device model | — | — | — | — | Proposal: llama.cpp or MediaPipe LLM inference running the compressed model |
7 · What needs to be done
Pilot one crawl dump 0 of 3 done
| Task | Status | Size | Skill |
|---|---|---|---|
| Run the pre-filter and language ID on one crawl dump and measure the yield | todo | M | data engineering |
| Hand-check 500 sampled documents for real Hinglish and junk | todo | S | Hindi reading |
| Go or no-go decision on whether the yield justifies scaling | todo | S | judgement |
Build the corpus 0 of 3 done
| Task | Status | Size | Skill |
|---|---|---|---|
| Clean, deduplicate and filter adult content and personal data | todo | L | data engineering |
| Label about 50,000 documents with an open model and train the quality scorer | todo | M | machine learning |
| Freeze and document the data versions | todo | S | data engineering |
Build the tests 0 of 1 done
| Task | Status | Size | Skill |
|---|---|---|---|
| Build the Hinglish held-out set and the hand-checked multiple-choice set | todo | M | Hindi, evaluation |
Run the comparisons 0 of 2 done
| Task | Status | Size | Skill |
|---|---|---|---|
| Train 150M models on each data version with two seeds | todo | M | model training |
| Confirm at 400M and run one 1B headline comparison | todo | L | model training |
Small model and release 0 of 2 done
| Task | Status | Size | Skill |
|---|---|---|---|
| Fine-tune and compress a 1B Hinglish model for a tablet | todo | M | machine learning |
| Publish the data, pipeline, results and a write-up | todo | M | writing |
8 · Roles
| Role | How many | Hours / week | What they do |
|---|---|---|---|
| Data pipeline engineer | 2 | 4–6 hours/week | Builds harvesting, cleaning, deduplication and versioning. |
| ML engineer | 1 | 4–6 hours/week | Runs the quality scorer, the small-model training and the comparisons. |
| Hindi and Hinglish reviewer | 2 | 2 hours/week | Hand-checks samples and the test set. |
| On-device engineer | 1 | 2–4 hours/week | Gets the compressed model running on an Android tablet. |
9 · What you'll learn
- How web-scale pre-training data is collected and cleaned.
- How language identification, deduplication and quality filtering work.
- How to run a fair data comparison with fixed compute and several seeds.
- How small language models are trained, fine-tuned and compressed.
- How to run a model on a phone or tablet.
Level-up questions
- How much real Hinglish is actually in the public web crawl, and how much is mislabelled as English?
- Does training on Hinglish improve Hinglish ability without hurting Hindi in its own script?
- Which language detector separates romanized Hindi from English and Urdu best?
- How small can the final model be and still hold a useful Hinglish conversation?