Robotics data benchmark
A fair test of which ways of cleaning robot demonstration data actually make robots succeed more.
1 · Purpose
Robots learn tasks from recorded demonstrations, and many papers propose ways to pick the best demonstrations. Each paper tests on its own setup, so nobody can say which method really works. We would keep the robot policy, training recipe and simulator fixed, change only which demonstrations go in, and measure how often the robot actually completes its tasks across several runs. Behind it sits a small data engine with lineage, so every trained policy traces back to the exact demonstrations it saw.
2 · Business benefits
3 · Status and how to join
Good first tasks this week:
- Run one LeRobot dataset through a schema check and list what is inconsistent.
- Write a script that injects idle frames into an episode.
- Summarise one curation paper in a page: what it scores and how it was tested.
4 · Requirements
What it must do
- Load public demonstration datasets in a standard format and pin their exact revision.
- Check each dataset against a schema and flag metadata that contradicts the data.
- Inject known kinds of bad data, such as jerky or idle episodes, to create a ground truth.
- Apply each curation method to choose a subset under the same data budget.
- Train the same small policy on each subset with the same recipe.
- Run closed-loop trials in simulation over several seeds and report success rates with spread.
- Publish a leaderboard where anyone can submit a new curation method.
How well it must do it
- Results report at least three seeds and their spread, never a single run.
- Every checkpoint is linked to its episode set, scores and dataset revision.
- The whole benchmark runs on a modest GPU budget so others can repeat it.
- Simulation results are clearly labelled as simulation, not real-robot results.
5 · Architecture, very high level
6 · Tech stack, per platform
| Component | Open stack | Azure | AWS | GCP |
|---|---|---|---|---|
| Data format | Proposal: LeRobotDataset v3 | — | — | — |
| Policy | Proposal: SmolVLA, with ACT or Diffusion Policy as cheap baselines | — | — | — |
| Simulation | Proposal: LIBERO and LIBERO-Plus, robomimic for the cheap first stage | — | — | — |
| Lineage | Proposal: OpenLineage events or MLflow runs | — | — | — |
| Leaderboard | Proposal: a static site from result files | — | — | — |
| Training and simulation | — | Proposal: Azure Machine Learning on GPU VMs | Proposal: EC2 GPU instances or SageMaker | Proposal: Vertex AI or GPU VMs |
| Storage | — | Proposal: Blob Storage | Proposal: S3 | — |
7 · What needs to be done
Fixed harness 0 of 2 done
| Task | Status | Size | Skill |
|---|---|---|---|
| Pick the policy, simulator and data budget and write them down | todo | S | robot learning |
| Reproduce a clean baseline success rate over three seeds | todo | M | robot learning |
Data engine 0 of 2 done
| Task | Status | Size | Skill |
|---|---|---|---|
| Build the loader, contract checker and lineage records | todo | M | data engineering |
| Build the corruption injector with five kinds of bad episodes | todo | M | data engineering |
Compare methods 0 of 2 done
| Task | Status | Size | Skill |
|---|---|---|---|
| Implement three published curation methods behind one interface | todo | L | machine learning |
| Run every method on the fixed harness over three seeds | todo | L | GPU operations |
Publish 0 of 2 done
| Task | Status | Size | Skill |
|---|---|---|---|
| Publish the leaderboard, data versions and a write-up | todo | M | writing |
| Document how to submit a new method | todo | S | writing |
8 · Roles
| Role | How many | Hours / week | What they do |
|---|---|---|---|
| Robot-learning lead | 1 | 4–6 hours/week | Owns the fixed harness and the fairness of the comparison. |
| Data engineer | 2 | 3–4 hours/week | Builds loading, contracts, corruption and lineage. |
| ML engineer | 1 | 3–4 hours/week | Implements curation methods and runs training. |
9 · What you'll learn
- How robots learn from demonstrations and how they are tested in simulation.
- Why low training loss can still mean a robot fails the task.
- How to design a fair benchmark with fixed compute and several seeds.
- How data lineage makes machine-learning results trustworthy.
Level-up questions
- Which simulator benchmark is hard enough that curation effects show, given that the standard one is nearly saturated?
- Do offline quality scores predict closed-loop success at all?
- How many seeds are enough to separate a real gain from noise?
- Which curation methods can run on a modest GPU budget?