Robotics data benchmark

A fair test of which ways of cleaning robot demonstration data actually make robots succeed more.

Not started · lead itAI Engineering

0 of 8 tasks done

1 · Purpose

Robots learn tasks from recorded demonstrations, and many papers propose ways to pick the best demonstrations. Each paper tests on its own setup, so nobody can say which method really works. We would keep the robot policy, training recipe and simulator fixed, change only which demonstrations go in, and measure how often the robot actually completes its tasks across several runs. Behind it sits a small data engine with lineage, so every trained policy traces back to the exact demonstrations it saw.

For: Robot-learning researchers, robotics startups, and data engineers curious about data quality for physical AI.

2 · Business benefits

One fair comparisonCuration methods are compared on the same harness for the first time.
Measured on real successScores come from the robot doing the task, not from training loss alone.
Less wasted data collectionTeams learn which demonstrations are worth recording and keeping.
ReproducibleEvery result traces back to the exact data version and seed.

3 · Status and how to join

Not started · lead it. Nobody has started this yet. Lead it, or join the first group. Lead: Open — looking for a lead.

Good first tasks this week:

  • Run one LeRobot dataset through a schema check and list what is inconsistent.
  • Write a script that injects idle frames into an episode.
  • Summarise one curation paper in a page: what it scores and how it was tested.

The code is not public yet; joiners get access.

4 · Requirements

What it must do

How well it must do it

5 · Architecture, very high level

Dataset loaderReads public demonstrationdatasets at a pinned revision.Contract checkerChecks schema and metadata andrejects broken episodes.Corruption injectorPlants known bad episodes somethods can be graded.Curation runnerApplies each method to pick asubset under one data budget.Policy trainerTrains the same small policy oneach subset.Simulation evaluatorRuns the robot in simulationand counts task successes perLineage and leaderboardRecords what produced eachresult and publishes theepisodes at a valid episodesepisodes with chosen subsetpolicy checkposuccess rates
Components and what each is responsible for, with no tool names: the tools come in the next section.

6 · Tech stack, per platform

ComponentOpen stackAzureAWSGCP
Data formatProposal: LeRobotDataset v3———
PolicyProposal: SmolVLA, with ACT or Diffusion Policy as cheap baselines———
SimulationProposal: LIBERO and LIBERO-Plus, robomimic for the cheap first stage———
LineageProposal: OpenLineage events or MLflow runs———
LeaderboardProposal: a static site from result files———
Training and simulation—Proposal: Azure Machine Learning on GPU VMsProposal: EC2 GPU instances or SageMakerProposal: Vertex AI or GPU VMs
Storage—Proposal: Blob StorageProposal: S3—

Bold with ✓ = built and running today. Everything else is the planned or possible binding: the components stay the same, only the tools change.

7 · What needs to be done

Fixed harness 0 of 2 done

TaskStatusSizeSkill
Pick the policy, simulator and data budget and write them downtodoSrobot learning
Reproduce a clean baseline success rate over three seedstodoMrobot learning

Data engine 0 of 2 done

TaskStatusSizeSkill
Build the loader, contract checker and lineage recordstodoMdata engineering
Build the corruption injector with five kinds of bad episodestodoMdata engineering

Compare methods 0 of 2 done

TaskStatusSizeSkill
Implement three published curation methods behind one interfacetodoLmachine learning
Run every method on the fixed harness over three seedstodoLGPU operations

Publish 0 of 2 done

TaskStatusSizeSkill
Publish the leaderboard, data versions and a write-uptodoMwriting
Document how to submit a new methodtodoSwriting

Size: S = an evening or two, M = about a week part-time, L = a few weeks.

8 · Roles

RoleHow manyHours / weekWhat they do
Robot-learning lead14–6 hours/weekOwns the fixed harness and the fairness of the comparison.
Data engineer23–4 hours/weekBuilds loading, contracts, corruption and lineage.
ML engineer13–4 hours/weekImplements curation methods and runs training.

9 · What you'll learn

Level-up questions

10 · Links