A manipulation policy that achieves impressive success rates in the lab where it was trained can fail on its first visit to a real site. The robot has not changed. The environment has: different lighting, a different bench height, unfamiliar objects, clutter that was not there before. This is domain shift, and for robotics it is one of the most persistent obstacles between a promising demo and a deployable system.
The most reliable defence is training data that already contains the variety the robot will meet. This article explains which kinds of diversity matter, why, and how to plan coverage without collecting endlessly.
Why models overfit to places
Learning systems take shortcuts. If every training example of “place the part in the bin” shows the same blue bin on the same grey floor under the same lights, a model can solve the task by recognising that scene rather than understanding bins. It scores well on held-out data from the same environment and fails the moment any of those cues change. The model was never forced to learn the general concept, because the data never separated the concept from the place.
The dimensions of environmental diversity
Site and layout
Kitchens, workshops, warehouses and building sites differ in layout, working height, reach distances and floor space. Even within one category — two garment factories, say — station layouts and workflows vary. Recording the same task in several sites is usually worth more than recording one site at length.
Lighting
Daylight through windows, overhead fluorescent tubes, harsh spotlights and dim corners all change colour, contrast and shadow. Night shifts and outdoor work add further variation. Vision models are notoriously sensitive to lighting they have not seen.
Objects and materials
The same task is performed on different instances: a different cup, a heavier sheet, a stiffer fabric, a worn tool. Shape, texture, transparency and deformability all change how an object looks and how it must be handled.
Clutter and background activity
Real sites contain irrelevant objects, partially finished work and other people moving through the frame. Clean demonstration benches teach a model to expect an empty scene, which it will rarely get.
People and technique
Different people perform the same task differently: left- or right-handed, fast or careful, with different grips and different orders of sub-steps. Diversity of operators gives a model a broader notion of what correct execution looks like.
Diversity is not the same as volume
Adding more hours from the same environment increases dataset size without increasing coverage. The useful measure is how many distinct conditions are represented, and how well they match the conditions of deployment. As argued in What actually makes training data usable, coverage of the cases a model gets wrong beats raw volume.
Planning coverage
- List the deployment environments and the factors that vary between them: site type, lighting, objects, clutter, operators.
- Decide which factors the model must be robust to and which can be fixed for now.
- Build a coverage matrix — tasks against environment conditions — and set a minimum number of episodes per cell.
- Collect across several sites per task, with several operators per site.
- Hold out at least one entire environment for evaluation, so test scores measure generalisation rather than memorisation.
Documenting diversity
Diversity is only useful if it is recorded. Each episode should carry metadata — environment type, site identifier, lighting condition, operator identifier (anonymised), objects involved — so teams can filter, rebalance and diagnose failures by condition. Without metadata, a dataset may be diverse in fact but impossible to analyse.
Real environments versus simulation
Domain randomisation in simulation — varying textures, lighting and object placement — is a valuable complement, and it is cheap. But simulated variety is limited to what the simulator models. Real environments contain variation nobody thought to randomise: the reflective film on a sheet of metal, the way fabric bunches on a sewing table, the dust on a site floor. Synthetic data works best alongside real data, a point covered in Synthetic data is a complement.
Conclusion
Robots generalise to the extent their training data forces them to. Diversity of site, lighting, objects, clutter and people is what forces a model to learn the task rather than the place. Planned deliberately and documented with metadata, it is one of the highest-leverage investments a robotics data programme can make.
Mvaak records real tasks across homes, workshops, factories and building sites for Physical AI teams. See Physical AI data and Egocentric Video Data, or read How Real-World Human Activities Help Train Robotics Systems.
Written by Mvaak — AI Data Services.
