The Reality of the Sim-to-Real Gap

Simulation promises unlimited data, zero hardware wear, parallelized training, and perfect reproducibility. In practice, every simulation is an approximation, and the gap between that approximation and reality is where policies go to die. Contact dynamics differ between simulated and real materials. Actuator friction, backlash, and latency are difficult to model. Rendered images look convincing to humans but activate neural network feature detectors differently than real camera images. Sensor noise, cable stiffness, table vibration -- dozens of small effects that simulation ignores or simplifies compound into a gap that untreated policies cannot cross.

But the gap is not uniform. Some tasks transfer easily; others are nearly impossible with current methods. Understanding where your task falls on this spectrum -- and applying the right techniques for your specific gap -- is what separates teams that successfully deploy sim-trained policies from teams that spend months in simulation only to fail on real hardware. For background on the theoretical foundations, see our companion article on sim-to-real transfer theory.

What Transfers Well and What Does Not

Transfers well: Locomotion on flat and moderately rough terrain, basic object grasping with parallel jaw grippers, navigation and obstacle avoidance, and reaching motions in free space. These tasks depend on dynamics that simulation models accurately (rigid body mechanics, basic friction) and visual features that transfer reasonably (object shapes, workspace geometry).

Transfers with effort: Pick-and-place of rigid objects, pushing and sliding manipulation, door and drawer opening, and simple assembly with generous tolerances. These tasks involve contact dynamics that simulation approximates but does not perfectly match. Successful transfer requires the techniques described below.

Does not transfer well (yet): Contact-rich manipulation of deformable objects (cloth folding, cable routing), precise assembly with sub-millimeter tolerances (connector mating, snap-fit insertion), and tasks involving granular materials, fluids, or soft bodies. The physics of these interactions is either too complex to simulate accurately or too sensitive to parameter errors for domain randomization to bridge the gap.

Tip 1: Use Systematic Domain Randomization

Domain randomization is the most widely validated technique for sim-to-real transfer, but "randomize everything" is not a strategy. Effective domain randomization targets the specific parameters that differ between your simulation and your real setup. Start by identifying your failure modes on real hardware (even 5-10 real trials give useful signal), then randomize the simulation parameters most likely to cause those failures.

For visual policies, effective randomization ranges include: camera position offset of plus or minus 10 cm from nominal, viewing angle variation of plus or minus 10 degrees, brightness variation of plus or minus 30%, hue shift of plus or minus 20%, saturation variation of plus or minus 15%, Gaussian blur with sigma 0-2 pixels, and random rectangular occlusions covering 5-20% of the image. These ranges make simulated images look unrealistically varied -- but that is exactly the point. The policy learns features that are invariant to all these variations, which means it also handles the real-world visual differences between simulation and your actual camera.

For physical parameters, prioritize: contact friction coefficients (randomize by plus or minus 50%), object mass (plus or minus 30%), joint damping (plus or minus 20%), and actuator latency (add 0-20 ms random delay). Randomize contact stiffness for any task involving sustained contact.

Tip 2: Invest in Real-to-Sim Calibration

Default simulation parameters -- joint stiffness, damping, friction, inertia tensors -- often differ from your real robot by 10-50%. Before training, spend 2-4 hours doing system identification: move each joint through its range and measure the actual torque-position-velocity relationship. Use these measured values as the center of your domain randomization distribution. This step alone often reduces sim-to-real error by 30-50% on contact tasks.

Calibrate your camera model as well. Measure the actual intrinsic parameters (focal length, principal point, distortion coefficients) and extrinsic parameters (position and orientation relative to the robot base) of your real cameras and set your simulation cameras to match. Visual policies are surprisingly sensitive to camera parameter mismatches -- a 5% error in focal length can cause a 10-15% drop in grasping accuracy.

Tip 3: Prefer Depth Over RGB for Visual Input

Photorealistic RGB rendering in simulation still does not match real-world cameras closely enough for direct zero-shot transfer on many tasks. Lighting models, material shaders, and shadow algorithms produce images that look similar to humans but differ in ways that neural network feature detectors are sensitive to. Depth images have a much smaller sim-to-real gap because depth is a direct representation of geometry, and simulation renders geometry accurately.

Teams that use depth as the primary visual input (with RGB as a secondary channel for semantic information) consistently report 20-40% improvement in zero-shot sim-to-real transfer on grasping tasks. If your task does not require color or texture information for object discrimination, use depth-only. If it does require color information (sorting objects by color, for example), use an RGB channel but apply aggressive visual domain randomization on it.

Tip 4: Use Privileged Information During Training

The privileged information technique -- sometimes called teacher-student training -- has become the standard approach for sim-to-real locomotion and is increasingly used for manipulation. The idea: train a teacher policy in simulation that has access to ground-truth state information that would be unavailable on real hardware (exact object pose, true friction coefficients, ground-truth contact locations). Then train a student policy that uses only the sensor observations available on the real robot to match the teacher's behavior through distillation.

This works because the teacher policy can solve the task optimally using perfect information, and the student learns to approximate that optimal behavior using only the noisy, partial observations it will have access to on real hardware. The teacher provides a much stronger training signal than raw reward alone. This technique was central to the success of quadruped locomotion transfer at ETH Zurich, Carnegie Mellon, and Unitree, and has been adapted for manipulation tasks involving contact sensing and force control.

Tip 5: Randomize Contact Parameters Specifically

For any task involving sustained contact -- insertion, sliding, pushing, surface following -- contact parameter randomization is critical and often under-emphasized. Randomize not just friction coefficients but also contact stiffness, contact damping, restitution (bounciness), and the solver iteration count for the contact model. The contact solver in simulation introduces artifacts (penetration, jitter, premature slip) that the real world does not have, and randomizing the solver parameters forces the policy to handle these artifacts rather than exploit them.

A practical approach: run your policy on 100 simulation trials with fixed contact parameters and record the contact force profiles. Then repeat with heavily randomized contact parameters. If the policy's success rate drops significantly with randomized contacts, it was exploiting specific simulation artifacts. Retrain with randomized contacts until the success rate is stable, then transfer to real hardware.

Tip 6: Start with Coarse Tasks Before Refining

Attempting to transfer a policy for a precision task directly from simulation is a recipe for frustration. Instead, decompose your task into a coarse version and a fine version. The coarse version -- approach the object, roughly align the gripper, make initial contact -- transfers well from simulation because it depends on geometry and trajectory planning, not precise contact dynamics. The fine version -- final alignment, insertion, controlled force application -- should be trained or fine-tuned on real data.

This hierarchical approach combines the volume of simulation with the fidelity of real data. The simulation policy handles the 80% of the task that involves free-space motion and rough positioning. A small amount of real data (50-200 demonstrations) handles the 20% that requires precise contact dynamics. This hybrid consistently outperforms both sim-only and real-only training when total data budget is limited.

Tip 7: Test Frequently on Real Hardware

The single most common mistake in sim-to-real projects is spending months optimizing in simulation before testing on real hardware. By the time the team discovers how their policy fails in reality, they have invested enormous effort optimizing for the wrong thing. Test on real hardware early and often -- every 1-2 weeks at minimum during active development.

Structure your real-hardware testing as systematic perturbation testing, not random evaluation. Test at 5-10 specific challenging positions: extreme workspace corners, objects near the edge of reachable space, objects at atypical heights or orientations. This structured evaluation reveals whether failures are concentrated at specific conditions (diagnosable and fixable) or randomly distributed (harder to fix, suggesting a fundamental gap). Log both simulation and real failure modes using the same categories and compare the distributions -- where they diverge points directly to the simulation parameters that need improvement.

Simulator Selection: Isaac Sim, MuJoCo, or Genesis

NVIDIA Isaac Sim (built on PhysX 5, integrated with Omniverse) is the leading choice for high-fidelity simulation as of 2026. Its GPU-accelerated physics enables thousands of parallel simulation instances, making reinforcement learning tractable for complex tasks. Isaac Sim also offers the best rendering quality for visual policy training. The main drawbacks are setup complexity, hardware requirements (high-end NVIDIA GPU), and the learning curve for the Omniverse ecosystem.

MuJoCo (now open-source from DeepMind) remains the standard for fast, accurate contact physics in research settings. It is faster per-environment than Isaac Sim, has a simpler API, and offers the most extensive ecosystem of pre-built environments and benchmarks. MuJoCo is the right choice when you need fast iteration on policy architecture and reward design and do not need photorealistic rendering. Its contact model is well-characterized and produces consistent results.

Genesis is a newer simulator that emphasizes speed and differentiability. It supports differentiable physics, enabling gradient-based optimization through the simulation, which can accelerate contact-rich task learning. Genesis is gaining adoption for tasks where differentiable simulation provides a clear advantage -- parameter optimization, trajectory optimization -- but its ecosystem is less mature than MuJoCo or Isaac Sim.

When to Skip Sim Entirely

Simulation is not always the right choice. Skip simulation and go directly to real data collection when: your task involves deformable objects or materials that are poorly simulated (cloth, cables, food); you have access to fast real-world data collection (SVRC's data services can collect 500+ episodes per day); your task requires fewer than 1,000 demonstrations; or when the effort to build an accurate simulation environment exceeds the effort to collect real data.

The decision framework is simple: estimate the cost of building and calibrating a simulation environment for your specific task (including engineering time, hardware for rendering, and the debugging time for sim-to-real transfer). Compare it to the cost of collecting the equivalent amount of real data. For many manipulation tasks in 2026, the real-data path is faster and more predictable. Simulation excels when you need millions of episodes (reinforcement learning), when the task transfers well (locomotion), or when real data collection is dangerous or expensive (surgical robotics, hazardous environments).

Start Your Transfer Pipeline

SVRC's RL environment service provides managed simulation environments with system identification and physics calibration for your specific hardware. For teams pursuing the hybrid approach, we also offer real-data collection through our data services to supplement your simulation training with the real-world demonstrations that close the final gap.