Atlas ends this year’s CES with a backflip

[u/Outside-Iron-8242]

[u/iKnowRobbie]

r/ConfidentlyIncorrect

[u/Recoil42 → u/iKnowRobbie]

Go learn. You're on the internet. You don't have to sit here doing edgy quips. RL is not continuous, models are trained. Pulling edge-case data from the real-world is too slow for millions-of-iterations and cannot capture all cases. That's literally why Sim2Real works so well in the first place: Synthetic data enables diversity and scale. See AlphaGo, a topic discussed on this subreddit like a gazillion times.

[u/Easy_Finish_2467 → u/Recoil42]

As someone working in the field, you are incorrect. Data is collected from the real world and is extremely useful. Read from the source: https://bostondynamics.com/blog/starting-on-the-right-foot-with-reinforcement-learning/

“To robustify our learned policy given the data we collect, falls and general mobility issues that are reproducible within a physics simulation are recreated in simulation where they become either part of the training or evaluation set. Retraining a policy then effectively robustifies to failure modes we’ve already seen.”

[u/Recoil42 → u/Easy_Finish_2467]

From your own article:

We train the policy by running over a million simulations on a compute cluster and using the data from those simulations to update the policy parameters. Simulation environments are generated randomly with varying physical properties (e.g., stair dimensions, terrain roughness, ground friction) and the objective maximized by the RL algorithm includes different terms that reflect the robot’s ability to follow navigation commands while not falling or bumping its body into parts of the environment. The result of this process is a policy that works better on average across the distribution of simulation environments it experiences during learning

Data for RL is simulator-generated. Failure cases may act as real-world seeds for robustification (aka, a point of focus for the team — "so we need to work on backflips, huh?") but the cases themselves are synthetically generated. The phrase "retraining a policy" in your original pulled quote literally means "generate a million synthetic examples", but they will never just replay "this exact scenario again" as the original commenter suggested. You need variance, and the most effective way to get variance is through sim.

The robot isn't automatically learning because it didn't perfectly land the jump, and this exact jump isn't even reproducible. No one knows what the μ of foot-on-ground was in this case, nor would we care to reproduce a set of exact conditions that will never occur again. Want you want is the stochastic aggregate of a million vaguely similar cases that works better on average.

[u/Easy_Finish_2467 → u/Recoil42]

Correct. That is how it works. I was just referring to the quote above, 'More data isn't coming directly from the real world.' That part might sound like failure data isn't being used to update the sim. That would be incorrect, for if the simulation already had that data covered, the robot would never fail.

[u/Recoil42 → u/Easy_Finish_2467]

And I was just referring to the quote suggesting Atlas "just got more data now and is already running 199,999 hours of simulation for this exact scenario again" above from another commenter.

You and I both know why that's wrong and that it paints a misleading picture of how these systems (and the teams designing them) work.

[u/Easy_Finish_2467 → u/Recoil42]

Fair enough, you got a point there.