Watching a robot on two wheeled legs makes it tempting to focus on wheel speed. Yet acceleration changes body attitude, leg support, and ground contact together. Balancing, tracking a velocity command, and handling disturbances form a continuous feedback problem.
While working through this project, I focused on what the policy learns and how its actions retain their meaning at deployment. This note follows the Isaac Gym, PPO, ONNX, and ROS pipeline from observations and rewards to the commands ultimately sent to joints.
Reinforcement learning faces a continuous control task
At each step, the environment provides robot observations and a motion command. The policy produces an action, physics advances, and the environment returns a reward and the next observation. The policy must learn from consequences over time rather than imitate an isolated reference pose.
velocity command + robot observation
-> policy -> leg / wheel commands
-> physics step -> reward + next observationVelocity tracking on flat ground, disturbance recovery, and complicated contact each present different learning problems. Distance traveled alone is insufficient: body oscillation, wheel slip, or jittering commands can accompany high short-term speed.
The training setup follows a legged_gym environment structure, using Isaac Gym for parallel physics and PPO for policy updates. Learning to Walk in Minutes provides background on massively parallel training. Its reported training times and robot performance are not measurements of this project.
The actor's inputs determine where it can be deployed
The project configuration specifies a 28-dimensional actor observation, a 148-dimensional critic observation, and a 7-dimensional action vector. These are interface dimensions, not a way to infer the robot's number of legs or motors.
| Interface | Configured dimension | Purpose |
|---|---|---|
| Actor observation | 28 | Inputs available when executing the policy |
| Critic observation | 148 | Expanded training inputs for value estimation |
| Policy action | 7 | Vector mapped into joint commands |
Actor inputs are organized around proprioception: attitude or gravity direction, angular velocity, joint state, commands, and action history. Their exact order, inclusion, and scaling must match the active configuration. History stacking adds a time dimension; a single 28-dimensional frame is not necessarily the encoder's complete input.
The critic has additional state and terrain-height samples. This is an asymmetric actor-critic design: training can use simulator information to improve value estimates, while the deployed action policy must not depend on unavailable ground truth.
History provides clues rather than direct measurements
An optional mlp_encoder encodes observation history for use alongside current inputs and commands. Changes in posture, joint response, and previous actions can provide clues about inertia, contact, and delay.
An encoder does not automatically become a validated velocity sensor. Whether base velocity comes from explicit state estimation or is represented implicitly through history needs a clear answer on both sides of deployment. Simulator ground-truth velocity cannot simply become a physical robot input.
How PPO turns a rollout into an update
PPO alternates environment sampling with multiple epochs of minibatch updates. The actor represents an action distribution, the critic estimates future return, and advantage estimates describe how an action compares with the current expectation.
Training commonly samples a Gaussian distribution for exploration. The clipped objective uses the new-to-old policy probability ratio and advantage to limit update incentives. This discourages excessive changes but does not guarantee more stable motion after every update.
sample rollout with the current policy
estimate returns and advantages with GAE
update actor with the clipped policy objective
fit current value predictions to return targets
apply entropy regularization and collect fresh dataThe rsl_rl PPO implementation provides a useful reference. Value loss trains current critic predictions; stored old predictions are references, rather than substitutes for the network output being optimized.
The project analysis records the following settings. They describe a training configuration, rather than universal optimal hyperparameters for wheeled-leg robots.
| Parameter | Configured value | What it affects |
|---|---|---|
| Parallel environments | 4096–8192 | Sampling scale and hardware load |
| Learning rate | 1e-3 | Update magnitude and scheduling |
| Discount gamma | 0.99 | Weight of future return |
| GAE lambda | 0.95 | Advantage bias and variance |
| Epochs per collected batch | 5 | Data reuse and policy drift |
| Mini-batches | 4 | Update batch partitioning |
| Entropy coefficient | 0.01 | Exploration pressure |
More environments increase sampling capacity. Wall-clock training time and final quality also depend on rollout length, simulation cost, optimization cost, and task difficulty. Environment count alone cannot establish a convergence time.
Rewards define how the robot is willing to move
Velocity-tracking rewards encourage the commanded linear and angular motion. Other terms shape its cost. The configuration includes penalties for action changes, joint acceleration, torques, and collisions.
| Reward or penalty | Intended behavior | Possible failure |
|---|---|---|
| Linear and angular tracking | Follow motion commands | Sacrifice body attitude for tracking |
| Vertical velocity penalty | Reduce unnecessary bouncing | Suppress necessary leg motion |
| Action-change penalty | Avoid abrupt command changes | Become sluggish if overweighted |
| Torque and joint acceleration | Limit effort and violent responses | Avoid moving if overweighted |
| Collision penalty | Reduce unwanted contact | Mislead learning if contacts are mislabeled |
I would inspect each reward component alongside posture, velocity error, and action traces. Total return can improve because one reward is easy to collect without improving disturbance rejection or terrain adaptation.
Rewards also depend on timescale. Changing physics timestep, policy interval, or control decimation can change a penalty's effect. Those settings belong in any comparison between policies.
Wheels and legs need different action semantics
The policy emits a normalized vector that deployment maps to physical commands. Leg joints generally use position targets with PD control, while wheels use angular-velocity targets. The same number cannot be passed unchanged to both actuator types.
This is a conceptual action adapter; the robot configuration defines actual indices, scales, and limits:
a = clip(policy_output, -1, 1)
leg_position_target = nominal_position + leg_scale * a_leg
wheel_velocity_target = wheel_scale * a_wheelAction order, nominal joint positions, position and velocity units, wheel direction, gains, and limits must agree. A single reversed wheel sign or mismatched leg zero position can turn a correct simulated action into different physical motion.
Exploration and deployment require separate decisions
During training, an actor can provide means and sample using standard deviations. Deployment must establish whether the exported graph returns means, sampled actions, or another representation. Gaussian exploration during training does not imply that the physical robot must receive the same random perturbations.
Deterministic means versus sampling is an explicit deployment choice to evaluate. Action clipping supplies numerical bounds; joint limits, mode checks, and exception handling remain necessary underneath it.
Put relevant variation into training
The project analysis includes randomization of friction, mass, execution delays, observation noise, and external pushes. The goal is to reduce dependence on one idealized environment by exposing the policy to a range of physical conditions.
Those ranges need physical justification. Narrow ranges can miss real errors; excessively broad ranges can fill training with implausible combinations. Evaluation should hold out parameter combinations and inspect tracking errors, falls, and saturation rather than rely only on training reward.
Actuator response is another source of mismatch. If an actuator network is introduced, it needs response data and independent validation. Finding a network file in an asset directory does not establish that motor dynamics have been modeled successfully.
Preserve policy meaning from ONNX to ROS
Export the actor needed for execution and retain its history encoder when required. The physical robot does not need the training critic or its privileged observations.
IMU / encoders -> state estimation
-> observation assembly and history
-> normalization -> encoder, if used -> ONNX actor
-> action mapping -> hybrid joint interfaceThe ROS update loop connects state estimation, mode selection, inference, and command submission. The model, observation order, scaling parameters, action mapping, and control period should form one deployment contract.
Some observations use fixed scales; other implementations use mean and variance statistics. Deployment must reproduce what training actually used. After ONNX export, compare training-framework and ONNX Runtime actions on identical observations, then check history-buffer behavior during resets and mode switches.
A state machine can select policies for normal motion, speed, stairs, or fall handling. A mode name does not establish a tested capability. Switching also requires compatible current posture, command targets, and history state.
Evaluate learning through separate measurements
I would separate simulation task performance, export equivalence, and deployment-loop behavior. These cover tracking and falls, action differences for identical inputs, and complete cycle latency and exception handling respectively.
Timing measurements should cover state acquisition through command submission, including tail latency and overruns. Fast MLP inference alone does not establish that the entire loop always meets its deadline. Motion evaluations need recorded terrain, commands, disturbance conditions, and repeated trials.
This note records the reinforcement-learning and deployment pipeline without an accompanying auditable hardware test table. It therefore does not present stable stair climbing, self-righting, or millisecond latency as measured outcomes. The useful questions to keep following are what the policy observes, what it optimizes, and what is ultimately sent to the motors.