From 846fbfaaabcdbd7a3a3e36cd03d9aae8b687dddf Mon Sep 17 00:00:00 2001 From: JackM323 Date: Fri, 31 Jul 2026 15:56:11 +0200 Subject: [PATCH] Readme reward/termination explanaition added --- README.md | 46 +++++++++++++++++++++++++++++++--------------- ml/run_train.py | 2 +- 2 files changed, 32 insertions(+), 16 deletions(-) diff --git a/README.md b/README.md index 6b82ea5..cdb10b7 100644 --- a/README.md +++ b/README.md @@ -70,12 +70,6 @@ python -m pip install --upgrade pip setuptools wheel pip install -r requirements.txt ``` -> **Linux X11 Headless Note:** If running PyBullet GUI on Linux gives an X11 server connection error (`cannot connect to X server`), ensure your display environment variable is set: -> ```bash -> export DISPLAY=:0 -> python main.py -> ``` - ### Windows (PowerShell) ```powershell @@ -85,12 +79,6 @@ python -m pip install --upgrade pip setuptools wheel pip install -r requirements.txt ``` -If PowerShell blocks script execution: -```powershell -Set-ExecutionPolicy -Scope Process -ExecutionPolicy RemoteSigned -.\.venv\Scripts\Activate.ps1 -``` - --- ## Usage Guide @@ -151,10 +139,38 @@ python ml/run_eval.py --model ml/checkpoints/ppo_joint_command.zip --episodes 3 --- +## Reward Function & Termination Mechanics (`ml/env.py`) + +### 1. Reward Function Formulation + +The per-robot step reward ($R_{\text{step}}$) incentivizes tracking directional velocity commands while maintaining body stability and smooth joint actuation. The total environment step reward is the sum of all individual active robot rewards: + +$$R_{\text{step}} = R_{\text{alive}} + R_{\text{tracking}} + R_{\text{rotation}} - P_{\text{stability}} - P_{\text{action}}$$ + +* **Alive Bonus ($R_{\text{alive}} = +0.1$):** A constant positive baseline awarded every timestep the robot remains upright. +* **Velocity Tracking ($R_{\text{tracking}} = v_{x,\text{cmd}} \cdot v_x + v_{y,\text{cmd}} \cdot v_y$):** Rewards linear movement in the target command direction ($v_x, v_y$). +* **Yaw Rotation Tracking ($R_{\text{rotation}} = \omega_{\text{cmd}} \cdot \omega_z$):** Rewards turning along the vertical yaw axis according to angular command $\omega_{\text{cmd}}$. +* **Body Stability Penalty ($P_{\text{stability}} = 0.2 \cdot (|\text{roll}| + |\text{pitch}|)$):** Penalizes tilting away from a level horizontal posture. +* **Action Energy Penalty ($P_{\text{action}} = 0.01 \cdot \sum a_i^2$):** Penalizes excessive joint delta actions to encourage smooth, energy-efficient leg movements and reduce jitter. + +--- + +### 2. Failure Detection & Termination Logic + +In multi-robot vectorized training (`JackBotEnv`), individual robot failures are handled independently to allow maximum simulation efficiency: + +* **Individual Failure Masking:** A robot is flagged as failed (`failed_robots_mask[idx] = True`) if either condition is met: + * **Severe Tilt:** Base roll or pitch orientation exceeds $0.7\text{ radians}$ ($\approx 40^\circ$). + * **Base Collapse:** Base height drops below $0.05\text{ meters}$ above the ground plane. +* **Visual Failure Feedback:** When a robot fails during GUI execution (`--gui`), its 3D URDF mesh immediately updates to a **semi-transparent dark gray** visual state (`COLOR_FAILED = [0.3, 0.3, 0.3, 0.6]`) to distinguish it from active learners. +* **Environment Termination (`terminated=True`):** The entire environment step resets when any Robot fails. +* **Environment Truncation (`truncated=True`):** Occurs when the episode reaches the maximum allowable step budget (`max_episode_steps = 3000`). + +--- + ## Environment Mechanics & Telemetry (`ml/env.py`) When training with `--gui`, `JackBotEnv` includes dynamic visual feedback mechanisms: -* **Failure Detection & Graying:** Robots are continuously evaluated for roll/pitch tilt ($> 0.7\text{ rad}$) or base collapse ($< 0.05\text{ m}$ height). When a robot fails, its state mask is flagged and its 3D mesh automatically turns **semi-transparent dark gray**. -* **Leader Crown ($\text{👑}$):** A floating crown indicator tracks and sits directly above the robot currently achieving the highest cumulative reward in the multi-robot grid. -* **Termination Threshold:** The environment episode terminates automatically when the percentage of failed robots exceeds the configured threshold (default: $30\%$). \ No newline at end of file +* **Metrics HUD:** A live on-screen text overlay tracking active episode count, total step rate (FPS), average base height, roll/pitch angles, and cumulative per-robot rewards. +* **Leader Crown ($\text{👑}$):** A floating crown debug indicator tracks and positions itself directly above the base of whichever robot is achieving the highest cumulative reward in the multi-robot grid. \ No newline at end of file diff --git a/ml/run_train.py b/ml/run_train.py index 045569f..971061b 100644 --- a/ml/run_train.py +++ b/ml/run_train.py @@ -26,7 +26,7 @@ def main(): parser.add_argument("--device", type=str, default="auto", help="Device to use: 'cpu', 'cuda', or 'auto'") parser.add_argument("--gui", action="store_true", help="Show the PyBullet GUI during training") parser.add_argument("--num-robots", type=int, default=1, help="Number of robots in the training environment") - parser.add_argument("--robot-spacing", type=float, default=0.5, help="Spacing between robots in meters") + parser.add_argument("--robot-spacing", type=float, default=1.5, help="Spacing between robots in meters") parser.add_argument("--start-pose", type=str, choices=["init_deg", "init90_deg"], default="init_deg", help="Initial robot pose at reset") args = parser.parse_args()