JackM323/JackBot
1
0
Fork 0
Code Issues 3 Pull Requests Actions Packages Projects 2 Releases Wiki Activity

Readme and comments and reward pdated

Browse Source 
readme was outdated
env rewards got a penalty for standing still while it should move instead of 0 reward
This commit is contained in:
JackM323
2026-09-13 13:28:24 +02:00
parent 0fe0a8697f
commit 14251aa415
8 changed files with 380 additions and 163 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+334 -151
View File
@@ -1,46 +1,50 @@
# JackBot — Hexapod Control, Simulation & RL Framework
JackBot is a modular 3D hexapod robot control and machine learning framework built in Python. It supports real-time kinematics, multi-input options (GUI, gamepads), hardware streaming (ESP32 / Arduino), and vectorized Reinforcement Learning (PPO) using PyBullet and Gymnasium.
JackBot is a modular Python framework for controlling a six-legged robot in simulation or on hardware. The current workspace reflects a matured control stack with a PyBullet-based simulator, a GUI-driven manual runtime, and a reinforcement-learning pipeline for PPO training and evaluation.
---
## What JackBot Is
JackBot is a Python-based hexapod robot project that combines:
JackBot is a Python-based hexapod project that combines:
* a robot control stack for a six-legged walking robot,
* a physics simulator (`simulation.py`) for testing in software before using real hardware, and
* a reinforcement learning pipeline that teaches the robot how to move through trial and error or pretraining from kinematics.
* a physics simulator (`simulation.py`) for testing in software before using real hardware,
* a reinforcement learning pipeline that trains locomotion policies with PPO and behavioral cloning,
* and a GUI/gamepad input layer for manual operation.
The project is a full control system that can:
The project can currently:
* run the robot in a PyBullet simulation,
* accept human commands from a GUI or gamepad,
* stream target joint positions to physical hardware (ESP32 / Arduino), and
* train and evaluate policy checkpoints using PPO and Behavioral Cloning (BC).
* accept commands from a GUI or gamepad,
* stream target joint positions to physical hardware (ESP32 / Arduino),
* train PPO models and evaluate saved checkpoints,
* generate kinematics-based teacher data for behavioral cloning.
---
## Why the Project Exists
A hexapod is hard to control manually because each leg has several joints and the robot has to maintain balance while moving. Instead of hard-coding every motion rule, this project uses the robot model and physics simulation as a testbed to learn or refine walking behavior.
A hexapod is hard to control manually because each leg has multiple joints and the robot must maintain balance while moving. The project uses a robot model and physics simulation as a testbed for learning motion strategies, refining kinematic control, and validating behavior before applying commands to real hardware.
In practice, the RL workflow looks like this:
In the current codebase, the practical workflow is:
1. The robot starts from a standing pose.
2. The policy receives sensory information (joint angles + command vector) from the simulation.
3. The policy chooses continuous target joint positions.
4. The simulation steps forward and computes physics.
5. The policy receives shaped rewards for surviving, following commands, and remaining stable.
6. Over many iterations, PPO improves the locomotion policy.
1. Start the robot in either simulation or hardware mode.
2. Feed motion commands from GUI/gamepad or a training environment.
3. Convert target foot positions into joint targets using inverse kinematics.
4. Apply these targets to the active backend.
5. Train or evaluate PPO policies against the simulated robot.
---
## Key Components
* **Robot abstraction (`Robot.py`)**: Wraps the robot body, state machine execution, joint control API, and backend selection.
* **Kinematics (`kinematics.py`)**: Turns target leg positions into actual joint angles through inverse kinematics.
* **Simulation (`simulation.py`)**: Consolidates PyBullet scene management, motor positioning, physics queries, and environment stepping into a single manager.
* **Inputs (`inputs/`)**: Interface for receiving control commands from a GUI, gamepad, or procedural random generators.
* **ML Subsystem (`ml/`)**: Contains Gymnasium environments (`env.py`), custom callbacks (`callbacks.py`), behavioral cloning pretraining (`pretrain_bc.py`), PPO training (`run_train.py`), and multi-phase evaluation scripts (`run_eval.py`, `run_eval_training.py`).
* **Robot abstraction (`Robot.py`)**: Central handler for the active backend, current joint state, IK/FK helpers, gait handling, and RL action application.
* **Kinematics (`kinematics.py`)**: Converts target foot positions into joint angles using IKPy chains for each leg.
* **Simulation (`simulation.py`)**: Manages PyBullet scene loading, stepping, joint actuation, base pose queries, and physics telemetry.
* **Inputs (`inputs/`)**: Receives commands from GUI sliders, Pygame gamepads, and random command generation.
* **ML Subsystem (`ml/`)**: Contains the Gymnasium environment (`env.py`), PPO training (`run_train.py`), evaluation (`run_eval.py`), BC pretraining (`pretrain_bc.py`), callbacks, and metrics overlays.
* **States (`states/`)**: Contains additional state-machine classes such as `IdleState`, `WalkingState`, and `WaveState`. These exist in the repository, but the current active runtime in `main.py` does not currently drive them through `Robot.tick()`.
---
@@ -48,49 +52,68 @@ In practice, the RL workflow looks like this:
```text
JackBot/
├── main.py # Manual control, GUI, and hardware streaming entry point
├── Robot.py # Robot wrapper, gait execution, and backend abstraction
├── config.py # Global settings and hardware/backend configuration
├── kinematics.py # Inverse kinematics mapping cartesian targets to joint angles
├── simulation.py # PyBullet simulation manager (handles engine, URDF loading, & physics)
├── robot_init.py # Neutral standing pose and initial joint definitions
├── DataTypes.py # Typed numpy structures for positions and joint angles
├── main.py # Manual runtime entry point
├── Robot.py # Unified robot wrapper + backend selection
├── config.py # Central runtime configuration
├── kinematics.py # IK/FK helpers
├── simulation.py # PyBullet scene / physics manager
├── robot_init.py # Initial joint definitions and center points
├── DataTypes.py # Typed arrays for positions / angles
├── JackBotUrdf.urdf # Robot URDF
│
├── states/ # Gait and state-machine behavior logic
├── states/ # State classes present in the project
│ ├── State.py
│ ├── IdleState.py
│ ├── WalkingState.py
│ └── WaveState.py
│ ├── WaveState.py
│ ├── ml_walking.py
│ └── __init__.py
│
├── inputs/ # Command input sources
├── inputs/ # Command input providers
│ ├── InputProvider.py
│ ├── PygameController.py
│ └── RandomeInputProvider.py
│ ├── RandomeInputProvider.py
│ └── RandomInputProvider.py
│
├── gui/ # Visual control frontend (Tkinter/CustomTkinter)
├── gui/ # GUI/dashboard control layer
│ └── MainWindow.py
│
├── EspCommunication.py # ESP32 socket communication layer
├── ArduinoCommunication.py # Arduino serial communication layer
├── JackBotUrdf.urdf # Robot URDF mesh and joint axis definition
├── ml/ # Gymnasium RL training + evaluation stack
│ ├── env.py # RL environment + reward logic
│ ├── callbacks.py # SB3 callbacks for logs/curriculum
│ ├── pretrain_bc.py # Behavior cloning pretraining
│ ├── run_train.py # PPO training entry point
│ ├── run_eval.py # Evaluation of saved checkpoints
│ ├── run_eval_training.py # Kinematics-mode benchmark script
│ ├── MetricsOverlay.py # 3D in-scene metrics HUD
│ └── checkpoints/ # Model checkpoints + saved runs
│
└── ml/ # Reinforcement Learning Subsystem
├── env.py # Gymnasium environment wrapping PyBullet & Robot control
├── callbacks.py # Custom SB3 callbacks for TensorBoard logging & Curriculum progression
├── pretrain_bc.py # Behavioral Cloning (BC) script to pre-train policy from IK teacher
├── run_train.py # Main training entry point (runs parallelized PPO via SubprocVecEnv)
├── run_eval.py # Phase-by-phase policy evaluator with CLI reports
├── run_eval_training.py # Benchmark reward system across phases using pure kinematics
└── MetricsOverlay.py # PyBullet HUD debug text and visual overlays
├── EspCommunication.py # ESP32 UDP communication layer
├── ArduinoCommunication.py # Arduino serial communication layer
├── Helper Scripts/ # Utility scripts
│ ├── FindCenterPoints.py
│ └── torqueCalc.py
│
├── requirements.txt
├── README.md
└── ...
```
### Important current-state note
The repository contains a few pieces that are still present but not fully connected to the current runtime:
* `states/WalkingState.py`, `states/WaveState.py`, and `states/ml_walking.py` exist, but `main.py` currently drives `Robot.tick()` directly instead of routing through the active `STATE_REGISTRY`.
* `inputs/RandomeInputProvider.py` exists, but the active runtime does not currently use it directly. The GUI resolves random walking behavior in `gui/MainWindow.py`.
* `WaveEmoteState` / `LaolaWaveEmoteState` are defined in `states/WaveState.py`, but they are not currently registered in `states/__init__.py`.
---
## System Requirements
* **Python 3.12** (Recommended)
* **Python:** 3.12 recommended
* **OS:** Windows 10/11 or Linux
* **Dependencies:** `pybullet`, `gymnasium`, `stable-baselines3`, `torch`, `numpy`, `pygame`, `ikpy`, `pyserial`, `matplotlib`
* **Dependencies:** `pybullet`, `gymnasium`, `stable-baselines3`, `torch`, `numpy`, `pygame`, `ikpy`, `pyserial`, `matplotlib`, `dearpygui`
---
@@ -117,128 +140,269 @@ pip install -r requirements.txt
---
### Usage Guide
## Usage Guide
## Manual Control & Hardware Streaming (`main.py`)
`main.py` is the operational entry point for driving the robot manually via GUI sliders or a gamepad, running either in 3D PyBullet simulation or connected to physical hardware.
`main.py` is the main operational entry point for driving the robot manually through the GUI or a connected gamepad.
To launch:
```bash
python main.py
```
#### Configuration (`config.py`)
Edit `config.py` prior to launching `main.py` to configure execution mode and connections:
### Configuration (`config.py`)
* **Backend Selection (`cfg.backend`):**
* `BackendType.SIMULATION`: Executes motion inside a 3D PyBullet window.
* `BackendType.ESP32`: Streams target joint angles over WiFi sockets to an ESP32 micro-controller (`cfg.esp32_ip`, `cfg.esp32_port`).
* `BackendType.ARDUINO`: Streams target joint angles over Serial to an Arduino (`cfg.port`, `cfg.baudrate`).
Before launching `main.py`, edit `config.py` to select the backend and connection targets.
* **Backend Selection (`cfg.backend`)**:
* `BackendType.SIMULATION`: run inside a PyBullet window
* `BackendType.ESP32`: stream motion over UDP to an ESP32
* `BackendType.ARDUINO`: stream motion over serial to an Arduino
Supported configuration values in the current code include:
* `backend`
* `urdf_path`
* `port`, `baudrate`
* `esp32_ip`, `esp32_port`
* `tick_rate_hz`, `step_duration`
* `step_height`, `step_length`
### GUI input sources
The current GUI (`gui/MainWindow.py`) exposes these input sources:
* `Gamepad`
* `GUI Sliders`
* `Random Walk`
The active `main.py` runtime resolves commands through `resolve_active_command(...)` and then passes the resulting `vector_dirmov` directly to `Robot.tick()`.
---
### Machine Learning Pipeline (`ml/`)
## Machine Learning Pipeline (`ml/`)
The machine learning system trains the hexapod to move using a combination of Behavioral Cloning (BC) (optional) and Proximal Policy Optimization (PPO) driven by a multi-phase curriculum.
The ML subsystem currently uses:
### ML Workflow Steps
* **Gymnasium** as the environment API
* **PyBullet** as the physics engine
* **Stable-Baselines3 PPO** as the learning algorithm
* **Curriculum phases** to gradually expose more difficult command spaces
The machine learning system trains the hexapod to move using a combination of Behavioral Cloning (BC) (optional) and Proximal Policy Optimization (PPO) driven by a multi-phase curriculum.
#### Methods implemented
### Current ML scripts
The current ML stack uses:
* `ml/run_train.py` — PPO training entry point
* `ml/run_eval.py` — phase-based model evaluation
* `ml/run_eval_training.py` — reward benchmark for kinematics mode
* `ml/pretrain_bc.py` — collect teacher data from the kinematics solver and pretrain a base policy
* `ml/env.py` — environment definition and reward logic
* `ml/callbacks.py` — reward logging and curriculum callbacks
* **Gymnasium** as the environment API.
* **PyBullet** as the physics engine.
* **Stable-Baselines3 PPO** as the learning algorithm.
* **Curriculum learning** to gradually unlock harder command regimes.
### Training flow
The key ideas are:
Training is launched with:
1. **Standing first**: the robot must stay upright and stable.
2. **Forward motion second**: the robot must learn to move in a commanded direction.
3. **Turning next**: after forward command-following is stable, the policy gets a yaw/turning challenge.
4. **Omni-direction later**: the full command space is gradually exposed only once the simpler skills are reliable.
This staged approach is important because learning to balance and walk all at once is difficult for a real hexapod.
#### Training flow
Training is launched through `ml/run_train.py` and hands off to `ml/train.py`.
Typical training starts as:
```bash
python ml/run_train.py --timesteps 100000 --model ml/checkpoints/ppo_joint_command
python ml/run_train.py --total-timesteps 1500000 --gui
```
Training uses the simulator as the environment, runs PPO updates, and periodically saves checkpoint models.
The current training script creates a vectorized PPO environment, optionally loads a pre-trained checkpoint, and saves results into `ml/checkpoints/` and `ml/logs/`.
#### Evaluation flow
### Evaluation flow
Evaluation uses `ml/run_eval.py` and `ml/evaluate.py`.
Evaluation is launched with:
A saved PPO model is loaded and run in deterministic evaluation mode. This is used to answer questions like:
* Does the policy actually move?
* Does it survive longer than before?
* Does it remain stable under the commanded motion?
Example:
```bash
python ml/run_eval.py --model ml/checkpoints/ppo_joint_command.zip --episodes 5 --gui
python ml/run_eval.py --model ml/checkpoints/jackbot_kinematics_base.zip --episodes-per-phase 5 --gui
```
This script runs a multi-phase deterministic evaluation suite with fixed command vectors for:
* `FORWARD`
* `TURN_AND_DIRECTION`
* `OMNI_DIRECTION`
* `FULL_COMMAND`
### Behavioral cloning pretraining
The repository also contains a behavioral cloning route:
```bash
python ml/pretrain_bc.py --num-samples 100000 --epochs 15 --save-path ml/checkpoints/jackbot_kinematics_base.zip
```
This script gathers `(observation, action)` data from the kinematics teacher and trains a PPO policy to act as a learned base model.
---
## What the Reward Is Trying to Teach
The reward function is not a single number with one purpose. It combines several terms so the policy learns to:
The reward function in `ml/env.py` is designed to teach several things at once:
* survive in the environment,
* move in the commanded direction,
* avoid unwanted sideways drift,
* stay upright without excessive tilt,
* avoid giant control jumps,
* keep the robot away from a low collapsed posture.
* survival and stability,
* following the commanded direction,
* avoiding lateral drift,
* maintaining base height,
* reducing abrupt control changes,
* staying close to a stable stance when the command is zero.
The main ideas are:
The environment uses several reward components, including:
* a small alive bonus for staying upright,
* directional movement rewards,
* penalties for drifting or standing still when a command is active,
* penalties for excessive tilt or collapse,
* a penalty for large abrupt control changes.
* height reward,
* stability reward,
* pose closeness reward,
* smoothness reward,
* linear velocity tracking,
* angular velocity tracking,
* jitter penalty,
* stand-by penalty when the command is zero.
This style of reward shaping encourages the policy to learn real locomotion rather than just freezing in place.
### Detailed reward structure
### Curriculum / phase progression
The reward is built step by step inside `JackBotEnv._compute_reward()` in `ml/env.py`. It is not a single binary success signal; it is a dense shaping signal that rewards good behavior continuously throughout the episode.
The environment uses a staged curriculum to expose command complexity gradually.
At each control step, the environment measures:
* **Phase 0**: basic standing / forward-focused behavior
* **Phase 1**: turning and directional regularization
* **Phase 2**: omni-directional movement
* **Phase 3**: full-command challenge
* current body height,
* roll and pitch angles,
* current joint configuration,
* measured linear and angular velocities,
* the active command vector `[vx, vy, omega]`,
* the difference between the current action and the previous actions.
The goal is to prevent the learner from being asked to master all difficult motion goals at once.
From these values, it computes a set of sub-rewards:
* `height reward`: a Gaussian-style reward based on how close the robot body is to the target height.
* `stability reward`: a reward for keeping roll/pitch low and the body well balanced.
* `pose closeness reward`: rewards staying near the default standing joint posture.
* `smoothness reward`: rewards actions that change gradually instead of abruptly.
* `linear velocity reward`: encourages the robot to move in the commanded direction and speed.
* `angular velocity reward`: rewards matching the commanded turning rate.
The environment then adds two kinds of correction terms:
* `jitter penalty`: subtracts a small amount when action changes are noisy or jerky.
* `stand_penalty`: when the command is effectively zero, penalizes unintended movement and yaw drift.
This makes the reward function behave as a soft guidance system: the agent gets a steady gradient that says “this is closer to what we want” or “this is worse than the desired behavior.”
### Standing mode vs. walking mode
The reward code handles two cases differently:
#### 1. Standing mode
When the command vector is close to zero (`cmd_norm < 0.05` and `abs(cmd_yaw) < 0.05`), the agent is not supposed to move much. In that case the reward focuses on:
* keeping the body at the correct height,
* staying stable,
* maintaining a clean posture,
* staying smooth.
A small `stand_penalty` is then applied to discourage unintended speed and yaw drift while the robot is supposed to hold position.
#### 2. Walking mode
When a non-zero command is active, the robot is rewarded for moving in the intended direction and turning at the requested rate. The reward then emphasizes:
* matching the commanded linear velocity,
* matching the commanded yaw rate,
* continuing to maintain height and stability,
* staying smooth in its control inputs.
If the command expects movement but the robot is effectively still, the code now applies a stronger `stillness_penalty` instead of a neutral reward. This means standing while the command says to move is explicitly discouraged.
### Why the reward is shaped this way
The goal is not just to teach the robot to stay alive. The reward is designed so that a PPO agent learns multiple useful habits at once:
* do not collapse or tip over,
* keep the body at a sensible height,
* follow directional commands,
* avoid unstable oscillations,
* avoid overreactive control jumps,
* stay near a normal standing pose when no motion is requested.
This is why the environment is not based on a single sparse reward such as “+1 for success, 0 otherwise.” Instead, it uses dense reward shaping so the policy receives useful feedback on every step.
### Are the penalties real penalties?
Yes — in the reward function they are real negative contributions. For example:
* `jitter_penalty` subtracts from the step reward when action changes are too abrupt,
* `stand_penalty` subtracts when the robot moves unnecessarily while standing,
* `stillness_penalty` subtracts when movement is commanded but the robot remains essentially frozen.
In the current implementation, a non-zero command that is not followed by meaningful motion now yields an explicit penalty instead of a neutral reward. This is the main behavior change requested for the training setup: standing while the command says to move is now actively discouraged.
The code does this:
```python
final_reward = step_reward + jitter_penalty + alive_bonus
```
That means:
* negative penalty terms can now reduce the reward below zero,
* the reward is no longer clipped to zero in this path,
* and the episode is still only ended by the separate failure check in `_update_robot_failure()`.
So the answer is:
* the penalties are real reward penalties,
* they are now strong enough to discourage command-mismatch behavior,
* and the actual terminal condition remains the failure check in `_update_robot_failure()`.
### What actually ends an episode?
The episode ends when the robot is considered failed, not when a reward penalty is applied. In `ml/env.py`, the environment marks the robot as failed if:
* it is too tilted (`roll` or `pitch` exceed the configured failure threshold), or
* it has collapsed below a minimum body-height threshold.
That is a hard termination condition. In other words:
* reward penalties discourage bad behavior,
* failure conditions stop the episode when the robot is clearly unstable or collapsed.
### Practical interpretation
A good mental model is:
* the reward function teaches the robot what “good locomotion” looks like,
* the failure check prevents the robot from continuing when it is physically broken or unstable,
* and the curriculum gradually increases the difficulty of the commands as the robot becomes more capable.
This combination is a common reinforcement-learning setup for locomotion: dense rewards shape the desired behavior, while hard failure conditions protect the training process from degenerate states.
This reward shaping encourages the robot to learn locomotion patterns rather than simply freezing in place.
---
## Curriculum / Phase Progression
The environment currently uses the following curriculum stages:
* **STAND_ONLY**
* **FORWARD**
* **TURN_AND_DIRECTION**
* **OMNI_DIRECTION**
* **FULL_COMMAND**
The training environment gradually advances phase complexity based on survival, stability, and movement metrics. `ml/callbacks.py` includes custom curriculum-handling logic for the online learning loop.
---
## How the Control Loop Works in Practice
A simple, human-readable pipeline is:
A simplified view of the current runtime is:
1. `main.py` or the training/evaluation wrapper creates the robot environment.
2. The simulation runs the robot in PyBullet.
3. The policy observes the latest joint states and command vectors.
4. PPO predicts a new action.
5. The action is applied to the robot joints.
6. The simulator steps forward one frame.
7. The reward is computed from the new state.
8. The policy improves based on that reward.
That loop is the essence of the machine learning part of the project.
1. `main.py` starts the joystick controller process and opens the GUI.
2. The GUI resolves active commands (`Gamepad`, slider, or random walk).
3. `Robot.tick()` receives the current motion vector and applies it through the active backend.
4. In simulation mode, `Simulation.step()` advances the PyBullet world.
5. In training/evaluation mode, `JackBotEnv.step()` computes reward, updates curriculum, and returns observations.
6. PPO uses the observation/action loop to improve the policy.
---
@@ -247,54 +411,73 @@ That loop is the essence of the machine learning part of the project.
This repository is useful because it combines several layers that are often separate:
* robot control and kinematics,
* physical simulation,
* physics simulation,
* command input sources,
* RL environment construction,
* PPO training and evaluation.
* PPO training and evaluation,
* hardware communication layers.
For a beginner, the easiest way to think about the repo is:
For a newcomer, the easiest way to think about the project is:
- `main.py` is for direct manual control,
- `ml/env.py` is the simulator-to-policy interface,
- `ml/train.py` is where training happens,
- `ml/evaluate.py` is where you check whether the learned policy is actually good.
- `main.py` is the manual control entry point,
- `Robot.py` is the core robot wrapper,
- `simulation.py` is the physics layer,
- `ml/env.py` is the environment interface for RL,
- `ml/run_train.py`, `ml/run_eval.py`, and `ml/pretrain_bc.py` are the main ML workflows.
---
## Reward Function & Termination Mechanics (`ml/env.py`)
## Current runtime notes and caveats
### 1. Reward Function Formulation
### State system status
The per-robot step reward ($R_{\text{step}}$) is shaped to reward tracked motion and survival while strongly discouraging stillness once a command is present. The current implementation uses the following structure:
The state classes are present in the repository, but the current runtime path is not using them as the main control loop:
$$R_{\text{step}} = R_{\text{alive}} + R_{\text{tracking}} + R_{\text{rotation}} - P_{\text{drift}} - P_{\text{still}}(t) - P_{\text{height}} - P_{\text{stability}} - P_{\text{large-delta}}$$
* `Robot.tick()` currently updates the robot directly from `vector_dirmov`
* the `STATE_REGISTRY` exists, but it is not the path used by the current `main.py` execution flow
* the state subsystem remains partially implemented and should be treated as a legacy or optional extension
* **Alive Bonus ($R_{\text{alive}} \approx +0.02$):** A small positive baseline awarded every timestep the robot remains upright.
* **Velocity Tracking ($R_{\text{tracking}}$):** Rewards linear movement aligned with the commanded direction vector.
* **Yaw Rotation Tracking ($R_{\text{rotation}}$):** Rewards turning in the commanded yaw direction.
* **Drift Penalty ($P_{\text{drift}}$):** Penalizes lateral motion that does not align with the commanded direction.
* **Age-Ramped Stillness Penalty ($P_{\text{still}}(t)$):** Penalizes remaining stationary when a command is active; the penalty ramps up as the episode gets older so the policy cannot settle into a frozen local optimum.
* **Height Penalty ($P_{\text{height}}$):** Penalizes falling below the measured post-settle standing height.
* **Body Stability Penalty ($P_{\text{stability}}$):** Penalizes roll and pitch tilt.
* **Large-Delta Penalty ($P_{\text{large-delta}}$):** Penalizes only large joint-control jumps, not normal continuous command usage, so the robot is free to use its joints continuously.
### Input source status
The repository currently includes both:
* `inputs/PygameController.py` for gamepad input
* `gui/MainWindow.py` for selecting `Gamepad`, `GUI Sliders`, and `Random Walk`
The standalone random input provider file is present, but it is not the path currently used by `main.py`.
### Hardware communication status
The hardware communication classes still exist:
* `EspCommunication.py` for ESP32 UDP
* `ArduinoCommunication.py` for Arduino serial
They are available via `cfg.backend`, but they are not the primary path in the current example workflows shown here.
---
### 2. Failure Detection & Termination Logic
## Helper Scripts
In multi-robot vectorized training (`JackBotEnv`), individual robot failures are handled independently to allow maximum simulation efficiency:
### `Helper Scripts/FindCenterPoints.py`
* **Individual Failure Masking:** A robot is flagged as failed (`failed_robots_mask[idx] = True`) if either condition is met:
* **Severe Tilt:** Base roll or pitch orientation exceeds $0.9\text{ radians}$.
* **Base Collapse:** Base height drops below a relative threshold derived from the measured settled standing height.
* **Visual Failure Feedback:** When a robot fails during GUI execution (`--gui`), its 3D URDF mesh immediately updates to a **semi-transparent dark gray** visual state (`COLOR_FAILED = [0.3, 0.3, 0.3, 0.6]`) to distinguish it from active learners.
* **Environment Termination (`terminated=True`):** The environment as a whole is marked as terminated when the failure ratio reaches the configured threshold.
* **Environment Truncation (`truncated=True`):** Occurs when the episode reaches the maximum allowable step budget (`max_episode_steps`).
This script appears to be a utility for analyzing kinematic center point data and related foot-placement experiments. It is not part of the active runtime path.
### `Helper Scripts/torqueCalc.py`
This is a stand-alone Tkinter utility for estimating servo torque requirements based on robot mass, leg dimensions, and safety factor. It is a design/support script rather than part of the main robot runtime.
---
## Environment Mechanics & Telemetry (`ml/env.py`)
## Bottom Line
When training with `--gui`, `JackBotEnv` includes dynamic visual feedback mechanisms:
The current codebase is a working hybrid of:
* **Metrics HUD:** A live on-screen text overlay tracking active episode count, total step rate (FPS), average base height, roll/pitch angles, and cumulative per-robot rewards.
* real-time robot control,
* PyBullet simulation,
* GUI/manual input,
* PPO-based RL training and evaluation,
* partial state-machine scaffolding,
* hardware communication support.
The information in this README has been updated to match the current files in the workspace, especially around the true ML entry points, active runtime flow, and the fact that some older state-helper modules are present but not currently wired into the main execution path.
ml/MetricsOverlay.py
+5 -1
View File
@@ -1,5 +1,9 @@
"""
ml/MetricsOverlay.py - Camera-Facing (Billboard) 3D Floating Text Overlay
ml/MetricsOverlay.py - 3D HUD overlay for live simulation telemetry.
This file provides a small PyBullet HUD renderer that draws live robot metrics in the
simulation scene. It is used during GUI runs to show the current phase, command vector,
reward information, and basic motion statistics without leaving the 3D view.
"""
from typing import List, Tuple, Optional, Dict
import numpy as np
ml/callbacks.py
+5 -2
View File
@@ -1,6 +1,9 @@
"""
ml/callbacks.py - Stable-Baselines3 Custom Callbacks for Logging & Curriculum Advancement
Fully compatible with SubprocVecEnv and DummyVecEnv.
ml/callbacks.py - Stable-Baselines3 callbacks for training diagnostics.
These callbacks extend SB3 training with two responsibilities: logging reward-component
statistics for TensorBoard/console output, and checking whether the curriculum should
advance to a harder set of commands based on recent training performance.
"""
import numpy as np
ml/env.py
+12 -4
View File
@@ -1,5 +1,10 @@
"""
ml/env.py - Gymnasium Environment for JackBot Hexapod RL Training
ml/env.py - Gymnasium environment for JackBot RL training and evaluation.
This file defines JackBotEnv, the main training/evaluation environment used by PPO.
It wraps the PyBullet simulation and Robot interfaces into a Gymnasium-compatible
step/reset loop, manages command sampling, curriculum progression, and reward
calculation, and exposes metrics that the training callbacks can log.
"""
import time
import math
@@ -363,7 +368,10 @@ class JackBotEnv(gym.Env):
target_speed = math.hypot(target_vx, target_vy)
if not is_moving:
total_reward = 0.0
# If the command says move but the robot stays effectively still,
# give a real penalty instead of a neutral reward.
stillness_penalty = -0.10
total_reward = stillness_penalty
else:
lin_vel_error = (filtered_vx - target_vx)**2 + (filtered_vy - target_vy)**2
r_lin_vel = math.exp(-25.0 * lin_vel_error)
@@ -371,7 +379,7 @@ class JackBotEnv(gym.Env):
if target_speed > 0.08 and raw_speed < 0.03:
r_lin_vel = 0.0
stillness_penalty = -0.1 # Softened from -0.25
stillness_penalty = -0.05
w_lin_vel, w_ang_vel, w_height, w_stability, w_smoothness = 0.55, 0.15, 0.10, 0.12, 0.08
total_reward = (
@@ -381,7 +389,7 @@ class JackBotEnv(gym.Env):
step_reward = float(total_reward / 10.0)
alive_bonus = 0.01
final_reward = max(0.0, step_reward + jitter_penalty + alive_bonus)
final_reward = step_reward + jitter_penalty + alive_bonus
self.last_reward_components = {
"height": float(r_height),
ml/pretrain_bc.py
+6 -1
View File
@@ -1,5 +1,10 @@
"""
ml/pretrain_bc.py - Behavioral Cloning from Kinematics Teacher
ml/pretrain_bc.py - Behavioral Cloning teacher-data pipeline.
This script collects observation/action pairs from the kinematics solver, then uses
those pairs as training data for a PPO policy. In practice it serves as a teacher-
student pretraining step: the kinematics controller generates sample trajectories, and
this file teaches the policy to imitate those behavior patterns before PPO training.
"""
import sys
import time
ml/run_eval.py
+6 -2
View File
@@ -1,6 +1,10 @@
"""
ml/run_eval.py - Phase-by-Phase Policy Evaluator for JackBot
Evaluates a trained model across all curriculum phases with fixed command vectors.
ml/run_eval.py - Evaluate a saved JackBot policy across fixed command phases.
This script loads a trained PPO checkpoint, instantiates the Gymnasium environment in
non-random mode, and runs deterministic evaluation episodes for several command
regimes. It is used to measure whether a policy can survive, move, and maintain
stability under forward, turning, lateral, and omni-direction commands.
"""
import argparse
ml/run_eval_training.py
+6 -1
View File
@@ -1,5 +1,10 @@
"""
ml/run_eval_training.py - Benchmark reward system across all curriculum phases.
ml/run_eval_training.py - Run a kinematics-mode reward benchmark.
This script repeatedly resets the JackBot environment in kinematics mode and applies
fixed command vectors for each curriculum phase. It is intended as a lightweight
benchmark to inspect reward components, movement quality, and survival behavior without
requiring an already-trained PPO model.
"""
import time
import numpy as np
ml/run_train.py
+6 -1
View File
@@ -1,4 +1,9 @@
"""Minimal training launcher for quick experiments.
"""PPO training launcher for JackBot.
This script is the main entry point for training a policy in the JackBot Gymnasium
environment. It creates a vectorized environment, optionally loads a pretrained base
model, runs Stable-Baselines3 PPO for a configured number of timesteps, and saves
checkpoints plus evaluation artifacts during training.
Usage:
python ml/run_train.py --total-timesteps 1500000 --gui