diff --git a/README.md b/README.md index b8a59fc..8496f4f 100644 --- a/README.md +++ b/README.md @@ -9,40 +9,38 @@ JackBot is a modular 3D hexapod robot control and machine learning framework bui JackBot is a Python-based hexapod robot project that combines: * a robot control stack for a six-legged walking robot, -* a physics simulator for testing in software before using real hardware, and -* a reinforcement learning pipeline that teaches the robot how to move through trial and error. +* a physics simulator (`simulation.py`) for testing in software before using real hardware, and +* a reinforcement learning pipeline that teaches the robot how to move through trial and error or pretraining from kinematics. -The project is not just one script or one model. It is a full control system that can: +The project is a full control system that can: * run the robot in a PyBullet simulation, * accept human commands from a GUI or gamepad, -* stream target joint positions to physical hardware, and -* train a policy using PPO so the robot can learn locomotion automatically. +* stream target joint positions to physical hardware (ESP32 / Arduino), and +* train and evaluate policy checkpoints using PPO and Behavioral Cloning (BC). ## Why the Project Exists -A hexapod is hard to control manually because each leg has several joints and the robot has to maintain balance while moving. Instead of hard-coding every motion rule, this project uses the robot model and physics simulation as a testbed to learn walking behavior. +A hexapod is hard to control manually because each leg has several joints and the robot has to maintain balance while moving. Instead of hard-coding every motion rule, this project uses the robot model and physics simulation as a testbed to learn or refine walking behavior. -In practice, the workflow looks like this: +In practice, the RL workflow looks like this: 1. The robot starts from a standing pose. -2. The policy receives sensory information from the simulation. -3. The policy chooses new joint target motions. -4. The simulation updates the physics. -5. The policy receives a reward for surviving and moving in the right direction. -6. Over many iterations, PPO improves the controller. +2. The policy receives sensory information (joint angles + command vector) from the simulation. +3. The policy chooses continuous target joint positions. +4. The simulation steps forward and computes physics. +5. The policy receives shaped rewards for surviving, following commands, and remaining stable. +6. Over many iterations, PPO improves the locomotion policy. --- ## Key Components -* **Robot abstraction (`Robot.py`)**: wraps the robot body, robot state, and joint control API. -* **Kinematics (`kinematics.py`)**: turns target leg poses into actual joint angles through inverse kinematics. -* **Simulation (`simulation.py`)**: creates and updates the PyBullet world where the robot can be tested safely. -* **Inputs (`inputs/`)**: lets the robot be controlled from either a GUI, a gamepad, or a generated target command. -* **ML environment (`ml/env.py`)**: provides the Gymnasium environment that the policy interacts with. -* **PPO training (`ml/train.py`)**: trains a policy using stable-baselines3. -* **Evaluation (`ml/evaluate.py`)**: loads a saved model and runs policy rollouts to inspect performance. +* **Robot abstraction (`Robot.py`)**: Wraps the robot body, state machine execution, joint control API, and backend selection. +* **Kinematics (`kinematics.py`)**: Turns target leg positions into actual joint angles through inverse kinematics. +* **Simulation (`simulation.py`)**: Consolidates PyBullet scene management, motor positioning, physics queries, and environment stepping into a single manager. +* **Inputs (`inputs/`)**: Interface for receiving control commands from a GUI, gamepad, or procedural random generators. +* **ML Subsystem (`ml/`)**: Contains Gymnasium environments (`env.py`), custom callbacks (`callbacks.py`), behavioral cloning pretraining (`pretrain_bc.py`), PPO training (`run_train.py`), and multi-phase evaluation scripts (`run_eval.py`, `run_eval_training.py`). --- @@ -50,40 +48,40 @@ In practice, the workflow looks like this: ```text JackBot/ -├── main.py # manual control and hardware streaming entry point -├── Robot.py # robot wrapper and backend abstraction -├── config.py # global settings and communication configuration -├── kinematics.py # inverse kinematics used to map motion commands to joints -├── simulation.py # PyBullet simulation shell for the robot -├── robot_init.py # neutral standing pose and initial joint values -├── DataTypes.py # typed data structures for positions and joint angles +├── main.py # Manual control, GUI, and hardware streaming entry point +├── Robot.py # Robot wrapper, gait execution, and backend abstraction +├── config.py # Global settings and hardware/backend configuration +├── kinematics.py # Inverse kinematics mapping cartesian targets to joint angles +├── simulation.py # PyBullet simulation manager (handles engine, URDF loading, & physics) +├── robot_init.py # Neutral standing pose and initial joint definitions +├── DataTypes.py # Typed numpy structures for positions and joint angles │ -├── states/ # gait/state-machine behavior logic +├── states/ # Gait and state-machine behavior logic │ ├── State.py │ ├── IdleState.py │ ├── WalkingState.py │ └── WaveState.py │ -├── inputs/ # command sources for the robot +├── inputs/ # Command input sources │ ├── InputProvider.py │ ├── PygameController.py │ └── RandomeInputProvider.py │ -├── gui/ # visual control frontend +├── gui/ # Visual control frontend (Tkinter/CustomTkinter) │ └── MainWindow.py │ -├── EspCommunication.py # ESP32 communication layer +├── EspCommunication.py # ESP32 socket communication layer ├── ArduinoCommunication.py # Arduino serial communication layer -├── JackBotUrdf.urdf # robot mesh and joint definition +├── JackBotUrdf.urdf # Robot URDF mesh and joint axis definition │ -└── ml/ # reinforcement learning subsystem - ├── env.py # Gymnasium environment used by PPO - ├── SimManager.py # PyBullet scene setup and stepping - ├── MetricsOverlay.py # HUD and visual overlays - ├── train.py # PPO training entry point - ├── run_train.py # command-line training launcher - ├── evaluate.py # policy evaluation loop - └── run_eval.py # CLI wrapper for evaluation +└── ml/ # Reinforcement Learning Subsystem + ├── env.py # Gymnasium environment wrapping PyBullet & Robot control + ├── callbacks.py # Custom SB3 callbacks for TensorBoard logging & Curriculum progression + ├── pretrain_bc.py # Behavioral Cloning (BC) script to pre-train policy from IK teacher + ├── run_train.py # Main training entry point (runs parallelized PPO via SubprocVecEnv) + ├── run_eval.py # Phase-by-phase policy evaluator with CLI reports + ├── run_eval_training.py # Benchmark reward system across phases using pure kinematics + └── MetricsOverlay.py # PyBullet HUD debug text and visual overlays ``` --- @@ -91,7 +89,7 @@ JackBot/ ## System Requirements * **Python 3.12** (Recommended) -* **OS:** Linux (Ubuntu/Debian) or Windows 10/11 +* **OS:** Windows 10/11 or Linux * **Dependencies:** `pybullet`, `gymnasium`, `stable-baselines3`, `torch`, `numpy`, `pygame`, `ikpy`, `pyserial`, `matplotlib` --- @@ -118,9 +116,9 @@ pip install -r requirements.txt --- -## Usage Guide +### Usage Guide -### 1. Manual Control & Hardware Streaming (`main.py`) +## Manual Control & Hardware Streaming (`main.py`) `main.py` is the operational entry point for driving the robot manually via GUI sliders or a gamepad, running either in 3D PyBullet simulation or connected to physical hardware. @@ -139,19 +137,11 @@ Edit `config.py` prior to launching `main.py` to configure execution mode and co --- -### 2. How the Machine Learning System Works +### Machine Learning Pipeline (`ml/`) -The machine learning subsystem teaches the robot to move by interacting with a simulation environment instead of relying on a manually written gait controller. +The machine learning system trains the hexapod to move using a combination of Behavioral Cloning (BC) (optional) and Proximal Policy Optimization (PPO) driven by a multi-phase curriculum. -At a high level: - -* The policy receives a vector of observations from the simulator. -* The policy outputs a set of continuous joint deltas. -* Those outputs are applied to the robot in the physics simulation. -* The environment computes a reward based on survival, movement, stability, and command-following quality. -* PPO updates the policy over many training iterations to improve the controller. - -#### What the policy sees and does +### ML Workflow Steps The policy input contains: