Shadow-Boxer is a research project focused on building a smart shadow-boxing learning module using a single camera. The module has two main components:
-
Smart Shadow-Boxing RL-trained Assistant
- Reproduces realistic shadow-boxing strategies given a human pose sequence.
- Serves as a real-time instructor for a human learning to shadow box in front of a camera.
-
RL-trained Manipulator Robot for Boxing
- Learns boxing movements and strategies to “hit” a learner with realistic strikes.
- Maintains safety by avoiding collisions with the user.
We trained, in an unsupervised manner, a Gaussian Mixture Model (GMM) using raw internet boxing videos to classify basic boxing actions from sequences of 2D poses. The GMM serves the following purposes:
- Classify boxing actions: By analyzing sequences of poses within a sliding window, the model works in real time (~2ms) to provide action labels.
- Provide a reward function: In a reinforcement learning (RL) context, the model’s likelihood is used to reward sequences that resemble real shadow boxing.
This repository will soon contain a dataset of 2D poses estimated from internet videos of experienced boxers performing shadow boxing. The keypoints were estimated using the RTMO pose estimator implemented through fast and easy-to-use rtmlib. Each sample consists of a series of frames, where each frame has 2D keypoint coordinates representing the pose of a boxer.
Once the GMM is trained, we use it inside an RL loop to train a human model to shadow box properly. This training is conducted in the Genesis simulator, where:
- The policy (controller) is rewarded based on the likelihood given by the GMM.
- Over training, the policy learns to produce sequences of poses that maximize the “boxer-like” score.
- The resulting policy can be deployed in real-time as a shadow-boxing instructor or used to train manipulator robot policies to box.
This project is licensed under the MIT License.