R2BC: Multi-Agent Imitation Learning from Single-Agent Demonstrations

Connor Mattson, Varun Raveendra, Ellen Novoseller, Nicholas Waytowich, Veron Lawhern, Daniel S. Brown

Setup

Requires Python 3.10.12

To install packages, run the following command in a virtual env

pip install -r requirements.txt

To add the required external submodules

git submodule add git@github.com:proroklab/rllib_differentiable_comms.git src/external_libs/rllib_differentiable_comms
git submodule add git@github.com:proroklab/HetGPPO.git src/external_libs/HetGPPO

Download Synthetic Demonstrator Policies

Use this anonymized Google Drive link to access the pretrained RL demonstrators that are used in the simulation experiments.

For each environment, modify the "expert_pth" attribute in the config file config/environments/[ENVIRONMENT_NAME].yaml line 9 to point to your downloaded trained RL model.

TL;DR

To test R2BC on the VMAS Navigation environment use the following command

python -m src.run_experiment navigation r2bc_decent --render

Primary Experiments (Figures 2-3)

Run the following.

bash src/experiments/shell_scripts/navigation.sh
bash src/experiments/shell_scripts/balance.sh
bash src/experiments/shell_scripts/buzz_wire.sh
bash src/experiments/shell_scripts/transport.sh

The resulting data can be used in the following scripts to generate the paper's plots

• analysis/mabc/plot_results.py
• analysis/mabc/plot_covariate_shift.py

Ablation Experiments (Figure 4)

bash src/experiments/shell_scripts/ablation.sh

The resulting data can be used in the following scripts to generate the paper's plots

• analysis/mabc/plot_ablation_bars.py

Human JBC Demonstrations

Humans can give offline demonstrations using the src.demonstrations.human package. Used as such

python -m src.demonstrations.human.collect_human_demos --scenario_name [SCENARIO_NAME] --n [N_AGENTS]

For a complete list of parameters, see

python -m src.demonstrations.human.collect_human_demos --help

Then, you'll need to modify the corresponding data paths in config/base.yaml under datasets.