RL-VSL agent embedded in RSUs — reproducibility package

This repository accompanies the paper "Reinforcement Learning Agent Embedded in Roadside Units to Implement an Adaptive Variable Speed Limit System in a Cooperative Intelligent Transportation Systems Environment". It contains the three-stage training pipeline (surrogate/DL pretraining → SUMO training → few-shot refinement), the evaluation code, the physics-based benchmark, and the post-processing notebooks used to produce the paper's figures.

Functional architecture

flowchart TD
    A["SUMO scenario<br/>CV_p.net.xml, CV.rou.xml, CV_p.sumocfg<br/>3-lane freeway, downstream shockwave"]

    subgraph S1["Stage 1 — baseline agent (single RSU, 6 km coverage)"]
        B1["training_main.py + training_simulation.py<br/>DQL training loop"]
        B2["testing_main.py + testing_simulation.py<br/>evaluation across seeds / MPR"]
        B3["Calibration runs<br/>model_5, model_6_gamma0.8, model_7_gamma0.7"]
    end

    subgraph S2["Stage 2 — state-reconstruction surrogate (DL)"]
        C1["dtabase_results copie.ipynb<br/>builds data_model.csv from data_base.csv"]
        C2["Test_ele.ipynb / RL_DL_Ele<br/>DL model: sparse state to complete state<br/>(6 km full coverage)"]
        C2b["Test_ele.ipynb (cov_2km)<br/>DL model for 2 km reduced-coverage RSU"]
        C3["training_settings/11xx.ini<br/>14-run hyperparameter grid (batch, epochs, lr, gamma)"]
    end

    subgraph S3["Stage 3 — few-shot fine-tuning"]
        D1["training_main.py<br/>refinement in full SUMO from pretrained agent"]
        D2["testing_main.py + testing_simulation.py<br/>evaluation of the fine-tuned agent"]
    end

    subgraph BENCH["Physics-based benchmark"]
        E1["LD.py / Loop_500.py<br/>SPECIALIST, loop-detector based VSL"]
    end

    subgraph POST["Post-processing"]
        F1["boxplot_res.ipynb<br/>fich_txt.ipynb<br/>KPI aggregation and figures"]
    end

    A --> B1
    A --> E1
    B1 --> B2
    B1 --> B3
    B1 --> C1
    C1 --> C2
    C1 -.-> C2b
    C2 --> C3
    C3 --> D1
    D1 --> D2
    B2 --> F1
    D2 --> F1
    E1 --> F1

Loading

Stage-by-stage notes

• Stage 1 — 01_stage1_baseline_agent/ (from RL_sumo/): trains the DQL agent from scratch directly in SUMO on a single RSU with 6 km coverage. Three calibration runs (model_5, model_6_gamma0.8, model_7_gamma0.7) compare discount factors, consistent with the γ = 0.7 vs γ = 0.8 comparison described in the paper. testing_main.py / testing_simulation.py evaluate a trained agent over multiple random seeds and CV market penetration rates.
• Stage 2 — 02_stage2_state_reconstruction_DL/: trains a deep-learning model that reconstructs a complete traffic state from partial observations.
  • full_coverage_6km/ (from RL_DL_sumo/): the reference DL model. The 14 files in RL_DL_Ele/training_settings/ (1111.ini … 1144.ini) encode a hyperparameter grid, mirroring the calibration table in the paper.
  • reduced_coverage_2km/ (from the earlier chap6_appren_etat.zip delivery): the same DL architecture trained to reconstruct state from a 2 km RSU window — this is the model referenced in Section 5.2 of the paper for reduced-coverage RSUs.
• Stage 3 — 03_stage3_finetuning_multiRSU/ (from RL_sumo_apresRLDLsumo/): same code structure as Stage 1, used to fine-tune the agent in full SUMO after pretraining — the reward_transfertlearning.png output in models/model_6/ confirms this is the transfer-learning refinement stage (paper Section 3.5, Step 3b).
• Benchmark (LD.py / Loop_500.py, inside each stage's sumo_network/): physics-based VSL control using virtual loop detectors (SPECIALIST-style), used as the "Full CV-based" comparison point against the RL agent.
• Post-processing (boxplot_res.ipynb, fich_txt.ipynb, inside each stage's sumo_network/): aggregate per-seed, per-MPR results into the KPI comparisons (TTS, Cv-TT) and figures.
• 04_misc/: TEST.py, a standalone scratch script not part of the main pipeline.

Directory layout

01_stage1_baseline_agent/
├── training_main.py, training_simulation.py, model.py, memory.py, utils.py, visualization.py
├── testing_main.py, testing_simulation.py, testing_settings.ini
├── training_settings.ini, model.json, episode_list.npy
├── fichiers_plus/               action-space construction
├── models/                      model_5, model_6_gamma0.8, model_7_gamma0.7 (weights, figures)
└── sumo_network/
    ├── CV.rou.xml, CV_p.net.xml, CV_p.sumocfg
    ├── LD.py, Loop_500.py, Loopdete.py       physics benchmark
    ├── MPR_0.py … MPR_100.py, config_listCV.py
    ├── run_*.sh                              CV/LD × gap/instan/stati variants, allseed/seed/MPR
    ├── boxplot_res.ipynb, fich_txt.ipynb      post-processing
    ├── xml/                                   SUMO xml<->csv tools
    └── stats/                                 example run output

02_stage2_state_reconstruction_DL/
├── full_coverage_6km/
│   ├── Test_ele.ipynb, dtabase_results copie.ipynb, data_frame.csv, data_model.csv
│   ├── apprentissage/models/best_model_2D.h5
│   └── RL_DL_Ele/
│       ├── training_main.py, training_simulation.py, model.py, memory.py, utils.py
│       ├── training_settings.ini, training_settings/1111.ini … 1144.ini
│       ├── DL_model/ (data_base.csv, xmean/xstd/ymean/ystd.csv, best_model_2D.h5)
│       └── models/model_1_gamma0.8, model_2
└── reduced_coverage_2km/
    ├── Test_ele.ipynb, data_base.csv
    ├── DL_model/
    └── models/best_model_2D.h5

03_stage3_finetuning_multiRSU/
└── (same layout as 01_stage1_baseline_agent/, applied to the fine-tuning stage)

04_misc/
└── TEST.py

Execution order

1. 01_stage1_baseline_agent/sumo_network/run_allseed.sh (or the individual run_seed.sh / run_MPR.sh / run_training_main.sh) → trains the baseline DQL agent and produces models/, stats/.
2. 01_stage1_baseline_agent/testing_main.py → evaluates the trained agent over 30 seeds × several MPR values.
3. 02_stage2_state_reconstruction_DL/full_coverage_6km/dtabase_results copie.ipynb then Test_ele.ipynb (both in full_coverage_6km/ and reduced_coverage_2km/) → train the DL state-reconstruction models.
4. RL_DL_Ele/training_main.py with each training_settings/11xx.ini → surrogate hyperparameter calibration (paper Fig. 4).
5. 03_stage3_finetuning_multiRSU/ → few-shot refinement of the pretrained agent in full SUMO, then evaluation via its own testing_main.py.
6. sumo_network/LD.py (either stage) → physics-based SPECIALIST benchmark.
7. sumo_network/boxplot_res.ipynb and fich_txt.ipynb → final KPI figures.