PIE Energy Collection - Production System

Status: ✅ Ready for Production
System: AMD EPYC 9554P 64-Core Processor
Dataset: PIE (3,546,816 unique executions)

---

Quick Start

1. Configure System Paths

nano config.env

# Update for your system:
PROJECT_ROOT="/your/path/to/green-code-gen"
SNIPER_ROOT="${PROJECT_ROOT}/sniper/sniper"

2. Test Locally

source config.env
head -10 ${PIE_DATASET}/execution_master.jsonl > test.jsonl

python3 energy_data_collection/sniper_execution_runner.py \
    --batch-file test.jsonl \
    --cache-dir /tmp/test \
    --sniper-config ${SNIPER_CONFIG}

Expected: ✅ Success in 7.7s: 0.033674J, 247,800 cycles

3. Submit Production

cd energy_data_collection
sbatch slurm_execution_master.sh

---

System Overview

Pipeline:

PIE (77K samples) → Execution Master (3.5M) → 440 Batches → SLURM → Results Cache

Key Features:

• ✅ High-precision energy (6 decimals: 0.033674 J)
• ✅ Auto-resumability (instant skip of completed)
• ✅ Data safety (trap logic on timeout)
• ✅ Accurate CPU model (64-core EPYC 9554P)

---

Configuration

Sniper Config: epyc_9554p.cfg

• 64 cores, 1.5 GHz, L1:32KB/L2:1MB/L3:256MB
• Tested vs epyc_9554p_realistic.cfg → identical for single-threaded

Energy Precision: Modified sniper/sniper/tools/mcpat.py:266

print('... %6.6f %sJ ...')  # Was %6.2f

---

Monitoring

# Job status
squeue -u $USER

# Progress
find pie_energy_cache/completed -name "*.done" | wc -l

# Check batch
python3 energy_data_collection/check_incomplete_jobs.py --batch-id 0

Result format:

{
  "execution_id": "p00000_23684839615f_68b329da",
  "result": {
    "energy_joules": 0.033674,
    "power_watts": 204.56,
    "cycles": 247800,
    "instructions": 260109
  }
}

---

Data Safety

Trap Logic Timeline:

• T-5min: SIGUSR1 warning
• T-2min: SIGTERM → graceful shutdown (60s grace period)
• T-0: Job ends, all completed data preserved in cache

Resumability: Rerun same batch → skips completed instantly (~74K exec/s)

---

Resources

Per Batch: ~8,061 executions, 75s budget/exec (3× margin), 7-day max
Total: 440 parallel batches, 3.5M executions, ~7 days wall time

---

File Structure

green-code-gen/
├── README.md                   # ← You are here
├── config.env                  # ← Configure first
├── energy_data_collection/
│   ├── README.md              # Script details
│   ├── slurm_execution_master.sh  # ← Submit this
│   └── sniper_execution_runner.py
├── PIE_Dataset/
│   ├── execution_master.jsonl # 3.5M executions
│   └── batches/               # 440 batch files
└── sniper/sniper/config/epyc_9554p.cfg

---

Troubleshooting

Config not found: Verify config.env paths, run source config.env
Timeouts: Increase SNIPER_TIMEOUT in config.env
Compilation errors: Check logs/*.err, verify module load StdEnv/2023

---

Next Steps

After completion:

1. Collect: tar -czf results.tar.gz pie_energy_cache/completed/
2. Stats: Count completions, energy distribution
3. Map back to train.jsonl samples

---

Documentation: See energy_data_collection/README.md for script details
Historical docs: See depr/ directories

Version: 1.0 (2025-10-20) - Production Release ✅