Cloud datacenters generate massive amounts of Dark Data—unmanaged, orphaned, and idle storage volumes that consume electricity, emit greenhouse gases, and increase operating costs.
This platform provides an end-to-end cloud-native solution that:
- Detects Dark Storage Volumes using a dual-model Machine Learning pipeline trained on real-world cloud disk traces (16,296 disks).
- Optimizes Decommissioning & Data Migration using metaheuristic algorithms (PSO, ACO, ARAS, and Carbon-Aware Scheduling).
- Quantifies Environmental & Financial Impact (kWh energy saved, kg CO2 reduced, USD cost reclaimed).
- Monitors Cloud Health in Real Time via Kubernetes microservices, Prometheus custom metrics, and a 12-panel Grafana dashboard.
flowchart TD
A[Cloud Storage Traces: 16,296 Disks] --> B[Feature Extraction & Preprocessing]
B --> C{Dual-Model ML Classifier}
C -->|13 Features| D[XGBoost Classifier: 97.85% Acc]
C -->|19 Features| E[Random Forest Classifier: 100% Acc]
D & E --> F[Dark Volume Detection Stream: 1,147 Volumes]
F --> G[Optimization Engine]
G --> G1[PSO: Energy Minimization]
G --> G2[ACO: CPU & IOPS Efficiency]
G --> G3[ARAS: Multi-Criteria Analysis]
G --> G4[Carbon-Aware: WattTime Grid API]
G1 & G2 & G3 & G4 --> H[Calculators: Storage, Energy, Cost, CO2]
H --> I[Kubernetes Microservices: FastAPI Pods]
I --> J[Prometheus Metrics Exporter]
J --> K[Grafana Multi-Algorithm Dashboard]
FOLDER STRUCTURE
dark-data-phase2/
│
├── 📁 phase1_artifacts/ # ← All your files go here
│ │
│ ├── 📁 models/ # Trained models
│ │ ├── model_mlbased.joblib # XGBoost (97.85% accuracy)
│ │ └── model_rulebased.joblib # Random Forest (100% accuracy)
│ │
│ ├── 📁 metadata/ # Model metadata
│ │ ├── feature_columns_mlbased.json # 13 features
│ │ ├── feature_columns_rulebased.json # 19 features
│ │ ├── feature_importance_mlbased.csv
│ │ ├── feature_importance_rulebased.csv
│ │ ├── model_comparison_dual.csv
│ │ └── model_metadata_dual.json
│ │
│ ├── 📁 data/ # Dataset
│ │ ├── training_features.csv # FULL 16,296 disks ← MAIN INPUT
│ │ └── detected_dark_volumes.csv # 1,147 pre-detected (optional)
│ │
│ └── README.md # Phase 1 summary
│
├── 📁 services/ # Phase 2 microservices
│ │
│ ├── 📁 inference/ # Batch prediction service
│ │ ├── Dockerfile
│ │ ├── requirements.txt
│ │ ├── batch_predictor.py # Run both models on 16K dataset
│ │ ├── dual_model_inference.py # Compare ML vs Rule-based
│ │ └── app.py # FastAPI endpoint
│ │
│ ├── 📁 optimization/ # Optimization algorithms
│ │ │
│ │ ├── 📁 pso/ # PSO - Energy optimization
│ │ │ ├── pso_ml.py # PSO on ML-based predictions
│ │ │ ├── pso_rule.py # PSO on Rule-based predictions
│ │ │ └── Dockerfile
│ │ │
│ │ ├── 📁 aco/ # ACO - CPU optimization
│ │ │ ├── aco_ml.py
│ │ │ ├── aco_rule.py
│ │ │ └── Dockerfile
│ │ │
│ │ ├── 📁 aras/ # ARAS - Multi-criteria
│ │ │ ├── aras_ml.py
│ │ │ ├── aras_rule.py
│ │ │ └── Dockerfile
│ │ │
│ │ ├── 📁 carbon/ # Carbon-Aware
│ │ │ ├── carbon_ml.py
│ │ │ ├── carbon_rule.py
│ │ │ └── Dockerfile
│ │ │
│ │ └── comparator.py # Compare all algorithms
│ │
│ ├── 📁 calculators/ # Metrics calculators
│ │ ├── energy_calculator.py # Calculate kWh
│ │ ├── carbon_calculator.py # Calculate kg CO2
│ │ ├── cost_calculator.py # Calculate USD
│ │ ├── resource_calculator.py # Calculate CPU/Memory util
│ │ ├── storage_calculator.py # Calculate storage saved
│ │ ├── metrics_aggregator.py # Aggregate all metrics
│ │ └── Dockerfile
│ │
│ ├── 📁 monitoring/ # Prometheus + Grafana
│ │ ├── prometheus.yml
│ │ ├── alerting_rules.yml
│ │ ├── dashboard.json # Grafana dashboard
│ │ └── datasource.yml
│ │
│ └── 📁 dashboard/ # Web dashboard (optional)
│ ├── app.py # Flask/FastAPI
│ ├── templates/
│ │ └── index.html
│ └── Dockerfile
│
├── 📁 k8s/ # Kubernetes manifests
│ ├── 00-namespace.yaml
│ ├── 01-configmap.yaml # Mount Phase 1 artifacts
│ ├── 02-persistent-volumes.yaml
│ ├── 10-inference-deployment.yaml
│ ├── 11-pso-ml-deployment.yaml
│ ├── 12-pso-rule-deployment.yaml
│ ├── 13-aco-ml-deployment.yaml
│ ├── 14-aco-rule-deployment.yaml
│ ├── 15-aras-ml-deployment.yaml
│ ├── 16-aras-rule-deployment.yaml
│ ├── 17-carbon-ml-deployment.yaml
│ ├── 18-carbon-rule-deployment.yaml
│ ├── 20-calculators-deployment.yaml
│ ├── 30-prometheus-deployment.yaml
│ ├── 31-grafana-deployment.yaml
│ └── 40-cronjobs.yaml # Periodic optimization runs
│
├── 📁 output/ # Generated results
│ ├── predictions_ml.csv # ML-based predictions (16K)
│ ├── predictions_rule.csv # Rule-based predictions (16K)
│ ├── dark_volumes_ml.csv # ML dark volumes (~1,147)
│ ├── dark_volumes_rule.csv # Rule dark volumes (~1,147)
│ │
│ ├── pso_ml_results.json # PSO on ML predictions
│ ├── pso_rule_results.json # PSO on Rule predictions
│ ├── aco_ml_results.json
│ ├── aco_rule_results.json
│ ├── aras_ml_results.json
│ ├── aras_rule_results.json
│ ├── carbon_ml_results.json
│ ├── carbon_rule_results.json
│ │
│ ├── comparison_report.json # Final comparison
│ └── metrics_summary.csv # All metrics in one place
│
├── 📁 scripts/ # Deployment scripts
│ ├── setup_phase2.sh # One-click setup
│ ├── deploy_docker.sh # Deploy with Docker Compose
│ ├── deploy_k8s.sh # Deploy to Kubernetes
│ └── test_all.py # Test all services
│
├── docker-compose.yml # Local Docker Compose setup
├── requirements.txt # Python dependencies
└── README.md # Phase 2 documentation