Skip to content

DavidVeksler/Server-Alerting-System

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

3 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Server Alerting System

License: MIT Python 3.6+

A lightweight Python server monitoring and alerting tool for WordOps-managed Ubuntu servers. It watches disk space, CPU load, memory/swap, PHP-FPM, MariaDB, SSL certificate expiration, Redis, and network connections, then sends real-time email notifications (via SMTP or the Mailgun API) whenever a threshold is crossed — with WordPress-aware root-cause analysis on high-load alerts. Built for sysadmins and DevOps engineers who want dependable uptime and performance alerting without standing up a full observability stack.

🚀 Features

  • Zero Dependencies: Uses only Python standard library + psutil
  • Dual Email Support: SMTP and Mailgun API integration
  • Anti-Spam Protection: Configurable cooldown periods prevent alert flooding
  • Comprehensive Monitoring: 9 critical system components monitored
  • Enhanced Load Analysis: Identifies specific WordPress sites causing high CPU usage
  • Stateful Alerting: Tracks alert history to prevent duplicate notifications
  • JSON Configuration: Easy-to-modify configuration file
  • Automated Execution: Cron-based scheduling with detailed logging

📊 Monitoring Capabilities

  • Disk Space: Monitor root filesystem usage with cleanup recommendations
  • CPU Load: Enhanced analysis with WordPress site identification and PHP-FPM worker details
  • Memory & Swap: Available memory and swap usage monitoring
  • PHP-FPM: Queue monitoring and worker analysis
  • MariaDB: Connection count and query monitoring
  • SSL Certificates: Expiration monitoring for domains
  • Redis: Memory usage monitoring
  • Network: Connection analysis for traffic spike detection

🔧 Enhanced Load Monitoring

The system provides detailed analysis when high server load is detected:

  • Process Analysis: Top CPU-consuming processes with runtime information
  • WordPress Site Identification: Pinpoints which specific WordPress sites are causing load
  • PHP-FPM Worker Analysis: Shows active workers and CPU usage per site
  • Database Impact: MariaDB CPU usage and active query count
  • I/O Analysis: Disk performance metrics and bottleneck detection
  • Network Traffic: Web connection monitoring for traffic spikes
  • Targeted Recommendations: Specific actions based on root cause analysis

📁 Project Structure

server-alerting-system/
├── README.md                    # This file
├── bin/                         # Executable scripts
│   ├── server_alerts.py         # Main monitoring script
│   └── alert_test.py           # Email testing utility
├── config/                      # Configuration examples
│   ├── server_alerts.json.example  # Default configuration
│   ├── mailgun.json.example     # Mailgun configuration example
│   └── smtp.json.example        # SMTP configuration example
├── docs/                        # Documentation
│   └── ALERTING_SYSTEM.md      # Comprehensive documentation
├── examples/                    # Usage examples
└── scripts/                     # Installation scripts
    ├── install.sh              # Installation script
    └── uninstall.sh            # Uninstallation script

🚀 Quick Start

1. Clone the Repository

git clone https://github.com/DavidVeksler/Server-Alerting-System.git
cd Server-Alerting-System

2. Install

sudo chmod +x scripts/install.sh
sudo ./scripts/install.sh

3. Configure Email Settings

Edit the configuration file:

sudo nano /etc/server_alerts.json

Option A: Mailgun API (Recommended)

{
  "email": {
    "method": "mailgun",
    "from_email": "alerts@yourdomain.com",
    "to_email": "admin@yourdomain.com",
    "mailgun_api_key": "your-api-key-here",
    "mailgun_domain": "yourdomain.com",
    "mailgun_region": "us"
  }
}

Option B: SMTP (Gmail Example)

{
  "email": {
    "method": "smtp",
    "smtp_server": "smtp.gmail.com",
    "smtp_port": 587,
    "use_tls": true,
    "username": "your-email@gmail.com",
    "password": "your-app-password",
    "from_email": "alerts@yourdomain.com",
    "to_email": "admin@yourdomain.com"
  }
}

4. Test Email Delivery

sudo python3 /usr/local/bin/alert_test.py

5. Monitor Logs

sudo tail -f /var/log/server_alerts.log

⚙️ Configuration

Threshold Tuning

The system monitors several metrics with configurable thresholds:

{
  "thresholds": {
    "disk_pct": 85,              # Disk usage percentage
    "load_1m": 2.5,              # 1-minute load average
    "mem_free_mb": 512,          # Available memory in MB
    "swap_pct": 50,              # Swap usage percentage
    "php_queue": 10,             # PHP-FPM queue length
    "db_threads": 30,            # MariaDB active connections
    "ssl_days": 14,              # SSL certificate expiration days
    "redis_memory_mb": 150       # Redis memory usage in MB
  }
}

Enabling/Disabling Checks

{
  "monitoring": {
    "enabled_checks": [
      "disk", "load", "memory", "swap", "php-fpm",
      "mariadb", "ssl", "redis"
    ],
    "primary_domain": "yourdomain.com",
    "alert_cooldown_minutes": 60
  }
}

📧 Sample Alert Output

When high server load is detected, you'll receive detailed information:

Subject: [SERVER ALERT] High system load: 3.2 (CRITICAL)

System load has exceeded the threshold of 2.5 (immediate attention required).

LOAD ANALYSIS:
Current load averages:
  1 minute:  3.20
  5 minutes: 2.80
  15 minutes: 2.10

System context:
  CPU cores: 2
  Load per core: 1.60 (>1.0 indicates overload)
  CPU usage: 85.5%
  I/O wait: 12.3%

TOP CPU-CONSUMING PROCESSES:
  1. php-fpm (PID 1234): 45.2% CPU, 8.1% RAM (running 2.3h)
  2. php-fpm (PID 1235): 38.7% CPU, 7.8% RAM (running 1.8h)
  3. mysqld (PID 1236): 25.1% CPU, 15.2% RAM (running 3.2h)

PHP-FPM ANALYSIS:
  Active workers using CPU: 8
  Total PHP-FPM processes: 15
  Combined PHP CPU usage: 156.3%
  Average CPU per active worker: 19.5%
  WordPress sites with activity:
    example.com: 5 workers, 89.2% CPU
    blog.example.com: 3 workers, 67.1% CPU

DATABASE ANALYSIS:
  MariaDB CPU usage: 25.1%
  Active database queries: 3

RECOMMENDED ACTIONS:
• PHP-FPM using 156.3% CPU - investigate WordPress performance
• Focus on example.com site - highest CPU usage (89.2%)
• Check for plugin issues, slow queries, or excessive requests
• Monitor in real-time: htop or top
• Check detailed logs: /var/log/cpu-monitoring.log

🛠️ Management Commands

Manual Test Run

sudo python3 /usr/local/bin/server_alerts.py

View Alert History

sudo grep "ALERT SENT" /var/log/server_alerts.log

Clear Alert State (Force Fresh Alerts)

sudo rm -f /var/tmp/server_alerts_state.json

View Current System Status

# Check load
uptime

# Check disk space
df -h

# Check memory
free -h

# Check PHP-FPM workers
ps aux | grep php-fpm | wc -l

# Check MariaDB connections
sudo mysql -e "SHOW STATUS LIKE 'Threads_connected';"

🔍 Troubleshooting

Email Delivery Issues

  1. SMTP Problems:

    # Test SMTP connectivity
    telnet smtp.gmail.com 587
    
    # Check postfix (if using local SMTP)
    sudo systemctl status postfix
  2. Mailgun Problems:

    # Test API connectivity
    curl -s --user 'api:YOUR_API_KEY' \
        https://api.mailgun.net/v3/YOUR_DOMAIN/messages \
        -F from='alerts@YOUR_DOMAIN' \
        -F to='admin@youremail.com' \
        -F subject='API Test' \
        -F text='Testing Mailgun API'

Configuration Issues

# Validate JSON syntax
sudo python3 -c "
import json
with open('/etc/server_alerts.json', 'r') as f:
    json.load(f)
print('Configuration is valid')
"

Permission Issues

# Fix file permissions
sudo chown root:root /usr/local/bin/server_alerts.py
sudo chmod 755 /usr/local/bin/server_alerts.py
sudo chown root:root /etc/server_alerts.json
sudo chmod 644 /etc/server_alerts.json

📋 Requirements

  • OS: Ubuntu 18.04+ (tested on Ubuntu 24.04.2 LTS)
  • Python: 3.6+
  • Packages: python3-psutil
  • Services: WordOps, nginx, PHP-FPM, MariaDB (optional: Redis)
  • Access: Root privileges for installation

🤝 Contributing

  1. Fork the repository
  2. Create a feature branch (git checkout -b feature/amazing-feature)
  3. Commit your changes (git commit -m 'Add amazing feature')
  4. Push to the branch (git push origin feature/amazing-feature)
  5. Open a Pull Request

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🆘 Support

  • Documentation: See docs/ALERTING_SYSTEM.md for comprehensive documentation
  • Issues: Report bugs and request features via GitHub Issues
  • Email: For critical issues or security concerns

🏆 Acknowledgments

  • Built for WordOps server management
  • Designed for Ubuntu server environments
  • Optimized for WordPress hosting scenarios

Version: 1.0
Compatibility: Ubuntu 24.04.2 LTS, Python 3.12+, WordOps 3.18+
Last Updated: 2025-07-27

About

Lightweight Python email alerting and monitoring tool for WordOps/Ubuntu servers — tracks disk, CPU load, memory, PHP-FPM, MariaDB, SSL expiry, and Redis, with SMTP/Mailgun notifications

Topics

Resources

License

Stars

0 stars

Watchers

0 watching

Forks

Releases

No releases published

Packages

 
 
 

Contributors