diff --git a/docs/privilegesecurediscovery/2.22/administration/onpremmaintenance/nps-d-health-check_1.md b/docs/privilegesecurediscovery/2.22/administration/onpremmaintenance/nps-d-health-check_1.md new file mode 100644 index 0000000000..3b205ae5c0 --- /dev/null +++ b/docs/privilegesecurediscovery/2.22/administration/onpremmaintenance/nps-d-health-check_1.md @@ -0,0 +1,225 @@ +--- +title: NPS-D Health Checks +--- + +# Perform a Health Check on NPS-D + +This article describes the standard health check procedure for Netwrix Privilege Secure Discovery +(NPS-D), formerly known as SecureONE. It covers application, database, disaster recovery (DR), +Docker, and operating system (OS) checks that confirm a healthy environment, and it highlights items +to follow up on with Support. + +This article groups the checks in the same order the intro lists them. Work through each group in turn, and +run the commands on every node in the cluster unless a step says otherwise. + +## Application Checks + +Confirm that the NPS-D services, nodes, and logging are healthy. + +### Services and Nodes + +On NPS-D 2.22 and higher, confirm all services and nodes are up and reporting correctly: + +```bash +s1 status +s1 nodes +``` + +On NPS-D 2.21.x and earlier, use the underlying Docker commands directly: + +```bash +sudo docker stack services +sudo docker node ls +``` + +Confirm every service shows the expected replica count (for example `1/1` or `6/6`) and every node +shows `Ready`/`Active`. Investigate a replica count mismatch or a `Down`/`Drain` node before +continuing. + +### Service Local Logging Files and Content + +Run on each node in the cluster: + +```bash +sudo ls -alh /secureone/data/logs/ | grep svc + +for i in $(sudo find /secureone/data/logs/ -iname "*.log*"); do + echo "File: $i" + sudo grep -iE "error|fail" "$i" | wc -l + sudo grep -iE "error|fail" "$i" | grep -o "message[^,]*" | sort | uniq -c | sort -nk1 | tail + echo "----------" +done +``` + +Confirm log files are present and actively receiving writes, and review any recurring error or +failure messages. + +:::note +The `find` pattern needs the wildcards (`*.log*`) to match filenames, and the `grep` pattern needs `-E` for `|` to work as an alternation between `error` and `fail`. +::: + +### Fluentd/Log Forwarding to SIEM + +If log forwarding isn't configured, the customer can create a Support ticket to enable it. Reference the SIEM log forwarding documentation: + +[Forward Logs to SIEM](https://docs.netwrix.com/docs/kb/privilegesecurediscovery/security-and-compliance/forward-logs-to-siem-fluentd) + +## Database Checks + +Confirm that MongoDB replication is healthy across the cluster. + +### MongoDB Replica Status + +For clustered environments, confirm a healthy database replication status. + +On NPS-D 2.22.x and higher, use the `s1` CLI helper: + +```bash +s1 database-status +``` + +This runs the same `rs.status()` and `rs.printSlaveReplicationInfo()` queries against MongoDB +internally and prints the parsed result, so you don't need to look up the container manually. + +On NPS-D 2.21.x and earlier: + +```bash +mEvl="sudo docker exec -it $(sudo docker ps | grep mongod | cut -d' ' -f1) mongo SecureONE --quiet --eval" +$mEvl 'rs.status()' | grep "name\|stateStr\|lastHeartbeatRecv\|lastHeartbeatMessage" | column -t +echo +$mEvl 'rs.printSlaveReplicationInfo()' +unset mEvl +``` + +Look for each node reporting a healthy `stateStr` (for example `PRIMARY` or `SECONDARY`) and recent +heartbeat timestamps. Delayed or missing heartbeats indicate replication issues to investigate +before proceeding. + +## Disaster Recovery (DR) Checks + +Confirm that backup and restore functions are configured and running. + +### DR Functions (Backup and Restore) + +If the environment has no DR configuration, skip this section and note it as N/A. + +Confirm the configuration of `backup_v1.sh` on PROD: + +```bash +sudo grep "FAHis\|DRIP\|HowManyBaks\|S1EnvType" /secureone/data/backup_v1.sh | grep -v "#" | head -3 +``` + +Confirm backups are running: + +```bash +ls -alh /secureone/data/db/backups/* +``` + +Check the crontab entry: + +```bash +sudo crontab -l | tail +``` + +Confirm the configuration of `restore_v1.sh` on DR: + +```bash +sudo grep "HowManyBaks\|S1EnvType" /secureone/data/restore_v1.sh | grep -v "#" | head -2 +``` + +Confirm restores are running and cleanup is occurring: + +```bash +ls -alh /secureone/data/db/restore/* +``` + +Review the latest restore log: + +```bash +sudo tail $(find /secureone/data/db/restore/ -name "restore*.log" | sort | tail -n1) +``` + +Check the crontab entry: + +```bash +sudo crontab -l | tail +``` + +## Docker Checks + +Confirm that the Docker engine, images, and networking are healthy on every node. + +### Docker Version + +```bash +docker --version +``` + +### Docker Daemon Status + +Run on each node in the cluster: + +```bash +sudo service docker status +``` + +### Local Docker/Service Images + +Run on each node in the cluster: + +```bash +sudo docker image ls +``` + +Confirm the expected images are present and versions are consistent across every node in the +cluster. A mismatched tag on one node (for example after a partial upgrade) is a common source of +crash loops. + +### APIPA (Link-Local) Networks + +Run on one node in the cluster: + +```bash +sudo docker network inspect docker_gwbridge bridge ingress s1_default | grep '"Name": "docker_gwbridge"\|"Name": "bridge"\|"Name": "ingress"\|"Name": "s1_default"\|Subnet\|Gateway' +``` + +Confirm none of the reported subnets fall in the 169.254.0.0/16 (Automatic Private IP Addressing, +or APIPA/link-local) range. Overlay networks that have fallen back to link-local addressing are a +sign of Docker Swarm networking issues. + +## OS Checks + +Confirm that disk space, time sync, and the operating system are healthy on every node. + +### Disk Space and Time Sync + +Run on each node in the cluster: + +```bash +date && echo "------" && df -h && echo "------" && timedatectl status +``` + +Confirm sufficient free disk space and that time sync is active and healthy on every node. If a +node shows an unsynced or unreachable NTP source, resolve that before continuing, since time drift +can cause replica set and TLS issues. + +### Ubuntu Version + +Run on each node in the cluster: + +```bash +lsb_release -a +``` + +Confirm the OS version on each node is a supported version for the installed NPS-D release. + +### Upgradable Packages + +Run on each node in the cluster: + +```bash +sudo apt update && apt list --upgradable +``` + +Confirm the number and nature of pending OS package updates. Flag any packages that are +significantly out of date or that address known security issues.