diff --git a/docs/Collecting Metrics/Ceph.mdx b/docs/Collecting Metrics/Ceph.mdx index 757b11333..25aa891d4 100644 --- a/docs/Collecting Metrics/Ceph.mdx +++ b/docs/Collecting Metrics/Ceph.mdx @@ -14,7 +14,11 @@ slug: "/collecting-metrics/ceph" # Monitor Ceph -Netdata gives you a complete operational view of Ceph by collecting four complementary telemetry surfaces: the MGR Prometheus module, official `ceph-exporter`, the NVMe-oF gateway exporter, and the Ceph Dashboard API. Deploy one Agent close to each Ceph node to monitor cluster state, daemon health, host resources, RGW traffic, and local hardware together. +Netdata gives you a complete operational view of Ceph by collecting four complementary telemetry surfaces—the MGR +Prometheus module, official `ceph-exporter`, the NVMe-oF gateway exporter, and the Ceph Dashboard API—and by running +authenticated S3 lifecycle checks from the client vantages that depend on object storage. +Deploy one Agent close to each Ceph node to monitor cluster state, daemon health, host resources, RGW traffic, +client-visible S3 correctness, and local hardware. ## What you can monitor @@ -23,6 +27,8 @@ Netdata gives you a complete operational view of Ceph by collecting four complem - CephFS/MDS, RBD, RBD Mirror, SMB, and client I/O telemetry exposed by your Ceph release. - Host-local daemon performance from official `ceph-exporter`. - RGW requests, Lua execution, notifications, queues, retries, and access logs. +- Authenticated S3 write, read, list, delete, payload-integrity, cleanup, and latency results from selected vantages. +- Directional multisite S3 replication, payload integrity, recovery-point objective, and delete-propagation results. - RGW endpoint availability and TLS certificate health. - NVMe-oF gateway, block-device, host, subsystem, and namespace telemetry from supported exporters. - Node hardware health, cooling, power, memory, processors, storage, and temperature reporting on Tentacle. @@ -63,6 +69,8 @@ Use this map to identify the Netdata surface that owns the operational question | RBD mirroring | Local/remote snapshot timestamp synchronization | MGR Prometheus | Identify mirrored images that have diverged | | RGW service health | Notifications, Lua execution, queue pressure, retries, aborted requests | MGR Prometheus | Inspect aggregate gateway behavior | | RGW request outcomes | Status classes, bytes, clients, and request duration | `web_log` | Analyze complete RGW access logs | +| Authenticated S3 correctness | PUT, GET integrity, LIST, DELETE, cleanup, and latency | `s3check` | Verify client-visible object operations | +| Multisite S3 replication | Directional payload integrity, visibility lag, RPO, and delete propagation | `s3check` | Verify the replication paths that client applications depend on | | RGW endpoint reachability | Unauthenticated HTTP liveness | `httpcheck` | Verify the selected endpoint is reachable | | RGW certificates | Certificate expiration and revocation | `x509check` | Track certificate lifecycle independently of RGW traffic | | Node health | CPU, memory, disk I/O, filesystems, network interfaces, and processes | Standard Netdata collectors | Continue using normal Agent monitoring for each Ceph node | @@ -100,6 +108,8 @@ Run one Agent on each Ceph node. Each Agent monitors the Ceph services and host | `ceph-exporter` | One job on each node whose daemon telemetry you need | Host-local daemon performance and daemon inventory | | NVMe-oF gateway exporter | One job on each gateway endpoint | Gateway-local runtime, block-device, host, subsystem, and namespace telemetry | | Ceph Dashboard API | One logical job per Ceph cluster | Dashboard API component integrity and Ceph investigation Functions | +| Authenticated S3 check | One job for each selected client vantage | Client-visible S3 object lifecycle and latency | +| Directional multisite S3 check | One explicit source-to-destination job per replication path | Client-visible replication correctness, RPO, and delete propagation | | Host collectors | Every Agent | Node disks, filesystems, network interfaces, processes, and logs | Use one stable job identity for the MGR surface. If the active MGR moves, update DNS or the reverse proxy to the current active endpoint rather than creating one job for every possible MGR. Multiple active MGR jobs for the same cluster create duplicate cluster alert owners. @@ -128,6 +138,37 @@ Enable the exporter in each Ceph NVMe-oF gateway deployment and collect every ga Enable the Ceph Dashboard module, secure it with TLS, and create a read-only Dashboard user. Configure one native Ceph collector job per cluster. The Dashboard collector complements the metric endpoints: it owns API component integrity and provides Ceph investigation Functions. +### Authenticated S3 checks + +Configure an `s3check` job for every client vantage whose object-storage behavior matters. Each job uses a dedicated +unversioned bucket and prefix, reconciles that prefix, performs one authenticated PUT, GET, LIST, DELETE, and +cleanup cycle, verifies the downloaded payload, and removes probe objects after interrupted cycles. Place jobs at each +site or RGW client path that requires a client-visible correctness signal. + +For multisite replication, set `mode: multisite` and configure one explicit source and destination. The source uses the +job's top-level S3 settings; the destination has its own endpoint, region, bucket, prefix, credentials, addressing, and +transport settings. Add bounded `source_site` and `destination.site` labels, then create one job for each direction you +want to verify—for example site-a to site-b and site-b to site-a. Netdata never probes every combination automatically. +The destination prefix identifies where the replicated probe key is expected. If source and destination prefixes differ, +the replication policy must map the source route namespace onto the destination prefix. + +After a multisite job deletes its exact source and destination probe keys, Netdata keeps the sanitized ownership journal until +the larger configured replication or delete deadline elapses. It then lists both owner-scoped namespaces, waits one more +collection interval, and repeats the lists in reverse endpoint order before releasing ownership. This bounded confirmation window +aligns object cleanup with the replication policy you configured. + +Configure endpoint addresses that resolve to distinct S3 services; Netdata rejects literal, default-port, and +virtual-host aliases for the same bucket, but it does not resolve DNS names to guess whether two services share one +gateway. + +A multisite job writes one small source object, verifies the destination object's SHA-256 digest, measures how long +client visibility takes, deletes the source, and optionally waits for the destination copy to disappear. It persists a +sanitized ownership journal across Agent restarts, reconciles both Agent-and-job-owned key namespaces before +new writes, and removes both objects when a visibility or delete deadline is reached. Set `rpo_threshold_ms`, +`replication_timeout_ms`, `delete_threshold_ms`, `delete_timeout_ms`, and `verify_delete` to match the replication +policy. Visibility and delete objectives must be at least one collection interval because Netdata polls each bounded +phase once per cycle; the two objective alerts are silent until you enable and tune them. Probe keys live in an Agent-and-job-owned namespace, so separate Agents, jobs, and reverse directions can coexist without reconciliation deleting one another’s active objects. + ## Supported releases Netdata's built-in Ceph profile recognizes the metric surfaces of: @@ -184,7 +225,9 @@ The built-in Ceph profile recognizes all three Prometheus interfaces. Alert owne - placement group, pool, capacity, and recovery conditions; - node-proxy hardware conditions exposed by MGR; - gateway-local NVMe-oF conditions exposed by each gateway-exporter job; -- RGW notification, Lua, request-fallback, queue-pressure, and multisite retry conditions. +- RGW notification, Lua, request-fallback, queue-pressure, and multisite retry conditions; +- authenticated S3 stage and multisite phase failures, plus configured latency objectives from each `s3check` job; +- directional multisite payload mismatches, RPO breaches, and delete-propagation objectives. The native Dashboard collector owns API component collection failures. Generic Netdata collectors own host-local and endpoint checks: @@ -215,7 +258,10 @@ The MGR profile provides aggregate RGW telemetry for requests, aborted requests, Collect the RGW JSON access log with `web_log` to analyze HTTP outcomes, bytes, clients, and latency. Configure RGW to emit its access log in JSON format and make that file available to the Agent. The Ceph JSON example maps request, status, size, and client fields, and declares `total_time` as a numeric custom field in milliseconds, preserving Ceph's exact duration field. -Use `httpcheck` for unauthenticated endpoint liveness and `x509check` for certificate expiration or revocation. +Use `s3check` for authenticated object lifecycle correctness, client-vantage latency, and directional multisite +replication. The MGR multisite counters show RGW replication work and retries; `s3check` proves what a client can +currently read at the destination and whether the payload is identical. Keep `httpcheck` for unauthenticated endpoint +liveness and `x509check` for certificate expiration or revocation. ## Investigation Functions @@ -272,6 +318,7 @@ For collector configuration details, see: - [Ceph](/docs/collecting-metrics/collectors/storage-and-filesystems/ceph) - [Ceph Prometheus](/docs/collecting-metrics/collectors/storage-and-filesystems/ceph-prometheus) +- [S3 Compatible Object Storage](/docs/collecting-metrics/collectors/synthetic-testing/s3-compatible-object-storage) For Agent deployment, streaming, retention, exporting, and notifications, see: diff --git a/docs/Collecting Metrics/Collectors/Collectors Page 2.mdx b/docs/Collecting Metrics/Collectors/Collectors Page 2.mdx index a8e7bb66e..b127c4b43 100644 --- a/docs/Collecting Metrics/Collectors/Collectors Page 2.mdx +++ b/docs/Collecting Metrics/Collectors/Collectors Page 2.mdx @@ -281,6 +281,9 @@ import { Grid, Box, GridPagination } from '@site/src/components/Grid_integration + + + @@ -539,9 +542,6 @@ import { Grid, Box, GridPagination } from '@site/src/components/Grid_integration - - - diff --git a/docs/Collecting Metrics/Collectors/Collectors Page 3.mdx b/docs/Collecting Metrics/Collectors/Collectors Page 3.mdx index 8e233a46b..449812c18 100644 --- a/docs/Collecting Metrics/Collectors/Collectors Page 3.mdx +++ b/docs/Collecting Metrics/Collectors/Collectors Page 3.mdx @@ -17,6 +17,9 @@ description: "Browse Netdata integrations for Collecting Metrics / Collectors an import { Grid, Box, GridPagination } from '@site/src/components/Grid_integrations'; + + + diff --git a/docs/Collecting Metrics/Collectors/Storage and Filesystems/Physical and Logical Disk Performance Metrics.mdx b/docs/Collecting Metrics/Collectors/Storage and Filesystems/Physical and Logical Disk Performance Metrics.mdx index fd5bbf67e..ff9473fa6 100644 --- a/docs/Collecting Metrics/Collectors/Storage and Filesystems/Physical and Logical Disk Performance Metrics.mdx +++ b/docs/Collecting Metrics/Collectors/Storage and Filesystems/Physical and Logical Disk Performance Metrics.mdx @@ -73,6 +73,7 @@ No action required. | Option | Description | Default | Required | |:-----|:------------|:--------|:---------:| | update every | Data collection frequency. | 1 | no | +| exclude space metrics on paths | Space-separated, case-insensitive Netdata simple patterns for logical-volume mount points to exclude from disk space collection. Patterns support `*` wildcards and `!` negative terms. For example, `*AssuredRecoveryTemp*` excludes temporary backup volumes with that path component. | | no | @@ -101,7 +102,20 @@ sudo ./edit-config netdata.conf ``` ##### Examples -There are no configuration examples. + +###### Exclude temporary backup volumes + +Exclude logical-volume disk space metrics for mount points that contain `AssuredRecoveryTemp`. + +
+Configuration + +```yaml +[plugin:windows:PerflibStorage] + exclude space metrics on paths = *AssuredRecoveryTemp* + +``` +
@@ -142,9 +156,10 @@ Labels: | Label | Description | |:-----------|:----------------| | mount_point | Drive letter or mount point path assigned by Windows (e.g., 'C:', 'D:'). | -| driver_type | Classification of the disk device (e.g., norootdir, removable, cdrom, ramdisk). | +| drive_type | Classification of the disk device (e.g., norootdir, removable, cdrom, ramdisk). | | filesystem | File system format used on the volume (e.g., NTFS, FAT32). | | rw_mode | Current read/write permissions status of the volume (read-only access, read and write access). | +| serial_number | Volume serial number in hexadecimal format. | Metrics: diff --git a/docs/Collecting Metrics/Collectors/Synthetic Testing/S3 Compatible Object Storage.mdx b/docs/Collecting Metrics/Collectors/Synthetic Testing/S3 Compatible Object Storage.mdx new file mode 100644 index 000000000..31f970197 --- /dev/null +++ b/docs/Collecting Metrics/Collectors/Synthetic Testing/S3 Compatible Object Storage.mdx @@ -0,0 +1,572 @@ +--- +custom_edit_url: "https://github.com/netdata/netdata/edit/master/src/go/plugin/go.d/collector/s3check/metadata.yaml" +sidebar_label: "S3 Compatible Object Storage" +learn_status: "Published" +toc_max_heading_level: "6" +toc_collapsible: "true" +learn_rel_path: "Collecting Metrics/Collectors/Synthetic Testing" +keywords: [s3, object storage, ceph, rgw, synthetic] +description: "Verify authenticated S3 operations and optional directional multisite replication." +message: "DO NOT EDIT THIS FILE DIRECTLY, IT IS GENERATED BY THE COLLECTOR'S metadata.yaml FILE" +sidebar_position: "120" +learn_link: "https://learn.netdata.cloud/docs/collecting-metrics/collectors/synthetic-testing/s3-compatible-object-storage" +slug: "/collecting-metrics/collectors/synthetic-testing/s3-compatible-object-storage" +--- + + +# S3 Compatible Object Storage + + +S3 Compatible Object Storage + + +Plugin: go.d.plugin +Module: s3check + +Maintained by Netdata + +## Overview + +Verify authenticated S3 operations and optional directional multisite replication. + +Each single-site job measures stage state, duration, latency objective, attempts, retries, and failures. Each +multisite job adds directional replication lag, RPO state, delete-propagation lag, and bounded request +accounting. Both probes are intentionally active and clean up the objects they create. + + +The collector uses the AWS SigV4 S3 client with explicit static credentials, endpoint, region, and addressing +configuration. It requires a dedicated unversioned bucket and prefix, checks bucket versioning before the job +starts and again before destructive operations, and never deletes versions or foreign-owned keys. Exact object +ownership is persisted before a write or reconciliation delete so restart and configuration changes cannot +abandon cleanup. + +Compatibility is contract-based rather than vendor-specific. The service must support SigV4 static credentials, +GetBucketVersioning, PutObject, GetObject, HeadObject, ListObjectsV2, and DeleteObject. An ordinary object +deletion must also remove the object completely without leaving hidden versions. Ceph RGW multisite is the +reference deployment model for directional replication checks. AWS S3 satisfies the single-site contract when it +is configured with virtual-host addressing and a bucket where versioning has never been enabled. AWS-native +cross-region or same-region replication is not a multisite source because AWS requires versioning on both +buckets. Other S3-compatible services can be used when they satisfy the same API and unversioned-bucket contract. + +Every single-site cycle generates a unique small payload and key, uploads it, reads and verifies its SHA-256 +digest, confirms LIST visibility, deletes it, and verifies that it disappeared. + +In multisite mode, one job represents exactly one source-to-destination direction. Netdata persists sanitized +pending-object state before the source write, polls destination visibility across bounded collection cycles, +verifies the SHA-256 digest, deletes the source, measures destination disappearance, and cleans both sites. +Daemon sync counters are not used as proof of client-visible correctness. + +After an interrupted write path the collector retries deletion; after a process restart it removes at most two +keys owned by this Agent and job per endpoint per cycle before creating a new object. Probe namespaces are derived +from the Agent machine GUID and job identity, so separate Agents, jobs, and reverse directions can probe the +same route without deleting one another's active objects. Raw S3 errors are classified +into bounded reasons. Credentials, signed headers, request IDs, endpoint details, and probe payloads are not +exposed as chart labels or stored in pending state. + + +This collector is supported on all platforms. + +This collector supports collecting metrics from multiple instances of this integration, including remote instances. + +No host capabilities are required. The source identity needs GetBucketVersioning, PutObject, GetObject, ListBucket, +and DeleteObject on its bucket. A multisite destination identity needs GetBucketVersioning, GetObject, +ListBucket, and DeleteObject on its bucket; it is not granted PutObject because the probe never writes to the +destination. + + +### Default Behavior + +#### Auto-Detection + +There is no auto-detection. A job starts only after an endpoint, region, bucket, and credentials are +configured. + + +#### Limits + +A successful single-site probe from a clean namespace makes nine S3 API calls: prefix LIST, pre-write bucket proof, +PUT, GET, LIST, pre-delete bucket proof, DELETE, HEAD, and bucket proof after DELETE. On the next collection, Netdata +performs one owner-prefix LIST to clear the probe quarantine and then starts the next nine-call probe, for ten calls in +that steady-state cycle. A still-present object is retried on a later cycle. Reconciling two stale keys also makes nine +calls. A multisite cycle makes at most eleven calls while it advances one persisted lifecycle. Each call uses the +configured timeout and bounded retry count, and the worst-case per-cycle deadline must fit inside update_every. + +The probe payload is 4 KiB. Prefix reconciliation lists this job's owner namespace before every new write +and deletes at most two exact keys from the persisted ownership journal per endpoint per cycle. Visibility and delete objectives must +be at least one collection interval. Multisite polling spans collection cycles +until the configured visibility or delete deadline; it does not continuously retry inside one cycle. +After multisite cleanup deletes the exact source and destination keys, Netdata waits for the larger configured +replication or delete deadline, lists both owner namespaces, waits one more collection interval, and repeats the lists in +reverse endpoint order before releasing the ownership journal. This bounded confirmation window follows the configured +replication policy and uses only owner-scoped requests. + + +#### Performance Impact + +Each single-site cycle performs one small object write, read, list request, delete, and metadata check. Each +multisite cycle follows the persisted lifecycle through as many immediately successful write, visibility, +delete, and cleanup steps as its bounded operation budget allows. Use dedicated +buckets, explicit directions, and intervals appropriate for the endpoints and replication policy. + + +## Setup + + +You can configure the **s3check** collector in two ways: + +| Method | Best for | How to | +|-----------------------|------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------| +| [**UI**](#via-ui) | Fast setup without editing files | Go to **Nodes → Configure this node → Collectors → Jobs**, search for **s3check**, then click **+** to add a job. | +| [**File**](#via-file) | If you prefer configuring via file, or need to automate deployments (e.g., with Ansible) | Edit `go.d/s3check.conf` and add a job. | + +:::important + +UI configuration requires paid Netdata Cloud plan. + +::: + + +### Prerequisites + +#### Create a dedicated unversioned bucket and prefix + +Create a bucket used only by Netdata. Bucket versioning and Object Lock must be disabled; both Enabled and +Suspended versioning states are rejected. Keep the configured prefix reserved for this collector. + + +#### Verify S3 compatibility and addressing + +Single-site checks require the documented API operations and a never-versioned dedicated bucket. AWS S3 uses +virtual-host addressing, so set path_style to no. Multisite checks also require replication that copies the +probe payload to the configured destination prefix without versioning either bucket. Ceph RGW multisite is +the reference deployment model for this contract. + + +#### Create restricted S3 credentials + +Create credentials for one dedicated probe identity. The identity needs S3 read and write access to the +selected bucket and permission to read its versioning status. Prefer Netdata secret references instead of +storing credentials directly in configuration files. + + +#### Choose the client vantage + +Run the job on the Netdata Agent whose network path represents the client view you need to test. For Ceph, +run jobs at each site or RGW vantage whose client-visible S3 correctness matters. + + + +### Configuration + +#### Options + +Every job requires an endpoint, region, bucket, access key ID, and secret access key. The default retry and +timeout budget is chosen to fit the default 120-second interval. + + +
+Config options + + + +| Group | Option | Description | Default | Required | +|:------|:-----|:------------|:--------|:---------:| +| **Base** | update_every | Probe interval in seconds. | 120 | no | +| | autodetection_retry | Recheck interval in seconds after a failed initial configured-job check; zero disables retries. | 0 | no | +| | mode | Run a single-endpoint lifecycle probe, or run one explicit source-to-destination multisite replication probe. | single | no | +| **Base / Multisite** | source_site | Bounded source-site label used by multisite charts and alerts. | | no | +| | destination | Explicit destination object; required in multisite mode and rejected in single-site mode. No other site is probed automatically. | | no | +| | destination.site | Bounded destination-site label used by multisite charts and alerts. | | no | +| | destination.endpoint | Absolute HTTP(S) destination S3 endpoint without a path, query, or credentials. | | no | +| | destination.region | S3 signing region for the destination endpoint. | | no | +| | destination.bucket | Dedicated unversioned destination bucket. | | no | +| | destination.prefix | Dedicated destination key prefix, ending with a slash. | netdata-s3check/ | no | +| | destination.path_style | Use destination path-style bucket addressing, common for Ceph RGW; set it to no for virtual-host addressing. | yes | no | +| | rpo_threshold_ms | Source-write to destination-visibility objective for the silent RPO alert. | 900000 | no | +| | replication_timeout_ms | Bounded source-write to destination-visibility polling deadline. | 1800000 | no | +| | delete_threshold_ms | Source-delete to destination-disappearance objective for the silent policy alert. | 300000 | no | +| | delete_timeout_ms | Bounded source-delete to destination-disappearance polling deadline. | 900000 | no | +| | verify_delete | Wait for destination disappearance after deleting the source; disabling this still removes the destination probe object but does not measure delete propagation. | yes | no | +| **Base** | endpoint | Absolute HTTP(S) S3 base endpoint without a path, query, or credentials. | | yes | +| | region | S3 signing region, for example us-east-1. | | yes | +| | bucket | Dedicated unversioned S3 bucket used by the probe. | | yes | +| | prefix | Dedicated key prefix, ending with a slash. | netdata-s3check/ | no | +| | path_style | Use path-style bucket addressing, common for Ceph RGW and many S3-compatible services; set it to no for AWS S3. | yes | no | +| | max_retries | Additional attempts allowed per S3 operation, from zero through two. | 1 | no | +| | latency_threshold_ms | Successful stage duration in milliseconds that raises the silent latency alert; zero disables it. | 0 | no | +| | vnode | Associates the job with a configured Virtual Node. | | no | +| **Auth** | access_key_id | S3 access key ID. Use a Netdata secret reference in configuration files. | | yes | +| | secret_access_key | S3 secret access key. Use a Netdata secret reference in configuration files. | | yes | +| | session_token | Optional static S3 session token. | | no | +| **Base / Auth** | destination.access_key_id | Destination S3 access key ID. Use a Netdata secret reference. | | no | +| | destination.secret_access_key | Destination S3 secret access key. Use a Netdata secret reference. | | no | +| | destination.session_token | Optional destination static S3 session token. | | no | +| **HTTP** | timeout | Timeout in seconds for one HTTP attempt. | 2 | no | +| | not_follow_redirects | Reject redirects so signed requests cannot be resent to an unconfigured origin. | yes | no | +| | proxy_url | HTTP proxy URL; empty uses standard proxy environment variables. If the URL contains credentials, use a Netdata secret reference. | | no | +| **TLS** | tls_skip_verify | Skip TLS certificate and hostname verification. This is insecure. | no | no | +| | tls_ca | Absolute path to a CA bundle. | | no | +| | tls_cert | Absolute path to a client certificate. | | no | +| | tls_key | Absolute path to a client private key. | | no | +| **Base / HTTP** | destination.timeout | Timeout in seconds for one destination HTTP attempt. | 2 | no | +| | destination.not_follow_redirects | Reject destination redirects so signed requests cannot be resent to an unconfigured origin. | yes | no | +| | destination.proxy_url | HTTP proxy URL for the destination; empty uses standard proxy environment variables. If the URL contains credentials, use a Netdata secret reference. | | no | +| **Base / TLS** | destination.tls_skip_verify | Skip destination TLS certificate and hostname verification. This is insecure. | no | no | +| | destination.tls_ca | Absolute path to a destination CA bundle. | | no | +| | destination.tls_cert | Absolute path to a destination client certificate. | | no | +| | destination.tls_key | Absolute path to a destination client private key. | | no | + + +
+ + +#### via UI + +Configure the **s3check** collector from the Netdata web interface: + +1. Go to **Nodes**. +2. Select the node **where you want the s3check data-collection job to run** and click the :gear: (**Configure this node**). That node will run the data collection. +3. The **Collectors → Jobs** view opens by default. +4. In the Search box, type _s3check_ (or scroll the list) to locate the **s3check** collector. +5. Click the **+** next to the **s3check** collector to add a new job. +6. Fill in the job fields, then click **Test** to verify the configuration and **Submit** to save. + - **Test** runs the job with the provided settings and shows whether data can be collected. + - If it fails, an error message appears with details (for example, connection refused, timeout, or command execution errors), so you can adjust and retest. + + +#### via File + +The configuration file name for this integration is `go.d/s3check.conf`. + +The file format is YAML. Generally, the structure is: + +```yaml +update_every: 1 +autodetection_retry: 0 +jobs: + - name: some_name1 + - name: some_name2 +``` +You can edit the configuration file using the [`edit-config`](/docs/netdata-agent/configuration#edit-configuration-files) script from the +Netdata [config directory](/docs/netdata-agent/configuration#locate-your-config-directory). + +```bash +cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata +sudo ./edit-config go.d/s3check.conf +``` + +##### Examples + +###### Ceph RGW authenticated S3 check + +Run one job from each client vantage whose authenticated S3 behavior matters. Replace the bucket, +endpoint, and secret references with values for a dedicated unversioned probe bucket. + + +
+Config + +```yaml +jobs: + - name: ceph_rgw_site_a + endpoint: https://rgw.example.net + region: us-east-1 + bucket: netdata-s3check + prefix: netdata-s3check/ + access_key_id: ${env:NETDATA_S3CHECK_ACCESS_KEY_ID} + secret_access_key: ${env:NETDATA_S3CHECK_SECRET_ACCESS_KEY} + path_style: yes + update_every: 120 + timeout: 2 + max_retries: 1 + latency_threshold_ms: 0 + +``` +
+ +###### AWS S3 lifecycle check + +Replace the Region endpoint, bucket, and secret references. Use a dedicated bucket where versioning has +never been enabled, and use virtual-host addressing by setting path_style to no. + + +
+Config + +```yaml +jobs: + - name: aws_s3 + endpoint: https://s3.us-east-1.amazonaws.com + region: us-east-1 + bucket: netdata-s3check + prefix: netdata-s3check/ + access_key_id: ${env:NETDATA_S3CHECK_AWS_ACCESS_KEY_ID} + secret_access_key: ${env:NETDATA_S3CHECK_AWS_SECRET_ACCESS_KEY} + path_style: no + update_every: 120 + timeout: 2 + max_retries: 1 + latency_threshold_ms: 0 + +``` +
+ +###### Ceph RGW directional multisite replication check + +Configure one explicit job for each replication direction that needs client-visible verification. Both +buckets must be dedicated and unversioned; the reverse direction is a separate job. The destination prefix +identifies where the replicated probe key is expected; if it differs, the replication policy must map the +source route namespace to it. + + +
+Config + +```yaml +jobs: + - name: ceph_rgw_site_a_to_site_b + mode: multisite + source_site: site-a + destination: + site: site-b + endpoint: https://rgw-site-b.example.net + region: us-east-1 + bucket: netdata-s3check-site-b + prefix: netdata-s3check/ + access_key_id: ${env:NETDATA_S3CHECK_SITE_B_ACCESS_KEY_ID} + secret_access_key: ${env:NETDATA_S3CHECK_SITE_B_SECRET_ACCESS_KEY} + path_style: yes + endpoint: https://rgw-site-a.example.net + region: us-east-1 + bucket: netdata-s3check-site-a + prefix: netdata-s3check/ + access_key_id: ${env:NETDATA_S3CHECK_SITE_A_ACCESS_KEY_ID} + secret_access_key: ${env:NETDATA_S3CHECK_SITE_A_SECRET_ACCESS_KEY} + path_style: yes + rpo_threshold_ms: 900000 + replication_timeout_ms: 1800000 + delete_threshold_ms: 300000 + delete_timeout_ms: 900000 + verify_delete: yes + +``` +
+ + + +## Alerts + + +The following alerts are available: + +| Alert name | On metric | Description | +|:------------|:----------|:------------| +| [ s3check_stage_failed ](https://github.com/netdata/netdata/blob/master/src/health/health.d/s3check.conf) | s3check.stage_status | The authenticated S3 lifecycle probe reported stage $\{label:stage} with bounded failure reason $\{label:reason}. Raw provider errors are intentionally not exposed. | +| [ s3check_stage_latency ](https://github.com/netdata/netdata/blob/master/src/health/health.d/s3check.conf) | s3check.stage_latency_status | Successful stage $\{label:stage} took at least the configured latency_threshold_ms. Tune the objective to the client vantage and workload. | +| [ s3check_multisite_phase_failed ](https://github.com/netdata/netdata/blob/master/src/health/health.d/s3check.conf) | s3check.multisite_phase_failure | The active S3 probe cannot write, verify, delete, reconcile, or clean up its configured ownership namespace. Inspect the phase status chart for the bounded phase and reason. | +| [ s3check_multisite_payload_mismatch ](https://github.com/netdata/netdata/blob/master/src/health/health.d/s3check.conf) | s3check.multisite_payload_mismatch | The destination site returned a different SHA-256 payload than the source probe wrote. The probe removes both objects and keeps the incident open until a later verified payload clears it. | +| [ s3check_multisite_replication_rpo_breach ](https://github.com/netdata/netdata/blob/master/src/health/health.d/s3check.conf) | s3check.multisite_rpo_status | The current directional probe from $\{label:source_site} to $\{label:destination_site} has not become client-visible within rpo_threshold_ms. Tune the objective to the replication policy. | +| [ s3check_multisite_delete_propagation_breach ](https://github.com/netdata/netdata/blob/master/src/health/health.d/s3check.conf) | s3check.multisite_delete_status | The destination copy from $\{label:source_site} to $\{label:destination_site} remains visible after the source delete for longer than delete_threshold_ms. Tune the objective to the replication policy. | + + + +## Metrics + +Metrics grouped by *scope*. + +The scope defines the instance that the metric belongs to. An instance is uniquely identified by a set of labels. + +Every stage is emitted on every cycle, including skipped stages. + + + +### Per stage + +One lifecycle stage of one configured S3 job. + +Labels: + +| Label | Description | +|:-----------|:----------------| +| stage | Lifecycle stage: setup, put, get, list, delete, or cleanup. | +| reason | Bounded result classification; raw provider errors are never exposed. | + +Metrics: + +| Metric | Description | Dimensions | Unit | +|:------|:------------|:----------|:----| +| s3check.stage_status | S3 Probe Stage Status | ok, failed, skipped | status | +| s3check.stage_duration | S3 Probe Stage Duration | duration | milliseconds | +| s3check.stage_latency_status | S3 Probe Stage Latency Objective Status | exceeded | status | + + +### Per stage requests + +Cumulative request accounting for one lifecycle stage of one configured S3 job. + +Labels: + +| Label | Description | +|:-----------|:----------------| +| stage | Lifecycle stage: setup, put, get, list, delete, or cleanup. | + +Metrics: + +| Metric | Description | Dimensions | Unit | +|:------|:------------|:----------|:----| +| s3check.stage_requests | S3 Probe Stage Requests | operations, attempts, retries, failures | requests | + + +### Per multisite route + +One explicit source-to-destination replication direction. + +Labels: + +| Label | Description | +|:-----------|:----------------| +| source_site | Bounded operator label for the source site. | +| destination_site | Bounded operator label for the destination site. | + +Metrics: + +| Metric | Description | Dimensions | Unit | +|:------|:------------|:----------|:----| +| s3check.multisite_phase_failure | S3 Multisite Phase Failure Status | failed | status | +| s3check.multisite_payload_mismatch | S3 Multisite Payload Mismatch Status | mismatch | status | +| s3check.multisite_replication_lag | S3 Multisite Replication Lag | lag | milliseconds | +| s3check.multisite_rpo_status | S3 Multisite Replication RPO Status | breached | status | +| s3check.multisite_delete_lag | S3 Multisite Delete Propagation Lag | lag | milliseconds | +| s3check.multisite_delete_status | S3 Multisite Delete Propagation Status | breached | status | + + +### Per multisite phase + +One lifecycle phase of one directional S3 multisite job. + +Labels: + +| Label | Description | +|:-----------|:----------------| +| source_site | Bounded operator label for the source site. | +| destination_site | Bounded operator label for the destination site. | +| phase | Directional phase: setup, source_put, replication_wait, source_delete, delete_wait, or cleanup. | +| reason | Bounded result classification; raw provider errors are never exposed. | + +Metrics: + +| Metric | Description | Dimensions | Unit | +|:------|:------------|:----------|:----| +| s3check.multisite_status | S3 Multisite Probe Phase Status | ok, waiting, failed, skipped | status | +| s3check.multisite_phase_duration | S3 Multisite Probe Phase Duration | duration | milliseconds | + + +### Per multisite phase requests + +Cumulative request accounting for one directional S3 multisite phase. + +Labels: + +| Label | Description | +|:-----------|:----------------| +| source_site | Bounded operator label for the source site. | +| destination_site | Bounded operator label for the destination site. | +| phase | Directional phase: setup, source_put, replication_wait, source_delete, delete_wait, or cleanup. | + +Metrics: + +| Metric | Description | Dimensions | Unit | +|:------|:------------|:----------|:----| +| s3check.multisite_phase_requests | S3 Multisite Probe Phase Requests | operations, attempts, retries, failures | requests | + + + +## Troubleshooting + +### Debug Mode + +**Important**: Debug mode is not supported for data collection jobs created via the UI using the Dyncfg feature. + +To troubleshoot issues with the `s3check` collector, run the `go.d.plugin` with the debug option enabled. The output +should give you clues as to why the collector isn't working. + +- Navigate to the `plugins.d` directory, usually at `/usr/libexec/netdata/plugins.d/`. If that's not the case on + your system, open `netdata.conf` and look for the `plugins` setting under `[directories]`. + + ```bash + cd /usr/libexec/netdata/plugins.d/ + ``` + +- Switch to the `netdata` user. + + ```bash + sudo -u netdata -s + ``` + +- Run the `go.d.plugin` to debug the collector: + + ```bash + ./go.d.plugin -d -m s3check + ``` + + To debug a specific job: + + ```bash + ./go.d.plugin -d -m s3check -j jobName + ``` + +### Getting Logs + +If you're encountering problems with the `s3check` collector, follow these steps to retrieve logs and identify potential issues: + +- **Run the command** specific to your system (systemd, non-systemd, or Docker container). +- **Examine the output** for any warnings or error messages that might indicate issues. These messages should provide clues about the root cause of the problem. + +#### System with systemd + +Use the following command to view logs generated since the last Netdata service restart: + +```bash +journalctl _SYSTEMD_INVOCATION_ID="$(systemctl show --value --property=InvocationID netdata)" --namespace=netdata --grep s3check +``` + +#### System without systemd + +Locate the collector log file, typically at `/var/log/netdata/collector.log`, and use `grep` to filter for collector's name: + +```bash +grep s3check /var/log/netdata/collector.log +``` + +**Note**: This method shows logs from all restarts. Focus on the **latest entries** for troubleshooting current issues. + +#### Docker Container + +If your Netdata runs in a Docker container named "netdata" (replace if different), use this command: + +```bash +docker logs netdata 2>&1 | grep s3check +``` + +### S3 check does not start + +Verify that the endpoint is absolute, the region and bucket are set, credentials resolve, and the account +can call GetBucketVersioning. In multisite mode both source and destination must pass this check. Netdata +rejects buckets whose versioning status is Enabled or Suspended. + + +### Cleanup remains pending + +The collector found stale probe keys. It reconciles the prefix before every new write, deletes at most +two collector-owned keys per cycle, and creates no new object until that prefix is clean. Confirm that +no other writer uses the prefix. A persisted ownership journal is rejected after its mode, source, destination, +endpoint, bucket, or prefix changes; restore the matching configuration so the pending object can be cleaned +safely. + + +### AWS multisite replication is rejected + +The multisite probe requires both buckets to remain never-versioned so an ordinary object deletion proves +complete cleanup. AWS-native replication requires versioning on both buckets and therefore does not satisfy +this contract. Use the AWS single-site lifecycle check, or select an S3-compatible replication implementation +that meets the documented unversioned-bucket contract. diff --git a/docs/Collecting Metrics/Collectors/Synthetic Testing/Site 24x7.mdx b/docs/Collecting Metrics/Collectors/Synthetic Testing/Site 24x7.mdx index 360ac4a18..820a22166 100644 --- a/docs/Collecting Metrics/Collectors/Synthetic Testing/Site 24x7.mdx +++ b/docs/Collecting Metrics/Collectors/Synthetic Testing/Site 24x7.mdx @@ -7,7 +7,7 @@ toc_collapsible: "true" learn_rel_path: "Collecting Metrics/Collectors/Synthetic Testing" description: "Monitor Site24x7 website and infrastructure monitoring metrics for efficient performance tracking and management." message: "DO NOT EDIT THIS FILE DIRECTLY, IT IS GENERATED BY THE COLLECTOR'S metadata.yaml FILE" -sidebar_position: "120" +sidebar_position: "130" learn_link: "https://learn.netdata.cloud/docs/collecting-metrics/collectors/synthetic-testing/site-24x7" slug: "/collecting-metrics/collectors/synthetic-testing/site-24x7" --- diff --git a/docs/Collecting Metrics/Collectors/Synthetic Testing/Synthetic Testing.mdx b/docs/Collecting Metrics/Collectors/Synthetic Testing/Synthetic Testing.mdx index 4feebf895..d4f9e68a3 100644 --- a/docs/Collecting Metrics/Collectors/Synthetic Testing/Synthetic Testing.mdx +++ b/docs/Collecting Metrics/Collectors/Synthetic Testing/Synthetic Testing.mdx @@ -45,6 +45,9 @@ import { Grid, Box, GridPagination } from '@site/src/components/Grid_integration + + + diff --git a/docs/Collecting Metrics/Collectors/Synthetic Testing/TCP UDP Endpoints.mdx b/docs/Collecting Metrics/Collectors/Synthetic Testing/TCP UDP Endpoints.mdx index 29a3b17ea..1d8087132 100644 --- a/docs/Collecting Metrics/Collectors/Synthetic Testing/TCP UDP Endpoints.mdx +++ b/docs/Collecting Metrics/Collectors/Synthetic Testing/TCP UDP Endpoints.mdx @@ -7,7 +7,7 @@ toc_collapsible: "true" learn_rel_path: "Collecting Metrics/Collectors/Synthetic Testing" description: "Collector for monitoring service availability and response time." message: "DO NOT EDIT THIS FILE DIRECTLY, IT IS GENERATED BY THE COLLECTOR'S metadata.yaml FILE" -sidebar_position: "130" +sidebar_position: "140" learn_link: "https://learn.netdata.cloud/docs/collecting-metrics/collectors/synthetic-testing/tcp-udp-endpoints" slug: "/collecting-metrics/collectors/synthetic-testing/tcp-udp-endpoints" --- diff --git a/docs/Collecting Metrics/Collectors/Synthetic Testing/Uptimerobot.mdx b/docs/Collecting Metrics/Collectors/Synthetic Testing/Uptimerobot.mdx index ffcc6c7ce..b98dab63c 100644 --- a/docs/Collecting Metrics/Collectors/Synthetic Testing/Uptimerobot.mdx +++ b/docs/Collecting Metrics/Collectors/Synthetic Testing/Uptimerobot.mdx @@ -7,7 +7,7 @@ toc_collapsible: "true" learn_rel_path: "Collecting Metrics/Collectors/Synthetic Testing" description: "Monitor UptimeRobot website uptime monitoring metrics for efficient website availability tracking and management." message: "DO NOT EDIT THIS FILE DIRECTLY, IT IS GENERATED BY THE COLLECTOR'S metadata.yaml FILE" -sidebar_position: "140" +sidebar_position: "150" learn_link: "https://learn.netdata.cloud/docs/collecting-metrics/collectors/synthetic-testing/uptimerobot" slug: "/collecting-metrics/collectors/synthetic-testing/uptimerobot" --- diff --git a/docs/Collecting Metrics/Collectors/Synthetic Testing/X.509 certificate.mdx b/docs/Collecting Metrics/Collectors/Synthetic Testing/X.509 certificate.mdx index d6d477cd5..8be3989d3 100644 --- a/docs/Collecting Metrics/Collectors/Synthetic Testing/X.509 certificate.mdx +++ b/docs/Collecting Metrics/Collectors/Synthetic Testing/X.509 certificate.mdx @@ -8,7 +8,7 @@ learn_rel_path: "Collecting Metrics/Collectors/Synthetic Testing" keywords: [x509, certificate] description: "This collectors monitors x509 certificates expiration time and revocation status." message: "DO NOT EDIT THIS FILE DIRECTLY, IT IS GENERATED BY THE COLLECTOR'S metadata.yaml FILE" -sidebar_position: "150" +sidebar_position: "160" learn_link: "https://learn.netdata.cloud/docs/collecting-metrics/collectors/synthetic-testing/x.509-certificate" slug: "/collecting-metrics/collectors/synthetic-testing/x.509-certificate" --- diff --git a/docs/Developer and Contributor Corner/External Plugins/go.d.plugin/go.d.plugin.mdx b/docs/Developer and Contributor Corner/External Plugins/go.d.plugin/go.d.plugin.mdx index 122e144cb..68b255592 100644 --- a/docs/Developer and Contributor Corner/External Plugins/go.d.plugin/go.d.plugin.mdx +++ b/docs/Developer and Contributor Corner/External Plugins/go.d.plugin/go.d.plugin.mdx @@ -136,6 +136,7 @@ All capabilities are set automatically during Netdata installation using the [of | [rethinkdb](https://github.com/netdata/netdata/tree/master/src/go/plugin/go.d/collector/rethinkdb) | RethinkDB | | [riakkv](https://github.com/netdata/netdata/tree/master/src/go/plugin/go.d/collector/riakkv) | Riak KV | | [rspamd](https://github.com/netdata/netdata/tree/master/src/go/plugin/go.d/collector/rspamd) | Rspamd | +| [s3check](https://github.com/netdata/netdata/tree/master/src/go/plugin/go.d/collector/s3check) | S3 Compatible Object Storage | | [samba](https://github.com/netdata/netdata/tree/master/src/go/plugin/go.d/collector/samba) | Samba | | [scaleio](https://github.com/netdata/netdata/tree/master/src/go/plugin/go.d/collector/scaleio) | Dell EMC ScaleIO | | [SNMP](https://github.com/netdata/netdata/blob/master/src/go/plugin/go.d/collector/snmp) | SNMP | diff --git a/docs/Welcome to Netdata/Monitor Anything.mdx b/docs/Welcome to Netdata/Monitor Anything.mdx index ea825e3b0..f01e01a36 100644 --- a/docs/Welcome to Netdata/Monitor Anything.mdx +++ b/docs/Welcome to Netdata/Monitor Anything.mdx @@ -594,6 +594,7 @@ Need a dedicated integration? [Submit a feature request](https://github.com/netd | [MQTT Blackbox](/docs/collecting-metrics/collectors/synthetic-testing/mqtt-blackbox) | Track MQTT message transport performance using blackbox testing methods. | | [Nagios Plugins and Custom Scripts](/docs/collecting-metrics/collectors/synthetic-testing/nagios-plugins-and-custom-scripts) | This collector runs [Nagios-compatible plugins](https://www.nagios-plugins.org/) and custom scripts in any language (Bash, PowerShell, Python, Go, etc.). | | [Ping](/docs/collecting-metrics/collectors/synthetic-testing/ping) | This module measures round-trip time and packet loss by sending ping messages to network hosts. | +| [S3 Compatible Object Storage](/docs/collecting-metrics/collectors/synthetic-testing/s3-compatible-object-storage) | Verify authenticated S3 operations and optional directional multisite replication. | | [Site 24x7](/docs/collecting-metrics/collectors/synthetic-testing/site-24x7) | Monitor Site24x7 website and infrastructure monitoring metrics for efficient performance tracking and management. | | [TCP/UDP Endpoints](/docs/collecting-metrics/collectors/synthetic-testing/tcp-udp-endpoints) | Collector for monitoring service availability and response time. | | [Uptimerobot](/docs/collecting-metrics/collectors/synthetic-testing/uptimerobot) | Monitor UptimeRobot website uptime monitoring metrics for efficient website availability tracking and management. | diff --git a/ingest/generated_map.yaml b/ingest/generated_map.yaml index a2a147999..4120ce5d4 100644 --- a/ingest/generated_map.yaml +++ b/ingest/generated_map.yaml @@ -5823,6 +5823,16 @@ meta_yaml: https://github.com/netdata/netdata/edit/master/src/go/plugin/go.d/collector/ping/metadata.yaml message: DO NOT EDIT THIS FILE DIRECTLY, IT IS GENERATED BY THE COLLECTOR'S metadata.yaml FILE +- custom_edit_url: https://github.com/netdata/netdata/edit/master/src/go/plugin/go.d/collector/s3check/README.md + sidebar_label: S3 Compatible Object Storage + learn_status: Published + learn_rel_path: Collecting Metrics/Collectors/Synthetic Testing + keywords: '[''s3'', ''object storage'', ''ceph'', ''rgw'', ''synthetic'']' + description: Verify authenticated S3 operations and optional directional multisite + replication. + meta_yaml: https://github.com/netdata/netdata/edit/master/src/go/plugin/go.d/collector/s3check/metadata.yaml + message: DO NOT EDIT THIS FILE DIRECTLY, IT IS GENERATED BY THE COLLECTOR'S metadata.yaml + FILE - custom_edit_url: https://github.com/netdata/netdata/edit/master/src/go/plugin/go.d/collector/prometheus/integrations/site_24x7.md sidebar_label: Site 24x7 learn_status: Published diff --git a/ingest/generated_sidebar_order.json b/ingest/generated_sidebar_order.json index d032c43dd..0012657a1 100644 --- a/ingest/generated_sidebar_order.json +++ b/ingest/generated_sidebar_order.json @@ -2,7 +2,7 @@ "schema_version": 1, "source": "netdata/docs/.map/map.yaml", "source_sha256": "99485c694e5d686dd5eec68f7ccf4e42defb0e993a0439c4b57e40a0bd9b3caf", - "source_corpus_sha256": "98e42804742d4fc1eb48bb205694632879993cc5214e0371a34361f37c9e6e52", + "source_corpus_sha256": "df3b8a5407c03904e0e6acebfccfac2c6ea987cbb2233a8cd80a536a29dcbc71", "order": [ { "parent_path": "Alerts & Notifications", @@ -1725,5 +1725,5 @@ "position": 170 } ], - "full_ingest_identity_sha256": "37af03fadb694153a66a8891cdded21973d0734d5ab556df999283137dbe892f" + "full_ingest_identity_sha256": "76fdcf42ef947c97d2f4f7f831e3077fcc9b03aff6bbc4c9b5e73b2f7909d737" } diff --git a/ingest/generated_sidebar_order.json.sha256 b/ingest/generated_sidebar_order.json.sha256 index 6cd2e7e7e..6d054423f 100644 --- a/ingest/generated_sidebar_order.json.sha256 +++ b/ingest/generated_sidebar_order.json.sha256 @@ -1 +1 @@ -e12745340e34cc0cc7e7ca4d7e79b8036c09abf385c33ce421ddfa90dfcd0644 generated_sidebar_order.json +2f6aba0d02aec3b04d890fe6c73b480aa278605addb8976e95ccc8d70a7e2b50 generated_sidebar_order.json diff --git a/ingest/one_commit_back_file-dict.yaml b/ingest/one_commit_back_file-dict.yaml index ff175ef2d..f173be2d5 100644 --- a/ingest/one_commit_back_file-dict.yaml +++ b/ingest/one_commit_back_file-dict.yaml @@ -1608,6 +1608,8 @@ learn_path: /docs/collecting-metrics/collectors/databases/riak-kv - custom_edit_url: https://github.com/netdata/netdata/edit/master/src/go/plugin/go.d/collector/rspamd/README.md learn_path: /docs/collecting-metrics/collectors/applications/rspamd +- custom_edit_url: https://github.com/netdata/netdata/edit/master/src/go/plugin/go.d/collector/s3check/README.md + learn_path: /docs/collecting-metrics/collectors/synthetic-testing/s3-compatible-object-storage - custom_edit_url: https://github.com/netdata/netdata/edit/master/src/go/plugin/go.d/collector/samba/README.md learn_path: /docs/collecting-metrics/collectors/storage-and-filesystems/samba - custom_edit_url: https://github.com/netdata/netdata/edit/master/src/go/plugin/go.d/collector/scaleio/README.md