S3 Compatible Object Storage

Plugin: go.d.plugin Module: s3check

Overview

Verify authenticated S3 operations and optional directional multisite replication.

Each single-site job measures stage state, duration, latency objective, attempts, retries, and failures. Each multisite job adds directional replication lag, RPO state, delete-propagation lag, and bounded request accounting. Both probes are intentionally active and clean up the objects they create.

The collector uses the AWS SigV4 S3 client with explicit static credentials, endpoint, region, and addressing configuration. It requires a dedicated unversioned bucket and prefix, checks bucket versioning before the job starts and again before destructive operations, and never deletes versions or foreign-owned keys. Exact object ownership is persisted before a write or reconciliation delete so restart and configuration changes cannot abandon cleanup.

Compatibility is contract-based rather than vendor-specific. The service must support SigV4 static credentials, GetBucketVersioning, PutObject, GetObject, HeadObject, ListObjectsV2, and DeleteObject. An ordinary object deletion must also remove the object completely without leaving hidden versions. Ceph RGW multisite is the reference deployment model for directional replication checks. AWS S3 satisfies the single-site contract when it is configured with virtual-host addressing and a bucket where versioning has never been enabled. AWS-native cross-region or same-region replication is not a multisite source because AWS requires versioning on both buckets. Other S3-compatible services can be used when they satisfy the same API and unversioned-bucket contract.

Every single-site cycle generates a unique small payload and key, uploads it, reads and verifies its SHA-256 digest, confirms LIST visibility, deletes it, and verifies that it disappeared.

In multisite mode, one job represents exactly one source-to-destination direction. Netdata persists sanitized pending-object state before the source write, polls destination visibility across bounded collection cycles, verifies the SHA-256 digest, deletes the source, measures destination disappearance, and cleans both sites. Daemon sync counters are not used as proof of client-visible correctness.

After an interrupted write path the collector retries deletion; after a process restart it removes at most two keys owned by this Agent and job per endpoint per cycle before creating a new object. Probe namespaces are derived from the Agent machine GUID and job identity, so separate Agents, jobs, and reverse directions can probe the same route without deleting one another's active objects. Raw S3 errors are classified into bounded reasons. Credentials, signed headers, request IDs, endpoint details, and probe payloads are not exposed as chart labels or stored in pending state.

This collector is supported on all platforms.

This collector supports collecting metrics from multiple instances of this integration, including remote instances.

No host capabilities are required. The source identity needs GetBucketVersioning, PutObject, GetObject, ListBucket, and DeleteObject on its bucket. A multisite destination identity needs GetBucketVersioning, GetObject, ListBucket, and DeleteObject on its bucket; it is not granted PutObject because the probe never writes to the destination.

Default Behavior

Auto-Detection

There is no auto-detection. A job starts only after an endpoint, region, bucket, and credentials are configured.

Limits

A successful single-site probe from a clean namespace makes nine S3 API calls: prefix LIST, pre-write bucket proof, PUT, GET, LIST, pre-delete bucket proof, DELETE, HEAD, and bucket proof after DELETE. On the next collection, Netdata performs one owner-prefix LIST to clear the probe quarantine and then starts the next nine-call probe, for ten calls in that steady-state cycle. A still-present object is retried on a later cycle. Reconciling two stale keys also makes nine calls. A multisite cycle makes at most eleven calls while it advances one persisted lifecycle. Each call uses the configured timeout and bounded retry count, and the worst-case per-cycle deadline must fit inside update_every.

The probe payload is 4 KiB. Prefix reconciliation lists this job's owner namespace before every new write and deletes at most two exact keys from the persisted ownership journal per endpoint per cycle. Visibility and delete objectives must be at least one collection interval. Multisite polling spans collection cycles until the configured visibility or delete deadline; it does not continuously retry inside one cycle. After multisite cleanup deletes the exact source and destination keys, Netdata waits for the larger configured replication or delete deadline, lists both owner namespaces, waits one more collection interval, and repeats the lists in reverse endpoint order before releasing the ownership journal. This bounded confirmation window follows the configured replication policy and uses only owner-scoped requests.

Performance Impact

Each single-site cycle performs one small object write, read, list request, delete, and metadata check. Each multisite cycle follows the persisted lifecycle through as many immediately successful write, visibility, delete, and cleanup steps as its bounded operation budget allows. Use dedicated buckets, explicit directions, and intervals appropriate for the endpoints and replication policy.

Setup

You can configure the s3check collector in two ways:

Method	Best for	How to
UI	Fast setup without editing files	Go to Nodes → Configure this node → Collectors → Jobs, search for s3check, then click + to add a job.
File	If you prefer configuring via file, or need to automate deployments (e.g., with Ansible)	Edit go.d/s3check.conf and add a job.

important

UI configuration requires paid Netdata Cloud plan.

Prerequisites

Create a dedicated unversioned bucket and prefix

Create a bucket used only by Netdata. Bucket versioning and Object Lock must be disabled; both Enabled and Suspended versioning states are rejected. Keep the configured prefix reserved for this collector.

Verify S3 compatibility and addressing

Single-site checks require the documented API operations and a never-versioned dedicated bucket. AWS S3 uses virtual-host addressing, so set path_style to no. Multisite checks also require replication that copies the probe payload to the configured destination prefix without versioning either bucket. Ceph RGW multisite is the reference deployment model for this contract.

Create restricted S3 credentials

Create credentials for one dedicated probe identity. The identity needs S3 read and write access to the selected bucket and permission to read its versioning status. Prefer Netdata secret references instead of storing credentials directly in configuration files.

Choose the client vantage

Run the job on the Netdata Agent whose network path represents the client view you need to test. For Ceph, run jobs at each site or RGW vantage whose client-visible S3 correctness matters.

Configuration

Options

Every job requires an endpoint, region, bucket, access key ID, and secret access key. The default retry and timeout budget is chosen to fit the default 120-second interval.

Config options

Group	Option	Description	Default	Required
Base	update_every	Probe interval in seconds.	120	no
	autodetection_retry	Recheck interval in seconds after a failed initial configured-job check; zero disables retries.	0	no
	mode	Run a single-endpoint lifecycle probe, or run one explicit source-to-destination multisite replication probe.	single	no
Base / Multisite	source_site	Bounded source-site label used by multisite charts and alerts.		no
	destination	Explicit destination object; required in multisite mode and rejected in single-site mode. No other site is probed automatically.		no
	destination.site	Bounded destination-site label used by multisite charts and alerts.		no
	destination.endpoint	Absolute HTTP(S) destination S3 endpoint without a path, query, or credentials.		no
	destination.region	S3 signing region for the destination endpoint.		no
	destination.bucket	Dedicated unversioned destination bucket.		no
	destination.prefix	Dedicated destination key prefix, ending with a slash.	netdata-s3check/	no
	destination.path_style	Use destination path-style bucket addressing, common for Ceph RGW; set it to no for virtual-host addressing.	yes	no
	rpo_threshold_ms	Source-write to destination-visibility objective for the silent RPO alert.	900000	no
	replication_timeout_ms	Bounded source-write to destination-visibility polling deadline.	1800000	no
	delete_threshold_ms	Source-delete to destination-disappearance objective for the silent policy alert.	300000	no
	delete_timeout_ms	Bounded source-delete to destination-disappearance polling deadline.	900000	no
	verify_delete	Wait for destination disappearance after deleting the source; disabling this still removes the destination probe object but does not measure delete propagation.	yes	no
Base	endpoint	Absolute HTTP(S) S3 base endpoint without a path, query, or credentials.		yes
	region	S3 signing region, for example us-east-1.		yes
	bucket	Dedicated unversioned S3 bucket used by the probe.		yes
	prefix	Dedicated key prefix, ending with a slash.	netdata-s3check/	no
	path_style	Use path-style bucket addressing, common for Ceph RGW and many S3-compatible services; set it to no for AWS S3.	yes	no
	max_retries	Additional attempts allowed per S3 operation, from zero through two.	1	no
	latency_threshold_ms	Successful stage duration in milliseconds that raises the silent latency alert; zero disables it.	0	no
	vnode	Associates the job with a configured Virtual Node.		no
Auth	access_key_id	S3 access key ID. Use a Netdata secret reference in configuration files.		yes
	secret_access_key	S3 secret access key. Use a Netdata secret reference in configuration files.		yes
	session_token	Optional static S3 session token.		no
Base / Auth	destination.access_key_id	Destination S3 access key ID. Use a Netdata secret reference.		no
	destination.secret_access_key	Destination S3 secret access key. Use a Netdata secret reference.		no
	destination.session_token	Optional destination static S3 session token.		no
HTTP	timeout	Timeout in seconds for one HTTP attempt.	2	no
	not_follow_redirects	Reject redirects so signed requests cannot be resent to an unconfigured origin.	yes	no
	proxy_url	HTTP proxy URL; empty uses standard proxy environment variables. If the URL contains credentials, use a Netdata secret reference.		no
TLS	tls_skip_verify	Skip TLS certificate and hostname verification. This is insecure.	no	no
	tls_ca	Absolute path to a CA bundle.		no
	tls_cert	Absolute path to a client certificate.		no
	tls_key	Absolute path to a client private key.		no
Base / HTTP	destination.timeout	Timeout in seconds for one destination HTTP attempt.	2	no
	destination.not_follow_redirects	Reject destination redirects so signed requests cannot be resent to an unconfigured origin.	yes	no
	destination.proxy_url	HTTP proxy URL for the destination; empty uses standard proxy environment variables. If the URL contains credentials, use a Netdata secret reference.		no
Base / TLS	destination.tls_skip_verify	Skip destination TLS certificate and hostname verification. This is insecure.	no	no
	destination.tls_ca	Absolute path to a destination CA bundle.		no
	destination.tls_cert	Absolute path to a destination client certificate.		no
	destination.tls_key	Absolute path to a destination client private key.		no

via UI

Configure the s3check collector from the Netdata web interface:

1. Go to Nodes.
2. Select the node where you want the s3check data-collection job to run and click the ⚙ (Configure this node). That node will run the data collection.
3. The Collectors → Jobs view opens by default.
4. In the Search box, type s3check (or scroll the list) to locate the s3check collector.
5. Click the + next to the s3check collector to add a new job.
6. Fill in the job fields, then click Test to verify the configuration and Submit to save.
   • Test runs the job with the provided settings and shows whether data can be collected.
   • If it fails, an error message appears with details (for example, connection refused, timeout, or command execution errors), so you can adjust and retest.

via File

The configuration file name for this integration is go.d/s3check.conf.

The file format is YAML. Generally, the structure is:

update_every: 1
autodetection_retry: 0
jobs:
  - name: some_name1
  - name: some_name2

You can edit the configuration file using the edit-config script from the Netdata config directory.

cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata
sudo ./edit-config go.d/s3check.conf

Examples

Ceph RGW authenticated S3 check

Run one job from each client vantage whose authenticated S3 behavior matters. Replace the bucket, endpoint, and secret references with values for a dedicated unversioned probe bucket.

Config

jobs:
  - name: ceph_rgw_site_a
    endpoint: https://rgw.example.net
    region: us-east-1
    bucket: netdata-s3check
    prefix: netdata-s3check/
    access_key_id: ${env:NETDATA_S3CHECK_ACCESS_KEY_ID}
    secret_access_key: ${env:NETDATA_S3CHECK_SECRET_ACCESS_KEY}
    path_style: yes
    update_every: 120
    timeout: 2
    max_retries: 1
    latency_threshold_ms: 0

AWS S3 lifecycle check

Replace the Region endpoint, bucket, and secret references. Use a dedicated bucket where versioning has never been enabled, and use virtual-host addressing by setting path_style to no.

Config

jobs:
  - name: aws_s3
    endpoint: https://s3.us-east-1.amazonaws.com
    region: us-east-1
    bucket: netdata-s3check
    prefix: netdata-s3check/
    access_key_id: ${env:NETDATA_S3CHECK_AWS_ACCESS_KEY_ID}
    secret_access_key: ${env:NETDATA_S3CHECK_AWS_SECRET_ACCESS_KEY}
    path_style: no
    update_every: 120
    timeout: 2
    max_retries: 1
    latency_threshold_ms: 0

Ceph RGW directional multisite replication check

Configure one explicit job for each replication direction that needs client-visible verification. Both buckets must be dedicated and unversioned; the reverse direction is a separate job. The destination prefix identifies where the replicated probe key is expected; if it differs, the replication policy must map the source route namespace to it.

Config

jobs:
  - name: ceph_rgw_site_a_to_site_b
    mode: multisite
    source_site: site-a
    destination:
      site: site-b
      endpoint: https://rgw-site-b.example.net
      region: us-east-1
      bucket: netdata-s3check-site-b
      prefix: netdata-s3check/
      access_key_id: ${env:NETDATA_S3CHECK_SITE_B_ACCESS_KEY_ID}
      secret_access_key: ${env:NETDATA_S3CHECK_SITE_B_SECRET_ACCESS_KEY}
      path_style: yes
    endpoint: https://rgw-site-a.example.net
    region: us-east-1
    bucket: netdata-s3check-site-a
    prefix: netdata-s3check/
    access_key_id: ${env:NETDATA_S3CHECK_SITE_A_ACCESS_KEY_ID}
    secret_access_key: ${env:NETDATA_S3CHECK_SITE_A_SECRET_ACCESS_KEY}
    path_style: yes
    rpo_threshold_ms: 900000
    replication_timeout_ms: 1800000
    delete_threshold_ms: 300000
    delete_timeout_ms: 900000
    verify_delete: yes

Alerts

The following alerts are available:

Alert name	On metric	Description
Alert name s3check_stage_failed	On metrics3check.stage_status	DescriptionThe authenticated S3 lifecycle probe reported stage ${label:stage} with bounded failure reason ${label:reason}. Raw provider errors are intentionally not exposed.
Alert name s3check_stage_latency	On metrics3check.stage_latency_status	DescriptionSuccessful stage ${label:stage} took at least the configured latency_threshold_ms. Tune the objective to the client vantage and workload.
Alert name s3check_multisite_phase_failed	On metrics3check.multisite_phase_failure	DescriptionThe active S3 probe cannot write, verify, delete, reconcile, or clean up its configured ownership namespace. Inspect the phase status chart for the bounded phase and reason.
Alert name s3check_multisite_payload_mismatch	On metrics3check.multisite_payload_mismatch	DescriptionThe destination site returned a different SHA-256 payload than the source probe wrote. The probe removes both objects and keeps the incident open until a later verified payload clears it.
Alert name s3check_multisite_replication_rpo_breach	On metrics3check.multisite_rpo_status	DescriptionThe current directional probe from ${label:source_site} to ${label:destination_site} has not become client-visible within rpo_threshold_ms. Tune the objective to the replication policy.
Alert name s3check_multisite_delete_propagation_breach	On metrics3check.multisite_delete_status	DescriptionThe destination copy from ${label:source_site} to ${label:destination_site} remains visible after the source delete for longer than delete_threshold_ms. Tune the objective to the replication policy.

Metrics

Metrics grouped by scope.

The scope defines the instance that the metric belongs to. An instance is uniquely identified by a set of labels.

Every stage is emitted on every cycle, including skipped stages.

Per stage

One lifecycle stage of one configured S3 job.

Labels:

Label	Description
stage	Lifecycle stage: setup, put, get, list, delete, or cleanup.
reason	Bounded result classification; raw provider errors are never exposed.

Metrics:

Metric	Description	Dimensions	Unit
s3check.stage_status	S3 Probe Stage Status	ok, failed, skipped	status
s3check.stage_duration	S3 Probe Stage Duration	duration	milliseconds
s3check.stage_latency_status	S3 Probe Stage Latency Objective Status	exceeded	status

Per stage requests

Cumulative request accounting for one lifecycle stage of one configured S3 job.

Labels:

Label	Description
stage	Lifecycle stage: setup, put, get, list, delete, or cleanup.

Metrics:

Metric	Description	Dimensions	Unit
s3check.stage_requests	S3 Probe Stage Requests	operations, attempts, retries, failures	requests

Per multisite route

One explicit source-to-destination replication direction.

Labels:

Label	Description
source_site	Bounded operator label for the source site.
destination_site	Bounded operator label for the destination site.

Metrics:

Metric	Description	Dimensions	Unit
s3check.multisite_phase_failure	S3 Multisite Phase Failure Status	failed	status
s3check.multisite_payload_mismatch	S3 Multisite Payload Mismatch Status	mismatch	status
s3check.multisite_replication_lag	S3 Multisite Replication Lag	lag	milliseconds
s3check.multisite_rpo_status	S3 Multisite Replication RPO Status	breached	status
s3check.multisite_delete_lag	S3 Multisite Delete Propagation Lag	lag	milliseconds
s3check.multisite_delete_status	S3 Multisite Delete Propagation Status	breached	status

Per multisite phase

One lifecycle phase of one directional S3 multisite job.

Labels:

Label	Description
source_site	Bounded operator label for the source site.
destination_site	Bounded operator label for the destination site.
phase	Directional phase: setup, source_put, replication_wait, source_delete, delete_wait, or cleanup.
reason	Bounded result classification; raw provider errors are never exposed.

Metrics:

Metric	Description	Dimensions	Unit
s3check.multisite_status	S3 Multisite Probe Phase Status	ok, waiting, failed, skipped	status
s3check.multisite_phase_duration	S3 Multisite Probe Phase Duration	duration	milliseconds

Per multisite phase requests

Cumulative request accounting for one directional S3 multisite phase.

Labels:

Label	Description
source_site	Bounded operator label for the source site.
destination_site	Bounded operator label for the destination site.
phase	Directional phase: setup, source_put, replication_wait, source_delete, delete_wait, or cleanup.

Metrics:

Metric	Description	Dimensions	Unit
s3check.multisite_phase_requests	S3 Multisite Probe Phase Requests	operations, attempts, retries, failures	requests

Troubleshooting

Debug Mode

Important: Debug mode is not supported for data collection jobs created via the UI using the Dyncfg feature.

To troubleshoot issues with the s3check collector, run the go.d.plugin with the debug option enabled. The output should give you clues as to why the collector isn't working.

• Navigate to the plugins.d directory, usually at /usr/libexec/netdata/plugins.d/. If that's not the case on your system, open netdata.conf and look for the plugins setting under [directories].

  cd /usr/libexec/netdata/plugins.d/

• Switch to the netdata user.

  sudo -u netdata -s

• Run the go.d.plugin to debug the collector:

  ./go.d.plugin -d -m s3check

  To debug a specific job:

  ./go.d.plugin -d -m s3check -j jobName

Getting Logs

If you're encountering problems with the s3check collector, follow these steps to retrieve logs and identify potential issues:

• Run the command specific to your system (systemd, non-systemd, or Docker container).
• Examine the output for any warnings or error messages that might indicate issues. These messages should provide clues about the root cause of the problem.

System with systemd

Use the following command to view logs generated since the last Netdata service restart:

journalctl _SYSTEMD_INVOCATION_ID="$(systemctl show --value --property=InvocationID netdata)" --namespace=netdata --grep s3check

System without systemd

Locate the collector log file, typically at /var/log/netdata/collector.log, and use grep to filter for collector's name:

grep s3check /var/log/netdata/collector.log

Note: This method shows logs from all restarts. Focus on the latest entries for troubleshooting current issues.

Docker Container

If your Netdata runs in a Docker container named "netdata" (replace if different), use this command:

docker logs netdata 2>&1 | grep s3check

S3 check does not start

Verify that the endpoint is absolute, the region and bucket are set, credentials resolve, and the account can call GetBucketVersioning. In multisite mode both source and destination must pass this check. Netdata rejects buckets whose versioning status is Enabled or Suspended.

Cleanup remains pending

The collector found stale probe keys. It reconciles the prefix before every new write, deletes at most two collector-owned keys per cycle, and creates no new object until that prefix is clean. Confirm that no other writer uses the prefix. A persisted ownership journal is rejected after its mode, source, destination, endpoint, bucket, or prefix changes; restore the matching configuration so the pending object can be cleaned safely.

AWS multisite replication is rejected

The multisite probe requires both buckets to remain never-versioned so an ordinary object deletion proves complete cleanup. AWS-native replication requires versioning on both buckets and therefore does not satisfy this contract. Use the AWS single-site lifecycle check, or select an S3-compatible replication implementation that meets the documented unversioned-bucket contract.