How to configure Prometheus and MinIO in k8s

Objective:

Provide a practical example on how to configure Prometheus and MinIO in k8s

Diagram:

Links:

• https://github.com/minio/wiki/wiki/How-to-configure-Prometheus-and-MinIO-in-k8s

• https://min.io/docs/minio/linux/operations/monitoring/collect-minio-metrics-using-prometheus.html

• https://community.atlassian.com/t5/Marketplace-Apps-Integrations/How-to-handle-insecure-SSL-certificate-sites-with-Prometheus-for/ba-p/1477574

• https://community.atlassian.com/t5/Marketplace-Apps-Integrations/How-to-handle-insecure-SSL-certificate-sites-with-Prometheus-for/ba-p/1477574?utm_source=dm&utm_medium=unpaid-social&utm_campaign=P:online*O:community*I:social_share*

1. Have MinIO Running and Pointing to Prometheus:

createcluster # Create a kind cluster for testing
installoperator # Install MinIO Operator with Kustomize
installtenant # Install MinIO Tenant with Kustomize

• MinIO Pod is able to contact Prometheus because Prometheus is running over a public address:

# My Public IP Address is being used for Prometheus:
Prometheus > 9091~9091	> 9091~9091	> TCP > 192.168.0.13	> Any

• In the Tenant Spec, you have to set the environment variable:

spec:
  env:
    - name: MINIO_PROMETHEUS_URL
      value: "http://72.140.145.27"

2. Generate the Scrape Configuration

• Using NodePort exposed 9000 end point.

mc alias set myminio https://localhost:30081 console console123 --insecure
mc admin prometheus generate myminio --insecure

Expected:

$ mc alias set myminio https://localhost:30081 console console123 --insecure
Added `myminio` successfully.
$ mc admin prometheus generate myminio --insecure
scrape_configs:
- job_name: minio-job
  bearer_token: eyJhbGciOiJIUzUxMiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJwcm9tZXRoZXVzIiwic3ViIjoiY29uc29sZSIsImV4cCI6NDgzNDQxNzM3OH0.T3P7jzQNFm5c0TgL4djMMx4IfWIBXBvz78fu_R0DMuWb11ydZvsLptA29gC3CpengKDw7DiurImTxarmGOnPiA
  metrics_path: /minio/v2/metrics/cluster
  scheme: https
  static_configs:
  - targets: ['localhost:30081']

3. Configure Prometheus:

File Example: /Users/cniackz/prometheus-2.36.0.darwin-amd64/prometheus.yml

• Added insecure_skip_verify: true to tell Prometheus not to check the ssl certificate.

# my global config
global:
  scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
    - static_configs:
        - targets:
          # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

scrape_configs:
- job_name: minio-job
  bearer_token: eyJhbGciOiJIUzUxMiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJwcm9tZXRoZXVzIiwic3ViIjoiY29uc29sZSIsImV4cCI6NDgzNDQxNzM3OH0.T3P7jzQNFm5c0TgL4djMMx4IfWIBXBvz78fu_R0DMuWb11ydZvsLptA29gC3CpengKDw7DiurImTxarmGOnPiA
  metrics_path: /minio/v2/metrics/cluster
  scheme: https
  static_configs:
  - targets: ['localhost:30081']
  tls_config:
    insecure_skip_verify: true

4. Start Prometheus with Above configuration:

• 192.168.0.13 is the IP address of my MacBook Pro in my net at home.

cd ~/prometheus-2.36.0.darwin-amd64
./prometheus --config.file=prometheus.yml --web.listen-address 192.168.0.13:9091

• My public IP address is exposing my Prometheus: http://72.140.145.27/graph?g0.expr=&g0.tab=1&g0.stacked=0&g0.show_exemplars=0&g0.range_input=1h

5. To verify Prometheus is reachable, look into Console:

• Exposed console via NodePort:

• Page: https://localhost:30082/tools/metrics

• User: console

• Password: console123

• Go into Monitoring>Metrics

6. Prometheus has its own page:

• Page: http://72.140.145.27
• List of Metrics: https://github.com/minio/minio/blob/master/docs/metrics/prometheus/list.md
• Example: minio_node_disk_total

minio_node_disk_total{instance="localhost:30081", job="minio-job", server="myminio-pool-0-0.myminio-hl.tenant-lite.svc.cluster.local:9000"} 2
minio_node_disk_total{instance="localhost:30081", job="minio-job", server="myminio-pool-0-1.myminio-hl.tenant-lite.svc.cluster.local:9000"} 2
minio_node_disk_total{instance="localhost:30081", job="minio-job", server="myminio-pool-0-2.myminio-hl.tenant-lite.svc.cluster.local:9000"} 2
minio_node_disk_total{instance="localhost:30081", job="minio-job", server="myminio-pool-0-3.myminio-hl.tenant-lite.svc.cluster.local:9000"} 2

Another way to scrape the metrics with curl:

curl -k \
	https://10.244.2.7:9000/minio/v2/metrics/cluster \
	-H "Authorization: Bearer eyJhbGciOiJIUzUxMiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJwcm9tZXRoZXVzIiwic3ViIjoiY29uc29sZSIsImV4cCI6NDgzNDQwNzg2NH0.VxQAW-sUfeBhpGTVsLcYbtkpV9h4IPa_djl7rfUyX42gowKGBs9drHsFlPQeZN0yA0TjH0gEddZmMiDZ3ATg8Q"

curl -k \
	https://localhost:30081/minio/v2/metrics/cluster \
	-H "Authorization: Bearer eyJhbGciOiJIUzUxMiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJwcm9tZXRoZXVzIiwic3ViIjoiY29uc29sZSIsImV4cCI6NDgzNDQwNzg2NH0.VxQAW-sUfeBhpGTVsLcYbtkpV9h4IPa_djl7rfUyX42gowKGBs9drHsFlPQeZN0yA0TjH0gEddZmMiDZ3ATg8Q"

curl -k \
	https://127.0.0.1:30081/minio/v2/metrics/cluster \
	-H "Authorization: Bearer eyJhbGciOiJIUzUxMiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJwcm9tZXRoZXVzIiwic3ViIjoiY29uc29sZSIsImV4cCI6NDgzNDQwNzg2NH0.VxQAW-sUfeBhpGTVsLcYbtkpV9h4IPa_djl7rfUyX42gowKGBs9drHsFlPQeZN0yA0TjH0gEddZmMiDZ3ATg8Q"

curl -k \
	https://192.168.0.13:30081/minio/v2/metrics/cluster \
	-H "Authorization: Bearer eyJhbGciOiJIUzUxMiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJwcm9tZXRoZXVzIiwic3ViIjoiY29uc29sZSIsImV4cCI6NDgzNDQwNzg2NH0.VxQAW-sUfeBhpGTVsLcYbtkpV9h4IPa_djl7rfUyX42gowKGBs9drHsFlPQeZN0yA0TjH0gEddZmMiDZ3ATg8Q"

curl -k \
	https://localhost:30081/minio/v2/metrics/cluster \
	-H "Authorization: Bearer eyJhbGciOiJIUzUxMiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJwcm9tZXRoZXVzIiwic3ViIjoiY29uc29sZSIsImV4cCI6NDgzNDQxNzM3OH0.T3P7jzQNFm5c0TgL4djMMx4IfWIBXBvz78fu_R0DMuWb11ydZvsLptA29gC3CpengKDw7DiurImTxarmGOnPiA"

1. -k to trust insecure connection
2. Authorization Token comes from the mc command that gives the scrape
3. IP adderss can comes from the MinIO Pod if you wish
4. In port 9000 is where MinIO is being executed
5. /minio/v2/metrics/cluster is where the metrics comes from
6. This way you can get metrics as well:

# HELP minio_audit_failed_messages Total number of messages that failed to send since start
# TYPE minio_audit_failed_messages counter
minio_audit_failed_messages{server="myminio-pool-0-0.myminio-hl.tenant-lite.svc.cluster.local:9000",target_id="sys_console_0"} 0
minio_audit_failed_messages{server="myminio-pool-0-1.myminio-hl.tenant-lite.svc.cluster.local:9000",target_id="sys_console_0"} 0
minio_audit_failed_messages{server="myminio-pool-0-2.myminio-hl.tenant-lite.svc.cluster.local:9000",target_id="sys_console_0"} 0
minio_audit_failed_messages{server="myminio-pool-0-3.myminio-hl.tenant-lite.svc.cluster.local:9000",target_id="sys_console_0"} 0
# HELP minio_audit_target_queue_length Number of unsent messages in queue for target
# TYPE minio_audit_target_queue_length gauge
minio_audit_target_queue_length{server="myminio-pool-0-0.myminio-hl.tenant-lite.svc.cluster.local:9000",target_id="sys_console_0"} 0
minio_audit_target_queue_length{server="myminio-pool-0-1.myminio-hl.tenant-lite.svc.cluster.local:9000",target_id="sys_console_0"} 0
minio_audit_target_queue_length{server="myminio-pool-0-2.myminio-hl.tenant-lite.svc.cluster.local:9000",target_id="sys_console_0"} 0
minio_audit_target_queue_length{server="myminio-pool-0-3.myminio-hl.tenant-lite.svc.cluster.local:9000",target_id="sys_console_0"} 0
# HELP minio_audit_total_messages Total number of messages sent since start
# TYPE minio_audit_total_messages counter
minio_audit_total_messages{server="myminio-pool-0-0.myminio-hl.tenant-lite.svc.cluster.local:9000",target_id="sys_console_0"} 2
minio_audit_total_messages{server="myminio-pool-0-1.myminio-hl.tenant-lite.svc.cluster.local:9000",target_id="sys_console_0"} 2
minio_audit_total_messages{server="myminio-pool-0-2.myminio-hl.tenant-lite.svc.cluster.local:9000",target_id="sys_console_0"} 6
minio_audit_total_messages{server="myminio-pool-0-3.myminio-hl.tenant-lite.svc.cluster.local:9000",target_id="sys_console_0"} 6
# HELP minio_cluster_capacity_raw_free_bytes Total free capacity online in the cluster
# TYPE minio_cluster_capacity_raw_free_bytes gauge
minio_cluster_capacity_raw_free_bytes{server="myminio-pool-0-0.myminio-hl.tenant-lite.svc.cluster.local:9000"} 2.8134309888e+11
# HELP minio_cluster_capacity_raw_total_bytes Total capacity online in the cluster
# TYPE minio_cluster_capacity_raw_total_bytes gauge
minio_cluster_capacity_raw_total_bytes{server="myminio-pool-0-0.myminio-hl.tenant-lite.svc.cluster.local:9000"} 4.7563505664e+11
# HELP minio_cluster_capacity_usable_free_bytes Total free usable capacity online in the cluster
# TYPE minio_cluster_capacity_usable_free_bytes gauge
minio_cluster_capacity_usable_free_bytes{server="myminio-pool-0-0.myminio-hl.tenant-lite.svc.cluster.local:9000"} 2.1100732416e+11
# HELP minio_cluster_capacity_usable_total_bytes Total usable capacity online in the cluster
# TYPE minio_cluster_capacity_usable_total_bytes gauge
minio_cluster_capacity_usable_total_bytes{server="myminio-pool-0-0.myminio-hl.tenant-lite.svc.cluster.local:9000"} 3.5672629248e+11
# HELP minio_cluster_disk_offline_total Total drives offline
# TYPE minio_cluster_disk_offline_total gauge
minio_cluster_disk_offline_total{server="myminio-pool-0-0.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
# HELP minio_cluster_disk_online_total Total drives online
# TYPE minio_cluster_disk_online_total gauge
minio_cluster_disk_online_total{server="myminio-pool-0-0.myminio-hl.tenant-lite.svc.cluster.local:9000"} 8
# HELP minio_cluster_disk_total Total drives
# TYPE minio_cluster_disk_total gauge
minio_cluster_disk_total{server="myminio-pool-0-0.myminio-hl.tenant-lite.svc.cluster.local:9000"} 8
# HELP minio_cluster_nodes_offline_total Total number of MinIO nodes offline
# TYPE minio_cluster_nodes_offline_total gauge
minio_cluster_nodes_offline_total{server="myminio-pool-0-0.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
# HELP minio_cluster_nodes_online_total Total number of MinIO nodes online
# TYPE minio_cluster_nodes_online_total gauge
minio_cluster_nodes_online_total{server="myminio-pool-0-0.myminio-hl.tenant-lite.svc.cluster.local:9000"} 4
# HELP minio_heal_objects_heal_total Objects healed in current self healing run
# TYPE minio_heal_objects_heal_total gauge
minio_heal_objects_heal_total{server="myminio-pool-0-0.myminio-hl.tenant-lite.svc.cluster.local:9000",type="object"} 6
minio_heal_objects_heal_total{server="myminio-pool-0-1.myminio-hl.tenant-lite.svc.cluster.local:9000",type="bucket"} 1
minio_heal_objects_heal_total{server="myminio-pool-0-1.myminio-hl.tenant-lite.svc.cluster.local:9000",type="object"} 6
minio_heal_objects_heal_total{server="myminio-pool-0-2.myminio-hl.tenant-lite.svc.cluster.local:9000",type="bucket"} 1
minio_heal_objects_heal_total{server="myminio-pool-0-2.myminio-hl.tenant-lite.svc.cluster.local:9000",type="object"} 6
minio_heal_objects_heal_total{server="myminio-pool-0-3.myminio-hl.tenant-lite.svc.cluster.local:9000",type="bucket"} 1
minio_heal_objects_heal_total{server="myminio-pool-0-3.myminio-hl.tenant-lite.svc.cluster.local:9000",type="object"} 6
# HELP minio_heal_objects_total Objects scanned in current self healing run
# TYPE minio_heal_objects_total gauge
minio_heal_objects_total{server="myminio-pool-0-0.myminio-hl.tenant-lite.svc.cluster.local:9000",type="bucket"} 1
minio_heal_objects_total{server="myminio-pool-0-0.myminio-hl.tenant-lite.svc.cluster.local:9000",type="bucket-metadata"} 6
minio_heal_objects_total{server="myminio-pool-0-1.myminio-hl.tenant-lite.svc.cluster.local:9000",type="bucket"} 1
minio_heal_objects_total{server="myminio-pool-0-1.myminio-hl.tenant-lite.svc.cluster.local:9000",type="bucket-metadata"} 6
minio_heal_objects_total{server="myminio-pool-0-1.myminio-hl.tenant-lite.svc.cluster.local:9000",type="object"} 1
minio_heal_objects_total{server="myminio-pool-0-2.myminio-hl.tenant-lite.svc.cluster.local:9000",type="bucket"} 1
minio_heal_objects_total{server="myminio-pool-0-2.myminio-hl.tenant-lite.svc.cluster.local:9000",type="bucket-metadata"} 6
minio_heal_objects_total{server="myminio-pool-0-2.myminio-hl.tenant-lite.svc.cluster.local:9000",type="object"} 1
minio_heal_objects_total{server="myminio-pool-0-3.myminio-hl.tenant-lite.svc.cluster.local:9000",type="bucket"} 1
minio_heal_objects_total{server="myminio-pool-0-3.myminio-hl.tenant-lite.svc.cluster.local:9000",type="bucket-metadata"} 6
# HELP minio_heal_time_last_activity_nano_seconds Time elapsed (in nano seconds) since last self healing activity. This is set to -1 until initial self heal activity
# TYPE minio_heal_time_last_activity_nano_seconds gauge
minio_heal_time_last_activity_nano_seconds{server="myminio-pool-0-0.myminio-hl.tenant-lite.svc.cluster.local:9000"} 4.14486732289e+11
minio_heal_time_last_activity_nano_seconds{server="myminio-pool-0-1.myminio-hl.tenant-lite.svc.cluster.local:9000"} 4.14980798914e+11
minio_heal_time_last_activity_nano_seconds{server="myminio-pool-0-2.myminio-hl.tenant-lite.svc.cluster.local:9000"} 4.52299376376e+11
minio_heal_time_last_activity_nano_seconds{server="myminio-pool-0-3.myminio-hl.tenant-lite.svc.cluster.local:9000"} 4.02913377006e+11
# HELP minio_inter_node_traffic_dial_avg_time Average time of internodes TCP dial calls
# TYPE minio_inter_node_traffic_dial_avg_time gauge
minio_inter_node_traffic_dial_avg_time{server="myminio-pool-0-0.myminio-hl.tenant-lite.svc.cluster.local:9000"} 1.375792e+06
minio_inter_node_traffic_dial_avg_time{server="myminio-pool-0-1.myminio-hl.tenant-lite.svc.cluster.local:9000"} 608599
minio_inter_node_traffic_dial_avg_time{server="myminio-pool-0-2.myminio-hl.tenant-lite.svc.cluster.local:9000"} 346794
minio_inter_node_traffic_dial_avg_time{server="myminio-pool-0-3.myminio-hl.tenant-lite.svc.cluster.local:9000"} 496133
# HELP minio_inter_node_traffic_dial_errors Total number of internode TCP dial timeouts and errors
# TYPE minio_inter_node_traffic_dial_errors counter
minio_inter_node_traffic_dial_errors{server="myminio-pool-0-0.myminio-hl.tenant-lite.svc.cluster.local:9000"} 7
minio_inter_node_traffic_dial_errors{server="myminio-pool-0-1.myminio-hl.tenant-lite.svc.cluster.local:9000"} 5
minio_inter_node_traffic_dial_errors{server="myminio-pool-0-2.myminio-hl.tenant-lite.svc.cluster.local:9000"} 140
minio_inter_node_traffic_dial_errors{server="myminio-pool-0-3.myminio-hl.tenant-lite.svc.cluster.local:9000"} 83
# HELP minio_inter_node_traffic_errors_total Total number of failed internode calls
# TYPE minio_inter_node_traffic_errors_total counter
minio_inter_node_traffic_errors_total{server="myminio-pool-0-0.myminio-hl.tenant-lite.svc.cluster.local:9000"} 6
minio_inter_node_traffic_errors_total{server="myminio-pool-0-1.myminio-hl.tenant-lite.svc.cluster.local:9000"} 4
minio_inter_node_traffic_errors_total{server="myminio-pool-0-2.myminio-hl.tenant-lite.svc.cluster.local:9000"} 7
minio_inter_node_traffic_errors_total{server="myminio-pool-0-3.myminio-hl.tenant-lite.svc.cluster.local:9000"} 21
# HELP minio_inter_node_traffic_received_bytes Total number of bytes received from other peer nodes
# TYPE minio_inter_node_traffic_received_bytes counter
minio_inter_node_traffic_received_bytes{server="myminio-pool-0-0.myminio-hl.tenant-lite.svc.cluster.local:9000"} 64297
minio_inter_node_traffic_received_bytes{server="myminio-pool-0-1.myminio-hl.tenant-lite.svc.cluster.local:9000"} 77409
minio_inter_node_traffic_received_bytes{server="myminio-pool-0-2.myminio-hl.tenant-lite.svc.cluster.local:9000"} 54431
minio_inter_node_traffic_received_bytes{server="myminio-pool-0-3.myminio-hl.tenant-lite.svc.cluster.local:9000"} 92280
# HELP minio_inter_node_traffic_sent_bytes Total number of bytes sent to the other peer nodes
# TYPE minio_inter_node_traffic_sent_bytes counter
minio_inter_node_traffic_sent_bytes{server="myminio-pool-0-0.myminio-hl.tenant-lite.svc.cluster.local:9000"} 106248
minio_inter_node_traffic_sent_bytes{server="myminio-pool-0-1.myminio-hl.tenant-lite.svc.cluster.local:9000"} 147226
minio_inter_node_traffic_sent_bytes{server="myminio-pool-0-2.myminio-hl.tenant-lite.svc.cluster.local:9000"} 175375
minio_inter_node_traffic_sent_bytes{server="myminio-pool-0-3.myminio-hl.tenant-lite.svc.cluster.local:9000"} 216857
# HELP minio_node_disk_free_bytes Total storage available on a drive
# TYPE minio_node_disk_free_bytes gauge
minio_node_disk_free_bytes{disk="/export0",server="myminio-pool-0-0.myminio-hl.tenant-lite.svc.cluster.local:9000"} 3.516788736e+10
minio_node_disk_free_bytes{disk="/export0",server="myminio-pool-0-1.myminio-hl.tenant-lite.svc.cluster.local:9000"} 3.516788736e+10
minio_node_disk_free_bytes{disk="/export0",server="myminio-pool-0-2.myminio-hl.tenant-lite.svc.cluster.local:9000"} 3.516788736e+10
minio_node_disk_free_bytes{disk="/export0",server="myminio-pool-0-3.myminio-hl.tenant-lite.svc.cluster.local:9000"} 3.516788736e+10
minio_node_disk_free_bytes{disk="/export1",server="myminio-pool-0-0.myminio-hl.tenant-lite.svc.cluster.local:9000"} 3.516788736e+10
minio_node_disk_free_bytes{disk="/export1",server="myminio-pool-0-1.myminio-hl.tenant-lite.svc.cluster.local:9000"} 3.516788736e+10
minio_node_disk_free_bytes{disk="/export1",server="myminio-pool-0-2.myminio-hl.tenant-lite.svc.cluster.local:9000"} 3.516788736e+10
minio_node_disk_free_bytes{disk="/export1",server="myminio-pool-0-3.myminio-hl.tenant-lite.svc.cluster.local:9000"} 3.516788736e+10
# HELP minio_node_disk_free_inodes Total free inodes
# TYPE minio_node_disk_free_inodes gauge
minio_node_disk_free_inodes{disk="/export0",server="myminio-pool-0-0.myminio-hl.tenant-lite.svc.cluster.local:9000"} 3.609993e+06
minio_node_disk_free_inodes{disk="/export0",server="myminio-pool-0-1.myminio-hl.tenant-lite.svc.cluster.local:9000"} 3.609993e+06
minio_node_disk_free_inodes{disk="/export0",server="myminio-pool-0-2.myminio-hl.tenant-lite.svc.cluster.local:9000"} 3.609993e+06
minio_node_disk_free_inodes{disk="/export0",server="myminio-pool-0-3.myminio-hl.tenant-lite.svc.cluster.local:9000"} 3.609993e+06
minio_node_disk_free_inodes{disk="/export1",server="myminio-pool-0-0.myminio-hl.tenant-lite.svc.cluster.local:9000"} 3.609993e+06
minio_node_disk_free_inodes{disk="/export1",server="myminio-pool-0-1.myminio-hl.tenant-lite.svc.cluster.local:9000"} 3.609993e+06
minio_node_disk_free_inodes{disk="/export1",server="myminio-pool-0-2.myminio-hl.tenant-lite.svc.cluster.local:9000"} 3.609993e+06
minio_node_disk_free_inodes{disk="/export1",server="myminio-pool-0-3.myminio-hl.tenant-lite.svc.cluster.local:9000"} 3.609993e+06
# HELP minio_node_disk_offline_total Total drives offline
# TYPE minio_node_disk_offline_total gauge
minio_node_disk_offline_total{server="myminio-pool-0-0.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
minio_node_disk_offline_total{server="myminio-pool-0-1.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
minio_node_disk_offline_total{server="myminio-pool-0-2.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
minio_node_disk_offline_total{server="myminio-pool-0-3.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
# HELP minio_node_disk_online_total Total drives online
# TYPE minio_node_disk_online_total gauge
minio_node_disk_online_total{server="myminio-pool-0-0.myminio-hl.tenant-lite.svc.cluster.local:9000"} 2
minio_node_disk_online_total{server="myminio-pool-0-1.myminio-hl.tenant-lite.svc.cluster.local:9000"} 2
minio_node_disk_online_total{server="myminio-pool-0-2.myminio-hl.tenant-lite.svc.cluster.local:9000"} 2
minio_node_disk_online_total{server="myminio-pool-0-3.myminio-hl.tenant-lite.svc.cluster.local:9000"} 2
# HELP minio_node_disk_total Total drives
# TYPE minio_node_disk_total gauge
minio_node_disk_total{server="myminio-pool-0-0.myminio-hl.tenant-lite.svc.cluster.local:9000"} 2
minio_node_disk_total{server="myminio-pool-0-1.myminio-hl.tenant-lite.svc.cluster.local:9000"} 2
minio_node_disk_total{server="myminio-pool-0-2.myminio-hl.tenant-lite.svc.cluster.local:9000"} 2
minio_node_disk_total{server="myminio-pool-0-3.myminio-hl.tenant-lite.svc.cluster.local:9000"} 2
# HELP minio_node_disk_total_bytes Total storage on a drive
# TYPE minio_node_disk_total_bytes gauge
minio_node_disk_total_bytes{disk="/export0",server="myminio-pool-0-0.myminio-hl.tenant-lite.svc.cluster.local:9000"} 5.945438208e+10
minio_node_disk_total_bytes{disk="/export0",server="myminio-pool-0-1.myminio-hl.tenant-lite.svc.cluster.local:9000"} 5.945438208e+10
minio_node_disk_total_bytes{disk="/export0",server="myminio-pool-0-2.myminio-hl.tenant-lite.svc.cluster.local:9000"} 5.945438208e+10
minio_node_disk_total_bytes{disk="/export0",server="myminio-pool-0-3.myminio-hl.tenant-lite.svc.cluster.local:9000"} 5.945438208e+10
minio_node_disk_total_bytes{disk="/export1",server="myminio-pool-0-0.myminio-hl.tenant-lite.svc.cluster.local:9000"} 5.945438208e+10
minio_node_disk_total_bytes{disk="/export1",server="myminio-pool-0-1.myminio-hl.tenant-lite.svc.cluster.local:9000"} 5.945438208e+10
minio_node_disk_total_bytes{disk="/export1",server="myminio-pool-0-2.myminio-hl.tenant-lite.svc.cluster.local:9000"} 5.945438208e+10
minio_node_disk_total_bytes{disk="/export1",server="myminio-pool-0-3.myminio-hl.tenant-lite.svc.cluster.local:9000"} 5.945438208e+10
# HELP minio_node_disk_used_bytes Total storage used on a drive
# TYPE minio_node_disk_used_bytes gauge
minio_node_disk_used_bytes{disk="/export0",server="myminio-pool-0-0.myminio-hl.tenant-lite.svc.cluster.local:9000"} 2.428649472e+10
minio_node_disk_used_bytes{disk="/export0",server="myminio-pool-0-1.myminio-hl.tenant-lite.svc.cluster.local:9000"} 2.428649472e+10
minio_node_disk_used_bytes{disk="/export0",server="myminio-pool-0-2.myminio-hl.tenant-lite.svc.cluster.local:9000"} 2.428649472e+10
minio_node_disk_used_bytes{disk="/export0",server="myminio-pool-0-3.myminio-hl.tenant-lite.svc.cluster.local:9000"} 2.428649472e+10
minio_node_disk_used_bytes{disk="/export1",server="myminio-pool-0-0.myminio-hl.tenant-lite.svc.cluster.local:9000"} 2.428649472e+10
minio_node_disk_used_bytes{disk="/export1",server="myminio-pool-0-1.myminio-hl.tenant-lite.svc.cluster.local:9000"} 2.428649472e+10
minio_node_disk_used_bytes{disk="/export1",server="myminio-pool-0-2.myminio-hl.tenant-lite.svc.cluster.local:9000"} 2.428649472e+10
minio_node_disk_used_bytes{disk="/export1",server="myminio-pool-0-3.myminio-hl.tenant-lite.svc.cluster.local:9000"} 2.428649472e+10
# HELP minio_node_file_descriptor_limit_total Limit on total number of open file descriptors for the MinIO Server process
# TYPE minio_node_file_descriptor_limit_total gauge
minio_node_file_descriptor_limit_total{server="myminio-pool-0-0.myminio-hl.tenant-lite.svc.cluster.local:9000"} 1.048576e+06
minio_node_file_descriptor_limit_total{server="myminio-pool-0-1.myminio-hl.tenant-lite.svc.cluster.local:9000"} 1.048576e+06
minio_node_file_descriptor_limit_total{server="myminio-pool-0-2.myminio-hl.tenant-lite.svc.cluster.local:9000"} 1.048576e+06
minio_node_file_descriptor_limit_total{server="myminio-pool-0-3.myminio-hl.tenant-lite.svc.cluster.local:9000"} 1.048576e+06
# HELP minio_node_file_descriptor_open_total Total number of open file descriptors by the MinIO Server process
# TYPE minio_node_file_descriptor_open_total gauge
minio_node_file_descriptor_open_total{server="myminio-pool-0-0.myminio-hl.tenant-lite.svc.cluster.local:9000"} 21
minio_node_file_descriptor_open_total{server="myminio-pool-0-1.myminio-hl.tenant-lite.svc.cluster.local:9000"} 20
minio_node_file_descriptor_open_total{server="myminio-pool-0-2.myminio-hl.tenant-lite.svc.cluster.local:9000"} 21
minio_node_file_descriptor_open_total{server="myminio-pool-0-3.myminio-hl.tenant-lite.svc.cluster.local:9000"} 18
# HELP minio_node_go_routine_total Total number of go routines running
# TYPE minio_node_go_routine_total gauge
minio_node_go_routine_total{server="myminio-pool-0-0.myminio-hl.tenant-lite.svc.cluster.local:9000"} 378
minio_node_go_routine_total{server="myminio-pool-0-1.myminio-hl.tenant-lite.svc.cluster.local:9000"} 373
minio_node_go_routine_total{server="myminio-pool-0-2.myminio-hl.tenant-lite.svc.cluster.local:9000"} 375
minio_node_go_routine_total{server="myminio-pool-0-3.myminio-hl.tenant-lite.svc.cluster.local:9000"} 371
# HELP minio_node_iam_last_sync_duration_millis Last successful IAM data sync duration in milliseconds
# TYPE minio_node_iam_last_sync_duration_millis gauge
minio_node_iam_last_sync_duration_millis{server="myminio-pool-0-0.myminio-hl.tenant-lite.svc.cluster.local:9000"} 10
minio_node_iam_last_sync_duration_millis{server="myminio-pool-0-1.myminio-hl.tenant-lite.svc.cluster.local:9000"} 26
minio_node_iam_last_sync_duration_millis{server="myminio-pool-0-2.myminio-hl.tenant-lite.svc.cluster.local:9000"} 16
minio_node_iam_last_sync_duration_millis{server="myminio-pool-0-3.myminio-hl.tenant-lite.svc.cluster.local:9000"} 10025
# HELP minio_node_iam_plugin_authn_service_failed_requests_minute When plugin authentication is configured, returns failed requests count in the last full minute
# TYPE minio_node_iam_plugin_authn_service_failed_requests_minute gauge
minio_node_iam_plugin_authn_service_failed_requests_minute{server="myminio-pool-0-0.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
minio_node_iam_plugin_authn_service_failed_requests_minute{server="myminio-pool-0-1.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
minio_node_iam_plugin_authn_service_failed_requests_minute{server="myminio-pool-0-2.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
minio_node_iam_plugin_authn_service_failed_requests_minute{server="myminio-pool-0-3.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
# HELP minio_node_iam_plugin_authn_service_last_fail_seconds When plugin authentication is configured, returns time (in seconds) since the last failed request to the service
# TYPE minio_node_iam_plugin_authn_service_last_fail_seconds gauge
minio_node_iam_plugin_authn_service_last_fail_seconds{server="myminio-pool-0-0.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
minio_node_iam_plugin_authn_service_last_fail_seconds{server="myminio-pool-0-1.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
minio_node_iam_plugin_authn_service_last_fail_seconds{server="myminio-pool-0-2.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
minio_node_iam_plugin_authn_service_last_fail_seconds{server="myminio-pool-0-3.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
# HELP minio_node_iam_plugin_authn_service_last_succ_seconds When plugin authentication is configured, returns time (in seconds) since the last successful request to the service
# TYPE minio_node_iam_plugin_authn_service_last_succ_seconds gauge
minio_node_iam_plugin_authn_service_last_succ_seconds{server="myminio-pool-0-0.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
minio_node_iam_plugin_authn_service_last_succ_seconds{server="myminio-pool-0-1.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
minio_node_iam_plugin_authn_service_last_succ_seconds{server="myminio-pool-0-2.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
minio_node_iam_plugin_authn_service_last_succ_seconds{server="myminio-pool-0-3.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
# HELP minio_node_iam_plugin_authn_service_succ_avg_rtt_ms_minute When plugin authentication is configured, returns average round-trip-time of successful requests in the last full minute
# TYPE minio_node_iam_plugin_authn_service_succ_avg_rtt_ms_minute gauge
minio_node_iam_plugin_authn_service_succ_avg_rtt_ms_minute{server="myminio-pool-0-0.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
minio_node_iam_plugin_authn_service_succ_avg_rtt_ms_minute{server="myminio-pool-0-1.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
minio_node_iam_plugin_authn_service_succ_avg_rtt_ms_minute{server="myminio-pool-0-2.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
minio_node_iam_plugin_authn_service_succ_avg_rtt_ms_minute{server="myminio-pool-0-3.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
# HELP minio_node_iam_plugin_authn_service_succ_max_rtt_ms_minute When plugin authentication is configured, returns maximum round-trip-time of successful requests in the last full minute
# TYPE minio_node_iam_plugin_authn_service_succ_max_rtt_ms_minute gauge
minio_node_iam_plugin_authn_service_succ_max_rtt_ms_minute{server="myminio-pool-0-0.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
minio_node_iam_plugin_authn_service_succ_max_rtt_ms_minute{server="myminio-pool-0-1.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
minio_node_iam_plugin_authn_service_succ_max_rtt_ms_minute{server="myminio-pool-0-2.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
minio_node_iam_plugin_authn_service_succ_max_rtt_ms_minute{server="myminio-pool-0-3.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
# HELP minio_node_iam_plugin_authn_service_total_requests_minute When plugin authentication is configured, returns total requests count in the last full minute
# TYPE minio_node_iam_plugin_authn_service_total_requests_minute gauge
minio_node_iam_plugin_authn_service_total_requests_minute{server="myminio-pool-0-0.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
minio_node_iam_plugin_authn_service_total_requests_minute{server="myminio-pool-0-1.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
minio_node_iam_plugin_authn_service_total_requests_minute{server="myminio-pool-0-2.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
minio_node_iam_plugin_authn_service_total_requests_minute{server="myminio-pool-0-3.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
# HELP minio_node_iam_since_last_sync_millis Time (in milliseconds) since last successful IAM data sync. This is set to 0 until the first sync after server start.
# TYPE minio_node_iam_since_last_sync_millis gauge
minio_node_iam_since_last_sync_millis{server="myminio-pool-0-0.myminio-hl.tenant-lite.svc.cluster.local:9000"} 414325
minio_node_iam_since_last_sync_millis{server="myminio-pool-0-1.myminio-hl.tenant-lite.svc.cluster.local:9000"} 418453
minio_node_iam_since_last_sync_millis{server="myminio-pool-0-2.myminio-hl.tenant-lite.svc.cluster.local:9000"} 452556
minio_node_iam_since_last_sync_millis{server="myminio-pool-0-3.myminio-hl.tenant-lite.svc.cluster.local:9000"} 397755
# HELP minio_node_iam_sync_failures Number of failed IAM data syncs since server start.
# TYPE minio_node_iam_sync_failures counter
minio_node_iam_sync_failures{server="myminio-pool-0-0.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
minio_node_iam_sync_failures{server="myminio-pool-0-1.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
minio_node_iam_sync_failures{server="myminio-pool-0-2.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
minio_node_iam_sync_failures{server="myminio-pool-0-3.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
# HELP minio_node_iam_sync_successes Number of successful IAM data syncs since server start.
# TYPE minio_node_iam_sync_successes counter
minio_node_iam_sync_successes{server="myminio-pool-0-0.myminio-hl.tenant-lite.svc.cluster.local:9000"} 1
minio_node_iam_sync_successes{server="myminio-pool-0-1.myminio-hl.tenant-lite.svc.cluster.local:9000"} 1
minio_node_iam_sync_successes{server="myminio-pool-0-2.myminio-hl.tenant-lite.svc.cluster.local:9000"} 1
minio_node_iam_sync_successes{server="myminio-pool-0-3.myminio-hl.tenant-lite.svc.cluster.local:9000"} 1
# HELP minio_node_ilm_expiry_pending_tasks Number of pending ILM expiry tasks in the queue
# TYPE minio_node_ilm_expiry_pending_tasks gauge
minio_node_ilm_expiry_pending_tasks{server="myminio-pool-0-0.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
minio_node_ilm_expiry_pending_tasks{server="myminio-pool-0-1.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
minio_node_ilm_expiry_pending_tasks{server="myminio-pool-0-2.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
minio_node_ilm_expiry_pending_tasks{server="myminio-pool-0-3.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
# HELP minio_node_ilm_transition_active_tasks Number of active ILM transition tasks
# TYPE minio_node_ilm_transition_active_tasks gauge
minio_node_ilm_transition_active_tasks{server="myminio-pool-0-0.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
minio_node_ilm_transition_active_tasks{server="myminio-pool-0-1.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
minio_node_ilm_transition_active_tasks{server="myminio-pool-0-2.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
minio_node_ilm_transition_active_tasks{server="myminio-pool-0-3.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
# HELP minio_node_ilm_transition_pending_tasks Number of pending ILM transition tasks in the queue
# TYPE minio_node_ilm_transition_pending_tasks gauge
minio_node_ilm_transition_pending_tasks{server="myminio-pool-0-0.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
minio_node_ilm_transition_pending_tasks{server="myminio-pool-0-1.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
minio_node_ilm_transition_pending_tasks{server="myminio-pool-0-2.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
minio_node_ilm_transition_pending_tasks{server="myminio-pool-0-3.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
# HELP minio_node_ilm_versions_scanned Total number of object versions checked for ilm actions since server start
# TYPE minio_node_ilm_versions_scanned counter
minio_node_ilm_versions_scanned{server="myminio-pool-0-0.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
minio_node_ilm_versions_scanned{server="myminio-pool-0-1.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
minio_node_ilm_versions_scanned{server="myminio-pool-0-2.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
minio_node_ilm_versions_scanned{server="myminio-pool-0-3.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
# HELP minio_node_io_rchar_bytes Total bytes read by the process from the underlying storage system including cache, /proc/[pid]/io rchar
# TYPE minio_node_io_rchar_bytes counter
minio_node_io_rchar_bytes{server="myminio-pool-0-0.myminio-hl.tenant-lite.svc.cluster.local:9000"} 3.110057e+06
minio_node_io_rchar_bytes{server="myminio-pool-0-1.myminio-hl.tenant-lite.svc.cluster.local:9000"} 3.212771e+06
minio_node_io_rchar_bytes{server="myminio-pool-0-2.myminio-hl.tenant-lite.svc.cluster.local:9000"} 3.606326e+06
minio_node_io_rchar_bytes{server="myminio-pool-0-3.myminio-hl.tenant-lite.svc.cluster.local:9000"} 3.591005e+06
# HELP minio_node_io_read_bytes Total bytes read by the process from the underlying storage system, /proc/[pid]/io read_bytes
# TYPE minio_node_io_read_bytes counter
minio_node_io_read_bytes{server="myminio-pool-0-0.myminio-hl.tenant-lite.svc.cluster.local:9000"} 712704
minio_node_io_read_bytes{server="myminio-pool-0-1.myminio-hl.tenant-lite.svc.cluster.local:9000"} 720896
minio_node_io_read_bytes{server="myminio-pool-0-2.myminio-hl.tenant-lite.svc.cluster.local:9000"} 901120
minio_node_io_read_bytes{server="myminio-pool-0-3.myminio-hl.tenant-lite.svc.cluster.local:9000"} 1.073152e+06
# HELP minio_node_io_wchar_bytes Total bytes written by the process to the underlying storage system including page cache, /proc/[pid]/io wchar
# TYPE minio_node_io_wchar_bytes counter
minio_node_io_wchar_bytes{server="myminio-pool-0-0.myminio-hl.tenant-lite.svc.cluster.local:9000"} 1.998748e+06
minio_node_io_wchar_bytes{server="myminio-pool-0-1.myminio-hl.tenant-lite.svc.cluster.local:9000"} 1.3259889e+07
minio_node_io_wchar_bytes{server="myminio-pool-0-2.myminio-hl.tenant-lite.svc.cluster.local:9000"} 1.3804296e+07
minio_node_io_wchar_bytes{server="myminio-pool-0-3.myminio-hl.tenant-lite.svc.cluster.local:9000"} 6.052303e+06
# HELP minio_node_io_write_bytes Total bytes written by the process to the underlying storage system, /proc/[pid]/io write_bytes
# TYPE minio_node_io_write_bytes counter
minio_node_io_write_bytes{server="myminio-pool-0-0.myminio-hl.tenant-lite.svc.cluster.local:9000"} 143360
minio_node_io_write_bytes{server="myminio-pool-0-1.myminio-hl.tenant-lite.svc.cluster.local:9000"} 143360
minio_node_io_write_bytes{server="myminio-pool-0-2.myminio-hl.tenant-lite.svc.cluster.local:9000"} 143360
minio_node_io_write_bytes{server="myminio-pool-0-3.myminio-hl.tenant-lite.svc.cluster.local:9000"} 159744
# HELP minio_node_process_cpu_total_seconds Total user and system CPU time spent in seconds
# TYPE minio_node_process_cpu_total_seconds counter
minio_node_process_cpu_total_seconds{server="myminio-pool-0-0.myminio-hl.tenant-lite.svc.cluster.local:9000"} 2.17
minio_node_process_cpu_total_seconds{server="myminio-pool-0-1.myminio-hl.tenant-lite.svc.cluster.local:9000"} 2.27
minio_node_process_cpu_total_seconds{server="myminio-pool-0-2.myminio-hl.tenant-lite.svc.cluster.local:9000"} 3.03
minio_node_process_cpu_total_seconds{server="myminio-pool-0-3.myminio-hl.tenant-lite.svc.cluster.local:9000"} 2.53
# HELP minio_node_process_resident_memory_bytes Resident memory size in bytes
# TYPE minio_node_process_resident_memory_bytes gauge
minio_node_process_resident_memory_bytes{server="myminio-pool-0-0.myminio-hl.tenant-lite.svc.cluster.local:9000"} 1.36466432e+08
minio_node_process_resident_memory_bytes{server="myminio-pool-0-1.myminio-hl.tenant-lite.svc.cluster.local:9000"} 1.35192576e+08
minio_node_process_resident_memory_bytes{server="myminio-pool-0-2.myminio-hl.tenant-lite.svc.cluster.local:9000"} 1.33226496e+08
minio_node_process_resident_memory_bytes{server="myminio-pool-0-3.myminio-hl.tenant-lite.svc.cluster.local:9000"} 1.303552e+08
# HELP minio_node_process_starttime_seconds Start time for MinIO process per node, time in seconds since Unix epoc
# TYPE minio_node_process_starttime_seconds gauge
minio_node_process_starttime_seconds{server="myminio-pool-0-0.myminio-hl.tenant-lite.svc.cluster.local:9000"} 1.68081601932e+09
minio_node_process_starttime_seconds{server="myminio-pool-0-1.myminio-hl.tenant-lite.svc.cluster.local:9000"} 1.68081601526e+09
minio_node_process_starttime_seconds{server="myminio-pool-0-2.myminio-hl.tenant-lite.svc.cluster.local:9000"} 1.68081601128e+09
minio_node_process_starttime_seconds{server="myminio-pool-0-3.myminio-hl.tenant-lite.svc.cluster.local:9000"} 1.68081600727e+09
# HELP minio_node_process_uptime_seconds Uptime for MinIO process per node in seconds
# TYPE minio_node_process_uptime_seconds gauge
minio_node_process_uptime_seconds{server="myminio-pool-0-0.myminio-hl.tenant-lite.svc.cluster.local:9000"} 444.715088497
minio_node_process_uptime_seconds{server="myminio-pool-0-1.myminio-hl.tenant-lite.svc.cluster.local:9000"} 448.780589624
minio_node_process_uptime_seconds{server="myminio-pool-0-2.myminio-hl.tenant-lite.svc.cluster.local:9000"} 452.769652334
minio_node_process_uptime_seconds{server="myminio-pool-0-3.myminio-hl.tenant-lite.svc.cluster.local:9000"} 456.769945712
# HELP minio_node_scanner_bucket_scans_finished Total number of bucket scans finished since server start
# TYPE minio_node_scanner_bucket_scans_finished counter
minio_node_scanner_bucket_scans_finished{server="myminio-pool-0-0.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
minio_node_scanner_bucket_scans_finished{server="myminio-pool-0-1.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
minio_node_scanner_bucket_scans_finished{server="myminio-pool-0-2.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
minio_node_scanner_bucket_scans_finished{server="myminio-pool-0-3.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
# HELP minio_node_scanner_bucket_scans_started Total number of bucket scans started since server start
# TYPE minio_node_scanner_bucket_scans_started counter
minio_node_scanner_bucket_scans_started{server="myminio-pool-0-0.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
minio_node_scanner_bucket_scans_started{server="myminio-pool-0-1.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
minio_node_scanner_bucket_scans_started{server="myminio-pool-0-2.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
minio_node_scanner_bucket_scans_started{server="myminio-pool-0-3.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
# HELP minio_node_scanner_directories_scanned Total number of directories scanned since server start
# TYPE minio_node_scanner_directories_scanned counter
minio_node_scanner_directories_scanned{server="myminio-pool-0-0.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
minio_node_scanner_directories_scanned{server="myminio-pool-0-1.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
minio_node_scanner_directories_scanned{server="myminio-pool-0-2.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
minio_node_scanner_directories_scanned{server="myminio-pool-0-3.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
# HELP minio_node_scanner_objects_scanned Total number of unique objects scanned since server start
# TYPE minio_node_scanner_objects_scanned counter
minio_node_scanner_objects_scanned{server="myminio-pool-0-0.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
minio_node_scanner_objects_scanned{server="myminio-pool-0-1.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
minio_node_scanner_objects_scanned{server="myminio-pool-0-2.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
minio_node_scanner_objects_scanned{server="myminio-pool-0-3.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
# HELP minio_node_scanner_versions_scanned Total number of object versions scanned since server start
# TYPE minio_node_scanner_versions_scanned counter
minio_node_scanner_versions_scanned{server="myminio-pool-0-0.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
minio_node_scanner_versions_scanned{server="myminio-pool-0-1.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
minio_node_scanner_versions_scanned{server="myminio-pool-0-2.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
minio_node_scanner_versions_scanned{server="myminio-pool-0-3.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
# HELP minio_node_syscall_read_total Total read SysCalls to the kernel. /proc/[pid]/io syscr
# TYPE minio_node_syscall_read_total counter
minio_node_syscall_read_total{server="myminio-pool-0-0.myminio-hl.tenant-lite.svc.cluster.local:9000"} 5592
minio_node_syscall_read_total{server="myminio-pool-0-1.myminio-hl.tenant-lite.svc.cluster.local:9000"} 6111
minio_node_syscall_read_total{server="myminio-pool-0-2.myminio-hl.tenant-lite.svc.cluster.local:9000"} 8160
minio_node_syscall_read_total{server="myminio-pool-0-3.myminio-hl.tenant-lite.svc.cluster.local:9000"} 6744
# HELP minio_node_syscall_write_total Total write SysCalls to the kernel. /proc/[pid]/io syscw
# TYPE minio_node_syscall_write_total counter
minio_node_syscall_write_total{server="myminio-pool-0-0.myminio-hl.tenant-lite.svc.cluster.local:9000"} 2970
minio_node_syscall_write_total{server="myminio-pool-0-1.myminio-hl.tenant-lite.svc.cluster.local:9000"} 4549
minio_node_syscall_write_total{server="myminio-pool-0-2.myminio-hl.tenant-lite.svc.cluster.local:9000"} 5732
minio_node_syscall_write_total{server="myminio-pool-0-3.myminio-hl.tenant-lite.svc.cluster.local:9000"} 3910
# HELP minio_notify_current_send_in_progress Number of concurrent async Send calls active to all targets
# TYPE minio_notify_current_send_in_progress gauge
minio_notify_current_send_in_progress{server="myminio-pool-0-0.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
minio_notify_current_send_in_progress{server="myminio-pool-0-1.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
minio_notify_current_send_in_progress{server="myminio-pool-0-2.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
minio_notify_current_send_in_progress{server="myminio-pool-0-3.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
# HELP minio_s3_requests_incoming_total Volatile number of total incoming S3 requests
# TYPE minio_s3_requests_incoming_total gauge
minio_s3_requests_incoming_total{server="myminio-pool-0-0.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
minio_s3_requests_incoming_total{server="myminio-pool-0-1.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
minio_s3_requests_incoming_total{server="myminio-pool-0-2.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
minio_s3_requests_incoming_total{server="myminio-pool-0-3.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
# HELP minio_s3_requests_rejected_auth_total Total number S3 requests rejected for auth failure
# TYPE minio_s3_requests_rejected_auth_total counter
minio_s3_requests_rejected_auth_total{server="myminio-pool-0-0.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
minio_s3_requests_rejected_auth_total{server="myminio-pool-0-1.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
minio_s3_requests_rejected_auth_total{server="myminio-pool-0-2.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
minio_s3_requests_rejected_auth_total{server="myminio-pool-0-3.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
# HELP minio_s3_requests_rejected_header_total Total number S3 requests rejected for invalid header
# TYPE minio_s3_requests_rejected_header_total counter
minio_s3_requests_rejected_header_total{server="myminio-pool-0-0.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
minio_s3_requests_rejected_header_total{server="myminio-pool-0-1.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
minio_s3_requests_rejected_header_total{server="myminio-pool-0-2.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
minio_s3_requests_rejected_header_total{server="myminio-pool-0-3.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
# HELP minio_s3_requests_rejected_invalid_total Total number S3 invalid requests
# TYPE minio_s3_requests_rejected_invalid_total counter
minio_s3_requests_rejected_invalid_total{server="myminio-pool-0-0.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
minio_s3_requests_rejected_invalid_total{server="myminio-pool-0-1.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
minio_s3_requests_rejected_invalid_total{server="myminio-pool-0-2.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
minio_s3_requests_rejected_invalid_total{server="myminio-pool-0-3.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
# HELP minio_s3_requests_rejected_timestamp_total Total number S3 requests rejected for invalid timestamp
# TYPE minio_s3_requests_rejected_timestamp_total counter
minio_s3_requests_rejected_timestamp_total{server="myminio-pool-0-0.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
minio_s3_requests_rejected_timestamp_total{server="myminio-pool-0-1.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
minio_s3_requests_rejected_timestamp_total{server="myminio-pool-0-2.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
minio_s3_requests_rejected_timestamp_total{server="myminio-pool-0-3.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
# HELP minio_s3_requests_waiting_total Number of S3 requests in the waiting queue
# TYPE minio_s3_requests_waiting_total gauge
minio_s3_requests_waiting_total{server="myminio-pool-0-0.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
minio_s3_requests_waiting_total{server="myminio-pool-0-1.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
minio_s3_requests_waiting_total{server="myminio-pool-0-2.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
minio_s3_requests_waiting_total{server="myminio-pool-0-3.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
# HELP minio_s3_traffic_received_bytes Total number of s3 bytes received
# TYPE minio_s3_traffic_received_bytes counter
minio_s3_traffic_received_bytes{server="myminio-pool-0-0.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
minio_s3_traffic_received_bytes{server="myminio-pool-0-1.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
minio_s3_traffic_received_bytes{server="myminio-pool-0-2.myminio-hl.tenant-lite.svc.cluster.local:9000"} 58
minio_s3_traffic_received_bytes{server="myminio-pool-0-3.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
# HELP minio_s3_traffic_sent_bytes Total number of s3 bytes sent
# TYPE minio_s3_traffic_sent_bytes counter
minio_s3_traffic_sent_bytes{server="myminio-pool-0-0.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
minio_s3_traffic_sent_bytes{server="myminio-pool-0-1.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
minio_s3_traffic_sent_bytes{server="myminio-pool-0-2.myminio-hl.tenant-lite.svc.cluster.local:9000"} 761
minio_s3_traffic_sent_bytes{server="myminio-pool-0-3.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
# HELP minio_software_commit_info Git commit hash for the MinIO release
# TYPE minio_software_commit_info gauge
minio_software_commit_info{commit="74040b457b50417b58eae7cb17c63428a0e2dd44",server="myminio-pool-0-0.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
minio_software_commit_info{commit="74040b457b50417b58eae7cb17c63428a0e2dd44",server="myminio-pool-0-1.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
minio_software_commit_info{commit="74040b457b50417b58eae7cb17c63428a0e2dd44",server="myminio-pool-0-2.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
minio_software_commit_info{commit="74040b457b50417b58eae7cb17c63428a0e2dd44",server="myminio-pool-0-3.myminio-hl.tenant-lite.svc.cluster.local:9000"} 0
# HELP minio_software_version_info MinIO Release tag for the server
# TYPE minio_software_version_info gauge
minio_software_version_info{server="myminio-pool-0-0.myminio-hl.tenant-lite.svc.cluster.local:9000",version="2023-03-24T21:41:23Z"} 0
minio_software_version_info{server="myminio-pool-0-1.myminio-hl.tenant-lite.svc.cluster.local:9000",version="2023-03-24T21:41:23Z"} 0
minio_software_version_info{server="myminio-pool-0-2.myminio-hl.tenant-lite.svc.cluster.local:9000",version="2023-03-24T21:41:23Z"} 0
minio_software_version_info{server="myminio-pool-0-3.myminio-hl.tenant-lite.svc.cluster.local:9000",version="2023-03-24T21:41:23Z"} 0