From 11b3b2b0ee707ec650dc5bf2994b1faae7321e6e Mon Sep 17 00:00:00 2001 From: Paul Gottschling Date: Thu, 24 Mar 2022 18:16:12 -0400 Subject: [PATCH] Metrics guide Add separate Tabs for self-hosted and Cloud editions --- docs/pages/setup/reference/metrics.mdx | 142 ++++++++++++++++++------- 1 file changed, 104 insertions(+), 38 deletions(-) diff --git a/docs/pages/setup/reference/metrics.mdx b/docs/pages/setup/reference/metrics.mdx index 69c8ff0cf30bb..7e4c6d78cbd25 100644 --- a/docs/pages/setup/reference/metrics.mdx +++ b/docs/pages/setup/reference/metrics.mdx @@ -75,6 +75,15 @@ Teleport is tracking. It is compatible with The following metrics are available: + + +While Teleport Cloud does not expose monitoring endpoints, you can use the following metrics to monitor your Teleport Nodes. + + + + + + | Name | Type | Component | Description | | - | - | - | - | | `audit_failed_disk_monitoring` | counter | Teleport Audit Log | Number of times disk monitoring failed. | @@ -110,46 +119,46 @@ The following metrics are available: | `firestore_events_backend_batch_write_requests` | counter | GCP Cloud Firestore | Number of batch write requests to Cloud Firestore events. | | `firestore_events_backend_batch_write_seconds` | histogram | GCP Cloud Firestore | Latency for Cloud Firestore events batch write operations. | | `gcs_event_storage_downloads` | counter | GCP GCS | Number of downloads from the GCS backend. | -| `gcs_event_storage_downloads_seconds` | histogram | Internal GoLang | Latency for GCS download operations. | -| `gcs_event_storage_uploads` | counter | Internal GoLang | Number of uploads to the GCS backend. | -| `gcs_event_storage_uploads_seconds` | histogram | Internal GoLang | Latency for GCS upload operations. | -| `go_gc_duration_seconds` | summary | Internal GoLang | A summary of the GC invocation durations. | -| `go_goroutines` | gauge | Internal GoLang | Number of goroutines that currently exist. | -| `go_info` | gauge | Internal GoLang | Information about the Go environment. | -| `go_memstats_alloc_bytes` | gauge | Internal GoLang | Number of bytes allocated and still in use. | -| `go_memstats_alloc_bytes_total` | counter | Internal GoLang | Total number of bytes allocated, even if freed. | -| `go_memstats_buck_hash_sys_bytes` | gauge | Internal GoLang | Number of bytes used by the profiling bucket hash table. | -| `go_memstats_frees_total` | counter | Internal GoLang | Total number of frees. | -| `go_memstats_gc_cpu_fraction` | gauge | Internal GoLang | The fraction of this program's available CPU time used by the GC since the program started. | -| `go_memstats_gc_sys_bytes` | gauge | Internal GoLang | Number of bytes used for garbage collection system metadata. | -| `go_memstats_heap_alloc_bytes` | gauge | Internal GoLang | Number of heap bytes allocated and still in use. | -| `go_memstats_heap_idle_bytes` | gauge | Internal GoLang | Number of heap bytes waiting to be used. | -| `go_memstats_heap_inuse_bytes` | gauge | Internal GoLang | Number of heap bytes that are in use. | -| `go_memstats_heap_objects` | gauge | Internal GoLang | Number of allocated objects. | -| `go_memstats_heap_released_bytes` | gauge | Internal GoLang | Number of heap bytes released to OS. | -| `go_memstats_heap_sys_bytes` | gauge | Internal GoLang | Number of heap bytes obtained from system. | -| `go_memstats_last_gc_time_seconds` | gauge | Internal GoLang | Number of seconds since 1970 of last garbage collection. | -| `go_memstats_lookups_total` | counter | Internal GoLang | Total number of pointer lookups. | -| `go_memstats_mallocs_total` | counter | Internal GoLang | Total number of mallocs. | -| `go_memstats_mcache_inuse_bytes` | gauge | Internal GoLang | Number of bytes in use by mcache structures. | -| `go_memstats_mcache_sys_bytes` | gauge | Internal GoLang | Number of bytes used for mcache structures obtained from system. | -| `go_memstats_mspan_inuse_bytes` | gauge | Internal GoLang | Number of bytes in use by mspan structures. | -| `go_memstats_mspan_sys_bytes` | gauge | Internal GoLang | Number of bytes used for mspan structures obtained from system. | -| `go_memstats_next_gc_bytes` | gauge | Internal GoLang | Number of heap bytes when next garbage collection will take place. | -| `go_memstats_other_sys_bytes` | gauge | Internal GoLang | Number of bytes used for other system allocations. | -| `go_memstats_stack_inuse_bytes` | gauge | Internal GoLang | Number of bytes in use by the stack allocator. | -| `go_memstats_stack_sys_bytes` | gauge | Internal GoLang | Number of bytes obtained from system for stack allocator. | -| `go_memstats_sys_bytes` | gauge | Internal GoLang | Number of bytes obtained from system. | -| `go_threads` | gauge | Internal GoLang | Number of OS threads created. | +| `gcs_event_storage_downloads_seconds` | histogram | Internal Golang | Latency for GCS download operations. | +| `gcs_event_storage_uploads` | counter | Internal Golang | Number of uploads to the GCS backend. | +| `gcs_event_storage_uploads_seconds` | histogram | Internal Golang | Latency for GCS upload operations. | +| `go_gc_duration_seconds` | summary | Internal Golang | A summary of the GC invocation durations. | +| `go_goroutines` | gauge | Internal Golang | Number of goroutines that currently exist. | +| `go_info` | gauge | Internal Golang | Information about the Go environment. | +| `go_memstats_alloc_bytes` | gauge | Internal Golang | Number of bytes allocated and still in use. | +| `go_memstats_alloc_bytes_total` | counter | Internal Golang | Total number of bytes allocated, even if freed. | +| `go_memstats_buck_hash_sys_bytes` | gauge | Internal Golang | Number of bytes used by the profiling bucket hash table. | +| `go_memstats_frees_total` | counter | Internal Golang | Total number of frees. | +| `go_memstats_gc_cpu_fraction` | gauge | Internal Golang | The fraction of this program's available CPU time used by the GC since the program started. | +| `go_memstats_gc_sys_bytes` | gauge | Internal Golang | Number of bytes used for garbage collection system metadata. | +| `go_memstats_heap_alloc_bytes` | gauge | Internal Golang | Number of heap bytes allocated and still in use. | +| `go_memstats_heap_idle_bytes` | gauge | Internal Golang | Number of heap bytes waiting to be used. | +| `go_memstats_heap_inuse_bytes` | gauge | Internal Golang | Number of heap bytes that are in use. | +| `go_memstats_heap_objects` | gauge | Internal Golang | Number of allocated objects. | +| `go_memstats_heap_released_bytes` | gauge | Internal Golang | Number of heap bytes released to OS. | +| `go_memstats_heap_sys_bytes` | gauge | Internal Golang | Number of heap bytes obtained from system. | +| `go_memstats_last_gc_time_seconds` | gauge | Internal Golang | Number of seconds since 1970 of last garbage collection. | +| `go_memstats_lookups_total` | counter | Internal Golang | Total number of pointer lookups. | +| `go_memstats_mallocs_total` | counter | Internal Golang | Total number of mallocs. | +| `go_memstats_mcache_inuse_bytes` | gauge | Internal Golang | Number of bytes in use by mcache structures. | +| `go_memstats_mcache_sys_bytes` | gauge | Internal Golang | Number of bytes used for mcache structures obtained from system. | +| `go_memstats_mspan_inuse_bytes` | gauge | Internal Golang | Number of bytes in use by mspan structures. | +| `go_memstats_mspan_sys_bytes` | gauge | Internal Golang | Number of bytes used for mspan structures obtained from system. | +| `go_memstats_next_gc_bytes` | gauge | Internal Golang | Number of heap bytes when next garbage collection will take place. | +| `go_memstats_other_sys_bytes` | gauge | Internal Golang | Number of bytes used for other system allocations. | +| `go_memstats_stack_inuse_bytes` | gauge | Internal Golang | Number of bytes in use by the stack allocator. | +| `go_memstats_stack_sys_bytes` | gauge | Internal Golang | Number of bytes obtained from system for stack allocator. | +| `go_memstats_sys_bytes` | gauge | Internal Golang | Number of bytes obtained from system. | +| `go_threads` | gauge | Internal Golang | Number of OS threads created. | | `heartbeat_connections_received_total` | counter | Teleport Auth | Number of times auth received a heartbeat connection. | | `heartbeat_connections_missed_total` | counter | Teleport Auth | Number of times auth did not receive a heartbeat from a node. | -| `process_cpu_seconds_total` | counter | Internal GoLang | Total user and system CPU time spent in seconds. | -| `process_max_fds` | gauge | Internal GoLang | Maximum number of open file descriptors. | -| `process_open_fds` | gauge | Internal GoLang | Number of open file descriptors. | -| `process_resident_memory_bytes` | gauge | Internal GoLang | Resident memory size in bytes. | -| `process_start_time_seconds` | gauge | Internal GoLang | Start time of the process since unix epoch in seconds. | -| `process_virtual_memory_bytes` | gauge | Internal GoLang | Virtual memory size in bytes. | -| `process_virtual_memory_max_bytes` | gauge | Internal GoLang | Maximum amount of virtual memory available in bytes. | +| `process_cpu_seconds_total` | counter | Internal Golang | Total user and system CPU time spent in seconds. | +| `process_max_fds` | gauge | Internal Golang | Maximum number of open file descriptors. | +| `process_open_fds` | gauge | Internal Golang | Number of open file descriptors. | +| `process_resident_memory_bytes` | gauge | Internal Golang | Resident memory size in bytes. | +| `process_start_time_seconds` | gauge | Internal Golang | Start time of the process since unix epoch in seconds. | +| `process_virtual_memory_bytes` | gauge | Internal Golang | Virtual memory size in bytes. | +| `process_virtual_memory_max_bytes` | gauge | Internal Golang | Maximum amount of virtual memory available in bytes. | | `promhttp_metric_handler_requests_in_flight` | gauge | prometheus | Current number of scrapes being served. | | `promhttp_metric_handler_requests_total` | counter | prometheus | Total number of scrapes by HTTP status code. | | `proxy_connection_limit_exceeded_total` | counter | Teleport Proxy | Number of connections that exceeded the proxy connection limit. | @@ -170,3 +179,60 @@ The following metrics are available: | `user_max_concurrent_sessions_hit_total` | counter | Teleport Node | Number of times a user exceeded their concurrent session limit. | | `watcher_events` | histogram | cache | Per resource size of events emitted. | | `watcher_event_sizes` | histogram | cache | Overall size of events emitted. | + + + + +| Name | Type | Component | Description | +| - | - | - | - | +| `go_gc_duration_seconds` | summary | Internal Golang | A summary of the GC invocation durations. | +| `go_goroutines` | gauge | Internal Golang | Number of goroutines that currently exist. | +| `go_info` | gauge | Internal Golang | Information about the Go environment. | +| `go_memstats_alloc_bytes` | gauge | Internal Golang | Number of bytes allocated and still in use. | +| `go_memstats_alloc_bytes_total` | counter | Internal Golang | Total number of bytes allocated, even if freed. | +| `go_memstats_buck_hash_sys_bytes` | gauge | Internal Golang | Number of bytes used by the profiling bucket hash table. | +| `go_memstats_frees_total` | counter | Internal Golang | Total number of frees. | +| `go_memstats_gc_cpu_fraction` | gauge | Internal Golang | The fraction of this program's available CPU time used by the GC since the program started. | +| `go_memstats_gc_sys_bytes` | gauge | Internal Golang | Number of bytes used for garbage collection system metadata. | +| `go_memstats_heap_alloc_bytes` | gauge | Internal Golang | Number of heap bytes allocated and still in use. | +| `go_memstats_heap_idle_bytes` | gauge | Internal Golang | Number of heap bytes waiting to be used. | +| `go_memstats_heap_inuse_bytes` | gauge | Internal Golang | Number of heap bytes that are in use. | +| `go_memstats_heap_objects` | gauge | Internal Golang | Number of allocated objects. | +| `go_memstats_heap_released_bytes` | gauge | Internal Golang | Number of heap bytes released to OS. | +| `go_memstats_heap_sys_bytes` | gauge | Internal Golang | Number of heap bytes obtained from system. | +| `go_memstats_last_gc_time_seconds` | gauge | Internal Golang | Number of seconds since 1970 of last garbage collection. | +| `go_memstats_lookups_total` | counter | Internal Golang | Total number of pointer lookups. | +| `go_memstats_mallocs_total` | counter | Internal Golang | Total number of mallocs. | +| `go_memstats_mcache_inuse_bytes` | gauge | Internal Golang | Number of bytes in use by mcache structures. | +| `go_memstats_mcache_sys_bytes` | gauge | Internal Golang | Number of bytes used for mcache structures obtained from system. | +| `go_memstats_mspan_inuse_bytes` | gauge | Internal Golang | Number of bytes in use by mspan structures. | +| `go_memstats_mspan_sys_bytes` | gauge | Internal Golang | Number of bytes used for mspan structures obtained from system. | +| `go_memstats_next_gc_bytes` | gauge | Internal Golang | Number of heap bytes when next garbage collection will take place. | +| `go_memstats_other_sys_bytes` | gauge | Internal Golang | Number of bytes used for other system allocations. | +| `go_memstats_stack_inuse_bytes` | gauge | Internal Golang | Number of bytes in use by the stack allocator. | +| `go_memstats_stack_sys_bytes` | gauge | Internal Golang | Number of bytes obtained from system for stack allocator. | +| `go_memstats_sys_bytes` | gauge | Internal Golang | Number of bytes obtained from system. | +| `go_threads` | gauge | Internal Golang | Number of OS threads created. | +| `process_cpu_seconds_total` | counter | Internal Golang | Total user and system CPU time spent in seconds. | +| `process_max_fds` | gauge | Internal Golang | Maximum number of open file descriptors. | +| `process_open_fds` | gauge | Internal Golang | Number of open file descriptors. | +| `process_resident_memory_bytes` | gauge | Internal Golang | Resident memory size in bytes. | +| `process_start_time_seconds` | gauge | Internal Golang | Start time of the process since unix epoch in seconds. | +| `process_virtual_memory_bytes` | gauge | Internal Golang | Virtual memory size in bytes. | +| `process_virtual_memory_max_bytes` | gauge | Internal Golang | Maximum amount of virtual memory available in bytes. | +| `promhttp_metric_handler_requests_in_flight` | gauge | prometheus | Current number of scrapes being served. | +| `promhttp_metric_handler_requests_total` | counter | prometheus | Total number of scrapes by HTTP status code. | +| `reversetunnel_connected_proxies` | gauge | Teleport | Number of known proxies being sought. | +| `rx` | counter | Teleport | Number of bytes received. | +| `server_interactive_sessions_total` | gauge | Teleport | Number of active sessions. | +| `teleport_build_info` | gauge | Teleport | Provides build information of Teleport including gitref (git describe --long --tags), Go version, and Teleport version. The value of this gauge will always be 1. | +| `teleport_cache_events` | counter | Teleport | Number of events received by a Teleport service cache. Teleport Node services cache incoming events related to their service. | +| `teleport_cache_stale_events` | counter | Teleport | Number of stale events received by a Teleport service cache. A high percentage of stale events can indicate a degraded backend. | +| `trusted_clusters` | gauge | Teleport | Number of tunnels per state. | +| `tx` | counter | Teleport | Number of bytes transmitted. | +| `user_max_concurrent_sessions_hit_total` | counter | Teleport Node | Number of times a user exceeded their concurrent session limit. | +| `watcher_events` | histogram | cache | Per resource size of events emitted. | +| `watcher_event_sizes` | histogram | cache | Overall size of events emitted. | + + + \ No newline at end of file