From f3ffb18878797fc98a81206a7d842bcd1089f615 Mon Sep 17 00:00:00 2001 From: Aishwarya Chakravarthy Date: Wed, 24 Jul 2024 09:58:39 -0400 Subject: [PATCH] [docs] Tablet metadata and ASH updates (#23221) * updates * review comments * removed pages from stable and addressed review comments * missed edits --- .../observability/active-session-history.md | 40 +- .../explore/observability/yb-local-tablets.md | 13 +- .../ycql-stat-statements.md | 11 +- .../stable/explore/observability/_index.md | 9 +- .../observability/active-session-history.md | 413 ------------------ .../explore/observability/yb-local-tablets.md | 84 ---- .../explore/query-1-performance/_index.md | 2 +- .../query-1-performance/pg-stat-statements.md | 6 +- .../ycql-stat-statements.md | 163 ------- .../advanced-features/views.md | 2 +- 10 files changed, 41 insertions(+), 702 deletions(-) delete mode 100644 docs/content/stable/explore/observability/active-session-history.md delete mode 100644 docs/content/stable/explore/observability/yb-local-tablets.md delete mode 100644 docs/content/stable/explore/query-1-performance/ycql-stat-statements.md diff --git a/docs/content/preview/explore/observability/active-session-history.md b/docs/content/preview/explore/observability/active-session-history.md index fa1dc6ecf39b..bb6dc01c3948 100644 --- a/docs/content/preview/explore/observability/active-session-history.md +++ b/docs/content/preview/explore/observability/active-session-history.md @@ -15,9 +15,9 @@ type: docs Active Session History (ASH) provides a current and historical view of system activity by sampling session activity in the database. A database session or connection is considered active if it is consuming CPU, or has an active RPC call that is waiting on one of the wait events. -ASH exposes session activity in the form of [SQL views](../../ysql-language-features/advanced-features/views/) so that you can run analytical queries, aggregations for analysis, and troubleshoot performance issues. To run ASH, you need to enable [YSQL](../../../api/ysql/) or [YCQL](../../../api/ycql/) for their respective sessions. +ASH exposes session activity in the form of [SQL views](../../ysql-language-features/advanced-features/views/) so that you can run analytical queries, aggregations for analysis, and troubleshoot performance issues. To run ASH queries, you need to enable [YSQL](../../../api/ysql/). -Currently, ASH is available for [YSQL](../../../api/ysql/), [YCQL](../../../api/ycql/), and [YB-TServer](../../../architecture/yb-tserver/) processes. ASH facilitates analysis by recording wait events related to YSQL, YCQL, or YB-TServer requests while they are being executed. These wait events belong to the categories including but not limited to _CPU_, _WaitOnCondition_, _Network_, and _Disk IO_. +Currently, ASH is available for [YSQL](../../../api/ysql/), [YCQL](../../../api/ycql/), and [YB-TServer](../../../architecture/yb-tserver/). ASH facilitates analysis by recording wait events related to YSQL, YCQL, or YB-TServer requests while they are being executed. These wait events belong to the categories including but not limited to _CPU_, _WaitOnCondition_, _Network_, and _Disk IO_. Analyzing the wait events and wait event types lets you troubleshoot, answer the following questions, and subsequently tune performance: @@ -31,9 +31,9 @@ To use ASH, enable and configure the following flags for each node of your clust | Flag | Description | | :--- | :---------- | -| allowed_preview_flags_csv | Pass the flags `ysql_yb_ash_enable_infra` and `ysql_yb_enable_ash` in this flag in CSV format. | +| allowed_preview_flags_csv | Set the value of this flag to include `ysql_yb_ash_enable_infra,ysql_yb_enable_ash`. | | ysql_yb_ash_enable_infra | Enable or disable ASH infrastructure.
Default: false. Changing this flag requires a VM restart. | -| ysql_yb_enable_ash | Works only in conjunction with the flag `ysql_yb_ash_enable_infra`. Start sampling and instrumentation (that is, periodically check and keep track) of YSQL and YCQL queries, and YB-TServer requests.
Default: false. Changing this flag doesn't require a VM restart. | +| ysql_yb_enable_ash | Works only in conjunction with the flag `ysql_yb_ash_enable_infra`. Setting this flag to true enables the collection of wait events for YSQL and YCQL queries, and YB-TServer requests.
Default: false. Changing this flag doesn't require a VM restart. | ### Additional flags @@ -41,18 +41,17 @@ You can also use the following flags based on your requirements. | Flag | Description | | :--- | :---------- | -| ysql_yb_ash_circular_buffer_size | Size (in KBs) of circular buffer where the samples are stored.
Default: 16000. Changing this flag requires a VM restart. | -| ysql_yb_ash_sampling_interval_ms | Time (in milliseconds) duration between two sampling events (ysql, ycql, yb-tserver).
Default: 1000. Changing this flag doesn't require a VM restart. | +| ysql_yb_ash_circular_buffer_size | Size (in KBs) of circular buffer where the samples are stored.
Default: 16*1024. Changing this flag requires a VM restart. | +| ysql_yb_ash_sampling_interval_ms | Sampling interval (in milliseconds).
Default: 1000. Changing this flag doesn't require a VM restart. | | ysql_yb_ash_sample_size | Maximum number of events captured per sampling interval.
Default: 500. Changing this flag doesn't require a VM restart. | ## Limitations Note that the following limitations are subject to change as the feature is in [Tech Preview](/preview/releases/versioning/#feature-maturity). -- ASH is available per node only. [Aggregations](../../../develop/learn/aggregations-ycql/) need to be done by you. +- ASH is available per node and is not aggregated across the cluster. - ASH is not available for [YB-Master](../../../architecture/yb-master/) processes. -- ASH is available only for foreground activities or queries from customer applications. -- ASH does not capture start and end time of wait events. +- ASH is available for queries and a few background activities like compaction and flushes. ASH support for other background activities will be added in future releases. ## Learn more @@ -95,11 +94,11 @@ To get more details about the various steps of a query execution, use the [Expla href="./pg-stat-activity/" icon="/images/section_icons/manage/diagnostics.png">}} - {{}} + icon="fa-solid fa-tablets">}} --> {{Default: false. Changing this flag requires a VM restart. | -| ysql_yb_enable_ash | Works only in conjunction with the flag `ysql_yb_ash_enable_infra`. Start sampling and instrumentation (that is, periodically check and keep track) of YSQL and YCQL queries, and YB-TServer requests.
Default: false. Changing this flag doesn't require a VM restart. | - -### Additional flags - -You can also use the following flags based on your requirements. - -| Flag | Description | -| :--- | :---------- | -| ysql_yb_ash_circular_buffer_size | Size (in KBs) of circular buffer where the samples are stored.
Default: 16000. Changing this flag requires a VM restart. | -| ysql_yb_ash_sampling_interval_ms | Time (in milliseconds) duration between two sampling events (ysql, ycql, yb-tserver).
Default: 1000. Changing this flag doesn't require a VM restart. | -| ysql_yb_ash_sample_size | Maximum number of events captured per sampling interval.
Default: 500. Changing this flag doesn't require a VM restart. | - -## Limitations - -Note that the following limitations are subject to change as the feature is in [Tech Preview](/preview/releases/versioning/#feature-maturity). - -- ASH is available per node only. [Aggregations](../../../develop/learn/aggregations-ycql/) need to be done by you. -- ASH is not available for [YB-Master](../../../architecture/yb-master/) processes. -- ASH is available only for foreground activities or queries from customer applications. -- ASH does not capture start and end time of wait events. - - - -## YSQL views - -ASH exposes the following views in each node to analyze and troubleshoot performance issues. - -### yb_active_session_history - -Get information on wait events for each normalized query, YSQL, or YCQL request. - -| Column | Type | Description | -| :----- | :--- | :---------- | -| root_request_id | UUID | A 16-byte UUID that is generated per request. Generated for queries at YSQL/YCQL layer. | -| rpc_request_id | integer | ID for internal requests, it is a monotonically increasing number for the lifetime of a YB-TServer. | -| wait_event_component | text | There are three components: YSQL, YCQL, and YB-TServer. | -| wait_event_class | text | Every wait event has a class associated with it. | -| wait_event | text | Provides insight into what the RPC is waiting on. | -| wait_event_type | text | Type of the wait event such as CPU, WaitOnCondition, Network, Disk IO, and so on. | -| wait_event_aux | text | Additional information for the wait event. For example, tablet ID for YB-TServer wait events. | -| top_level_node_id | UUID | 16-byte YB-TServer UUID of the YSQL/YCQL node where the query is being executed. | -| query_id | bigint | Query ID as seen on the `/statements` endpoint. This can be used to join with [pg_stat_statements](../../query-1-performance/pg-stat-statements/)/[ycql_stat_statements](../../query-1-performance/ycql-stat-statements/). A known constant for background activities. For example, _flush_ is 2, _compaction_ is 3, and so on. | -| ysql_session_id | bigint | YSQL session identifier. Zero for YCQL and background activities. | -| client_node_ip | text | IP address of the client which sent the query to YSQL/YCQL. Null for background activities. | -| sample_weight | float | If in any sampling interval there are too many events, YugabyteDB only collects `yb_ash_sample_size` samples/events. Based on how many were sampled, weights are assigned to the collected events.

For example, if there are 200 events, but only 100 events are collected, each of the collected samples will have a weight of (200 / 100) = 2.0 | - -## Wait events - -List of wait events by the following request types. - -### YSQL - -These are only the wait events introduced by YugabyteDB, however some of the following [wait events](https://www.postgresql.org/docs/current/monitoring-stats.html) inherited from PostgreSQL might also show up in the [yb_active_session_history](#yb-active-session-history) view. - -| Class | Wait Event | Wait Event Type | Wait Event Aux | Description | -| :--------------- |:---------- | :-------------- |:--- | :---------- | -| TServer Wait | StorageRead | Network | | Waiting for a DocDB read operation | -| TServer Wait | CatalogRead | Network | | Waiting for a catalog read operation | -| TServer Wait | IndexRead | Network | | Waiting for a secondary index read operation | -| TServer Wait | StorageFlush | Network | | Waiting for a storage flush request | -| YSQLQuery | QueryProcessing| CPU | | Doing CPU work | -| YSQLQuery | yb_ash_metadata | LWLock | | Waiting to update ASH metadata for a query | -| Timeout | YBTxnConflictBackoff | Timeout | | Waiting due to conflict in DocDB | - -### YB-TServer - -| Class | Wait Event | Wait Event Type | Wait Event Aux | Description | -|:---------------- | :--------- |:--------------- | :--- | :---------- | -| Common | OnCpu_Passive | CPU | | Waiting for a thread to pick it up | -| Common | OnCpu_Active | CPU | | RPC is being actively processed on a thread | -| Common | ResponseQueued | Network | | Waiting for response to be transferred | -| Tablet | AcquiringLocks | Lock | \| Taking row-wise locks. May need to wait for other rpcs to release the lock. | -| Tablet | MVCC_WaitForSafeTime | Lock | \| Waiting for the SafeTime to be at least the desired read-time. | -| Tablet | BackfillIndex_WaitForAFreeSlot | Lock | \ | Waiting for a slot to open if there are too many backfill requests at the same time. | -| Tablet | CreatingNewTablet | I/O | \| Creating a new tablet may involve writing metadata files, causing I/O wait. | -| Tablet | WaitOnConflictingTxn | Lock | \| Waiting for the conflicting transactions to complete. | -| Consensus | WAL_Append | I/O | \| Persisting Wal edits | -| Consensus | WAL_Sync | I/O | \| Persisting Wal edits | -| Consensus | Raft_WaitingForReplication | Network | \| Waiting for Raft replication | -| Consensus | Raft_ApplyingEdits | Lock/CPU | \| Applying the edits locally | -| RocksDB | BlockCacheReadFromDisk | I/O | \| Populating block cache from disk | -| RocksDB | Flush | I/O | \ | Doing RocksDB flush | -| RocksDB | Compaction | I/O | \| Doing RocksDB compaction | -| RocksDB | RateLimiter | I/O | | Slow down due to rate limiter throttling access to disk | - -### YCQL - -| Class | Wait Event | Wait Event Type | Wait Event Aux | Description | -| :--------------- |:---------- | :-------------- |:--- | :---------- | -| YCQLQuery | YCQL_Parse | CPU | | CQL call is being actively processed | -| YCQLQuery | YCQL_Read | Network | \ | Waiting for DocDB read operation | -| YCQLQuery | YCQL_Write | Network | \ | Waiting for DocDB write operation | -| YBClient | LookingUpTablet | Network | | Looking up tablet information | -| YBClient | YBCSyncLeaderMasterRpc | Network | | Waiting on an RPC to the master/master-service | -| YBClient | YBCFindMasterProxy | Network | | Waiting on establishing the proxy to master leader | - - -## Examples - -{{% explore-setup-single %}} - -Make sure you have an active ysqlsh session (`./bin/ysqlsh`) to run the following examples. - -### Distribution of wait events for each query_id - -Check the distribution of wait events for each query_id, only for the last 20 minutes. - -```sql -SELECT - query_id, - wait_event_component, - wait_event, - wait_event_type, - COUNT(*) -FROM - yb_active_session_history -WHERE - sample_time >= current_timestamp - interval '20 minutes' -GROUP BY - query_id, - wait_event_component, - wait_event, - wait_event_type -ORDER BY - query_id, - wait_event_component, - wait_event_type; -``` - -```output - query_id | wait_event_component | wait_event | wait_event_type | count - ---------------------+----------------------+------------------------------------+-----------------+------------ - -4157456334073660389 | YSQL | CatalogRead | Network | 3 - -1970690938654296136 | TServer | Raft_ApplyingEdits | Cpu | 54 - -1970690938654296136 | TServer | OnCpu_Active | Cpu | 107 - -1970690938654296136 | TServer | OnCpu_Passive | Cpu | 144 - -1970690938654296136 | TServer | RocksDB_NewIterator | DiskIO | 6 - -1970690938654296136 | TServer | ConflictResolution_ResolveConficts | Network | 18 - -1970690938654296136 | TServer | Raft_WaitingForReplication | Network | 194 - -1970690938654296136 | TServer | Rpc_Done | WaitOnCondition | 18 - -1970690938654296136 | TServer | MVCC_WaitForSafeTime | WaitOnCondition | 5 - -1970690938654296136 | YSQL | QueryProcessing | Cpu | 1023 - 0 | TServer | OnCpu_Passive | Cpu | 10 - 0 | TServer | OnCpu_Active | Cpu | 9 - 6107501747146929242 | TServer | OnCpu_Active | Cpu | 208 - 6107501747146929242 | TServer | RocksDB_NewIterator | DiskIO | 5 - 6107501747146929242 | TServer | MVCC_WaitForSafeTime | WaitOnCondition | 10 - 6107501747146929242 | TServer | Rpc_Done | WaitOnCondition | 15 - 6107501747146929242 | YSQL | QueryProcessing | Cpu | 285 - 6107501747146929242 | YSQL | StorageRead | Network | 658 - 6107501747146929242 | YSQL | CatalogRead | Network | 1 -``` - -### Distribution of wait events for each query - -As ASH's query_id is the same as [pg_stat_statement's](../../query-1-performance/pg-stat-statements/) queryid, you can join with pg_stat_statements to view the distribution of wait events for each query. This may help in finding what's wrong with a particular query, or determine where most of the time is being spent on. In this example, you can see that in YB-TServer, most of the time is spent on the wait event `ConflictResolution_WaitOnConflictingTxns` which suggests that there are a lot of conflicts in the [DocDB storage layer](../../../architecture/docdb/). - -```sql -SELECT - SUBSTRING(query, 1, 50) AS query, - wait_event_component, - wait_event, - wait_event_type, - COUNT(*) -FROM - yb_active_session_history -JOIN - pg_stat_statements -ON - query_id = queryid -WHERE - sample_time >= current_timestamp - interval '20 minutes' -GROUP BY - query, - wait_event_component, - wait_event, - wait_event_type -ORDER BY - query, - wait_event_component, - wait_event_type; -``` - -```output - query | wait_event_component | wait_event | wait_event_type | count ------------------------------------------------+----------------------+------------------------------------------+-----------------+------------ - UPDATE test_table set v = v + $1 where k = $2 | TServer | OnCpu_Passive | Cpu | 46 - UPDATE test_table set v = v + $1 where k = $2 | TServer | Raft_ApplyingEdits | Cpu | 34 - UPDATE test_table set v = v + $1 where k = $2 | TServer | OnCpu_Active | Cpu | 39 - UPDATE test_table set v = v + $1 where k = $2 | TServer | RocksDB_NewIterator | DiskIO | 3 - UPDATE test_table set v = v + $1 where k = $2 | TServer | ConflictResolution_ResolveConficts | Network | 99 - UPDATE test_table set v = v + $1 where k = $2 | TServer | Raft_WaitingForReplication | Network | 38 - UPDATE test_table set v = v + $1 where k = $2 | TServer | ConflictResolution_WaitOnConflictingTxns | WaitOnCondition | 1359 - UPDATE test_table set v = v + $1 where k = $2 | TServer | Rpc_Done | WaitOnCondition | 5 - UPDATE test_table set v = v + $1 where k = $2 | TServer | LockedBatchEntry_Lock | WaitOnCondition | 141 - UPDATE test_table set v = v + $1 where k = $2 | YSQL | QueryProcessing | Cpu | 1929 -``` - -### Detect a hot shard - -In this example, you can see that a particular tablet is getting a lot of requests as compared to the other tablets. The `wait_event_aux` field contains the `tablet_id` in case of YB-TServer events. - -```sql -SELECT - wait_event_aux AS tablet_id, - COUNT(*) -FROM - yb_active_session_history -WHERE - wait_event_component = 'TServer' AND - wait_event_aux IS NOT NULL -GROUP BY - wait_event_aux -ORDER BY - count DESC; -``` - -```output - tablet_id | count ------------------+------- - 09f26a0bb117411 | 33129 - a1d82ef77aa64a8 | 5235 - 31bc90e0c59e4da | 2431 - 7b49c915e7fe4f1 | 1518 - 6b6a264711a84d2 | 403 - 96948dbb19674cb | 338 - e112a0dd35994e5 | 320 - f901168f334f432 | 315 - bddebf9b7d9b485 | 310 - 04a37ec2cecf49e | 70 - 70f6e424970c44c | 66 - 77bdebc4f7e3400 | 65 - b4ae6f1115fc4a9 | 63 - 8674a0708cba422 | 63 - 9cf4fc4a834040d | 61 - e66879054249434 | 61 - c2cfa997bf63463 | 59 - 9d64f3479792499 | 58 - e70fd34078e84fe | 58 - 8ea1aa0f2e4749a | 56 - b3bbaec3014f4f1 | 53 - d3bbd37828ab422 | 53 - 542c6f91ff6a403 | 52 - 27780cde5a1b445 | 50 - 4a64a9f25e414ce | 44 - 09bb0274a41146a | 5 - d58c56ce3fc7458 | 4 - 0350744dad944bd | 4 - 219fd39bafee44a | 3 -``` - -You can join with `yb_local_tablets` to get more information about the table type, table_name, and partition keys. As the `wait_event_aux` has only the first 15 characters of the `tablet_id` as a string, you have to join with only the first 15 characters from `yb_local_tablets`. - -```sql -SELECT - tablet_id, - table_type, - namespace_name, - table_name, - partition_key_start, - partition_key_end, - COUNT(*) -FROM - yb_active_session_history -JOIN - yb_local_tablets -ON - wait_event_aux = SUBSTRING(tablet_id, 1, 15) -GROUP BY - tablet_id, - table_type, - namespace_name, - table_name, - partition_key_start, - partition_key_end -ORDER BY - table_name, - count DESC; -``` - -```output - tablet_id | table_type | namespace_name | table_name | partition_key_start | partition_key_end | count -----------------------------------+------------+----------------+--------------+---------------------+-------------------+------- - 09f26a0bb117411fa068df13420ea643 | YSQL | yugabyte | test_table | | \x2aaa | 33129 - d58c56ce3fc74584bd2eb892cea51e2a | YSQL | yugabyte | test_table | \x5555 | \x8000 | 10 - 0350744dad944bd9ba70c91432c5d8e2 | YSQL | yugabyte | test_table | \xaaaa | \xd555 | 9 - 219fd39bafee44a59117c4089a2f71bf | YSQL | yugabyte | test_table | \x2aaa | \x5555 | 8 - 09bb0274a41146abbb6fe70e41b4f3c1 | YSQL | yugabyte | test_table | \xd555 | | 7 - a1d82ef77aa64a85987eef4aa2322c11 | System | system | transactions | \xf555 | | 7973 - 31bc90e0c59e4da5b36e38e9e48554a0 | System | system | transactions | \x9555 | \xa000 | 5169 - 7b49c915e7fe4f13afdee958028d4446 | System | system | transactions | \xb555 | \xc000 | 4256 - 6b6a264711a84d25b3226f78b9dd4d6f | System | system | transactions | | \x0aaa | 403 - 96948dbb19674cb596c7f69177533000 | System | system | transactions | \x7555 | \x8000 | 338 - e112a0dd35994e5990b25c4ae8a00eb6 | System | system | transactions | \x5555 | \x6000 | 320 - f901168f334f43289007ee8385fedb67 | System | system | transactions | \x8aaa | \x9555 | 315 - bddebf9b7d9b4858b4e502b8a1d93e01 | System | system | transactions | \x6aaa | \x7555 | 310 - 04a37ec2cecf49e5bdb974ee80bb5c97 | System | system | transactions | \x2000 | \x2aaa | 73 - 70f6e424970c44c391edbbc18225ccb0 | System | system | transactions | \x0aaa | \x1555 | 69 - 77bdebc4f7e340068ad3d8c6ccf74b35 | System | system | transactions | \xeaaa | \xf555 | 69 - b4ae6f1115fc4a94b2d9e7aa514b5f28 | System | system | transactions | \xe000 | \xeaaa | 67 - 8674a0708cba4228bd1d435a642134bb | System | system | transactions | \xaaaa | \xb555 | 66 - 8ea1aa0f2e4749a4802986fefbf54d43 | System | system | transactions | \xd555 | \xe000 | 66 - e70fd34078e84fef945f28fe6b004309 | System | system | transactions | \x8000 | \x8aaa | 64 - e66879054249434da01b3e7a314b983c | System | system | transactions | \x4000 | \x4aaa | 63 - 9d64f34797924998911922563d3ad0ec | System | system | transactions | \xa000 | \xaaaa | 63 - 9cf4fc4a834040df91987b26bf9831ed | System | system | transactions | \x4aaa | \x5555 | 63 - c2cfa997bf634637880137870cee25a1 | System | system | transactions | \xcaaa | \xd555 | 61 - d3bbd37828ab4225a68bf4dd54924138 | System | system | transactions | \x1555 | \x2000 | 56 - b3bbaec3014f4f1aa01e04b42c5f40c7 | System | system | transactions | \x3555 | \x4000 | 56 - 542c6f91ff6a403399ddf46e4d5f29bb | System | system | transactions | \x6000 | \x6aaa | 56 - 27780cde5a1b445d83ff83c865b9bea5 | System | system | transactions | \xc000 | \xcaaa | 51 - 4a64a9f25e414ce7a649be2e6d24c7ee | System | system | transactions | \x2aaa | \x3555 | 46 -``` - -You can see that only a single tablet of the table `test_table` is getting most of the requests, and you have the partition range of that particular tablet. - -### Distribution of the type of the wait events for each component - -Detect where most time is being spent to help understand if the issue is related to disk IO, network operations, waiting for a condition or lock, or intense CPU work. - -```sql -SELECT - wait_event_component, - wait_event_type, - COUNT(*) -FROM - yb_active_session_history -GROUP BY - wait_event_component, - wait_event_type -ORDER BY - wait_event_component, - count; -``` - -```output - wait_event_component | wait_event_type | count -----------------------+-----------------+------- - TServer | WaitOnCondition | 47 - TServer | Network | 910 - TServer | Cpu | 2665 - TServer | DiskIO | 8193 - YSQL | LWLock | 1 - YSQL | Cpu | 1479 - YSQL | Network | 4575 -``` - -### Detect which client/application is sending the most amount of queries - -Suppose you have a rogue application which is sending a lot of queries, you can identify the application using the following example query: - -```sql -SELECT - client_node_ip, - COUNT(*) -FROM - yb_active_session_history -WHERE - client_node_ip IS NOT NULL -GROUP BY - client_node_ip; -``` - -```output - client_node_ip | count ------------------+------- - 127.0.0.2:56471 | 92 - 127.0.0.1:56473 | 106 - 127.0.0.1:56477 | 91 - 127.0.0.2:56481 | 51 - 127.0.0.3:56475 | 53 - 127.0.0.3:56485 | 18 - 127.0.0.3:56479 | 10997 -``` diff --git a/docs/content/stable/explore/observability/yb-local-tablets.md b/docs/content/stable/explore/observability/yb-local-tablets.md deleted file mode 100644 index a9e662051162..000000000000 --- a/docs/content/stable/explore/observability/yb-local-tablets.md +++ /dev/null @@ -1,84 +0,0 @@ ---- -title: View YSQL/YCQL and tablet metadata with yb_local_tablets -linkTitle: YQL and tablet metadata -description: View tablet metadata for YSQL and YCQL statements, and system tablets executed by a server. -headerTitle: View YSQL/YCQL and tablet metadata with yb_local_tablets -menu: - stable: - identifier: yb-local-tablets - parent: explore-observability - weight: 320 -type: docs ---- - -Use the YugabyteDB `yb_local_tablets` view to see metadata about the [YSQL](../../../api/ysql/) and [YCQL](../../../api/ycql/) statements, and system [tablets](../../../architecture/key-concepts/#tablet) of a node. - -This view is also present on `:9000/tablets` in the form of a YSQL view, and can be joined with YCQL wait events in the [yb_active_session_history](../../observability/active-session-history/#yb-active-session-history) view using the tablet ID. For example, see [Detect a hot shard](../../observability/active-session-history/#detect-a-hot-shard). - -The columns of the `yb_local_tablets` view are described in the following table. - -| Column | Type | Description | -| :----- | :--- | :---------- | -| tablet_id | text | 16 byte UUID of the tablet. | -| table_id | text | 16 byte UUID of the table which the tablet is part of. | -| table_type | text | Type of the table. Can be YSQL, YCQL, System, or Unknown. | -| namespace_name | text | Name of the database or the keyspace. | -| ysql_schema_name | text | YSQL schema name. Empty for YCQL, System, and Unknown table types. | -| table_name | text | Name of the table which the tablet is part of. | -| partition_key_start| bytea | Start key of the partition (inclusive). | -| partition_key_end | bytea | End key of the partition (exclusive).| - -## Examples - -{{% explore-setup-single %}} - -Note that as this view is accessible via YSQL, run your examples using [ysqlsh](../../../admin/ysqlsh/#starting-ysqlsh). - -### Describe the columns in the view - -```sql -yugabyte=# \d yb_local_tablets -``` - -```output - View "pg_catalog.yb_local_tablets" - Column | Type | Collation | Nullable | Default ----------------------+-------+-----------+----------+--------- - tablet_id | text | | | - table_id | text | | | - table_type | text | | | - namespace_name | text | | | - ysql_schema_name | text | | | - table_name | text | | | - partition_key_start | bytea | | | - partition_key_end | bytea | | | -``` - -### Get basic information - -This example includes a YCQL table, a [hash-partitioned](../../../architecture/docdb-sharding/sharding/#hash-sharding) YSQL table, and a [range-partitioned](../../../architecture/docdb-sharding/sharding/#range-sharding) YSQL table. - -```sql -yugabyte=# SELECT * FROM yb_local_tablets ORDER BY table_name, partition_key_start ASC NULLS FIRST; -``` - -```output - tablet_id | table_id | table_type | namespace_name | ysql_schema_name | table_name | partition_key_start | partition_key_end -----------------------------------+----------------------------------+------------+-----------------+------------------+-------------------------+------------------------+------------------------ - 230de13ea3f045c2bc817046c96bfb9e | db82083fb39e47b0976b99f3612fa144 | YCQL | ybdemo_keyspace | | cassandrakeyvalue | | \x8000 - cb8ef7044b094709870d421fccd568a4 | db82083fb39e47b0976b99f3612fa144 | YCQL | ybdemo_keyspace | | cassandrakeyvalue | \x8000 | - 76010b63fc714389ab97b432d9db78ac | 000033c1000030008000000000004000 | YSQL | postgres | public | postgresqlkeyvalue | | \x8000 - a5913f11706c4d8a80d74b7001dfe157 | 000033c1000030008000000000004000 | YSQL | postgres | public | postgresqlkeyvalue | \x8000 | - 110ae7c832e7418bbfb56222a3e6a7ca | 000033c3000030008000000000004006 | YSQL | yugabyte | public | range_partitioned_table | | \x48800000015361000021 - 746f84ac4a894b9c914fd4a89d5f89fc | 000033c3000030008000000000004006 | YSQL | yugabyte | public | range_partitioned_table | \x48800000015361000021 | \x48800000025362000021 - f584ca3aa57e43278fd4b5042ab116be | 000033c3000030008000000000004006 | YSQL | yugabyte | public | range_partitioned_table | \x48800000025362000021 | \x48800000035363000021 - 6d566b767f0347879934338e1642f58e | 000033c3000030008000000000004006 | YSQL | yugabyte | public | range_partitioned_table | \x48800000035363000021 | - bc90fa993cc340458d7d4500213e5aed | 4c9c54fb3fcc47dcb29e58899afc5e21 | System | system | | transactions | | \x2000 - d106f1c5039a4127bf1bee83c5c3fec8 | 4c9c54fb3fcc47dcb29e58899afc5e21 | System | system | | transactions | \x2000 | \x4000 - 045af4a5aa744e1fb06e41f4af134ee0 | 4c9c54fb3fcc47dcb29e58899afc5e21 | System | system | | transactions | \x4000 | \x6000 - 0f63fe4806824a4ab17b0fd9cf144b8a | 4c9c54fb3fcc47dcb29e58899afc5e21 | System | system | | transactions | \x6000 | \x8000 - a7251899b197456fbec72f7cc64cc7ad | 4c9c54fb3fcc47dcb29e58899afc5e21 | System | system | | transactions | \x8000 | \xa000 - 46035a0bc4144f8ea372c93dc5d3a8b6 | 4c9c54fb3fcc47dcb29e58899afc5e21 | System | system | | transactions | \xa000 | \xc000 - 1a13e5b16aa841608390c56e63deab20 | 4c9c54fb3fcc47dcb29e58899afc5e21 | System | system | | transactions | \xc000 | \xe000 - 5b452227726444a78d1c84aaaf44f5c0 | 4c9c54fb3fcc47dcb29e58899afc5e21 | System | system | | transactions | \xe000 | -``` diff --git a/docs/content/stable/explore/query-1-performance/_index.md b/docs/content/stable/explore/query-1-performance/_index.md index 3f747234da4e..947d3a0216d0 100644 --- a/docs/content/stable/explore/query-1-performance/_index.md +++ b/docs/content/stable/explore/query-1-performance/_index.md @@ -110,7 +110,7 @@ For more information, refer to [Optimizing YSQL queries using pg_hint_plan](./pg {{}} diff --git a/docs/content/stable/explore/query-1-performance/pg-stat-statements.md b/docs/content/stable/explore/query-1-performance/pg-stat-statements.md index 6d83dcc05476..d65985493a62 100644 --- a/docs/content/stable/explore/query-1-performance/pg-stat-statements.md +++ b/docs/content/stable/explore/query-1-performance/pg-stat-statements.md @@ -11,10 +11,10 @@ menu: type: docs --- -{{}} + Databases can be resource-intensive, consuming a lot of memory CPU, IO, and network resources. Optimizing your SQL can be very helpful in minimizing resource utilization. The `pg_stat_statements` module helps you track planning and execution statistics for all the SQL statements executed by a server. It is installed by default. @@ -454,4 +454,4 @@ yugabyte=# select pg_stat_statements_reset(); ## Learn more - [Latency histogram and P99 latencies](../../../yugabyte-platform/alerts-monitoring/latency-histogram/) in YugabyteDB Anywhere -- [Active Session History](../../observability/active-session-history) + diff --git a/docs/content/stable/explore/query-1-performance/ycql-stat-statements.md b/docs/content/stable/explore/query-1-performance/ycql-stat-statements.md deleted file mode 100644 index bd20067660c3..000000000000 --- a/docs/content/stable/explore/query-1-performance/ycql-stat-statements.md +++ /dev/null @@ -1,163 +0,0 @@ ---- -title: Get query statistics using ycql_stat_statements -linkTitle: Get query statistics -description: Track planning and execution statistics for all YCQL statements executed by a server. -headerTitle: Get query statistics using ycql_stat_statements -menu: - stable: - identifier: ycql-stat-statements - parent: query-tuning - weight: 200 -type: docs ---- - -{{}} -{{}} -{{}} -{{}} - -Databases can be resource-intensive, consuming a lot of memory, CPU, IO, and network resources. By optimizing your CQL, you can minimize resource use. The `ycql_stat_statements` module helps you track planning and execution statistics for all the YCQL statements executed by a server. - -This view provides YCQL statement metrics (similar to pg_stat_statements) that are also present on `:12000/statements`, accessible via YSQL. The view can be joined with YCQL wait events in the [yb_active_session_history](../../observability/active-session-history/#yb-active-session-history) view using the query ID. - -This view is added in an extension `yb_ycql_utils`, which is not enabled by default. - -The columns of the `ycql_stat_statements` view are described in the following table. - -| Column | Type | Description | -| :----- | :--- | :---------- | -| queryid | int8 | Hash code to identify identical normalized queries. | -| query | text | Text of a representative statement. | -| is_prepared | bool | Indicates whether the statement is a prepared statement or an unprepared query. | -| calls | int8 | Number of times the statement is executed.| -| total_time | float8 | Total time spent executing the statement, in milliseconds. | -| min_time | float8 | Minimum time spent executing the statement, in milliseconds. | -| max_time | float8 | Maximum time spent executing the statement, in milliseconds. | -| mean_time | float8 | Mean time spent executing the statement, in milliseconds. | -| stddev_time | float8 | Population standard deviation of time spent executing the statement, in milliseconds. | - -## Examples - -{{% setup/local %}} - -Note that as this view is accessible via YSQL, run your examples using [ysqlsh](../../../admin/ysqlsh/#starting-ysqlsh). - -### Describe the columns in the view - -1. Create the extension `yb_ycql_utils` as follows: - - ```sql - yugabyte=# CREATE EXTENSION yb_ycql_utils; - ``` - -1. Get the view description as follows: - - ```sql - yugabyte=# \d ycql_stat_statements - ``` - - ```output - View "public.ycql_stat_statements" - Column | Type | Collation | Nullable | Default - -------------+------------------+-----------+----------+--------- - queryid | bigint | | | - query | text | | | - is_prepared | boolean | | | - calls | bigint | | | - total_time | double precision | | | - min_time | double precision | | | - max_time | double precision | | | - mean_time | double precision | | | - stddev_time | double precision | | | - ``` - -### Get basic information - -The following example uses a [YCQL workload generator](../../../benchmark/key-value-workload-ycql/) to run YCQL queries in the background and then uses YSQL to query `ycql_stat_statements`. - -Note that there is no need to create the extension before running the queries, as the view directly fetches the data from YB–TServer. - -```sql -yugabyte=# SELECT * FROM ycql_stat_statements; -``` - -```output - queryid | query | is_prepared | calls | total_time | min_time | max_time | mean_time | stddev_time -----------------------+-------------------------------------------------------------------------------------------------------------------------+-------------+---------+------------------+-----------+-----------+-------------------+-------------------- - 8473086508688080607 | CREATE KEYSPACE IF NOT EXISTS ybdemo_keyspace WITH REPLICATION = {'class': 'SimpleStrategy', 'replication_factor' : 1}; | t | 1 | 0.778 | 0.778 | 0.778 | 0.778 | 0 - -8368694706463025697 | USE ybdemo_keyspace; | t | 1 | 0.1905 | 0.1905 | 0.1905 | 0.1905 | 0 - 2594328841729435159 | CREATE TABLE IF NOT EXISTS CassandraKeyValue (k varchar, v blob, primary key (k)); | t | 1 | 72.598458 | 72.598458 | 72.598458 | 72.598458 | 0 - 1957997667337333449 | SELECT k, v FROM CassandraKeyValue WHERE k = ?; | t | 1316141 | 376623.670250017 | 0.045125 | 17.026083 | 0.286157539541749 | 0.336173036868407 - -1081940071252633265 | INSERT INTO CassandraKeyValue (k, v) VALUES (?, ?); | t | 80525 | 48712.6266890002 | 0.107417 | 38.126 | 0.604937928457004 | 0.501434959216858 - 5685694520060019787 | SELECT * FROM system.peers_v2 | f | 1 | 0.611459 | 0.611459 | 0.611459 | 0.611459 | 0 - -8693369126900706654 | SELECT * FROM system_schema.functions WHERE keyspace_name = 'ybdemo_keyspace' | f | 1 | 0.408416 | 0.408416 | 0.408416 | 0.408416 | 0 - 7231358282794359932 | SELECT * FROM system_schema.views WHERE keyspace_name = 'ybdemo_keyspace' AND view_name = 'cassandrakeyvalue' | f | 1 | 0.341 | 0.341 | 0.341 | 0.341 | 0 - -6069774349418914791 | SELECT * FROM system.local WHERE key='local' | f | 1 | 1.088167 | 1.088167 | 1.088167 | 1.088167 | 0 - -5046967885002247753 | SELECT peer, rpc_address, schema_version, host_id FROM system.peers | f | 1 | 0.301083 | 0.301083 | 0.301083 | 0.301083 | 0 - -3349549932189089002 | SELECT * FROM system_schema.columns WHERE keyspace_name = 'ybdemo_keyspace' | f | 1 | 1.400792 | 1.400792 | 1.400792 | 1.400792 | 0 - 6930116125454979846 | SELECT * FROM system_schema.views | f | 1 | 1.129625 | 1.129625 | 1.129625 | 1.129625 | 0 - 7981072946573997034 | select cluster_name from system.local where key = 'local' | f | 4 | 1.650293 | 0.300084 | 0.522709 | 0.41257325 | 0.0795915597358633 - 4163559541422844425 | SELECT * FROM system_schema.keyspaces WHERE keyspace_name = 'ybdemo_keyspace' | f | 1 | 0.375458 | 0.375458 | 0.375458 | 0.375458 | 0 - -1896671018756022147 | SELECT * FROM system_schema.functions | f | 1 | 0.7635 | 0.7635 | 0.7635 | 0.7635 | 0 - 477300852678741015 | SELECT * FROM system.peers | f | 1 | 1.051666 | 1.051666 | 1.051666 | 1.051666 | 0 - -5984255118081173147 | SELECT * FROM system_schema.aggregates WHERE keyspace_name = 'ybdemo_keyspace' | f | 1 | 0.507375 | 0.507375 | 0.507375 | 0.507375 | 0 - -199636290905897800 | SELECT * FROM system_schema.tables WHERE keyspace_name = 'ybdemo_keyspace' AND table_name = 'cassandrakeyvalue' | f | 1 | 1.229417 | 1.229417 | 1.229417 | 1.229417 | 0 - 6202644009413539627 | SELECT * FROM system_schema.types | f | 1 | 0.942792 | 0.942792 | 0.942792 | 0.942792 | 0 - 6660660976596803555 | SELECT peer, rpc_address, schema_version, host_id FROM system.peers | f | 1 | 0.4265 | 0.4265 | 0.4265 | 0.4265 | 0 - -4656374775045675304 | SELECT * FROM system_schema.tables | f | 1 | 1.838917 | 1.838917 | 1.838917 | 1.838917 | 0 - -2256941656319582329 | SELECT * FROM system_schema.columns | f | 1 | 3.208791 | 3.208791 | 3.208791 | 3.208791 | 0 - -2674195503457906853 | SELECT * FROM system_schema.aggregates | f | 1 | 1.452334 | 1.452334 | 1.452334 | 1.452334 | 0 - -2285418643723910393 | SELECT * FROM system_schema.tables WHERE keyspace_name = 'ybdemo_keyspace' | f | 1 | 1.251458 | 1.251458 | 1.251458 | 1.251458 | 0 - 194015117456500066 | SELECT keyspace_name, table_name, start_key, end_key, replica_addresses FROM system.partitions | f | 3 | 2.742959 | 0.253125 | 1.297125 | 0.914319666666667 | 0.469474504586659 - -3362784747732104326 | SELECT schema_version, host_id FROM system.local WHERE key='local' | f | 1 | 0.421417 | 0.421417 | 0.421417 | 0.421417 | 0 - -4290033807176898337 | SELECT * FROM system_schema.types WHERE keyspace_name = 'ybdemo_keyspace' | f | 1 | 0.346209 | 0.346209 | 0.346209 | 0.346209 | 0 - 1413414562899452953 | SELECT * FROM system_schema.indexes | f | 1 | 2.569667 | 2.569667 | 2.569667 | 2.569667 | 0 - -3220527242581763013 | SELECT * FROM system_schema.keyspaces | f | 1 | 1.031458 | 1.031458 | 1.031458 | 1.031458 | 0 - -4060076456160928053 | SELECT * FROM system_schema.indexes WHERE keyspace_name = 'ybdemo_keyspace' | f | 1 | 1.297167 | 1.297167 | 1.297167 | 1.297167 | 0 - 8278745859691011170 | SELECT * FROM system_schema.columns WHERE keyspace_name = 'ybdemo_keyspace' AND table_name = 'cassandrakeyvalue' | f | 1 | 1.280542 | 1.280542 | 1.280542 | 1.280542 | 0 - -5564581055929365977 | SELECT schema_version, host_id FROM system.local WHERE key='local' | f | 1 | 0.311208 | 0.311208 | 0.311208 | 0.311208 | 0 - -1076592564131011035 | SELECT * FROM system_schema.indexes WHERE keyspace_name = 'ybdemo_keyspace' AND table_name = 'cassandrakeyvalue' | f | 1 | 1.201042 | 1.201042 | 1.201042 | 1.201042 | 0 - 6806527679817077701 | SELECT * FROM system_schema.views WHERE keyspace_name = 'ybdemo_keyspace' | f | 1 | 0.421375 | 0.421375 | 0.421375 | 0.421375 | 0 -``` - -### Top 10 time consuming queries - -```sql -yugabyte=# SELECT query FROM ycql_stat_statements ORDER BY mean_time DESC LIMIT 10; -``` - -```output - query -------------------------------------------------------------------------------------------------------------------------- - SELECT * FROM system.local WHERE key='local' - SELECT * FROM system_schema.columns - SELECT * FROM system_schema.tables - SELECT * FROM system.peers_v2 - SELECT * FROM system_schema.indexes - SELECT * FROM system_schema.columns WHERE keyspace_name = 'ybdemo_keyspace' - SELECT * FROM system_schema.views - CREATE KEYSPACE IF NOT EXISTS ybdemo_keyspace WITH REPLICATION = {'class': 'SimpleStrategy', 'replication_factor' : 1}; - SELECT * FROM system_schema.indexes WHERE keyspace_name = 'ybdemo_keyspace' - SELECT * FROM system_schema.columns WHERE keyspace_name = 'ybdemo_keyspace' AND table_name = 'cassandrakeyvalue' -``` - -### Top 10 response-time outlier queries - -```sql -yugabyte=# SELECT query FROM ycql_stat_statements ORDER BY stddev_time DESC LIMIT 10; -``` - -```output - query ------------------------------------------------------------------------------------------------- - SELECT * FROM system.local WHERE key='local' - SELECT * FROM system.peers_v2 - SELECT * FROM system_schema.tables - INSERT INTO CassandraKeyValue (k, v) VALUES (?, ?); - SELECT * FROM system_schema.columns - select cluster_name from system.local where key = 'local' - SELECT k, v FROM CassandraKeyValue WHERE k = ?; - SELECT * FROM system_schema.indexes - SELECT keyspace_name, table_name, start_key, end_key, replica_addresses FROM system.partitions - SELECT * FROM system_schema.aggregates -``` \ No newline at end of file diff --git a/docs/content/stable/explore/ysql-language-features/advanced-features/views.md b/docs/content/stable/explore/ysql-language-features/advanced-features/views.md index 1b8160437723..2f29fee9bb9f 100644 --- a/docs/content/stable/explore/ysql-language-features/advanced-features/views.md +++ b/docs/content/stable/explore/ysql-language-features/advanced-features/views.md @@ -216,7 +216,7 @@ employee_no | name ## Read more -- [Active Session History](../../../observability/active-session-history) + For detailed documentation on materialized views, refer to the following topics: