`row` emits the row without any additional metadata fields in the message. *You can only use `row` with Kafka sinks or sinkless changefeeds. `row` does not support [`avro` format](#format).
`wrapped` emits the full message including any metadata fields. See [Responses](changefeed-messages.html#responses) for more detail on message format.
Default: `envelope=wrapped` +New in v23.1: `execution_locality` | Key-value pairs | Restricts the execution of a changefeed to nodes that match the defined locality filter requirements, e.g., `WITH execution_locality = 'region=us-west-1a,cloud=aws'`.
See [Run a changefeed job by locality](changefeeds-in-multi-region-deployments.html#run-a-changefeed-job-by-locality) for usage and reference detail. `format` | `json` / `avro` / `csv`* | Format of the emitted record. For mappings of CockroachDB types to Avro types, [see the table](changefeed-messages.html#avro-types) and detail on [Avro limitations](changefeed-messages.html#avro-limitations).
*`format=csv` works only in combination with [`initial_scan = 'only'`](#initial-scan). You cannot combine `format=csv` with the [`diff`](#diff-opt) or [`resolved`](#resolved-option) options. Changefeeds use the same CSV format as the [`EXPORT`](export.html) statement. See [Export data with changefeeds](export-data-with-changefeeds.html) for details using these options to create a changefeed as an alternative to `EXPORT`.
Default: `format=json`. `full_table_name` | N/A | Use fully qualified table name in topics, subjects, schemas, and record output instead of the default table name. This can prevent unintended behavior when the same table name is present in multiple databases.
**Note:** This option cannot modify existing table names used as topics, subjects, etc., as part of an [`ALTER CHANGEFEED`](alter-changefeed.html) statement. To modify a topic, subject, etc., to use a fully qualified table name, create a new changefeed with this option.
Example: `CREATE CHANGEFEED FOR foo... WITH full_table_name` will create the topic name `defaultdb.public.foo` instead of `foo`. +New in v23.1: `gc_protect_expires_after` | [Duration string](https://pkg.go.dev/time#ParseDuration) | Automatically expires protected timestamp records that are older than the defined duration. In the case where a changefeed job remains paused, `gc_protect_expires_after` will trigger the underlying protected timestamp record to expire and cancel the changefeed job to prevent accumulation of protected data. Use with [`protect_data_from_gc_on_pause`](#protect-pause) to limit the amount of time a changefeed job will remain paused protecting change data.
See [Garbage collection and changefeeds](changefeed-messages.html#garbage-collection-and-changefeeds) for more detail on protecting changefeed data. `initial_scan` | `yes`/`no`/`only` | Control whether or not an initial scan will occur at the start time of a changefeed. Only one `initial_scan` option (`yes`, `no`, or `only`) can be used. If none of these are set, an initial scan will occur if there is no [`cursor`](#cursor-option), and will not occur if there is one. This preserves the behavior from previous releases. With `initial_scan = 'only'` set, the changefeed job will end with a successful status (`succeeded`) after the initial scan completes. You cannot specify `yes`, `no`, `only` simultaneously.
If used in conjunction with `cursor`, an initial scan will be performed at the cursor timestamp. If no `cursor` is specified, the initial scan is performed at `now()`.
Although the [`initial_scan` / `no_initial_scan`](../v21.2/create-changefeed.html#initial-scan) syntax from previous versions is still supported, you cannot combine the previous and current syntax.
Default: `initial_scan = 'yes'` `kafka_sink_config` | [`STRING`](string.html) | Set fields to configure the required level of message acknowledgement from the Kafka server, the version of the server, and batching parameters for Kafka sinks. Set the message file compression type. See [Kafka sink configuration](changefeed-sinks.html#kafka-sink-configuration) for more detail on configuring all the available fields for this option.
Example: `CREATE CHANGEFEED FOR table INTO 'kafka://localhost:9092' WITH kafka_sink_config='{"Flush": {"MaxMessages": 1, "Frequency": "1s"}, "RequiredAcks": "ONE"}'` New in v23.1: `key_column` | `'column'` | Overrides the key used in [message metadata](changefeed-messages.html). This changes the key hashed to determine downstream partitions. In sinks that support partitioning by message, CockroachDB uses the [32-bit FNV-1a](https://en.wikipedia.org/wiki/Fowler%E2%80%93Noll%E2%80%93Vo_hash_function) hashing algorithm to determine which partition to send to.
**Note:** `key_column` does not preserve ordering of messages from CockroachDB to the downstream sink, therefore you must also include the [`unordered`](#unordered) option in your changefeed creation statement. It does not affect per-key [ordering guarantees](changefeed-messages.html#ordering-guarantees) or the output of [`key_in_value`](#key-in-value).
See the [Define a key to determine the changefeed sink partition](#define-a-key-to-determine-the-changefeed-sink-partition) example. `key_in_value` | N/A | Make the [primary key](primary-key.html) of a deleted row recoverable in sinks where each message has a value but not a key (most have a key and value in each message). `key_in_value` is automatically used for [cloud storage sinks](changefeed-sinks.html#cloud-storage-sink), [webhook sinks](changefeed-sinks.html#webhook-sink), and [GC Pub/Sub sinks](changefeed-sinks.html#google-cloud-pub-sub). {% include {{ page.version.version }}/cdc/cloud-storage-external-connection.md %} -`metrics_label` | [`STRING`](string.html) | This is an **experimental** feature. Define a metrics label to which the metrics for one or multiple changefeeds increment. All changefeeds also have their metrics aggregated.
The maximum length of a label is 128 bytes. There is a limit of 1024 unique labels.
`WITH metrics_label=label_name`
For more detail on usage and considerations, see [Using changefeed metrics labels](monitor-and-debug-changefeeds.html#using-changefeed-metrics-labels). +`metrics_label` | [`STRING`](string.html) | Define a metrics label to which the metrics for one or multiple changefeeds increment. All changefeeds also have their metrics aggregated.
The maximum length of a label is 128 bytes. There is a limit of 1024 unique labels.
`WITH metrics_label=label_name`
For more detail on usage and considerations, see [Using changefeed metrics labels](monitor-and-debug-changefeeds.html#using-changefeed-metrics-labels). `min_checkpoint_frequency` | [Duration string](https://pkg.go.dev/time#ParseDuration) | Controls how often nodes flush their progress to the [coordinating changefeed node](change-data-capture-overview.html#how-does-an-enterprise-changefeed-work). Changefeeds will wait for at least the specified duration before a flush to the sink. This can help you control the flush frequency of higher latency sinks to achieve better throughput. If this is set to `0s`, a node will flush as long as the high-water mark has increased for the ranges that particular node is processing. If a changefeed is resumed, then `min_checkpoint_frequency` is the amount of time that changefeed will need to catch up. That is, it could emit duplicate messages during this time.
**Note:** [`resolved`](#resolved-option) messages will not be emitted more frequently than the configured `min_checkpoint_frequency` (but may be emitted less frequently). Since `min_checkpoint_frequency` defaults to `30s`, you **must** configure `min_checkpoint_frequency` to at least the desired `resolved` message frequency if you require `resolved` messages more frequently than `30s`.
**Default:** `30s` `mvcc_timestamp` | N/A | Include the [MVCC](architecture/storage-layer.html#mvcc) timestamp for each emitted row in a changefeed. With the `mvcc_timestamp` option, each emitted row will always contain its MVCC timestamp, even during the changefeed's initial backfill. `on_error` | `pause` / `fail` | Use `on_error=pause` to pause the changefeed when encountering **non**-retryable errors. `on_error=pause` will pause the changefeed instead of sending it into a terminal failure state. **Note:** Retryable errors will continue to be retried with this option specified.
Use with [`protect_data_from_gc_on_pause`](#protect-pause) to protect changes from [garbage collection](configure-replication-zones.html#gc-ttlseconds).
Default: `on_error=fail` -`protect_data_from_gc_on_pause` | N/A | When a [changefeed is paused](pause-job.html), ensure that the data needed to [resume the changefeed](resume-job.html) is not garbage collected. If `protect_data_from_gc_on_pause` is **unset**, pausing the changefeed will release the existing protected timestamp records. It is also important to note that pausing and adding `protect_data_from_gc_on_pause` to a changefeed will not protect data if the [garbage collection](configure-replication-zones.html#gc-ttlseconds) window has already passed.
Use with [`on-error=pause`](#on-error) to protect changes from garbage collection when encountering non-retryable errors.
See [Garbage collection and changefeeds](changefeed-messages.html#garbage-collection-and-changefeeds) for more detail on protecting changefeed data.
**Note:** If you use this option, changefeeds that are left paused for long periods of time can prevent garbage collection. +`protect_data_from_gc_on_pause` | N/A | When a [changefeed is paused](pause-job.html), ensure that the data needed to [resume the changefeed](resume-job.html) is not garbage collected. If `protect_data_from_gc_on_pause` is **unset**, pausing the changefeed will release the existing protected timestamp records. It is also important to note that pausing and adding `protect_data_from_gc_on_pause` to a changefeed will not protect data if the [garbage collection](configure-replication-zones.html#gc-ttlseconds) window has already passed.
Use with [`on-error=pause`](#on-error) to protect changes from garbage collection when encountering non-retryable errors.
See [Garbage collection and changefeeds](changefeed-messages.html#garbage-collection-and-changefeeds) for more detail on protecting changefeed data.
**Note:** If you use this option, changefeeds that are left paused for long periods of time can prevent garbage collection. Use with the [`gc_protect_expires_after`](#gc-protect-expire) option to set a limit for protected data and for how long a changefeed will remain paused. `resolved` | [Duration string](https://pkg.go.dev/time#ParseDuration) | Emits [resolved timestamp](changefeed-messages.html#resolved-def) events per changefeed in a format dependent on the connected sink. Resolved timestamp events do not emit until all ranges in the changefeed have progressed to a specific point in time.
Set an optional minimal duration between emitting resolved timestamps. Example: `resolved='10s'`. This option will **only** emit a resolved timestamp event if the timestamp has advanced and at least the optional duration has elapsed. If unspecified, all resolved timestamps are emitted as the high-water mark advances.
**Note:** If you require `resolved` message frequency under `30s`, then you **must** set the [`min_checkpoint_frequency`](#min-checkpoint-frequency) option to at least the desired `resolved` frequency. This is because `resolved` messages will not be emitted more frequently than `min_checkpoint_frequency`, but may be emitted less frequently. `schema_change_events` | `default` / `column_changes` | The type of schema change event that triggers the behavior specified by the `schema_change_policy` option:
- `default`: Include all [`ADD COLUMN`](alter-table.html#add-column) events for columns that have a non-`NULL` [`DEFAULT` value](default-value.html) or are [computed](computed-columns.html), and all [`DROP COLUMN`](alter-table.html#drop-column) events.
- `column_changes`: Include all schema change events that add or remove any column.
Default: `schema_change_events=default` `schema_change_policy` | `backfill` / `nobackfill` / `stop` | The behavior to take when an event specified by the `schema_change_events` option occurs:
- `backfill`: When [schema changes with column backfill](changefeed-messages.html#schema-changes-with-column-backfill) are finished, output all watched rows using the new schema.
- `nobackfill`: For [schema changes with column backfill](changefeed-messages.html#schema-changes-with-column-backfill), perform no logical backfills. The changefeed will still emit any duplicate records for the table being altered, but will not emit the new schema records.
- `stop`: [schema changes with column backfill](changefeed-messages.html#schema-changes-with-column-backfill), wait for all data preceding the schema change to be resolved before exiting with an error indicating the timestamp at which the schema change occurred. An `error: schema change occurred at
` will display in the `cockroach.log` file.
Default: `schema_change_policy=backfill` diff --git a/v23.1/create-schedule-for-backup.md b/v23.1/create-schedule-for-backup.md index efeb81642ab..a5efdece8ea 100644 --- a/v23.1/create-schedule-for-backup.md +++ b/v23.1/create-schedule-for-backup.md @@ -94,7 +94,10 @@ The data being backed up will not be eligible for garbage collection until a suc You can also use the `exclude_data_from_backup` option with a scheduled backup as a way to prevent protected timestamps from prolonging garbage collection on a table. See the example [Exclude a table's data from backups](take-full-and-incremental-backups.html#exclude-a-tables-data-from-backups) for usage information. -We recommend monitoring for your backup schedule to alert for failed backups. See [Set up monitoring for the backup schedule](manage-a-backup-schedule.html#set-up-monitoring-for-the-backup-schedule) for more detail. +We recommend monitoring for your backup schedule to alert for failed backups: + +- See the [Backup and Restore Monitoring](backup-and-restore-monitoring.html) page for a general overview and list of metrics available for backup, scheduled backup, and restore jobs. +- See [Set up monitoring for the backup schedule](manage-a-backup-schedule.html#set-up-monitoring-for-the-backup-schedule) for metrics and monitoring backup schedules specifically. ## View and control backup schedules diff --git a/v23.1/data-types.md b/v23.1/data-types.md index dad01cb4c20..516af5987bd 100644 --- a/v23.1/data-types.md +++ b/v23.1/data-types.md @@ -30,6 +30,8 @@ Type | Description | Example [`STRING`](string.html) | A string of Unicode characters. | `'a1b2c3'` [`TIME`
`TIMETZ`](time.html) | `TIME` stores a time of day in UTC.
`TIMETZ` converts `TIME` values with a specified time zone offset from UTC. | `TIME '01:23:45.123456'`
`TIMETZ '01:23:45.123456-5:00'` [`TIMESTAMP`
`TIMESTAMPTZ`](timestamp.html) | `TIMESTAMP` stores a date and time pairing in UTC.
`TIMESTAMPTZ` converts `TIMESTAMP` values with a specified time zone offset from UTC. | `TIMESTAMP '2016-01-25 10:10:10'`
`TIMESTAMPTZ '2016-01-25 10:10:10-05:00'` +[`TSQUERY`](tsquery.html) | New in v23.1: A list of lexemes and operators used in [full-text search](full-text-search.html). | `'list' & 'lexem' & 'oper' & 'use' & 'full' & 'text' & 'search'` +[`TSVECTOR`](tsvector.html) | New in v23.1: A list of lexemes with optional integer positions and weights used in [full-text search](full-text-search.html). | `'full':13 'integ':7 'lexem':4 'list':2 'option':6 'posit':8 'search':15 'text':14 'use':11 'weight':10` [`UUID`](uuid.html) | A 128-bit hexadecimal value. | `7f9c24e8-3b12-4fef-91e0-56a2d5a246ec` ## Data type conversions and casts diff --git a/v23.1/error-handling-and-troubleshooting.md b/v23.1/error-handling-and-troubleshooting.md index 27bd42b58c8..7c7dddf22e0 100644 --- a/v23.1/error-handling-and-troubleshooting.md +++ b/v23.1/error-handling-and-troubleshooting.md @@ -23,21 +23,9 @@ Take a look at [Troubleshoot SQL Behavior](query-behavior-troubleshooting.html). ## Transaction retry errors -Messages with [the PostgreSQL error code `40001` and the string `restart transaction`](common-errors.html#restart-transaction) indicate that a transaction failed because it [conflicted with another concurrent or recent transaction accessing the same data](performance-best-practices-overview.html#transaction-contention). The transaction needs to be retried by the client. +Messages with the error code `40001` and the string `restart transaction` are known as [*transaction retry errors*](transaction-retry-error-reference.html). These indicate that a transaction failed due to [contention](performance-best-practices-overview.html#understanding-and-avoiding-transaction-contention) with another concurrent or recent transaction attempting to write to the same data. The transaction needs to be retried by the client. -If your language's client driver or ORM implements transaction retry logic internally (e.g., if you are using Python and [SQLAlchemy with the CockroachDB dialect](build-a-python-app-with-cockroachdb-sqlalchemy.html)), then you do not need to handle this logic from your application. - -If your driver or ORM does not implement this logic, then you will need to implement a retry loop in your application. - -{% include {{page.version.version}}/misc/client-side-intervention-example.md %} - -{{site.data.alerts.callout_info}} -If a consistently high percentage of your transactions are resulting in [transaction retry errors with the error code `40001` and the string `restart transaction`](common-errors.html#restart-transaction), then you may need to evaluate your [schema design](schema-design-overview.html) and data access patterns to find and remove sources of contention. For more information about contention, see [Transaction Contention](performance-best-practices-overview.html#transaction-contention). - -For more information about what is causing a specific transaction retry error code, see the [Transaction Retry Error Reference](transaction-retry-error-reference.html#transaction-retry-error-reference). -{{site.data.alerts.end}} - -For more information about transaction retry errors, see [Client-side retry handling](transaction-retry-error-reference.html#client-side-retry-handling). +{% include {{ page.version.version }}/performance/transaction-retry-error-actions.md %} ## Unsupported SQL features diff --git a/v23.1/follower-reads.md b/v23.1/follower-reads.md index 853454e3285..5f30ad37288 100644 --- a/v23.1/follower-reads.md +++ b/v23.1/follower-reads.md @@ -98,7 +98,7 @@ A _bounded staleness read_ is a historical read that uses a dynamic, system-dete Use bounded staleness follower reads when you: -- Need minimally stale reads from the nearest replica without blocking on [conflicting transactions](transactions.html#transaction-contention). This is possible because the historical timestamp is chosen dynamically and the least stale timestamp that can be served locally without blocking is used. +- Need minimally stale reads from the nearest replica without blocking on [conflicting transactions](performance-best-practices-overview.html#transaction-contention). This is possible because the historical timestamp is chosen dynamically and the least stale timestamp that can be served locally without blocking is used. - Can confine the read to a single statement that meets the [bounded staleness limitations](#bounded-staleness-read-limitations). - Need higher availability than is provided by [exact staleness reads](#exact-staleness-reads). Specifically, what we mean by availability in this context is: - The ability to serve a read with low latency from a local replica rather than a leaseholder. diff --git a/v23.1/full-text-search.md b/v23.1/full-text-search.md new file mode 100644 index 00000000000..0bfff629dd3 --- /dev/null +++ b/v23.1/full-text-search.md @@ -0,0 +1,472 @@ +--- +title: Full-Text Search +summary: Full-text searches using TSVECTOR and TSQUERY enable natural-language searches on documents with ranked results. +toc: true +docs_area: develop +--- + +{% include_cached new-in.html version="v23.1" %} A full-text search is used to perform natural-language searches on documents such as articles, websites, or other written formats. + +This page describes how to perform full-text searches using the provided [built-in functions](functions-and-operators.html#full-text-search-functions). + +{{site.data.alerts.callout_info}} +Some PostgreSQL syntax and features are unsupported. For details, see [Unsupported features](#unsupported-features). +{{site.data.alerts.end}} + +## How does full-text search work? + +In the PostgreSQL terminology, a *document* is a natural-language text [converted to a data type](#process-a-document) that is searchable using [specially formatted queries](#form-a-query). A document is typically stored within a single database row or concatenated from multiple fields. + +A full-text search has the following advantages over pattern matching with `LIKE` and `ILIKE`: + +- A full-text search can specify a [text search configuration](#text-search-configuration) that enables language-specific searches. +- The results of a full-text search can be [ranked](#rank-search-results). +- A full-text search can be accelerated using a [full-text index](#full-text-indexes). +- `LIKE` and `ILIKE` are only fast for prefix searches or when indexed with a [trigram index](trigram-indexes.html). + +{{site.data.alerts.callout_success}} +{% include {{ page.version.version }}/sql/use-case-trigram-indexes.md %} +{{site.data.alerts.end}} + +### Process a document + +To make a document searchable, convert it to the [`TSVECTOR`](tsvector.html) data type. A `TSVECTOR` value consists of individual *lexemes*, which are normalized strings used for text matching. Each lexeme also includes a list of integer positions that indicate where the lexeme existed in the original document. + +The `to_tsvector()` [built-in function](functions-and-operators.html#full-text-search-functions) converts a string input into a `TSVECTOR` value: + +{% include_cached copy-clipboard.html %} +~~~ sql +SELECT to_tsvector('How do trees get on the internet?'); +~~~ + +~~~ + to_tsvector +--------------------------------- + 'get':4 'internet':7 'tree':3 +~~~ + +This `TSVECTOR` consists of the lexemes `get`, `internet`, and `tree`. Normalization removes the following from the input: + +- Derivatives of words, which are reduced using a [stemming](https://en.wikipedia.org/wiki/Stemming) algorithm. In this example, "trees" is normalized to `tree`. +- *Stop words*. These are words that are considered not useful for indexing and searching, based on the [text search configuration](#text-search-configuration). This example does not specify a configuration, and `english` is used by default. "How", "do", "on", and "the" are identified as stop words. +- Punctuation and capitalization. + +In the preceding output, the integers indicate that `get` is in the fourth position, `internet` is in the seventh position, and `tree` is in the third position in the input. + +### Form a query + +A full-text search attempts to match a *query* to a document. A full-text search query has the [`TSQUERY`](tsquery.html) data type. Like `TSVECTOR`, a `TSQUERY` value consists of individual *lexemes*, which are normalized strings used for text matching. Lexemes in a `TSQUERY` are separated with any combination of `&` (AND), `|` (OR), `<->` (FOLLOWED BY), or `!` (NOT) operators. + +- The `to_tsquery()` [built-in function](functions-and-operators.html#full-text-search-functions) normalizes a `TSQUERY` input. The input must also be formatted as a `TSQUERY`, or the statement will error. + + {% include_cached copy-clipboard.html %} + ~~~ sql + SELECT to_tsquery('How & do & trees & get & on & the & internet?'); + ~~~ + + ~~~ + to_tsquery + ------------------------------- + 'tree' & 'get' & 'internet' + ~~~ + +- The `plainto_tsquery()` [built-in function](functions-and-operators.html#full-text-search-functions) converts a string input into a `TSQUERY` value, and separates the lexemes with `&` (AND): + + {% include_cached copy-clipboard.html %} + ~~~ sql + SELECT plainto_tsquery('How do trees get on the internet?'); + ~~~ + + ~~~ + plainto_tsquery + ------------------------------- + 'tree' & 'get' & 'internet' + ~~~ + +- The `phraseto_tsquery()` [built-in function](functions-and-operators.html#full-text-search-functions) converts a string input into a `TSQUERY` value, and separates the lexemes with `<->` (FOLLOWED BY): + + ~~~ sql + SELECT phraseto_tsquery('How do trees get on the internet?'); + ~~~ + + ~~~ + phraseto_tsquery + ----------------------------------- + 'tree' <-> 'get' <3> 'internet' + ~~~ + + In the preceding output, `<->` (equivalent to `<1>`) indicates that `get` must follow `tree` in a matching `TSVECTOR`. `<3>` further indicates that `get` and `internet` must be separated by **two** lexemes in a matching `TSVECTOR`. This resulted from converting the stop words "on" and "the" in the input. + + To match this query, a document must therefore contain phrases such as "get tree" and "get {word} {word} internet". + +Queries and documents are matched using the [`@@` comparison operator](#comparisons). For usage examples, see [Match queries to documents](#match-queries-to-documents). + +### Rank search results + +You can rank the results of a full-text search. + +The `ts_rank()` [built-in function](functions-and-operators.html#full-text-search-functions) outputs a search rank based on the frequency of matching lexemes. In the following example, two lexemes match: + +~~~ sql +SELECT ts_rank(to_tsvector('How do trees get on the internet?'), plainto_tsquery('how to get internet')); +~~~ + +{% include_cached copy-clipboard.html %} +~~~ + ts_rank +-------------- + 0.09735848 +~~~ + +In this example, three lexemes match, resulting in a higher rank: + +~~~ sql +SELECT ts_rank(to_tsvector('How do trees get on the internet?'), plainto_tsquery('wow, do trees get internet?')); +~~~ + +{% include_cached copy-clipboard.html %} +~~~ + ts_rank +-------------- + 0.26426345 +~~~ + +{{site.data.alerts.callout_info}} +Because a rank must be calculated for each matching document, ranking a full-text search can incur a performance overhead if there are many matching documents. +{{site.data.alerts.end}} + +For more information about using `ts_rank()`, see the [PostgreSQL documentation](https://www.postgresql.org/docs/15/textsearch-controls.html#TEXTSEARCH-RANKING). + +## Comparisons + +Full-text searches support the following comparison operator: + +- **matching**: [`@@`](functions-and-operators.html#operators). This operator is set between a `TSQUERY` and `TSVECTOR`, and returns `true` if the lexemes match. The `TSQUERY` and `TSVECTOR` can be specified in any order. + +For usage examples, see [Match queries to documents](#match-queries-to-documents). + +## Full-text indexes + +{{site.data.alerts.callout_info}} +You can perform full-text searches without a full-text index. However, an index will drastically improve search performance when searching a large number of documents. +{{site.data.alerts.end}} + +To create a full-text index, use the [`CREATE INDEX`](create-index.html) syntax that defines an [inverted index](inverted-indexes.html), specifying a `TSVECTOR` column. + +- Using the PostgreSQL-compatible syntax: + + ~~~ sql + CREATE INDEX {optional name} ON {table} USING GIN ({column}); + ~~~ + + {{site.data.alerts.callout_info}} + GIN and GiST indexes are implemented identically on CockroachDB. `GIN` and `GIST` are therefore synonymous when defining a full-text index. + {{site.data.alerts.end}} + +- Using `CREATE INVERTED INDEX`: + + ~~~ sql + CREATE INVERTED INDEX {optional name} ON {table} ({column}); + ~~~ + +For more ways to define full-text indexes, see [Create a full-text index with an expression](#create-a-full-text-index-with-an-expression) and [Create a full-text index with a stored computed column](#create-a-full-text-index-with-a-stored-computed-column). + +## Text search configuration + +A *text search configuration* determines how inputs are parsed into `TSVECTOR` and `TSQUERY` values. This includes a dictionary that is used to identify derivatives of words, as well as stop words to exclude when normalizing [documents](#process-a-document) and [queries](#form-a-query). + +The supported dictionaries are English, Danish, Dutch, Finnish, French, German, Hungarian, Italian, Norwegian, Portuguese, Russian, Spanish, Swedish, and Turkish. An additional `simple` dictionary does not perform stemming or stopwording when normalizing [documents](#process-a-document) or [queries](#form-a-query). + +You can specify a text search configuration as the first parameter when calling any of the [built-in functions](functions-and-operators.html#full-text-search-functions) to [process a document](#process-a-document) or [form a query](#form-a-query). For example: + +{% include_cached copy-clipboard.html %} +~~~ sql +SELECT to_tsvector('swedish', 'Hur får träd tillgång till internet?'); +~~~ + +~~~ + to_tsvector +---------------------------------------------- + 'får':2 'internet':6 'tillgång':4 'träd':3 +~~~ + +If you do not specify a configuration when calling the function, the value of the [`default_text_search_config`](set-vars.html#default-text-search-config) session variable is used. This defaults to `english` and can be changed as follows: + +{% include_cached copy-clipboard.html %} +~~~ sql +SET default_text_search_config = swedish; +~~~ + +For more information about text search configurations, see the [PostgreSQL documentation](https://www.postgresql.org/docs/current/textsearch-intro.html#TEXTSEARCH-INTRO-CONFIGURATIONS). + +{{site.data.alerts.callout_info}} +At this time, only the dictionary can be specified in a text search configuration. See [Unsupported features](#unsupported-features). +{{site.data.alerts.end}} + +## Examples + +### Match queries to documents + +Use the `@@` operator to match a query (`TSQUERY`) to a searchable document (`TSVECTOR`). In the following example, because the `TSQUERY` comprises the lexemes `get` and `internet`, which are both contained in the `TSVECTOR`, the output will be `true`: + +{% include_cached copy-clipboard.html %} +~~~ sql +SELECT to_tsvector('How do trees get on the internet?') @@ to_tsquery('How & to & get & internet?'); +~~~ + +~~~ + ?column? +------------ + t +~~~ + +Use the `plainto_tsquery()` [built-in function](functions-and-operators.html#full-text-search-functions) to match text to a searchable document. This search is equivalent to the preceding example: + +{% include_cached copy-clipboard.html %} +~~~ sql +SELECT to_tsvector('How do trees get on the internet?') @@ plainto_tsquery('How to get internet?'); +~~~ + +~~~ + ?column? +------------ + t +~~~ + +Use the `phraseto_tsquery()` [built-in function](functions-and-operators.html#full-text-search-functions) to match text phrases to a searchable document. Because `phraseto_tsquery()` separates the lexemes `get` and `internet` with the `<->` (FOLLOWED BY) operator, and the document does not contain a phrase like "get internet", the output will be `false`: + +{% include_cached copy-clipboard.html %} +~~~ sql +SELECT to_tsvector('How do trees get on the internet?') @@ phraseto_tsquery('How to get internet?'); +~~~ + +~~~ + ?column? +------------ + f +~~~ + +For an example of how text matching is used on a table, see [Perform a full-text search with ranked results](#perform-a-full-text-search-with-ranked-results). + +### Create a full-text index with an expression + +You can create an [expression index](expression-indexes.html) on a `STRING` column, using [`to_tsvector()`](#process-a-document) to convert the value to `TSVECTOR`. + +Given the table: + +{% include_cached copy-clipboard.html %} +~~~ sql +CREATE TABLE t (a STRING); +~~~ + +Create an expression index that converts column `a` to `TSVECTOR`: + +{% include_cached copy-clipboard.html %} +~~~ sql +CREATE INDEX ON t USING GIN (to_tsvector('english', a)); +~~~ + +{{site.data.alerts.callout_info}} +When using a [full-text search function](functions-and-operators.html#full-text-search-functions) in an expression index, you **must** specify a [text search configuration](#text-search-configuration). In the preceding example, the `english` configuration is specified. +{{site.data.alerts.end}} + +### Create a full-text index with a stored computed column + +You can create a full-text index on a [stored computed column](computed-columns.html) that has a `TSVECTOR` data type. + +Given the table: + +{% include_cached copy-clipboard.html %} +~~~ sql +CREATE TABLE t (a STRING); +~~~ + +Add a new `TSVECTOR` column that is computed from `a` using [`to_tsvector()`](#process-a-document): + +{% include_cached copy-clipboard.html %} +~~~ sql +ALTER TABLE t ADD COLUMN b TSVECTOR + AS (to_tsvector('english', a)) STORED; +~~~ + +{{site.data.alerts.callout_info}} +When using a [full-text search function](functions-and-operators.html#full-text-search-functions) in a stored generated column, you **must** specify a [text search configuration](#text-search-configuration). In the preceding example, the `english` configuration is specified. +{{site.data.alerts.end}} + +View the columns on the table: + +{% include_cached copy-clipboard.html %} +~~~ sql +SHOW COLUMNS FROM t; +~~~ + +~~~ + column_name | data_type | is_nullable | column_default | generation_expression | indices | is_hidden +--------------+-----------+-------------+----------------+---------------------------+---------------------+------------ + a | STRING | t | NULL | | {t_pkey} | f + rowid | INT8 | f | unique_rowid() | | {t_expr_idx,t_pkey} | t + b | TSVECTOR | t | NULL | to_tsvector('english', a) | {t_pkey} | f +(3 rows) +~~~ + +Create an inverted index on the `TSVECTOR` column: + +~~~ sql +CREATE INDEX ON t USING GIN (b); +~~~ + +### Perform a full-text search with ranked results + +1. Create a table with `STRING` columns: + + {% include_cached copy-clipboard.html %} + ~~~ sql + CREATE TABLE dadjokes (opener STRING, response STRING); + ~~~ + +1. Populate the table with sample values. These are the documents that you will search: + + {% include_cached copy-clipboard.html %} + ~~~ sql + INSERT INTO dadjokes (opener, response) VALUES + ('How do trees get on the internet?', 'They log on.'), + ('What do you call a pony with a sore throat?', 'A little horse.'), + ('What would a bathroom for fancy cats be called?', 'The glitter box.'), + ('Why did the scarecrow win an award?', 'It was outstanding in its field.'), + ('What kind of tree fits in your hand?', 'A palm tree.'), + ('What was a better invention than the first telephone?', 'The second one.'), + ('Where do you learn to make banana splits?', 'At sundae school.'), + ('How did the hipster burn the roof of his mouth?', 'He ate the pizza before it was cool.'), + ('What did one wall say to the other wall?', 'Meet you at the corner.'), + ('When does a joke become a dad joke?', 'When it becomes apparent.'); + ~~~ + +1. You can use `LIKE` or `ILIKE` to search for text. However, the results will be unranked: + + {% include_cached copy-clipboard.html %} + ~~~ sql + SELECT opener, response + FROM dadjokes + WHERE opener LIKE '%tree%' OR response LIKE '%tree%'; + ~~~ + + ~~~ + opener | response + ---------------------------------------+--------------- + How do trees get on the internet? | They log on. + What kind of tree fits in your hand? | A palm tree. + (2 rows) + ~~~ + +1. Create a full-text index on the concatenation of both table columns, specifying a [text search configuration](#text-search-configuration) (in this case, `english`), as is mandatory when [defining an expression index](#create-a-full-text-index-with-an-expression): + + ~~~ sql + CREATE INDEX ON dadjokes USING GIN (to_tsvector('english', opener || response)); + ~~~ + + {{site.data.alerts.callout_info}} + Because inverted joins on `TSVECTOR` values are not yet supported, this index won't be used to accelerate the SQL queries in this example. See [Unsupported features](#unsupported-features). + {{site.data.alerts.end}} + +1. Search the table for a query (in this case, `tree`), and rank the results. + + In the following statement, [`to_tsvector()`](#process-a-document) makes the table values searchable, [`to_tsquery()`](#form-a-query) forms the query, and [`ts_rank()`](#rank-search-results) calculates the search rankings: + + ~~~ sql + SELECT opener, response, ts_rank(joke, query) AS rank + FROM dadjokes, to_tsvector(opener || response) joke, to_tsquery('tree') query + WHERE query @@ joke + ORDER BY rank DESC + LIMIT 10; + ~~~ + + ~~~ + opener | response | rank + ---------------------------------------+--------------+-------------- + What kind of tree fits in your hand? | A palm tree. | 0.075990885 + How do trees get on the internet? | They log on. | 0.06079271 + (2 rows) + ~~~ + + The frequency of the `tree` lexeme in each row determines the difference in the rankings. + +1. Search the table for the query `calling`, and rank the results: + + ~~~ sql + SELECT opener, response, ts_rank(joke, query) AS rank + FROM dadjokes, to_tsvector(opener || response) joke, to_tsquery('calling') query + WHERE query @@ joke + ORDER BY rank DESC + LIMIT 10; + ~~~ + + ~~~ + opener | response | rank + --------------------------------------------------+------------------+------------- + What would a bathroom for fancy cats be called? | The glitter box. | 0.06079271 + What do you call a pony with a sore throat? | A little horse. | 0.06079271 + (2 rows) + ~~~ + + Unlike pattern matching with `LIKE` and `ILIKE`, a full-text search for `calling` produced matches. This is because [`to_tsvector()`](#process-a-document) and [`to_tsquery()`](#form-a-query) each normalized derivatives of the word "call" in their respective inputs to the lexeme `call`, using the default `english` [text search configuration](#text-search-configuration). + +1. Use [`plainto_tsquery()`](#form-a-query) to convert text input to a search query: + + ~~~ sql + SELECT opener, response, ts_rank(joke, query) AS rank + FROM dadjokes, to_tsvector(opener || response) joke, plainto_tsquery('no more joking, dad') query + WHERE query @@ joke + ORDER BY rank DESC + LIMIT 10; + ~~~ + + ~~~ + opener | response | rank + --------------------------------------+---------------------------+------------- + When does a joke become a dad joke? | When it becomes apparent. | 0.18681315 + (1 row) + ~~~ + +1. Alternatively, use [`phraseto_tsquery()`](#form-a-query) to search for matching text phrases (in this example, "joke dad"): + + ~~~ sql + SELECT opener, response, ts_rank(joke, query) AS rank + FROM dadjokes, to_tsvector(opener || response) joke, phraseto_tsquery('no more joking, dad') query + WHERE query @@ joke + ORDER BY rank DESC + LIMIT 10; + ~~~ + + ~~~ + opener | response | rank + ---------+----------+------- + (0 rows) + ~~~ + +## Unsupported features + +Some PostgreSQL syntax and features are unsupported. These include, but are not limited to: + +- Aspects of [text search configurations](#text-search-configuration) other than the specified dictionary. +- `websearch_to_tsquery()` built-in function. +- `tsquery_phrase()` built-in function. +- `ts_rank_cd()` built-in function. +- `setweight()` built-in function. +- Inverted joins on `TSVECTOR` values. +- `tsvector || tsvector` comparisons. +- `tsquery || tsquery` comparisons. +- `tsquery && tsquery` comparisons. +- `tsquery <-> tsquery` comparisons. +- `!! tsquery` comparisons. +- `tsquery @> tsquery` and `tsquery <@ tsquery` comparisons. + +For full details, see the [tracking issue](https://github.com/cockroachdb/cockroach/issues/41288). + +## See also + +- PostgreSQL documentation on [Full Text Search](https://www.postgresql.org/docs/current/textsearch.html) +- [`TSVECTOR`](tsvector.html) +- [`TSQUERY`](tsquery.html) +- [Inverted indexes](inverted-indexes.html) +- [Indexes](indexes.html) +- [SQL Statements](sql-statements.html) \ No newline at end of file diff --git a/v23.1/inverted-indexes.md b/v23.1/inverted-indexes.md index ec583035801..ccf8b5a4d8f 100644 --- a/v23.1/inverted-indexes.md +++ b/v23.1/inverted-indexes.md @@ -10,10 +10,11 @@ Generalized inverted indexes, or GIN indexes, store mappings from values within CockroachDB stores the contents of the following data types in GIN indexes: -- [JSONB](jsonb.html) -- [Arrays](array.html) +- [`JSONB`](jsonb.html) +- [`ARRAY`](array.html) - [Spatial data (`GEOMETRY` and `GEOGRAPHY` types)](spatial-indexes.html) -- [Strings (using trigram indexes)](trigram-indexes.html) +- [`TSVECTOR` (for full-text search)](tsvector.html) +- [`STRING` (using trigram indexes)](trigram-indexes.html) {{site.data.alerts.callout_success}}For a hands-on demonstration of using GIN indexes to improve query performance on a
JSONB column, see the JSON tutorial.{{site.data.alerts.end}}
@@ -60,7 +61,7 @@ This lets you search based on subcomponents.
### Creation
-You can use GIN indexes to improve the performance of queries using `JSONB` or `ARRAY` columns. You can create them:
+You can use GIN indexes to improve the performance of queries using [`JSONB`](jsonb.html), [`ARRAY`](array.html), [`TSVECTOR`](tsvector.html) columns (for [full-text searches](full-text-search.html)), or [`STRING`](string.html) (for [fuzzy searches using trigrams](trigram-indexes.html)). You can create them:
- Using the PostgreSQL-compatible syntax [`CREATE INDEX ... USING GIN`](create-index.html):
@@ -68,18 +69,28 @@ You can use GIN indexes to improve the performance of queries using `JSONB` or `
CREATE INDEX {optional name} ON {table} USING GIN ({column});
~~~
- You can also specify the `jsonb_ops` or `array_ops` opclass (for `JSONB` and `ARRAY` columns, respectively) using the syntax:
+ Also specify an opclass when [creating a trigram index](trigram-indexes.html#creation):
~~~ sql
CREATE INDEX {optional name} ON {table} USING GIN ({column} {opclass});
~~~
+ {{site.data.alerts.callout_success}}
+ You can also use the preceding syntax to specify the `jsonb_ops` or `array_ops` opclass (for `JSONB` and `ARRAY` columns, respectively).
+ {{site.data.alerts.end}}
+
- While creating the table, using the syntax [`CREATE INVERTED INDEX`](create-table.html#create-a-table-with-secondary-and-gin-indexes):
~~~ sql
CREATE INVERTED INDEX {optional name} ON {table} ({column});
~~~
+ Also specify an opclass when [creating a trigram index](trigram-indexes.html#creation):
+
+ ~~~ sql
+ CREATE INVERTED INDEX {optional name} ON {table} ({column} {opclass});
+ ~~~
+
### Selection
If a query contains a filter against an indexed `JSONB` or `ARRAY` column that uses any of the [supported operators](#comparisons), the GIN index is added to the set of index candidates.
@@ -193,7 +204,7 @@ CREATE TABLE users (
## Examples
-### Create a table with GIN index on a JSONB column
+### Create a table with GIN index on a `JSONB` column
In this example, let's create a table with a `JSONB` column and a GIN index:
@@ -269,7 +280,7 @@ Now, run a query that filters on the `JSONB` column:
(2 rows)
~~~
-### Add a GIN index to a table with an array column
+### Add a GIN index to a table with an `ARRAY` column
In this example, let's create a table with an `ARRAY` column first, and add the GIN index later:
@@ -335,7 +346,7 @@ Now, let’s add a GIN index to the table and run a query that filters on the `A
(2 rows)
~~~
-### Create a table with a partial GIN index on a JSONB column
+### Create a table with a partial GIN index on a `JSONB` column
In the same `users` table from [Create a table with GIN index on a JSONB column](#create-a-table-with-gin-index-on-a-jsonb-column), create a partial GIN index for online users.
@@ -369,10 +380,14 @@ SELECT * FROM users@idx_online_users WHERE user_profile->'online' = 'true' AND u
(1 row)
~~~
-### Create a trigram index on a STRING column
+### Create a trigram index on a `STRING` column
For an example showing how to create a trigram index on a [`STRING`](string.html) column, see [Trigram Indexes](trigram-indexes.html#examples).
+### Create a full-text index on a `TSVECTOR` column
+
+For an example showing how to create a full-text index on a [`TSVECTOR`](tsvector.html) column, see [Full-Text Search](full-text-search.html#examples).
+
### Inverted join examples
{% include {{ page.version.version }}/sql/inverted-joins.md %}
diff --git a/v23.1/jsonb.md b/v23.1/jsonb.md
index 80aeb78ba98..80a38d131d3 100644
--- a/v23.1/jsonb.md
+++ b/v23.1/jsonb.md
@@ -460,7 +460,7 @@ SELECT '100.50'::JSONB::DECIMAL;
(1 row)
~~~
-You use the [`parse_timestamp` function](functions-and-operators.html) to parse strings in `TIMESTAMP` format.
+You can use the [`parse_timestamp` function](functions-and-operators.html) to parse strings in `TIMESTAMP` format.
{% include_cached copy-clipboard.html %}
~~~ sql
diff --git a/v23.1/logging-use-cases.md b/v23.1/logging-use-cases.md
index 4272c75969f..d5974b8eb77 100644
--- a/v23.1/logging-use-cases.md
+++ b/v23.1/logging-use-cases.md
@@ -394,7 +394,7 @@ I210323 20:02:12.095253 59168 10@util/log/event_log.go:32 ⋮ [n1,client=‹[::1
- Preceding the `=` character is the `crdb-v2` event metadata. See the [reference documentation](log-formats.html#format-crdb-v2) for details on the fields.
- `ApplicationName` shows that the events originated from an application named `bank`. You can use this field to filter the logging output by application.
-- `ErrorText` shows that this query encountered a type of [transaction retry error](transaction-retry-error-reference.html#retry_write_too_old). For details on transaction retry errors and how to resolve them, see the [Transaction retry error reference](transaction-retry-error-reference.html).
+- `ErrorText` shows that this query encountered a [type of transaction retry error](transaction-retry-error-reference.html#retry_write_too_old). For details on transaction retry errors and how to resolve them, see the [Transaction Retry Error Reference](transaction-retry-error-reference.html#actions-to-take).
- `NumRetries` shows that the transaction was retried once before succeeding.
{{site.data.alerts.callout_info}}
diff --git a/v23.1/manage-a-backup-schedule.md b/v23.1/manage-a-backup-schedule.md
index e69197d0ca6..ecd59a16796 100644
--- a/v23.1/manage-a-backup-schedule.md
+++ b/v23.1/manage-a-backup-schedule.md
@@ -40,17 +40,19 @@ Further guidance on connecting to Amazon S3, Google Cloud Storage, Azure Storage
## Set up monitoring for the backup schedule
-We recommend that you [monitor your backup schedule with Prometheus](monitoring-and-alerting.html#prometheus-endpoint), and alert when there are anomalies such as backups that have failed or no backups succeeding over a certain amount of time— at which point, you can inspect schedules by running [`SHOW SCHEDULES`](show-schedules.html).
+We recommend that you [monitor your backup schedule with Prometheus](monitoring-and-alerting.html#prometheus-endpoint), and alert when there are anomalies such as backups that have failed or no backups succeeding over a certain amount of time—at which point, you can inspect schedules by running [`SHOW SCHEDULES`](show-schedules.html).
Metrics for scheduled backups fall into two categories:
- Backup schedule-specific metrics, aggregated across all schedules:
- - `schedules_BACKUP_started`: A counter for the total number of backups started by a schedule
- - `schedules_BACKUP_succeeded`: A counter for the number of backups started by a schedule that succeeded
- - `schedules_BACKUP_failed`: A counter for the number of backups started by a schedule that failed
+ - `schedules.BACKUP.started`: The total number of backups started by a schedule.
+ - `schedules.BACKUP.succeeded`: The number of backups started by a schedule that succeeded.
+ - `schedules.BACKUP.failed`: The number of backups started by a schedule that failed.
- When `schedules_BACKUP_failed` increments, run [`SHOW SCHEDULES`](show-schedules.html) to check which schedule is affected and to inspect the error in the `status` column.
+ When `schedules.BACKUP.failed` increments, run [`SHOW SCHEDULES`](show-schedules.html) to check which schedule is affected and to inspect the error in the `status` column.
+ - {% include_cached new-in.html version="v23.1" %} `schedules.BACKUP.protected_age_sec`: The age of the oldest [protected timestamp](architecture/storage-layer.html#protected-timestamps) record protected by backup schedules.
+ - {% include_cached new-in.html version="v23.1" %} `schedules.BACKUP.protected_record_count`: The number of [protected timestamp](architecture/storage-layer.html#protected-timestamps) records held by backup schedules.
- Scheduler-specific metrics:
diff --git a/v23.1/metrics.md b/v23.1/metrics.md
index a0763850111..cf7f85eb6de 100644
--- a/v23.1/metrics.md
+++ b/v23.1/metrics.md
@@ -5,7 +5,7 @@ toc: false
docs_area: reference.metrics
---
-As part of normal operation, CockroachDB continuously records metrics that track performance, latency, usage, and many other runtime indicators. These metrics are often useful in diagnosing problems, troubleshooting performance, or planning cluster infrastructure modifications. This page documents locations where metrics are exposed for analysis, and includes the full list of available metrics in CockroachDB.
+As part of normal operation, CockroachDB continuously records metrics that track performance, latency, usage, and many other runtime indicators. These metrics are often useful in diagnosing problems, troubleshooting performance, or planning cluster infrastructure modifications. This page documents locations where metrics are exposed for analysis.
## Available metrics
diff --git a/v23.1/monitor-and-debug-changefeeds.md b/v23.1/monitor-and-debug-changefeeds.md
index 5bd4de1facf..4a15388054c 100644
--- a/v23.1/monitor-and-debug-changefeeds.md
+++ b/v23.1/monitor-and-debug-changefeeds.md
@@ -20,7 +20,7 @@ The following define the categories of non-retryable errors:
- The changefeed cannot convert the data to the specified [output format](changefeed-messages.html). For example, there are [Avro](changefeed-messages.html#avro) types that changefeeds do not support, or a [CDC query](cdc-queries.html) is using an unsupported or malformed expression.
- The terminal error happens as part of established changefeed behavior. For example, you have specified the [`schema_change_policy=stop` option](create-changefeed.html#schema-policy) and a schema change happens.
-We recommend monitoring changefeeds with [Prometheus](monitoring-and-alerting.html#prometheus-endpoint) to avoid accumulation of garbage after a changefeed encounters an error. See [Garbage collection and changefeeds](changefeed-messages.html#garbage-collection-and-changefeeds) for more detail on how changefeeds interact with protected timestamps and garbage collection. In addition, see the [Recommended changefeed metrics to track](#recommended-changefeed-metrics-to-track) section for the essential metrics to track on a changefeed.
+We recommend monitoring changefeeds with [Prometheus](monitoring-and-alerting.html#prometheus-endpoint) to avoid accumulation of garbage after a changefeed encounters an error. See [Garbage collection and changefeeds](changefeed-messages.html#garbage-collection-and-changefeeds) for more detail on how changefeeds interact with [protected timestamps](architecture/storage-layer.html#protected-timestamps) and garbage collection. In addition, see the [Recommended changefeed metrics to track](#recommended-changefeed-metrics-to-track) section for the essential metrics to track on a changefeed.
## Monitor a changefeed
@@ -59,6 +59,17 @@ By default, changefeeds will retry errors with [some exceptions](#changefeed-ret
- `changefeed.error_retries`: The total number of retryable errors encountered by all changefeeds.
- `changefeed.failures`: The total number of changefeed jobs that have failed.
+#### Protected timestamp and garbage collection monitoring
+
+[Protected timestamps](architecture/storage-layer.html#protected-timestamps) will protect changefeed data from garbage collection in particular scenarios, but if a changefeed lags too far behind, the protected changes could cause data storage issues. See [Garbage collection and changefeeds](changefeed-messages.html#garbage-collection-and-changefeeds) for detail on when changefeed data is protected from garbage collection.
+
+{% include_cached new-in.html version="v23.1" %} You can monitor changefeed jobs for [protected timestamp](architecture/storage-layer.html#protected-timestamps) usage. We recommend setting up monitoring for the following metrics:
+
+- `jobs.changefeed.protected_age_sec`: Tracks the age of the oldest [protected timestamp](architecture/storage-layer.html#protected-timestamps) record protected by changefeed jobs. We recommend monitoring if `protected_age_sec` is greater than [`gc.ttlseconds`](configure-replication-zones.html#gc-ttlseconds). As `protected_age_sec` increases, garbage accumulation increases. Garbage collection will not progress on a table, database, or cluster if the protected timestamp record is present.
+- `jobs.changefeed.currently_paused`: Tracks the number of changefeed jobs currently considered [paused](pause-job.html). Since paused changefeed jobs can accumulate garbage, it is important to monitor the number of changefeeds paused.
+- `jobs.changefeed.expired_pts_records`: Tracks the number of expired [protected timestamp](architecture/storage-layer.html#protected-timestamps) records owned by changefeed jobs. You can monitor this metric in conjunction with the [`gc_protect_expires_after` option](create-changefeed.html#gc-protect-expire).
+- `jobs.changefeed.protected_record_count`: Tracks the number of [protected timestamp](architecture/storage-layer.html#protected-timestamps) records held by changefeed jobs.
+
### Using changefeed metrics labels
{{site.data.alerts.callout_info}}
@@ -160,7 +171,7 @@ I190312 18:56:53.537686 585 vendor/github.com/Shopify/sarama/client.go:170 [kaf
{% include_cached copy-clipboard.html %}
~~~ sql
-> SHOW CHANGEFEED JOBS;
+SHOW CHANGEFEED JOBS;
~~~
~~~
diff --git a/v23.1/node-shutdown.md b/v23.1/node-shutdown.md
index 3959aba829d..e1188d541e5 100644
--- a/v23.1/node-shutdown.md
+++ b/v23.1/node-shutdown.md
@@ -345,6 +345,8 @@ If the rebalancing stalls during decommissioning, replicas that have yet to move
Do **not** terminate the node process, delete the storage volume, or remove the VM before a `decommissioning` node has [changed its membership status](#status-change) to `decommissioned`. Prematurely terminating the process will prevent the node from rebalancing all of its range replicas onto other nodes gracefully, cause transient query errors in client applications, and leave the remaining ranges under-replicated and vulnerable to loss of [quorum](architecture/replication-layer.html#overview) if another node goes down.
{{site.data.alerts.end}}
+{% include {{page.version.version}}/prod-deployment/decommission-pre-flight-checks.md %}
+
### Terminate the node process
@@ -577,6 +579,15 @@ You can use [`cockroach node drain`](cockroach-node.html) to drain a node separa
- Use the correct topology pattern for your cluster.
-
-
- Your application is experiencing degraded performance with the following transaction retry errors:
-
-
-
SQLSTATE: 40001
- RETRY_WRITE_TOO_OLD
- RETRY_SERIALIZABLE
-
- The SQL Statement Contention dashboard in the DB Console is showing spikes over time. -
- The SQL Statement Errors graph in the DB Console is showing spikes in retries over time. -
Waiting status.SQLSTATE: 40001 and a transaction retry error message.crdb_internal.transaction_contention_events table indicates that your transactions have experienced contention.- Your application is experiencing transaction contention.
-
+
- The Hot Ranges page (DB Console) displays a higher-than-expected QPS for a range. +
- The Key Visualizer (DB Console) shows ranges with much higher-than-average write rates for the cluster. +
- Your cluster has hot spots.
-
@@ -77,15 +84,113 @@ This section provides solutions for common performance issues in your applicatio
### Transaction contention
-[Transaction contention](performance-best-practices-overview.html#transaction-contention) occurs when transactions issued from multiple clients at the same time operate on the same data. This can cause transactions to wait on each other (like when many people try to check out with the same cashier at a store) and decrease performance.
+[Transaction contention](performance-best-practices-overview.html#transaction-contention) is a state of conflict that occurs when:
+
+- A [transaction](transactions.html) is unable to complete due to another concurrent or recent transaction attempting to write to the same data. This is also called *lock contention*.
+- A transaction is [automatically retried](transactions.html#automatic-retries) because it could not be placed into a [serializable ordering](demo-serializable.html) among all of the currently-executing transactions. If the automatic retry is not possible or fails, a [*transaction retry error*](transaction-retry-error-reference.html) is emitted to the client, requiring the client application to [retry the transaction](transaction-retry-error-reference.html#client-side-retry-handling).
#### Indicators that your application is experiencing transaction contention
-{% include {{page.version.version}}/performance/contention-indicators.md %}
+##### Waiting transaction
+
+These are indicators that a transaction is trying to access a row that has been ["locked"](architecture/transaction-layer.html#writing) by another, concurrent write transaction.
+
+- The **Active Executions** table on the **Transactions** page ([{{ site.data.products.db }} Console](../cockroachcloud/transactions-page.html) or [DB Console](ui-transactions-page.html#active-executions-table)) shows transactions with `Waiting` in the **Status** column. You can sort the table by **Time Spent Waiting**.
+- Querying the [`crdb_internal.cluster_locks`](crdb-internal.html#cluster_locks) table shows transactions where [`granted`](crdb-internal.html#cluster-locks-columns) is `false`.
+
+These are indicators that lock contention occurred in the past:
+
+- Querying the [`crdb_internal.transaction_contention_events`](crdb-internal.html#transaction_contention_events) table indicates that your transactions have experienced lock contention.
+
+ - This is also shown in the **Transaction Executions** view on the **Insights** page ([{{ site.data.products.db }} Console](../cockroachcloud/insights-page.html#transaction-executions-view) and [DB Console](ui-insights-page.html#transaction-executions-view)). Transaction executions will display the **High Contention** insight.
+ {{site.data.alerts.callout_info}}
+ {% include {{ page.version.version }}/performance/sql-trace-txn-enable-threshold.md %}
+ {{site.data.alerts.end}}
+
+- The **SQL Statement Contention** graph ([{{ site.data.products.db }} Console](../cockroachcloud/metrics-page.html#sql-statement-contention) and [DB Console](ui-sql-dashboard.html#sql-statement-contention)) is showing spikes over time.
+
+
+If a long-running transaction is waiting due to [lock contention](performance-best-practices-overview.html#transaction-contention):
+
+1. [Identify the blocking transaction](#identify-conflicting-transactions).
+1. Evaluate whether you can cancel the transaction. If so, [cancel it](#cancel-a-blocking-transaction) to unblock the waiting transaction.
+1. Optimize the transaction to [reduce further contention](#reduce-transaction-contention). In particular, break down larger transactions such as [bulk deletes](bulk-delete-data.html) into smaller ones to have transactions hold locks for a shorter duration, and use [historical reads](as-of-system-time.html) when possible to reduce conflicts with other writes.
+
+If lock contention occurred in the past, you can [identify the transactions and objects that experienced lock contention](#identify-transactions-and-objects-that-experienced-lock-contention).
+
+##### Transaction retry error
+
+These are indicators that a transaction has failed due to [contention](performance-best-practices-overview.html#transaction-contention).
+
+- A [transaction retry error](transaction-retry-error-reference.html) with `SQLSTATE: 40001`, the string [`restart transaction`](common-errors.html#restart-transaction), and an error code such as [`RETRY_WRITE_TOO_OLD`](transaction-retry-error-reference.html#retry_write_too_old) or [`RETRY_SERIALIZABLE`](transaction-retry-error-reference.html#retry_serializable), is emitted to the client.
+- An event with `TransactionRetryWithProtoRefreshError` is emitted to the CockroachDB [logs](logging-use-cases.html#example-slow-sql-query).
+
+These are indicators that transaction retries occurred in the past:
+
+- The **Transaction Restarts** graph ([{{ site.data.products.db }} Console](../cockroachcloud/metrics-page.html#transaction-restarts) and [DB Console](ui-sql-dashboard.html#transaction-restarts) is showing spikes in transaction retries over time.
+
+{% include {{ page.version.version }}/performance/transaction-retry-error-actions.md %}
#### Fix transaction contention problems
-{% include {{ page.version.version }}/performance/statement-contention.md %}
+Identify the transactions that are in conflict, and unblock them if possible. In general, take steps to [reduce transaction contention](#reduce-transaction-contention).
+
+In addition, implement [client-side retry handling](transaction-retry-error-reference.html#client-side-retry-handling) so that your application can respond to [transaction retry errors](transaction-retry-error-reference.html) that are emitted when CockroachDB cannot [automatically retry](transactions.html#automatic-retries) a transaction.
+
+##### Identify conflicting transactions
+
+- In the **Active Executions** table on the **Transactions** page ([{{ site.data.products.db }} Console](../cockroachcloud/transactions-page.html) or [DB Console](ui-transactions-page.html#active-executions-table)), look for a **waiting** transaction (`Waiting` status).
+ {{site.data.alerts.callout_success}}
+ If you see many waiting transactions, a single long-running transaction may be blocking transactions that are, in turn, blocking others. In this case, sort the table by **Time Spent Waiting** to find the transaction that has been waiting for the longest amount of time. Unblocking this transaction may unblock the other transactions.
+ {{site.data.alerts.end}}
+ Click the transaction's execution ID and view the following transaction execution details:
+ 
+ - **Last Retry Reason** shows the last [transaction retry error](#transaction-retry-error) received for the transaction, if applicable.
+ - The details of the **blocking** transaction, directly below the **Contention Insights** section. Click the blocking transaction to view its details.
+
+##### Cancel a blocking transaction
+
+1. [Identify the **blocking** transaction](#identify-conflicting-transactions) and view its transaction execution details.
+1. Click its **Session ID** to open the **Session Details** page.
+ 
+1. Click **Cancel Statement** to cancel the **Most Recent Statement** and thus the transaction, or click **Cancel Session** to cancel the session issuing the transaction.
+
+##### Identify transactions and objects that experienced lock contention
+
+To identify transactions that experienced [lock contention](performance-best-practices-overview.html#transaction-contention) in the past:
+
+- In the **Transaction Executions** view on the **Insights** page ([{{ site.data.products.db }} Console](../cockroachcloud/insights-page.html#transaction-executions-view) and [DB Console](ui-insights-page.html#transaction-executions-view)), look for a transaction with the **High Contention** insight. Click the transaction's execution ID and view the transaction execution details, including the details of the blocking transaction.
+- Visit the **Transactions** page ([{{ site.data.products.db }} Console](../cockroachcloud/transactions-page.html) and [DB Console](ui-transactions-page.html)) and sort transactions by **Contention Time**.
+
+To view tables and indexes that experienced [contention](performance-best-practices-overview.html#transaction-contention):
+
+- Query the [`crdb_internal.transaction_contention_events`](crdb-internal.html#transaction_contention_events) table to view [transactions that have blocked other transactions](crdb-internal.html#transaction-contention-example).
+- Query the [`crdb_internal.cluster_contended_tables`](crdb-internal.html#cluster_contended_tables) table to [view all tables that have experienced contention](crdb-internal.html#view-all-tables-that-have-experienced-contention).
+- Query the [`crdb_internal.cluster_contended_indexes`](crdb-internal.html#cluster_contended_indexes) table to [view all indexes that have experienced contention](crdb-internal.html#view-all-indexes-that-have-experienced-contention).
+- Query the [`crdb_internal.cluster_contention_events`](crdb-internal.html#cluster_contention_events) table
+to [view the tables, indexes, and transactions with the most time under contention](crdb-internal.html#view-the-tables-indexes-with-the-most-time-under-contention).
+
+##### Reduce transaction contention
+
+[Contention](performance-best-practices-overview.html#transaction-contention) is often reported after it has already resolved. Therefore, preventing contention before it affects your cluster's performance is a more effective approach:
+
+{% include {{ page.version.version }}/performance/reduce-contention.md %}
+
+### Hot spots
+
+[Hot spots](performance-best-practices-overview.html#hot-spots) are a symptom of *resource contention* and can create problems as requests increase, including excessive [transaction contention](#transaction-contention).
+
+#### Indicators that your cluster has hot spots
+
+- The **CPU Percent** graph on the [**Hardware**](ui-hardware-dashboard.html) and [**Overload**](ui-overload-dashboard.html) dashboards (DB Console) shows spikes in CPU usage.
+- The **Hot Ranges** list on the [**Hot Ranges** page](ui-hot-ranges-page.html) (DB Console) displays a higher-than-expected QPS for a range.
+- The [**Key Visualizer**](ui-key-visualizer.html) (DB Console) shows [ranges with much higher-than-average write rates](ui-key-visualizer.html#identifying-hot-spots) for the cluster.
+
+If you find hot spots, use the [**Range Report**](ui-hot-ranges-page.html#range-report) and [**Key Visualizer**](ui-key-visualizer.html) to identify the ranges with excessive traffic. Then take steps to [reduce hot spots](#reduce-hot-spots).
+
+#### Reduce hot spots
+
+{% include {{ page.version.version }}/performance/reduce-hot-spots.md %}
### Statements with full table scans
diff --git a/v23.1/query-behavior-troubleshooting.md b/v23.1/query-behavior-troubleshooting.md
index 3d83963e9aa..f5c0fc2065d 100644
--- a/v23.1/query-behavior-troubleshooting.md
+++ b/v23.1/query-behavior-troubleshooting.md
@@ -17,42 +17,9 @@ For a developer-centric overview of optimizing SQL statement performance, see [O
When you experience a hanging or stuck query and the cluster is healthy (i.e., no [unavailable ranges](ui-replication-dashboard.html#unavailable-ranges), [network partitions](cluster-setup-troubleshooting.html#network-partition), etc), the cause could be a long-running transaction holding [write intents](architecture/transaction-layer.html#write-intents) open against the same rows as your query.
-Such long-running queries can hold intents open for (practically) unlimited durations. If your query tries to access those rows, it may have to wait for that transaction to complete (by [committing](commit-transaction.html) or [rolling back](rollback-transaction.html)) before it can make progress.
+Such long-running queries can hold intents open for (practically) unlimited durations. If your query tries to access those rows, it may have to wait for that transaction to complete (by [committing](commit-transaction.html) or [rolling back](rollback-transaction.html)) before it can make progress. Until the transaction is committed or rolled back, the chances of concurrent transactions internally retrying and throwing a retry error increase.
-This situation is hard to diagnose via the [Transactions](ui-transactions-page.html) and [Statements](ui-statements-page.html) pages in the [DB Console](ui-overview.html) since [contention](performance-best-practices-overview.html#transaction-contention) is only reported after the conflict has been resolved (which in this scenario may be never).
-
-In these cases, you will need to take the following steps.
-
-1. [Find long running transactions](#step-1-find-long-running-transactions)
-1. [Find client sessions for those transactions](#step-2-find-the-client-session)
-1. [Cancel the transaction or session](#step-3-cancel-the-transaction-or-session)
-
-#### Step 1. Find long-running transactions
-
-Run the following query against the [`crdb_internal.cluster_transactions`](crdb-internal.html#cluster_transactions) table to list transactions that have been running longer than 10 minutes.
-
-{% include_cached copy-clipboard.html %}
-~~~ sql
-SELECT now() - start AS dur, * FROM crdb_internal.cluster_transactions WHERE now() - start > '10m'::INTERVAL ORDER BY dur DESC LIMIT 10
-~~~
-
-For each row in the results, if the `txn_string` column shows `lock=true` (or `seq > 0`), the transaction associated with that row is a writing transaction, and its open write intents will block access for other transactions.
-If the query returns lots of transactions, it is often the case than a single transaction is blocking others, and those may be blocking yet others. Try to look for the oldest, longest-running transaction and cancel that one first; that may be sufficient to unblock all of the others.
-
-#### Step 2. Find the client session
-
-Next, find the client session owning the long-running transaction by querying the [`crdb_internal.cluster_sessions`](crdb-internal.html#cluster_sessions) table. You will need the value of the `id` column from the query in the previous step.
-This step is necessary if you want to cancel the entire session the transaction is associated with.
-
-~~~ sql
-SELECT * FROM crdb_internal.cluster_sessions WHERE kv_txn = {id_column_from_previous_query}
-~~~
-
-#### Step 3. Cancel the transaction or session
-
-Finally, cancel the longest-running transaction you found in [Step 1](#step-1-find-long-running-transactions) using [`CANCEL QUERY`](cancel-query.html) and check if that resolves the problem.
-
-If you want to cancel the whole session for that transaction, use [`CANCEL SESSION`](cancel-session.html) using the session ID you found in [Step 2](#step-2-find-the-client-session).
+Refer to the performance tuning recipe for [identifying and unblocking a waiting transaction](performance-recipes.html#waiting-transaction).
### Identify slow queries
diff --git a/v23.1/recommended-production-settings.md b/v23.1/recommended-production-settings.md
index d04d897c4e8..c717960a1fb 100644
--- a/v23.1/recommended-production-settings.md
+++ b/v23.1/recommended-production-settings.md
@@ -670,4 +670,6 @@ When running CockroachDB on Kubernetes, making the following minimal customizati
For more information and additional customization suggestions, see our full detailed guide to [CockroachDB Performance on Kubernetes](kubernetes-performance.html).
-{% include common/transaction-retries.md %}
+## Transaction retries
+
+When several transactions try to modify the same underlying data concurrently, they may experience [contention](performance-best-practices-overview.html#transaction-contention) that leads to [transaction retries](transactions.html#transaction-retries). To avoid failures in production, your application should be engineered to handle transaction retries using [client-side retry handling](transaction-retry-error-reference.html#client-side-retry-handling).
\ No newline at end of file
diff --git a/v23.1/schema-design-table.md b/v23.1/schema-design-table.md
index 4f84636e873..1a6405e91cb 100644
--- a/v23.1/schema-design-table.md
+++ b/v23.1/schema-design-table.md
@@ -211,7 +211,7 @@ Here are some best practices to follow when selecting primary key columns:
- Avoid defining primary keys over a single column of sequential data.
- Querying a table with a primary key on a single sequential column (e.g., an auto-incrementing [`INT`](int.html) column, or a [`TIMESTAMP`](timestamp.html) value) can result in single-range [hot spots](performance-best-practices-overview.html#hot-spots) that negatively affect performance, or cause [transaction contention](transactions.html#transaction-contention).
+ Querying a table with a primary key on a single sequential column (e.g., an auto-incrementing [`INT`](int.html) column, or a [`TIMESTAMP`](timestamp.html) value) can result in single-range [hot spots](performance-best-practices-overview.html#hot-spots) that negatively affect performance, or cause [transaction contention](performance-best-practices-overview.html#transaction-contention).
If you are working with a table that *must* be indexed on sequential keys, use [hash-sharded indexes](hash-sharded-indexes.html). For details about the mechanics and performance improvements of hash-sharded indexes in CockroachDB, see our [Hash Sharded Indexes Unlock Linear Scaling for Sequential Workloads](https://www.cockroachlabs.com/blog/hash-sharded-indexes-unlock-linear-scaling-for-sequential-workloads/) blog post.
diff --git a/v23.1/show-jobs.md b/v23.1/show-jobs.md
index f677ad830e1..c8cda4bc9c6 100644
--- a/v23.1/show-jobs.md
+++ b/v23.1/show-jobs.md
@@ -101,6 +101,10 @@ Status | Description
`revert-failed` | Job encountered a non-retryable error when reverting the changes. It is necessary to manually clean up a job with this status.
`retrying` | Job is retrying another job that failed.
+{{site.data.alerts.callout_info}}
+We recommend monitoring paused jobs to protect historical data from [garbage collection](architecture/storage-layer.html#garbage-collection), or potential data accumulation in the case of [changefeeds](changefeed-messages.html#garbage-collection-and-changefeeds). See [Monitoring paused jobs](pause-job.html#monitoring-paused-jobs) for detail on metrics to track paused jobs and [protected timestamps](architecture/storage-layer.html#protected-timestamps).
+{{site.data.alerts.end}}
+
## Examples
### Show jobs
diff --git a/v23.1/show-ranges.md b/v23.1/show-ranges.md
index 3d92344750d..4086361435f 100644
--- a/v23.1/show-ranges.md
+++ b/v23.1/show-ranges.md
@@ -5,9 +5,13 @@ toc: true
docs_area: reference.sql
---
-The `SHOW RANGES` [statement](sql-statements.html) shows information about the [ranges](architecture/overview.html#architecture-range) that comprise the data for a table, index, or entire database. This information is useful for verifying how SQL data maps to underlying ranges, and where the replicas for ranges are located. If `SHOW RANGES` displays `NULL` for both the start and end keys of a range, the range is empty and has no splits.
+The `SHOW RANGES` [statement](sql-statements.html) shows information about the [ranges](architecture/overview.html#architecture-range) that comprise the data for a table, index, database, or the current catalog. This information is useful for verifying how SQL data maps to underlying [ranges](architecture/overview.html#architecture-range), and where the [replicas](architecture/glossary.html#replica) for those ranges are located.
{{site.data.alerts.callout_info}}
+{% include {{page.version.version}}/sql/show-ranges-output-deprecation-notice.md %}
+{{site.data.alerts.end}}
+
+{{site.data.alerts.callout_success}}
To show range information for a specific row in a table or index, use the [`SHOW RANGE ... FOR ROW`](show-range-for-row.html) statement.
{{site.data.alerts.end}}
@@ -25,96 +29,388 @@ To use the `SHOW RANGES` statement, a user must either be a member of the [`admi
Parameter | Description
----------|------------
-[`table_name`](sql-grammar.html#table_name) | The name of the table you want range information about.
-[`table_index_name`](sql-grammar.html#table_index_name) | The name of the index you want range information about.
-[`database_name`](sql-grammar.html#database_name) | The name of the database you want range information about.
+[`table_name`](sql-grammar.html#table_name) | The name of the [table](show-tables.html) you want [range](architecture/overview.html#architecture-range) information about.
+[`table_index_name`](sql-grammar.html#table_index_name) | The name of the [index](indexes.html) you want [range](architecture/overview.html#architecture-range) information about.
+[`database_name`](sql-grammar.html#database_name) | The name of the [database](show-databases.html) you want [range](architecture/overview.html#architecture-range) information about.
+[`opt_show_ranges_options`](sql-grammar.html#show_ranges_options) | The [options](#options) used to configure what fields appear in the [response](#response).
+
+## Options
+
+The following [options](sql-grammar.html#show_ranges_options) are available to affect the output. Multiple options can be passed at once, separated by commas.
+
+- `TABLES`: List [tables](show-tables.html) contained per [range](architecture/overview.html#architecture-range).
+- `INDEXES`: List [indexes](indexes.html) contained per [range](architecture/overview.html#architecture-range).
+- `DETAILS`: Add [range](architecture/overview.html#architecture-range) size, [leaseholder](architecture/glossary.html#leaseholder) and other details. Note that this incurs a large computational overhead because it needs to fetch data across nodes.
+- `KEYS`: Include binary [start and end keys](#start-key).
## Response
-The following fields are returned for each partition:
+The specific fields in the response vary depending on the values passed as [options](#options). The following fields may be returned:
-Field | Description
-------|------------
-`table_name` | The name of the table.
-`start_key` | The start key for the range.
-`end_key` | The end key for the range.
-`range_id` | The range ID.
-`range_size_mb` | The size of the range.
-`lease_holder` | The node that contains the range's [leaseholder](architecture/overview.html#architecture-range).
-`lease_holder_locality` | The [locality](cockroach-start.html#locality) of the leaseholder.
-`replicas` | The nodes that contain the range [replicas](architecture/overview.html#architecture-range).
-`replica_localities` | The [locality](cockroach-start.html#locality) of the range.
+Field | Description | Emitted for option(s)
+------|-------------|----------------------
+`start_key` | The start key for the [range](architecture/overview.html#architecture-range). | Always emitted.
+`end_key` | The end key for the [range](architecture/overview.html#architecture-range). | Always emitted.
+`raw_start_key` | The start key for the [range](architecture/overview.html#architecture-range), displayed as a [hexadecimal byte value](sql-constants.html#string-literals-with-character-escapes). | `KEYS`
+`raw_end_key` | The end key for the [range](architecture/overview.html#architecture-range), displayed as a [hexadecimal byte value](sql-constants.html#string-literals-with-character-escapes). | `KEYS`
+`range_id` | The internal [range](architecture/overview.html#architecture-range) ID. | Always emitted.
+`voting_replicas` | The [nodes](architecture/glossary.html#node) that contain the range's voting replicas (that is, the replicas that participate in [Raft](architecture/replication-layer.html#raft) elections). | Always emitted.
+`non_voting_replicas` | The [nodes](architecture/glossary.html#node) that contain the range's [non-voting replicas](architecture/replication-layer.html#non-voting-replicas). | Always emitted.
+`replicas` | The [nodes](architecture/glossary.html#node) that contain the range's [replicas](architecture/glossary.html#replica). | Always emitted.
+`replica_localities` | The [localities](cockroach-start.html#locality) of the range's [replicas](architecture/glossary.html#replica). | Always emitted.
+`range_size` | The size of the [range](architecture/overview.html#architecture-range) in bytes. | `DETAILS`
+`range_size_mb` | The size of the [range](architecture/overview.html#architecture-range) in MiB. | `DETAILS`
+`lease_holder` | The [node](architecture/glossary.html#node) that contains the range's [leaseholder](architecture/glossary.html#leaseholder). | `DETAILS`
+`lease_holder_locality` | The [locality](cockroach-start.html#locality) of the range's [leaseholder](architecture/glossary.html#leaseholder). | `DETAILS`
+`learner_replicas` | The _learner replicas_ of the range. A learner replica is a replica that has just been added to a range, and is thus in an interim state. It accepts messages but doesn't vote in [Raft](architecture/replication-layer.html#raft) elections. This means it doesn't affect quorum and thus doesn't affect the stability of the range, even if it's very far behind. | Always emitted.
+`split_enforced_until` | The time a [range split](architecture/distribution-layer.html#range-splits) is enforced until. This can be set using [`ALTER TABLE ... SPLIT AT`](alter-table.html#split-at) using the [`WITH EXPIRATION` clause](alter-table.html#set-the-expiration-on-a-split-enforcement). Example: `2262-04-11 23:47:16.854776` (this is a default value which means "never"). | Always emitted.
+`schema_name` | The name of the [schema](create-schema.html) this [range](architecture/overview.html#architecture-range) holds data for. | `TABLES`, `INDEXES`
+`table_name` | The name of the [table](create-table.html) this [range](architecture/overview.html#architecture-range) holds data for. | `TABLES`, `INDEXES`
+`table_id` | The internal ID of the [table](create-table.html) this [range](architecture/overview.html#architecture-range) holds data for. | `TABLES`, `INDEXES`
+`table_start_key` | The start key of the first [range](architecture/overview.html#architecture-range) that holds data for this table. | `TABLES`
+`table_end_key` | The end key of the last [range](architecture/overview.html#architecture-range) that holds data for this table. | `TABLES`
+`raw_table_start_key` | The start key of the first [range](architecture/overview.html#architecture-range) that holds data for this table, expressed as [`BYTES`](bytes.html). | `TABLES`, `KEYS`
+`raw_table_end_key` | The end key of the last [range](architecture/overview.html#architecture-range) that holds data for this table, expressed as [`BYTES`](bytes.html). | `TABLES`, `KEYS`
+`index_name` | The name of the [index](indexes.html) this [range](architecture/overview.html#architecture-range) holds data for. | `INDEXES`
+`index_id` | The internal ID of the [index](indexes.html) this [range](architecture/overview.html#architecture-range) holds data for. | `INDEXES`
+`index_start_key` | The start key of the first [range](architecture/overview.html#architecture-range) of [index](indexes.html) data. | `INDEXES`
+`index_end_key` | The end key of the last [range](architecture/overview.html#architecture-range) of [index](indexes.html) data. | `INDEXES`
+`raw_index_start_key` | The start key of the first [range](architecture/overview.html#architecture-range) of [index](indexes.html) data, expressed as [`BYTES`](bytes.html). | `INDEXES`, `KEYS`
+`raw_index_end_key` | The end key of the last [range](architecture/overview.html#architecture-range) of [index](indexes.html) data, expressed as [`BYTES`](bytes.html). | `INDEXES`, `KEYS`
+
+## Examples
{{site.data.alerts.callout_info}}
-If both `start_key` and `end_key` show `NULL`, the range is empty and has no splits.
+{% include {{page.version.version}}/sql/show-ranges-output-deprecation-notice.md %}
{{site.data.alerts.end}}
-## Examples
-
{% include {{page.version.version}}/sql/movr-statements-geo-partitioned-replicas.md %}
-### Show ranges for a table (primary index)
+### Show ranges for a database
+
+- [Show ranges for a database (without options)](#show-ranges-for-a-database-without-options)
+- [Show ranges for a database (with tables, keys, details)](#show-ranges-for-a-database-with-tables-keys-details)
+- [Show ranges for a database (with tables)](#show-ranges-for-a-database-with-tables)
+- [Show ranges for a database (with indexes)](#show-ranges-for-a-database-with-indexes)
+- [Show ranges for a database (with details)](#show-ranges-for-a-database-with-details)
+- [Show ranges for a database (with keys)](#show-ranges-for-a-database-with-keys)
+
+{% include_cached copy-clipboard.html %}
+~~~ sql
+SHOW DATABASES;
+~~~
+
+~~~
+ database_name | owner | primary_region | secondary_region | regions | survival_goal
+----------------+-------+----------------+------------------+---------+----------------
+ defaultdb | root | NULL | NULL | {} | NULL
+ movr | demo | NULL | NULL | {} | NULL
+ postgres | root | NULL | NULL | {} | NULL
+ system | node | NULL | NULL | {} | NULL
+(4 rows)
+~~~
+
+#### Show ranges for a database (without options)
{% include_cached copy-clipboard.html %}
~~~ sql
-> WITH x as (SHOW RANGES FROM TABLE vehicles) SELECT * FROM x WHERE "start_key" NOT LIKE '%Prefix%';
+SHOW RANGES FROM DATABASE movr;
~~~
+
+~~~
+ start_key | end_key | range_id | replicas | replica_localities | voting_replicas | non_voting_replicas | learner_replicas | split_enforced_until
+-----------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------+----------+----------+------------------------------------------------------------------------------------+-----------------+---------------------+------------------+-----------------------------
+ /Table/106 | /Table/106/1/"amsterdam" | 70 | {1,6,8} | {"region=us-east1,az=b","region=us-west1,az=c","region=europe-west1,az=c"} | {1,6,8} | {} | {} | NULL
+ /Table/106/1/"amsterdam" | /Table/106/1/"amsterdam"/"\xb333333@\x00\x80\x00\x00\x00\x00\x00\x00#" | 71 | {7,8,9} | {"region=europe-west1,az=b","region=europe-west1,az=c","region=europe-west1,az=d"} | {9,7,8} | {} | {} | NULL
+ ...
+ /Table/111/1/"washington dc"/PrefixEnd | /Max | 309 | {3,5,9} | {"region=us-east1,az=d","region=us-west1,az=b","region=europe-west1,az=d"} | {3,5,9} | {} | {} | NULL
+(178 rows)
+~~~
+
+#### Show ranges for a database (with tables, keys, details)
+
+{% include_cached copy-clipboard.html %}
+~~~ sql
+SHOW RANGES FROM DATABASE movr WITH TABLES, KEYS, DETAILS;
+~~~
+
+~~~
+ start_key | end_key | raw_start_key | raw_end_key | range_id | schema_name | table_name | table_id | table_start_key | table_end_key | raw_table_start_key | raw_table_end_key | range_size_mb | lease_holder | lease_holder_locality | replicas | replica_localities | voting_replicas | non_voting_replicas | learner_replicas | split_enforced_until | range_size
+-----------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------+----------+-------------+----------------------------+----------+-----------------+----------------------+---------------------+-------------------+----------------------------+--------------+--------------------------+----------+------------------------------------------------------------------------------------+-----------------+---------------------+------------------+----------------------------+-------------
+ /Table/106 | /Table/106/1/"amsterdam" | \xf2 | \xf28912616d7374657264616d0001 | 174 | public | users | 106 | /Table/106 | /Table/107 | \xf2 | \xf3 | 0 | 3 | region=us-east1,az=d | {3,6,9} | {"region=us-east1,az=d","region=us-west1,az=c","region=europe-west1,az=d"} | {3,9,6} | {} | {} | NULL | 0
+ /Table/106/1/"amsterdam" | /Table/106/1/"amsterdam"/"\xb333333@\x00\x80\x00\x00\x00\x00\x00\x00#" | \xf28912616d7374657264616d0001 | \xf28912616d7374657264616d000112b333333333334000ff8000ff00ff00ff00ff00ff00ff230001 | 175 | public | users | 106 | /Table/106 | /Table/107 | \xf2 | \xf3 | 0.00011900000000000000000 | 3 | region=us-east1,az=d | {3,7,8} | {"region=us-east1,az=d","region=europe-west1,az=b","region=europe-west1,az=c"} | {3,7,8} | {} | {} | NULL | 119
+ ...
+ /Table/111/1/"washington dc"/PrefixEnd | /Max | \xf66f891277617368696e67746f6e2064630002 | \xffff | 295 | public | user_promo_codes | 111 | /Table/111 | /Table/112 | \xf66f | \xf670 | 0 | 8 | region=europe-west1,az=c | {3,4,8} | {"region=us-east1,az=d","region=us-west1,az=a","region=europe-west1,az=c"} | {3,8,4} | {} | {} | NULL | 0
+(145 rows)
+~~~
+
+#### Show ranges for a database (with tables)
+
+{% include_cached copy-clipboard.html %}
+~~~ sql
+SHOW RANGES FROM DATABASE movr WITH TABLES;
+~~~
+
+~~~
+ start_key | end_key | range_id | schema_name | table_name | table_id | table_start_key | table_end_key | replicas | replica_localities | voting_replicas | non_voting_replicas | learner_replicas | split_enforced_until
+-----------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------+----------+-------------+----------------------------+----------+-----------------+----------------------+----------+------------------------------------------------------------------------------------+-----------------+---------------------+------------------+-----------------------------
+ /Table/106 | /Table/106/1/"amsterdam" | 67 | public | users | 106 | /Table/106 | /Table/107 | {1,4,9} | {"region=us-east1,az=b","region=us-west1,az=a","region=europe-west1,az=d"} | {1,9,4} | {} | {} | NULL
+ /Table/106/1/"amsterdam" | /Table/106/1/"amsterdam"/"\xb333333@\x00\x80\x00\x00\x00\x00\x00\x00#" | 68 | public | users | 106 | /Table/106 | /Table/107 | {7,8,9} | {"region=europe-west1,az=b","region=europe-west1,az=c","region=europe-west1,az=d"} | {7,9,8} | {} | {} | NULL
+ ...
+ /Table/111/1/"washington dc"/PrefixEnd | /Max | 311 | public | user_promo_codes | 111 | /Table/111 | /Table/112 | {1,5,7} | {"region=us-east1,az=b","region=us-west1,az=b","region=europe-west1,az=b"} | {1,7,5} | {} | {} | NULL
+(178 rows)
+~~~
+
+#### Show ranges for a database (with indexes)
+
+{% include_cached copy-clipboard.html %}
+~~~ sql
+SHOW RANGES FROM DATABASE movr WITH INDEXES;
+~~~
+
+~~~
+ start_key | end_key | range_id | schema_name | table_name | table_id | index_name | index_id | index_start_key | index_end_key | replicas | replica_localities | voting_replicas | non_voting_replicas | learner_replicas | split_enforced_until
+-----------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------+----------+-------------+----------------------------+----------+-----------------------------------------------+----------+-----------------+---------------+----------+------------------------------------------------------------------------------------+-----------------+---------------------+------------------+-----------------------------
+ /Table/106 | /Table/106/1/"amsterdam" | 70 | public | users | 106 | users_pkey | 1 | /Table/106/1 | /Table/106/2 | {1,6,8} | {"region=us-east1,az=b","region=us-west1,az=c","region=europe-west1,az=c"} | {1,6,8} | {} | {} | NULL
+ /Table/106/1/"amsterdam" | /Table/106/1/"amsterdam"/"\xb333333@\x00\x80\x00\x00\x00\x00\x00\x00#" | 71 | public | users | 106 | users_pkey | 1 | /Table/106/1 | /Table/106/2 | {7,8,9} | {"region=europe-west1,az=b","region=europe-west1,az=c","region=europe-west1,az=d"} | {9,7,8} | {} | {} | NULL
+ ...
+ /Table/111/1/"washington dc"/PrefixEnd | /Max | 309 | public | user_promo_codes | 111 | user_promo_codes_pkey | 1 | /Table/111/1 | /Table/111/2 | {3,5,9} | {"region=us-east1,az=d","region=us-west1,az=b","region=europe-west1,az=d"} | {3,5,9} | {} | {} | NULL
+(179 rows)
+~~~
+
+#### Show ranges for a database (with details)
+
+{% include_cached copy-clipboard.html %}
+~~~ sql
+SHOW RANGES FROM DATABASE movr WITH DETAILS;
+~~~
+
+~~~
+ start_key | end_key | range_id | range_size_mb | lease_holder | lease_holder_locality | replicas | replica_localities | voting_replicas | non_voting_replicas | learner_replicas | split_enforced_until | range_size
+-----------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------+----------+----------------------------+--------------+--------------------------+----------+------------------------------------------------------------------------------------+-----------------+---------------------+------------------+----------------------------+-------------
+ /Table/106 | /Table/106/1/"amsterdam" | 70 | 0 | 1 | region=us-east1,az=b | {1,6,8} | {"region=us-east1,az=b","region=us-west1,az=c","region=europe-west1,az=c"} | {1,6,8} | {} | {} | NULL | 0
+ /Table/106/1/"amsterdam" | /Table/106/1/"amsterdam"/"\xb333333@\x00\x80\x00\x00\x00\x00\x00\x00#" | 71 | 0.00011800000000000000000 | 9 | region=europe-west1,az=d | {7,8,9} | {"region=europe-west1,az=b","region=europe-west1,az=c","region=europe-west1,az=d"} | {9,7,8} | {} | {} | NULL | 118
+ ...
+ /Table/111/1/"washington dc"/PrefixEnd | /Max | 309 | 0 | 9 | region=europe-west1,az=d | {3,5,9} | {"region=us-east1,az=d","region=us-west1,az=b","region=europe-west1,az=d"} | {3,5,9} | {} | {} | NULL | 0
+(178 rows)
+~~~
+
+#### Show ranges for a database (with keys)
+
+{% include_cached copy-clipboard.html %}
+~~~ sql
+SHOW RANGES FROM DATABASE movr WITH KEYS;
+~~~
+
+~~~
+ start_key | end_key | raw_start_key | raw_end_key | range_id | replicas | replica_localities | voting_replicas | non_voting_replicas | learner_replicas | split_enforced_until
+-----------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------+----------+----------+------------------------------------------------------------------------------------+-----------------+---------------------+------------------+-----------------------------
+ /Table/106 | /Table/106/1/"amsterdam" | \xf2 | \xf28912616d7374657264616d0001 | 70 | {1,6,8} | {"region=us-east1,az=b","region=us-west1,az=c","region=europe-west1,az=c"} | {1,6,8} | {} | {} | NULL
+ /Table/106/1/"amsterdam" | /Table/106/1/"amsterdam"/"\xb333333@\x00\x80\x00\x00\x00\x00\x00\x00#" | \xf28912616d7374657264616d0001 | \xf28912616d7374657264616d000112b333333333334000ff8000ff00ff00ff00ff00ff00ff230001 | 71 | {7,8,9} | {"region=europe-west1,az=b","region=europe-west1,az=c","region=europe-west1,az=d"} | {9,7,8} | {} | {} | NULL
+ ...
+ /Table/111/1/"washington dc"/PrefixEnd | /Max | \xf66f891277617368696e67746f6e2064630002 | \xffff | 309 | {3,5,9} | {"region=us-east1,az=d","region=us-west1,az=b","region=europe-west1,az=d"} | {3,5,9} | {} | {} | NULL
+(178 rows)
+~~~
+
+### Show ranges for a table
+
+- [Show ranges for a table (without options)](#show-ranges-for-a-table-without-options)
+- [Show ranges for a table (with indexes, keys, details)](#show-ranges-for-a-table-with-indexes-keys-details)
+- [Show ranges for a table (with indexes)](#show-ranges-for-a-table-with-indexes)
+- [Show ranges for a table (with details)](#show-ranges-for-a-table-with-details)
+- [Show ranges for a table (with keys)](#show-ranges-for-a-table-with-keys)
+
+{% include_cached copy-clipboard.html %}
+~~~ sql
+SHOW TABLES;
+~~~
+
~~~
- start_key | end_key | range_id | range_size_mb | lease_holder | lease_holder_locality | replicas | replica_localities
-+------------------+----------------------------+----------+---------------+--------------+--------------------------+----------+------------------------------------------------------------------------------------+
- /"new york" | /"new york"/PrefixEnd | 58 | 0.000304 | 2 | region=us-east1,az=c | {1,2,5} | {"region=us-east1,az=b","region=us-east1,az=c","region=us-west1,az=b"}
- /"washington dc" | /"washington dc"/PrefixEnd | 102 | 0.000173 | 2 | region=us-east1,az=c | {1,2,3} | {"region=us-east1,az=b","region=us-east1,az=c","region=us-east1,az=d"}
- /"boston" | /"boston"/PrefixEnd | 63 | 0.000288 | 3 | region=us-east1,az=d | {1,2,3} | {"region=us-east1,az=b","region=us-east1,az=c","region=us-east1,az=d"}
- /"seattle" | /"seattle"/PrefixEnd | 97 | 0.000295 | 4 | region=us-west1,az=a | {4,5,6} | {"region=us-west1,az=a","region=us-west1,az=b","region=us-west1,az=c"}
- /"los angeles" | /"los angeles"/PrefixEnd | 55 | 0.000156 | 5 | region=us-west1,az=b | {4,5,6} | {"region=us-west1,az=a","region=us-west1,az=b","region=us-west1,az=c"}
- /"san francisco" | /"san francisco"/PrefixEnd | 71 | 0.000309 | 6 | region=us-west1,az=c | {1,5,6} | {"region=us-east1,az=b","region=us-west1,az=b","region=us-west1,az=c"}
- /"amsterdam" | /"amsterdam"/PrefixEnd | 59 | 0.000305 | 9 | region=europe-west1,az=d | {7,8,9} | {"region=europe-west1,az=b","region=europe-west1,az=c","region=europe-west1,az=d"}
- /"paris" | /"paris"/PrefixEnd | 62 | 0.000299 | 9 | region=europe-west1,az=d | {7,8,9} | {"region=europe-west1,az=b","region=europe-west1,az=c","region=europe-west1,az=d"}
- /"rome" | /"rome"/PrefixEnd | 67 | 0.000168 | 9 | region=europe-west1,az=d | {7,8,9} | {"region=europe-west1,az=b","region=europe-west1,az=c","region=europe-west1,az=d"}
-(9 rows)
+ schema_name | table_name | type | owner | estimated_row_count | locality
+--------------+----------------------------+-------+-------+---------------------+-----------
+ public | promo_codes | table | demo | 1000 | NULL
+ public | rides | table | demo | 500 | NULL
+ public | user_promo_codes | table | demo | 5 | NULL
+ public | users | table | demo | 50 | NULL
+ public | vehicle_location_histories | table | demo | 1000 | NULL
+ public | vehicles | table | demo | 15 | NULL
+(6 rows)
+~~~
+
+#### Show ranges for a table (without options)
+
+{% include_cached copy-clipboard.html %}
+~~~ sql
+SHOW RANGES FROM TABLE movr.users;
+~~~
+
+~~~
+ start_key | end_key | range_id | replicas | replica_localities | voting_replicas | non_voting_replicas | learner_replicas | split_enforced_until
+--------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------+----------+----------+------------------------------------------------------------------------------------+-----------------+---------------------+------------------+-----------------------------
+ …/Note that casting a string to a `TSQUERY` will not normalize the tokens into lexemes. To do so, [use `to_tsquery()`, `plainto_tsquery()`, or `phraseto_tsquery()`](#convert-string-to-tsquery). +`TSVECTOR` | Requires supported [`TSVECTOR`](tsvector.html) string format, e.g., `'Requires supported TSVECTOR string format.'`.
Note that casting a string to a `TSVECTOR` will not normalize the tokens into lexemes. To do so, [use `to_tsvector()`](#convert-string-to-tsvector). `UUID` | Requires supported [`UUID`](uuid.html) string format, e.g., `'63616665-6630-3064-6465-616462656562'`. ### `STRING` vs. `BYTES` @@ -176,12 +178,12 @@ In this case, [`LENGTH(string)`](functions-and-operators.html#string-and-byte-fu A literal entered through a SQL client will be translated into a different value based on the type: -+ `BYTES` give a special meaning to the pair `\x` at the beginning, and translates the rest by substituting pairs of hexadecimal digits to a single byte. For example, `\xff` is equivalent to a single byte with the value of 255. For more information, see [SQL Constants: String literals with character escapes](sql-constants.html#string-literals-with-character-escapes). ++ `BYTES` gives a special meaning to the pair `\x` at the beginning, and translates the rest by substituting pairs of hexadecimal digits to a single byte. For example, `\xff` is equivalent to a single byte with the value of 255. For more information, see [SQL Constants: String literals with character escapes](sql-constants.html#string-literals-with-character-escapes). + `STRING` does not give a special meaning to `\x`, so all characters are treated as distinct Unicode code points. For example, `\xff` is treated as a `STRING` with length 4 (`\`, `x`, `f`, and `f`). ### Cast hexadecimal digits to `BIT` - You can cast a `STRING` value of hexadecimal digits prefixed by `x` or `X` to a `BIT` value. +You can cast a `STRING` value of hexadecimal digits prefixed by `x` or `X` to a `BIT` value. For example: @@ -243,6 +245,58 @@ For example: (1 row) ~~~ +### Convert `STRING` to `TIMESTAMP` + +You can use the [`parse_timestamp()` function](functions-and-operators.html) to parse strings in `TIMESTAMP` format. + +{% include_cached copy-clipboard.html %} +~~~ sql +SELECT parse_timestamp ('2022-05-28T10:53:25.160Z'); +~~~ + +~~~ + parse_timestamp +-------------------------- +2022-05-28 10:53:25.16 +(1 row) +~~~ + +### Convert `STRING` to `TSVECTOR` + +You can use the [`to_tsvector()` function](functions-and-operators.html#full-text-search-functions) to parse strings in [`TSVECTOR`](tsvector.html) format. This will normalize the tokens into lexemes, and will add an integer position to each lexeme. + +{% include_cached copy-clipboard.html %} +~~~ sql +SELECT to_tsvector('How do trees get on the internet?'); +~~~ + +~~~ + to_tsvector +--------------------------------- + 'get':4 'internet':7 'tree':3 +~~~ + +For more information on usage, see [Full-Text Search](full-text-search.html). + +### Convert `STRING` to `TSQUERY` + +You can use the [`to_tsquery()`, `plainto_tsquery()`, and `phraseto_tsquery()` functions](functions-and-operators.html#full-text-search-functions) to parse strings in [`TSQUERY`](tsquery.html) format. This will normalize the tokens into lexemes. + +When using `to_tsquery()`, the string input must be formatted as a [`TSQUERY`](tsquery.html#syntax), with operators separating tokens. + +{% include_cached copy-clipboard.html %} +~~~ sql +SELECT to_tsquery('How & do & trees & get & on & the & internet?'); +~~~ + +~~~ + to_tsquery +------------------------------- + 'tree' & 'get' & 'internet' +~~~ + +For more information on usage, see [Full-Text Search](full-text-search.html). + ## See also - [Data Types](data-types.html) diff --git a/v23.1/transaction-retry-error-example.md b/v23.1/transaction-retry-error-example.md index a1be5ee8e10..7080d61bd75 100644 --- a/v23.1/transaction-retry-error-example.md +++ b/v23.1/transaction-retry-error-example.md @@ -5,13 +5,40 @@ toc: true docs_area: reference.transaction_retry_error_example --- -When a [transaction](transactions.html) is unable to complete due to [contention](architecture/overview.html#architecture-overview-contention) with another concurrent or recent transaction attempting to write to the same data, CockroachDB will [automatically attempt to retry the failed transaction](transactions.html#automatic-retries) without involving the client (i.e., silently). If the automatic retry is not possible or fails, a [transaction retry error](transaction-retry-error-reference.html)is emitted to the client. +When a [transaction](transactions.html) is unable to complete due to [contention](performance-best-practices-overview.html#transaction-contention) with another concurrent or recent transaction attempting to write to the same data, CockroachDB will [automatically attempt to retry the failed transaction](transactions.html#automatic-retries) without involving the client (i.e., silently). If the automatic retry is not possible or fails, a [transaction retry error](transaction-retry-error-reference.html) is emitted to the client. This page presents an [example of an application's transaction retry logic](#client-side-retry-handling-example), as well as a manner by which that logic can be [tested and verified](#testing-transaction-retry-logic) against your application's needs. ## Client-side retry handling example -{% include {{page.version.version}}/misc/client-side-intervention-example.md %} +The Python-like pseudocode below shows how to implement an application-level retry loop; it does not require your driver or ORM to implement [advanced retry handling logic](advanced-client-side-transaction-retries.html), so it can be used from any programming language or environment. In particular, your retry loop must: + +- Raise an error if the `max_retries` limit is reached +- Retry on `40001` error codes +- [`COMMIT`](commit-transaction.html) at the end of the `try` block +- Implement [exponential backoff](https://en.wikipedia.org/wiki/Exponential_backoff) logic as shown below for best performance + +~~~ python +while true: + n++ + if n == max_retries: + throw Error("did not succeed within N retries") + try: + # add logic here to run all your statements + conn.exec('COMMIT') + break + catch error: + if error.code != "40001": + throw error + else: + # This is a retry error, so we roll back the current transaction + # and sleep for a bit before retrying. The sleep time increases + # for each failed transaction. Adapted from + # https://colintemple.com/2017/03/java-exponential-backoff/ + conn.exec('ROLLBACK'); + sleep_ms = int(((2**n) * 100) + rand( 100 - 1 ) + 1) + sleep(sleep_ms) # Assumes your sleep() takes milliseconds +~~~ ## Testing transaction retry logic diff --git a/v23.1/transaction-retry-error-reference.md b/v23.1/transaction-retry-error-reference.md index 1287083fade..af786bd9c0b 100644 --- a/v23.1/transaction-retry-error-reference.md +++ b/v23.1/transaction-retry-error-reference.md @@ -5,7 +5,7 @@ toc: true docs_area: reference.transaction_retry_error_reference --- -When a [transaction](transactions.html) is unable to complete due to [contention](architecture/overview.html#architecture-overview-contention) with another concurrent or recent transaction attempting to write to the same data, CockroachDB will [automatically attempt to retry the failed transaction](transactions.html#automatic-retries) without involving the client (i.e., silently). If the automatic retry is not possible or fails, a _transaction retry error_ is emitted to the client. +When a [transaction](transactions.html) is unable to complete due to [contention](performance-best-practices-overview.html#understanding-and-avoiding-transaction-contention) with another concurrent or recent transaction attempting to write to the same data, CockroachDB will [automatically attempt to retry the failed transaction](transactions.html#automatic-retries) without involving the client (i.e., silently). If the automatic retry is not possible or fails, a _transaction retry error_ is emitted to the client. Transaction retry errors fall into two categories: @@ -26,11 +26,7 @@ The main reason why CockroachDB cannot auto-retry every serialization error with ## Actions to take -In most cases, the correct actions to take when encountering transaction retry errors are: - -1. Update your application to support [client-side retry handling](#client-side-retry-handling) when transaction retry errors are encountered. - -1. Adjust your application logic to [minimize transaction retry errors](#minimize-transaction-retry-errors) in the first place. +{% include {{ page.version.version }}/performance/transaction-retry-error-actions.md %} ### Client-side retry handling @@ -65,32 +61,24 @@ For a conceptual example of application-defined retry logic, and testing that lo ### Minimize transaction retry errors -In addition to the steps described in [Client-side retry handling](#client-side-retry-handling), which detail how to configure your application to restart a failed transaction, there are also a number of changes you can make to your application logic to increase the chance that CockroachDB can [automatically retry](transactions.html#automatic-retries) a failed transaction, and to reduce the number of transaction retry errors that reach the client application in the first place: - -1. Limit the number of affected rows by following [performance-tuning best practices](apply-statement-performance-rules.html) (e.g., query performance tuning, index design and maintenance, etc.). Not only will transactions run faster and hold locks for a shorter duration, but the chances of [read invalidation](architecture/transaction-layer.html#read-refreshing) when the transaction’s [timestamp is pushed](architecture/transaction-layer.html#timestamp-cache) due to a conflicting write is decreased due to a smaller read set (i.e., a smaller number of rows read). - -1. Break down larger transactions into smaller ones (e.g., [bulk deletes](bulk-delete-data.html)) to have transactions hold locks for a shorter duration. This will also decrease the likelihood of [pushed timestamps](architecture/transaction-layer.html#timestamp-cache) and retry errors. For instance, as the size of writes (number of rows written) decreases, the chances of the (bulk delete) transaction’s timestamp getting bumped by concurrent reads decreases. - -1. Design your applications to reduce network round trips by [sending statements in transactions as a single batch](transactions.html#batched-statements) (e.g., using [common table expressions](common-table-expressions.html)). Batching allows CockroachDB to [automatically retry](transactions.html#automatic-retries) a transaction when [previous reads are invalidated](architecture/transaction-layer.html#read-refreshing) at a [pushed timestamp](architecture/transaction-layer.html#timestamp-cache). When a multi-statement transaction is not batched, and takes more than a single round trip, CockroachDB cannot automatically retry the transaction. - -1. Limit the size of the result sets of your transactions to under 16KB, so that CockroachDB is more likely to [automatically retry](transactions.html#automatic-retries) when [previous reads are invalidated](architecture/transaction-layer.html#read-refreshing) at a [pushed timestamp](architecture/transaction-layer.html#timestamp-cache). When a transaction returns a result set over 16KB, even if that transaction has been sent as a single batch, CockroachDB cannot automatically retry the transaction. +In addition to the steps described in [Client-side retry handling](#client-side-retry-handling), which detail how to configure your application to restart a failed transaction, there are also a number of changes you can make to your application logic to reduce the number of transaction retry errors that reach the client application in the first place. -1. Use [`SELECT FOR UPDATE`](select-for-update.html) to aggressively lock rows that will later be updated in the transaction. Locking (blocking) earlier in the transaction will not allow other concurrent write transactions to conflict which leads to a situation where we would return out-of-date information subsequently returning a retry error ([`RETRY_WRITE_TOO_OLD`](#retry_write_too_old)). See [When and why to use `SELECT FOR UPDATE` in CockroachDB](https://www.cockroachlabs.com/blog/when-and-why-to-use-select-for-update-in-cockroachdb/) for more information. +Reduce failed transactions caused by [timestamp pushes](architecture/transaction-layer.html#timestamp-cache) or [read invalidation](architecture/transaction-layer.html#read-refreshing): -1. Use historical reads ([`SELECT ... AS OF SYSTEM TIME`](as-of-system-time.html)), preferably [bounded staleness reads](follower-reads.html#when-to-use-bounded-staleness-reads) or [exact staleness with follower reads](follower-reads.html#run-queries-that-use-exact-staleness-follower-reads) when possible to reduce conflicts with other writes. This reduces the likelihood of conflicts as fewer writes will happen at the historical timestamp. More specifically, writes’ timestamps are less likely to be pushed by historical reads as they would [when the read has a higher priority level](architecture/transaction-layer.html#transaction-conflicts). +{% include {{ page.version.version }}/performance/reduce-contention.md %} -1. If applicable to your workload, assign [column families](column-families.html#default-behavior) and separate columns that are frequently read and written into separate columns. Transactions will operate on disjoint column families and reduce the likelihood of conflicts. +Increase the chance that CockroachDB can [automatically retry](transactions.html#automatic-retries) a failed transaction: -1. As a last resort, consider adjusting the [closed timestamp interval](architecture/transaction-layer.html#closed-timestamps) using the `kv.closed_timestamp.target_duration` [cluster setting](cluster-settings.html) to reduce the likelihood of long-running write transactions having their [timestamps pushed](architecture/transaction-layer.html#timestamp-cache). This setting should be carefully adjusted if **no other mitigations are available** because there can be downstream implications (e.g., Historical reads, change data capture feeds, Stats collection, handling zone configurations, etc.). For example, a transaction _A_ is forced to refresh (i.e., change its timestamp) due to hitting the maximum [_closed timestamp_](architecture/transaction-layer.html#closed-timestamps) interval (closed timestamps enable [Follower Reads](follower-reads.html#how-stale-follower-reads-work) and [Change Data Capture (CDC)](change-data-capture-overview.html)). This can happen when transaction _A_ is a long-running transaction, and there is a write by another transaction to data that _A_ has already read. See the reference entry for [`RETRY_SERIALIZABLE`](#retry_serializable) for more information. +{% include {{ page.version.version }}/performance/increase-server-side-retries.md %} ## Transaction retry error reference -Note that your application's retry logic does not need to distinguish between the different types of serialization errors. They are listed here for reference during advanced troubleshooting. +Note that your application's retry logic does not need to distinguish between the different types of serialization errors. They are listed here for reference during [advanced troubleshooting](performance-recipes.html#transaction-contention). - [RETRY_WRITE_TOO_OLD](#retry_write_too_old) - [RETRY_SERIALIZABLE](#retry_serializable) - [RETRY_ASYNC_WRITE_FAILURE](#retry_async_write_failure) -- [ReadWithinUncertaintyInterval](#readwithinuncertaintyinterval) +- [ReadWithinUncertaintyIntervalError](#readwithinuncertaintyintervalerror) - [RETRY_COMMIT_DEADLINE_EXCEEDED](#retry_commit_deadline_exceeded) - [ABORT_REASON_ABORTED_RECORD_FOUND](#abort_reason_aborted_record_found) - [ABORT_REASON_CLIENT_REJECT](#abort_reason_client_reject) @@ -182,7 +170,7 @@ The `RETRY_ASYNC_WRITE_FAILURE` error occurs when some kind of problem with your See [Minimize transaction retry errors](#minimize-transaction-retry-errors) for the full list of recommended remediations. -### ReadWithinUncertaintyInterval +### ReadWithinUncertaintyIntervalError ``` TransactionRetryWithProtoRefreshError: ReadWithinUncertaintyIntervalError: @@ -213,7 +201,7 @@ The solution is to do one of the following: 1. If you [trust your clocks](operational-faqs.html#what-happens-when-node-clocks-are-not-properly-synchronized), you can try lowering the [`--max-offset` option to `cockroach start`](cockroach-start.html#flags), which provides an upper limit on how long a transaction can continue to restart due to uncertainty. {{site.data.alerts.callout_info}} -Uncertainty errors are a form of transaction conflict. For more information about transaction conflicts, see [Transaction conflicts](architecture/transaction-layer.html#transaction-conflicts). +Uncertainty errors are a sign of transaction conflict. For more information about transaction conflicts, see [Transaction conflicts](architecture/transaction-layer.html#transaction-conflicts). {{site.data.alerts.end}} See [Minimize transaction retry errors](#minimize-transaction-retry-errors) for the full list of recommended remediations. diff --git a/v23.1/transactions.md b/v23.1/transactions.md index ed4fc980fc0..6837d917ec5 100644 --- a/v23.1/transactions.md +++ b/v23.1/transactions.md @@ -57,29 +57,28 @@ To handle errors in transactions, you should check for the following types of se Type | Description -----|------------ -**Retry Errors** | Errors with [the code `40001` or string `restart transaction`](common-errors.html#restart-transaction), which indicate that a transaction failed because it could not be placed in a serializable ordering of transactions by CockroachDB. This is often due to [contention](#transaction-contention): conflicts with another concurrent or recent transaction accessing the same data. In such cases, the transaction needs to be retried by the client as described in [client-side retry handling](transaction-retry-error-reference.html#client-side-retry-handling). For a reference listing all of the retry error codes emitted by CockroachDB, see the [Transaction Retry Error Reference](transaction-retry-error-reference.html#transaction-retry-error-reference). +**Transaction Retry Errors** | Errors with the code `40001` and string `restart transaction`, which indicate that a transaction failed because it could not be placed in a [serializable ordering](demo-serializable.html) of transactions by CockroachDB. For details on transaction retry errors and how to resolve them, see the [Transaction Retry Error Reference](transaction-retry-error-reference.html#actions-to-take). **Ambiguous Errors** | Errors with the code `40003` which indicate that the state of the transaction is ambiguous, i.e., you cannot assume it either committed or failed. How you handle these errors depends on how you want to resolve the ambiguity. For information about how to handle ambiguous errors, see [here](common-errors.html#result-is-ambiguous). **SQL Errors** | All other errors, which indicate that a statement in the transaction failed. For example, violating the `UNIQUE` constraint generates a `23505` error. After encountering these errors, you can either issue a [`COMMIT`](commit-transaction.html) or [`ROLLBACK`](rollback-transaction.html) to abort the transaction and revert the database to its state before the transaction began.
If you want to attempt the same set of statements again, you must begin a completely new transaction. ## Transaction retries -Transactions may require retries if they experience deadlock or [read/write contention](performance-best-practices-overview.html#transaction-contention) with other concurrent transactions which cannot be resolved without allowing potential [serializable anomalies](https://en.wikipedia.org/wiki/Serializability). - -To mitigate read-write contention and reduce the need for transaction retries, use the following techniques: - -- Perform reads using [`AS OF SYSTEM TIME`](performance-best-practices-overview.html#use-as-of-system-time-to-decrease-conflicts-with-long-running-queries). -- Use [`SELECT FOR UPDATE`](select-for-update.html) to order transactions by controlling concurrent access to one or more rows of a table. This reduces retries in scenarios where a transaction performs a read and then updates the same row it just read. +Transactions may require retries due to [contention](performance-best-practices-overview.html#understanding-and-avoiding-transaction-contention) with another concurrent or recent transaction attempting to write to the same data. There are two cases in which transaction retries can occur: -- [Automatic retries](#automatic-retries), which CockroachDB processes for you. -- [Client-side retry handling](transaction-retry-error-reference.html#client-side-retry-handling), which your application must handle. +- [Automatic retries](#automatic-retries), which CockroachDB silently processes for you. +- [Client-side retries](transaction-retry-error-reference.html#client-side-retry-handling), which your application must handle after receiving a [*transaction retry error*](transaction-retry-error-reference.html). + +To reduce the need for transaction retries, see [Reduce transaction contention](performance-best-practices-overview.html#reduce-transaction-contention). ### Automatic retries -CockroachDB automatically retries individual statements (implicit transactions) and transactions sent from the client as a single batch, as long as the size of the results being produced for the client, including protocol overhead, is less than 16KiB by default. Once that buffer overflows, CockroachDB starts streaming results back to the client, at which point automatic retries cannot be performed any more. As long as the results of a single statement or batch of statements are known to stay clear of this limit, the client does not need to worry about transaction retries. +CockroachDB automatically retries individual statements (implicit transactions) and [transactions sent from the client as a single batch](#batched-statements), as long as the size of the results being produced for the client, including protocol overhead, is less than 16KiB by default. Once that buffer overflows, CockroachDB starts streaming results back to the client, at which point automatic retries cannot be performed any more. As long as the results of a single statement or batch of statements are known to stay clear of this limit, the client does not need to worry about transaction retries. -You can change the results buffer size for all new sessions using the `sql.defaults.results_buffer.size` [cluster setting](cluster-settings.html), or for a specific session using the `results_buffer_size` [session variable](set-vars.html). Decreasing the buffer size can increase the number of transaction retry errors a client receives, whereas increasing the buffer size can increase the delay until the client receives the first result row. +You can increase the occurrence of automatic retries as a way to [minimize transaction retry errors](transaction-retry-error-reference.html#minimize-transaction-retry-errors): + +{% include {{ page.version.version }}/performance/increase-server-side-retries.md %} {% include {{page.version.version}}/sql/sql-defaults-cluster-settings-deprecation-notice.md %} @@ -134,31 +133,6 @@ The [`enable_implicit_transaction_for_batch_statements` session variable](set-va In the event [bounded staleness reads](follower-reads.html#bounded-staleness-reads) are used along with either the [`with_min_timestamp` function or the `with_max_staleness` function](functions-and-operators.html#date-and-time-functions) and the `nearest_only` parameter is set to `true`, the query will throw an error if it can't be served by a nearby replica. -### Client-side intervention - -Your application should include client-side retry handling when the statements are sent individually, such as: - -~~~ -> BEGIN; - -> UPDATE products SET inventory = 0 WHERE sku = '8675309'; - -> INSERT INTO orders (customer, status) VALUES (1, 'new'); - -> COMMIT; -~~~ - -To indicate that a transaction must be retried, CockroachDB signals an error with the `SQLSTATE` error code `40001` (serialization error) and an error message that begins with the string `"restart transaction"`. These errors **cannot** be [resolved automatically](#automatic-retries), and require client-side intervention: - -- See [client-side retry handling](transaction-retry-error-reference.html#client-side-retry-handling) for a list of actions to take when encountering transaction retry errors. -- See [Transaction retry error reference](transaction-retry-error-reference.html#transaction-retry-error-reference) for a complete list of transaction retry error codes. - -## Transaction contention - -Transactions in CockroachDB [lock](crdb-internal.html#cluster_locks) data resources that are written during their execution. When a pending write from one transaction conflicts with a write of a concurrent transaction, the concurrent transaction must wait for the earlier transaction to complete before proceeding. When a dependency cycle is detected between transactions, the transaction with the higher priority aborts the dependent transaction to avoid deadlock, which must be [retried](#client-side-intervention). - -For more details about transaction contention and best practices for avoiding contention, see [Transaction Contention](performance-best-practices-overview.html#transaction-contention). - ## Nested transactions CockroachDB supports the nesting of transactions using [savepoints](savepoint.html). These nested transactions are also known as sub-transactions. Nested transactions can be rolled back without discarding the state of the entire surrounding transaction. diff --git a/v23.1/tsquery.md b/v23.1/tsquery.md new file mode 100644 index 00000000000..e1d5d4827e8 --- /dev/null +++ b/v23.1/tsquery.md @@ -0,0 +1,63 @@ +--- +title: TSQUERY +summary: The TSQUERY data type a list of lexemes separated by operators, and is used in full-text search. +toc: true +docs_area: reference.sql +--- + +The `TSQUERY` [data type](data-types.html) stores a list of lexemes separated by operators. `TSQUERY` values are used in [full-text search](full-text-search.html). + +## Syntax + +A `TSQUERY` comprises individual lexemes and operators in the form: `'These' & 'lexemes' & 'are' & 'not' & 'normalized' & 'lexemes.'`. + +The operators in a `TSQUERY` are used to [match a `TSQUERY` to a `TSVECTOR`](full-text-search.html#match-queries-to-documents). Valid `TSQUERY` operators are: + +- `&` (AND). Given `'one' & 'two'`, both `one` and `two` must be present in the matching `TSVECTOR`. +- `|` (OR). Given `'one' | 'two'`, either `one` or `two` must be present in the matching `TSVECTOR`. +- `!` (NOT). Given `'one' & ! 'two'`, `one` must be present and `two` must **not** be present in the matching `TSVECTOR`. +- `<->` (FOLLOWED BY). Given `'one' <-> 'two'`, `one` must be followed by `two` in the matching `TSVECTOR`. + - `<->` is equivalent to `<1>`. You can specify an integer `