Skip to content

Commit

Permalink
[BACKPORT pg15-cherrypicks] all: Bulk port from master - 96
Browse files Browse the repository at this point in the history
Summary:
 82bca83 [DOC-453] ASH wait event table updates for 2.21.1 (#23595)
 011830f [PLAT-14809][YBA CLI]Support Master Key Rotation
 bd39e84 [PLAT-15088] Add DB version check for connection pooling
 9853daf [PLAT-14808][YBA CLI]Support edit KMS operations
 b02f6f7 [doc][yba] KMS and expiring tokens (#23754)
 62d30d5 [PLAT-15091][PLAT-14092] Improve HA logging and backup metric
 6cc5f6a Add more documentation on allowed preview flags (#23826)
 a28b3ec [#22301] docdb: Improve handling of large responses
 bc28ee8 [#23752] DocDB: Integrating hnswlib into the vector indexing framework and hnsw_tool
 c40fff2 copy Develop changes to stable (#23812)
 bf7c0b0 [#23828] docdb: Fix compile error in lwproto-test.cc
 Excluded: 9c1cc23 [#23707] Add table name to /metrics JSON endpoint
 3de6206 [PLAT-15151]: Update autoflags based checks for xCluster/DR
 b7bc45e Revert "[PLAT-13800] skip setting imageBundle reference for YBM based clusters using machineImage"
 35add07 [PLAT-14888]: Store PITR parmas during create DR and allow users to edit them.
 3ba74e8 Fix docs for xcluster DDL flow for YCQL & onprem provider ssh user (#23827)
 981415e [#23820] DocDB: Use shared memory for executing read and write pg client queries in release
 f20edbb Moving Driver reference to Develop section (#23683)
 a3f3bfe [PLAT-15209] Do not set optimized mem & tablet split gflags for YBM
 a95dc94 [PLAT-14801] add v2 API to list YBC Gflags metadata
 06d2b8d [PLAT-15194]CLI | All outputs are returned as List instead of JSON for commands like describe
 0375a68 [#23700] CDCSDK: Perform table removal from CDC stream via background thread
 5d2a151 [#23797] YSQL: Stabilise PgExplainAnalyzeModifyTable tests when running with Connection Manager
 e62dcfd sbt Swaggergen on master to fix Uts

Test Plan: Jenkins: rebase: pg15-cherrypicks

Reviewers: jason, tfoucher

Differential Revision: https://phorge.dev.yugabyte.com/D37901
  • Loading branch information
yugabyte-ci authored and jaki committed Sep 9, 2024
1 parent 263fbb3 commit e727edc
Show file tree
Hide file tree
Showing 390 changed files with 7,443 additions and 3,332 deletions.
2 changes: 2 additions & 0 deletions .github/vale-styles/Yugabyte/spelling-exceptions.txt
Original file line number Diff line number Diff line change
Expand Up @@ -205,6 +205,7 @@ downvoted
downvotes
Dpl
Dreamweaver
Ebean
Ecto
Elasticsearch
enablement
Expand Down Expand Up @@ -425,6 +426,7 @@ Nokogiri
Northwind
noteable
noteables
Npgsql
npm
nullability
nullable
Expand Down
1 change: 1 addition & 0 deletions CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -607,6 +607,7 @@ include_directories(src)

include_directories("src/inline-thirdparty/usearch")
include_directories("src/inline-thirdparty/fp16")
include_directories("src/inline-thirdparty/hnswlib")


enable_testing()
Expand Down
2 changes: 1 addition & 1 deletion docs/content/preview/architecture/design-goals.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,7 @@ YugabyteDB has been designed with operational simplicity in mind, providing feat

## Heterogeneous workload support

Depending on the use case, the database may need to support diverse workloads, such as [transactional processing](../../benchmark/tpcc/), [analytical queries](../../sample-data/retail-analytics/), [real-time data ingestion](../../tutorials/azure/azure-event-hubs/), [time-series](../../develop/data-modeling/common-patterns/timeseries/), and [key-value](../../benchmark/key-value-workload-ycql/) workloads.
Depending on the use case, the database may need to support diverse workloads, such as [transactional processing](../../benchmark/tpcc/), [analytical queries](../../sample-data/retail-analytics/), [real-time data ingestion](/preview/tutorials/azure/azure-event-hubs/), [time-series](../../develop/data-modeling/common-patterns/timeseries/), and [key-value](../../benchmark/key-value-workload-ycql/) workloads.

## Transaction isolation levels

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,7 @@ To set up unidirectional transactional replication, do the following:
Checkpointing of yugabyte completed. Bootstrap is not required for setting up xCluster replication
```

1. If needed, perform a full copy of the database on the Primary to the Standby using distributed backup and restore. See [Distributed snapshots for YSQL](../../../manage/backup-restore/snapshot-ysql/). Otherwise, create the necessary schema objects (tables and indexes) on the Standby.
1. If needed, perform a full copy of the database on the Primary to the Standby using distributed backup and restore. See [Distributed snapshots for YSQL](../../../../manage/backup-restore/snapshot-ysql/). Otherwise, create the necessary schema objects (tables and indexes) on the Standby.

1. Set up the xCluster replication group.

Expand Down Expand Up @@ -117,7 +117,7 @@ list_xcluster_outbound_replication_groups [namespace_id]
You should see output similar to the following:

```output
1 Outbound Replication Groups found:
1 Outbound Replication Groups found:
[repl_group1]
```

Expand All @@ -132,7 +132,7 @@ list_universe_replications [namespace_id]
You should see output similar to the following:

```output
1 Universe Replication Groups found:
1 Universe Replication Groups found:
[repl_group1]
```

Expand Down
2 changes: 1 addition & 1 deletion docs/content/preview/develop/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,5 +77,5 @@ To learn more about how to use browser-based IDEs, see [Cloud-native development
Yugabyte provides multiple step-by-step guides for building scalable and fault-tolerant applications with YugabyteDB using your favorite programming language, services, and frameworks, including Kafka, Gen-AI, and more.

{{<lead link="../tutorials/">}}
For step-by-step guides for various frameworks, see [Tutorials](../tutorials/).
For step-by-step guides for various frameworks, see [Tutorials](/preview/tutorials/).
{{</lead>}}
16 changes: 9 additions & 7 deletions docs/content/preview/drivers-orms/csharp/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,8 @@ The following projects can be used to implement C# applications using the Yugaby

| Project | Documentation and Guides | Latest Driver Version | Supported YugabyteDB Version |
| :------ | :----------------------- | :-------------------- | :--------------------------- |
| YugabyteDB C# Driver for YSQL [Recommended] | [Documentation](ysql/) <br /> [Reference](../../reference/drivers/csharp/yb-npgsql-reference/) | [8.0.3-yb-1](https://www.nuget.org/packages/NpgsqlYugabyteDB/) | 2.8 and later |
| PostgreSQL Npgsql Driver | [Documentation](postgres-npgsql/) <br /> [Reference](../../reference/drivers/csharp/postgres-npgsql-reference/) | [6.0.3](https://www.nuget.org/packages/Npgsql/) | 2.6 and later |
| YugabyteDB C# Driver for YSQL [Recommended] | [Documentation](ysql/) <br /> [Reference](yb-npgsql-reference/) | [8.0.3-yb-1](https://www.nuget.org/packages/NpgsqlYugabyteDB/) | 2.8 and later |
| PostgreSQL Npgsql Driver | [Documentation](postgres-npgsql/) <br /> [Reference](postgres-npgsql-reference/) | [6.0.3](https://www.nuget.org/packages/Npgsql/) | 2.6 and later |
| YugabyteDB C# Driver for YCQL | [Documentation](ycql/) | [3.6.0](https://github.com/yugabyte/cassandra-csharp-driver/releases/tag/3.6.0) | |

| Project | Documentation and Guides | Example Apps |
Expand All @@ -32,17 +32,18 @@ The following projects can be used to implement C# applications using the Yugaby

Learn how to establish a connection to a YugabyteDB database and begin basic CRUD operations by referring to [Connect an app](ysql/) or [Use an ORM](entityframework/).

For reference documentation, including using projects with SSL, refer to the [drivers and ORMs reference](../../reference/drivers/csharp/yb-npgsql-reference/) pages.

## Prerequisites

To develop C# applications for YugabyteDB, you need the following:

- **.NET SDK**\
- **.NET SDK**

Install .NET SDK 6.0 or later. To download it for your supported OS, visit [Download .NET](https://dotnet.microsoft.com/en-us/download).

- **Create a C# project**\
- **Create a C# project**

For ease-of-use, use an integrated development environment (IDE) such as Visual Studio. To download and install Visual Studio, visit the [Visual Studio Downloads](https://visualstudio.microsoft.com/downloads/) page.

- To create a C# project in Visual Studio, select [Console Application](https://docs.microsoft.com/en-us/dotnet/core/tutorials/with-visual-studio?pivots=dotnet-6-0) as template when creating a new project.
- If you are not using an IDE, use the following dotnet command:

Expand All @@ -51,9 +52,10 @@ To develop C# applications for YugabyteDB, you need the following:
```

- **YugabyteDB cluster**

- Create a free cluster on YugabyteDB Aeon. Refer to [Use a cloud cluster](../../quick-start-yugabytedb-managed/). Note that YugabyteDB Aeon requires SSL.
- Alternatively, set up a standalone YugabyteDB cluster by following the steps in [Install YugabyteDB](../../quick-start/).

## Next step

- [Connect an app](ysql/)
[Connect an app](ysql/)
Original file line number Diff line number Diff line change
Expand Up @@ -3,13 +3,15 @@ title: PostgreSQL Npgsql Driver
headerTitle: C# Drivers
linkTitle: C# Drivers
description: C# PostgreSQL Npgsql Driver for YSQL
headcontent: C# Drivers for YSQL
badges: ysql
aliases:
- /preview/reference/drivers/csharp/postgres-npgsql-reference/
menu:
preview:
name: C# Drivers
identifier: ref-postgres-npgsql-driver
parent: drivers
weight: 620
identifier: ref-2-postgres-npgsql-driver
parent: csharp-drivers
weight: 100
type: docs
---

Expand All @@ -31,11 +33,11 @@ type: docs

</ul>

Npgsql is an open source ADO.NET Data Provider for PostgreSQL; it allows programs written in C#, Visual Basic, and F# to access the YugabyteDB server. Npgsql is based on [libpq](#libpq) and supports the [SCRAM-SHA-256 authentication method](../../../../secure/authentication/password-authentication/#scram-sha-256).
Npgsql is an open source ADO.NET Data Provider for PostgreSQL; it allows programs written in C#, Visual Basic, and F# to access the YugabyteDB server. Npgsql is based on [libpq](#libpq) and supports the [SCRAM-SHA-256 authentication method](../../../secure/authentication/password-authentication/#scram-sha-256).

For details on Npgsql, refer to the [Npgsql documentation](https://www.npgsql.org/doc/).

For building a sample C# application with Npgsql, see [Connect an application](../../../../drivers-orms/csharp/postgres-npgsql/).
For building a sample C# application with Npgsql, see [Connect an application](../postgres-npgsql/).

## Download the driver dependency

Expand Down
11 changes: 3 additions & 8 deletions docs/content/preview/drivers-orms/csharp/postgres-npgsql.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,9 @@ linkTitle: Connect an app
description: Connect a C# application using PostgreSQL Npgsql driver
menu:
preview:
identifier: postgres-npgsql-driver
identifier: csharp-2-postgres-npgsql-driver
parent: csharp-drivers
weight: 420
weight: 500
type: docs
---

Expand Down Expand Up @@ -113,7 +113,7 @@ var connStringBuilder = new NpgsqlConnectionStringBuilder();

YugabyteDB Aeon clusters require SSL. Refer to [Connect applications](../../../yugabyte-cloud/cloud-connect/connect-applications/) for instructions on how to obtain the cluster connection parameters and download the CA certificate.

Refer to [Configure SSL/TLS](../../../reference/drivers/csharp/postgres-npgsql-reference/#configure-ssl-tls) for more information on Npgsql default and supported SSL modes, and examples for setting up your connection strings when using SSL.
Refer to [Configure SSL/TLS](../postgres-npgsql-reference/#configure-ssl-tls) for more information on Npgsql default and supported SSL modes, and examples for setting up your connection strings when using SSL.

### Step 3: Write your application

Expand Down Expand Up @@ -290,8 +290,3 @@ John 35 CSharp + SSL
```

If you receive no output or an error, check the parameters in the connection string.

## Learn more

- [PostgreSQL Npgsql driver reference](../../../reference/drivers/csharp/postgres-npgsql-reference/)
- Build C# applications using [EntityFramework ORM](../entityframework)
Original file line number Diff line number Diff line change
Expand Up @@ -3,13 +3,16 @@ title: YugabyteDB Npgsql Smart Driver
headerTitle: C# Drivers
linkTitle: C# Drivers
description: C# Npgsql Smart Driver for YSQL
headcontent: C# Drivers for YSQL
badges: ysql
aliases:
- /preview/reference/drivers/csharp/
- /preview/reference/drivers/csharp/yb-npgsql-reference/
menu:
preview:
name: C# Drivers
identifier: ref-yb-npgsql-driver
parent: drivers
weight: 600
identifier: ref-1-yb-npgsql-driver
parent: csharp-drivers
weight: 100
type: docs
---

Expand All @@ -31,12 +34,12 @@ type: docs

</ul>

Yugabyte Npgsql smart driver is a .NET driver for [YSQL](../../../../api/ysql/) based on [PostgreSQL Npgsql driver](https://github.com/npgsql/npgsql/tree/main/src/Npgsql), with additional connection load balancing features.
Yugabyte Npgsql smart driver is a .NET driver for [YSQL](../../../api/ysql/) based on [PostgreSQL Npgsql driver](https://github.com/npgsql/npgsql/tree/main/src/Npgsql), with additional connection load balancing features.

For more information on the Yugabyte Npgsql smart driver, see the following:

- [YugabyteDB smart drivers for YSQL](../../../../drivers-orms/smart-drivers/)
- [CRUD operations](../../../../drivers-orms/csharp/ysql/)
- [YugabyteDB smart drivers for YSQL](../../smart-drivers/)
- [CRUD operations](../ysql/)
- [GitHub repository](https://github.com/yugabyte/npgsql)
- [Smart Driver architecture](https://github.com/yugabyte/yugabyte-db/blob/master/architecture/design/smart-driver.md)

Expand Down Expand Up @@ -79,7 +82,7 @@ var connStringBuilder = "Host=127.0.0.1,127.0.0.2,127.0.0.3;Port=5433;Database=y
NpgsqlConnection conn = new NpgsqlConnection(connStringBuilder)
```

You can specify [multiple hosts](../../../../drivers-orms/csharp/ysql/#use-multiple-addresses) in the connection string in case the primary address fails. After the driver establishes the initial connection, it fetches the list of available servers from the universe, and performs load balancing of subsequent connection requests across these servers.
You can specify [multiple hosts](../ysql/#use-multiple-addresses) in the connection string in case the primary address fails. After the driver establishes the initial connection, it fetches the list of available servers from the universe, and performs load balancing of subsequent connection requests across these servers.

To specify topology keys, you set the `Topology Keys` property to comma separated values, as per the following example:

Expand Down
2 changes: 1 addition & 1 deletion docs/content/preview/drivers-orms/csharp/ycql.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ linkTitle: Connect an app
description: Connect a C# application using YugabyteDB YCQL driver
menu:
preview:
identifier: csharp-driver-ycql
identifier: csharp-3-driver-ycql
parent: csharp-drivers
weight: 500
type: docs
Expand Down
6 changes: 2 additions & 4 deletions docs/content/preview/drivers-orms/csharp/ysql.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,9 @@ linkTitle: Connect an app
description: Connect a C# application using Npgsql Smart Driver
menu:
preview:
identifier: csharp-driver-ysql
identifier: csharp-1-driver-ysql
parent: csharp-drivers
weight: 400
weight: 500
type: docs
---

Expand Down Expand Up @@ -204,5 +204,3 @@ John 35 CSharp

- [YugabyteDB smart drivers for YSQL](../../smart-drivers/)
- [Smart Driver architecture](https://github.com/yugabyte/yugabyte-db/blob/master/architecture/design/smart-driver.md)
- [YugabyteDB Npgsql Smart Driver reference](../../../reference/drivers/csharp/yb-npgsql-reference/)
- Build C# applications using [Entity Framework ORM](../entityframework)
4 changes: 3 additions & 1 deletion docs/content/preview/drivers-orms/elixir/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,10 +28,12 @@ Learn how to establish a connection to a YugabyteDB database and begin basic CRU

To develop Elixir applications for YugabyteDB, you need the following:

- **Elixir**\
- **Elixir**

Install the latest versions of [Elixir, Erlang VM, IEx and Mix](https://elixir-lang.org/docs.html) (tested with Elixir 1.17.1 and Erlang/OTP 26 erts-14.2.5).

- **YugabyteDB cluster**

- Create a free cluster on YugabyteDB Aeon. Refer to [Use a cloud cluster](../../quick-start-yugabytedb-managed/). Note that YugabyteDB Aeon requires SSL.
- Alternatively, set up a standalone YugabyteDB cluster by following the steps in [Install YugabyteDB](../../quick-start/).

Expand Down
24 changes: 13 additions & 11 deletions docs/content/preview/drivers-orms/go/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,35 +19,37 @@ The following projects can be used to implement Golang applications using the Yu

| Project | Documentation and Guides | Latest Driver Version | Supported YugabyteDB Version |
| ------- | ------------------------ | ------------------------ | ---------------------|
| YugabyteDB PGX Driver [Recommended] | [Documentation](yb-pgx/)<br /> [Reference](../../reference/drivers/go/yb-pgx-reference/) | [v4](https://pkg.go.dev/github.com/yugabyte/pgx/) | 2.8 and above
| PGX Driver | [Documentation](pgx/)<br />[Reference](../../reference/drivers/go/pgx-reference/) | [v4](https://pkg.go.dev/github.com/jackc/pgx/) | 2.8 and above
| PQ Driver | [Documentation](pq/)<br />[Reference](../../reference/drivers/go/pq-reference/) | [v1.10.2](https://github.com/lib/pq/releases/tag/v1.10.2/) | 2.6 and above
| YugabyteDB PGX Driver [Recommended] | [Documentation](yb-pgx/)<br /> [Reference](yb-pgx-reference/) | [v4](https://pkg.go.dev/github.com/yugabyte/pgx/) | 2.8 and above |
| PGX Driver | [Documentation](pgx/)<br />[Reference](pgx-reference/) | [v4](https://pkg.go.dev/github.com/jackc/pgx/) | 2.8 and above |
| PQ Driver | [Documentation](pq/)<br />[Reference](pq-reference/) | [v1.10.2](https://github.com/lib/pq/releases/tag/v1.10.2/) | 2.6 and above |
| YugabyteDB Go Driver for YCQL | [Documentation](ycql) | [3.16.3](https://github.com/yugabyte/gocql) | |

| Project | Documentation and Guides | Example Apps |
| ------- | ------------------------ | ------------- |
| GORM [Recommended] | [Documentation](gorm/) <br/> [Hello World](../orms/go/ysql-gorm)| [GORM App](https://github.com/YugabyteDB-Samples/orm-examples/tree/master/golang/gorm)
| GO-PG | [Documentation](pg/) |
| GORM [Recommended] | [Documentation](gorm/) <br/> [Hello World](../orms/go/ysql-gorm)| [GORM App](https://github.com/YugabyteDB-Samples/orm-examples/tree/master/golang/gorm) |
| GO-PG | [Documentation](pg/) | |

Learn how to establish a connection to a YugabyteDB database and begin basic CRUD operations by referring to [Connect an app](yb-pgx/) or [Use an ORM](gorm/).

For reference documentation, including using projects with SSL, refer to the [drivers and ORMs reference](../../reference/drivers/go/pgx-reference/) pages.

## Prerequisites

To develop Golang applications for YugabyteDB, you need the following:

- **Go**\
Install the latest Go (1.16 or later) on your system.\
- **Go**

Install the latest Go (1.16 or later) on your system.

Run `go --version` in a terminal to check your version of Go. To install Go, visit [Go Downloads](https://golang.org/dl/).

- **Create a Go project**\
- **Create a Go project**

For ease-of-use, use an integrated development environment (IDE) such as Visual Studio. To download and install Visual Studio, visit the [Visual Studio Downloads](https://visualstudio.microsoft.com/downloads/) page.

- **YugabyteDB cluster**

- Create a free cluster on YugabyteDB Aeon. Refer to [Use a cloud cluster](../../quick-start-yugabytedb-managed/). Note that YugabyteDB Aeon requires SSL.
- Alternatively, set up a standalone YugabyteDB cluster by following the steps in [Install YugabyteDB](../../quick-start/).

## Next step

- [Connect an app](yb-pgx/)
[Connect an app](yb-pgx/)
10 changes: 5 additions & 5 deletions docs/content/preview/drivers-orms/go/pg.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ menu:
preview:
identifier: pg-orm
parent: go-drivers
weight: 610
weight: 600
type: docs
---

Expand Down Expand Up @@ -84,11 +84,11 @@ db := pg.Connect(opt)

#### Use SSL

For a YugabyteDB Aeon cluster, or a YugabyteDB cluster with SSL/TLS enabled, set the following SSL-related environment variables at the client side. SSL/TLS is enabled by default for client-side authentication. Refer to [Configure SSL/TLS](../../../reference/drivers/go/yb-pgx-reference/#configure-ssl-tls) for the default and supported modes.
For a YugabyteDB Aeon cluster, or a YugabyteDB cluster with SSL/TLS enabled, set the following SSL-related environment variables at the client side. SSL/TLS is enabled by default for client-side authentication. Refer to [Configure SSL/TLS](yb-pgx-reference/#configure-ssl-tls) for the default and supported modes.

```sh
$ export PGSSLMODE=verify-ca
$ export PGSSLROOTCERT=~/root.crt # Here, the CA certificate file is downloaded as `root.crt` under home directory. Modify your path accordingly.
$ export PGSSLROOTCERT=~/root.crt # CA certificate file is downloaded as `root.crt` under home directory. Modify your path accordingly.
```

| Environment Variable | Description |
Expand Down Expand Up @@ -224,7 +224,7 @@ func main() {

#### Using pg.Options()

If the password contains these special characters (#, %, ^), the driver may fail to parse the URL. In such a case, use pg.Options() instead of pg.ParseURL() to initialize the Options in `ybsql_hello_world.go`. The standard PG environment variables except PGPASSWORD and PGSSLROOTCERT are implicitly read by the driver. Set the PG variables as follows (replace the values as appropriate for YugabyteDB Aeon):
If the password contains these special characters (#, %, ^), the driver may fail to parse the URL. In such a case, use `pg.Options()` instead of `pg.ParseURL()` to initialize the Options in `ybsql_hello_world.go`. The standard PG environment variables except PGPASSWORD and PGSSLROOTCERT are implicitly read by the driver. Set the PG variables as follows (replace the values as appropriate for YugabyteDB Aeon):

```sh
$ export PGHOST=127.0.0.1
Expand All @@ -234,7 +234,7 @@ $ export PGPASSWORD=password#with%special^chars
$ export PGDATABASE=yugabyte
```

To use pg.Options(), replace the main function in your file with the following:
To use `pg.Options()`, replace the main function in your file with the following:

```go
/* Modify the main() from the ybsql_hello_world.go script by replacing the first few lines and enabling pg.Options() */
Expand Down
Loading

0 comments on commit e727edc

Please sign in to comment.