[JSON-API] Backport of connection pool for 1.11.x , configurable db options needs forward porting #11615

akshayshirahatti-da · 2021-11-09T14:22:53Z

Backport of Hikari connection pool for 1.11.x plus addition of a configurable pool size parameter to jdbcConfig.

Forward porting of configurable db options for main is on #11621

CHANGELOG_BEGIN
[JSON-API] Backport of Hikari connection pool for 1.11.x plus addition of a configurable pool size parameter to jdbcConfig.
CHANGELOG_END

Pull Request Checklist

Read and understand the contribution guidelines
Include appropriate tests
Set a descriptive title and thorough description
Add a reference to the issue this PR will solve, if appropriate
Include changelog additions in one or more commit message bodies between the CHANGELOG_BEGIN and CHANGELOG_END tags
Normal production system change, include purpose of change in description
If you mean to change the status of a component, please make sure you keep the Component Status page up to date.

NOTE: CI is not automatically run on non-members pull-requests for security
reasons. The reviewer will have to comment with /AzurePipelines run to
trigger the build.

…igurable connection pool size CHANGELOG_BEGIN Backport of Hikari connection pool for 1.11.x plus addition of a configurable pool size parameter to jdbcConfig. CHANGELOG_END

cocreature

Nice! Thanks for tackling this so quickly

ledger-service/http-json/src/main/scala/com/digitalasset/http/Config.scala

cocreature · 2021-11-09T14:35:18Z

ledger-service/http-json/src/main/scala/com/digitalasset/http/Main.scala

@@ -82,7 +82,7 @@ object Main extends StrictLogging {
      case _ =>
    }

-    val serviceF: Future[HttpService.Error \/ ServerBinding] =
+    val serviceF: Future[HttpService.Error \/ (ServerBinding, Option[ContractDao])] =


maybe worth having an httpservice class that contains the server binding & optional dao and abstracts away the details of closing? Not as part of the backport but separately

ledger-service/http-json/src/main/scala/com/digitalasset/http/dbbackend/ContractDao.scala

S11001001

Please retitle; if we think this PR is only a backport, that will constitute a regression in main. (In other words, needs-forward-port)

ledger-service/http-json/src/main/scala/com/digitalasset/http/Config.scala

ledger-service/http-json/src/main/scala/com/digitalasset/http/dbbackend/ContractDao.scala

…ments

Changelog --------- ``` - [JSON-API] added metrics to separately track: - time taken to update query-store ACS (from ledger) - lookup times for the query store [HTTP/JSON API] - metrics reporting can now be chosen via the command line option --metrics-reporter (currently hidden), valid options are console, csv://, graphite:// and prometheus:// - metrics reporting interval can also now be chosen via a command line option --metrics-reporting-interval (currently hidden) - [Ledger HTTP Json Service] Logging output can now be in JSON either via providing the cli option `--log-encoder json` or via setting the env var `LOG_FORMAT_JSON=true` [HTTP-JSON] - fixed that json log output could not be enabled via cli options except via usage of env vars - [Ledger HTTP Json service] Logging can now be configured via the `--log-level` cli argument. Valid values are `error`, `warn`, `info` (default), `debug`, `trace` - [Ledger HTTP Json Service] Logging now also tells service name if log level was changed. http-json: - add contextual id as logging context to distinguish different application runs in logs - add request id as logging context to distinguish different http requests within an application run - add for non-static endpoints trace logs which show how long processing it took in ns - [JSON API] Fix an error where transactions that delete a large number of contracts resulted in stackoverflows with the PostgreSQL backend and database errors with Oracle. - [JSON API] While updating the contract table for a query, if the DB appears to be slow, JSON API will slow down its own inserts and deletes at some point rather than construct ever-larger INSERT and DELETE batch commands. See `issue #11589 <https://github.com/digital-asset/daml/pull/11589>`__. Backport of Hikari connection pool for 1.11.x plus addition of configurable connection pool properties to jdbcConfig string, the properties are listed below poolSize -- specifies the max pool size for the database connection pool minIdle -- specifies the min idle connections for database connection pool connectionTimeout -- long value, specifies the connection timeout for database connection pool idleTimeout -- long value, specifies the idle timeout for the database connection pool [Ledger API] Retry the interpretation of a command in case of a race with other transactions. This fix drastically reduces the likelihood of the error "Could not find a suitable ledger time after 0 retries". For every update in the index db log the full context at the INFO level. [Integration Kit] The state of the participant indexer can now be checked via the GRPC health endpoint ``` Commits ------- ``` c679b4a [1.11.x] Metrics for investigating json api performance (#11769) 437c3d4 Bump Windows postgres (#11806) 382168b Add metrics to the http json service (#9923) (#11768) b7fc305 [Backport] Add cli option & system property to enable json only logging for the json api (#11717) 17cacb0 [Backport] Add logging command line option to ledger http service (#9581) (#11699) 56531e8 Backport: Support deletion of a large number of contracts (#11646) 6bf19fc Backport: limit contract insertion/deletion batching on backpressure (#11647) eed4ad3 Backport signing tool changes (#11651) ee7547e [JSON-API] Backport of connection pool for 1.11.x , configurable db options needs forward porting (#11615) 632ba5a Backport: Avoid collision in execution log postfix (#11644) cf4f1a1 Backport: add tar to dev-env (#10173) (#11636) b78fa8c Backport: Restart the submission interpretation in case of a race [DPP-737] (#11578) 5c3220d Fix build of release/1.11.x branch (#11582) cce7c79 Log context of all updates written to the database (#10057) a9d0473 Adds Indexer state to GRPC health checks [DPP-434] (#9951) (#9961) ``` changelog_begin changelog_end

akshayshirahatti-da requested a review from cocreature November 9, 2021 14:22

akshayshirahatti-da requested review from aherrmann-da, S11001001 and stefanobaghino-da as code owners November 9, 2021 14:22

backport of Hikari connection pool for 1.11.x plus addition of a conf…

a9c3910

…igurable connection pool size CHANGELOG_BEGIN Backport of Hikari connection pool for 1.11.x plus addition of a configurable pool size parameter to jdbcConfig. CHANGELOG_END

akshayshirahatti-da force-pushed the backport_connection_pool branch from a3b2f08 to a9c3910 Compare November 9, 2021 14:32

cocreature approved these changes Nov 9, 2021

View reviewed changes

S11001001 reviewed Nov 9, 2021

View reviewed changes

Add more configurable parameters and cleanup based on code review com…

6e7d3e3

…ments

akshayshirahatti-da changed the title ~~[JSON-API] Backport of connection pool for 1.11.x~~ [JSON-API] Backport of connection pool for 1.11.x , configurable db options needs forward porting Nov 9, 2021

make scala 2.13.3 builds pass, more additions to CliSpec test

d9d134e

akshayshirahatti-da enabled auto-merge (squash) November 11, 2021 05:45

Merge branch 'release/1.11.x' into backport_connection_pool

246d5cc

akshayshirahatti-da merged commit ee7547e into release/1.11.x Nov 11, 2021

akshayshirahatti-da deleted the backport_connection_pool branch November 11, 2021 06:36

akshayshirahatti-da linked an issue Nov 15, 2021 that may be closed by this pull request

[FEATURE REQUEST] Backport database connection pooling to 1.11.x #11596

Closed

akshayshirahatti-da mentioned this pull request Nov 15, 2021

[FEATURE REQUEST] Backport database connection pooling to 1.11.x #11596

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[JSON-API] Backport of connection pool for 1.11.x , configurable db options needs forward porting #11615

[JSON-API] Backport of connection pool for 1.11.x , configurable db options needs forward porting #11615

akshayshirahatti-da commented Nov 9, 2021 •

edited

Loading

cocreature left a comment

cocreature Nov 9, 2021

S11001001 left a comment

[JSON-API] Backport of connection pool for 1.11.x , configurable db options needs forward porting #11615

[JSON-API] Backport of connection pool for 1.11.x , configurable db options needs forward porting #11615

Conversation

akshayshirahatti-da commented Nov 9, 2021 • edited Loading

Pull Request Checklist

cocreature left a comment

Choose a reason for hiding this comment

cocreature Nov 9, 2021

Choose a reason for hiding this comment

S11001001 left a comment

Choose a reason for hiding this comment

akshayshirahatti-da commented Nov 9, 2021 •

edited

Loading