Skip to content

Commit

Permalink
[BACKPORT 2.18][PLAT-14524] Up-version pekko to fix TLSActor infinite…
Browse files Browse the repository at this point in the history
… loop

Summary:
We observe high CPU usage from YBA Java process on yba-installer portal with really high memory allocation rate.
It will continue until YBA restart in case it happened.
The issue is related to some edge case in Akka <-> JDK interaction in terms of TLS. Seems like related to TLS1.3 somehow as well: akka/akka#29922
They implemented the workaround to limit the infinite loop here, and it was working: akka/akka#31947
This fix was backported in Pekko: apache/pekko@9662048
But, if you look closer at case NEED_WRAP => code in both commits - Pekko code also has flushToUser() call, before this infinite loop check.
And the issue here is that flushToUser will reset the counter they use for infinite loop check..
So, the fix does not work in Pekko,
Luckily, later they had another commit in Pekko, which should fix the above issue: apache/pekko@1e41829
So, we basically need to up-version pekko.

Original diffs: https://phorge.dev.yugabyte.com/D36239, https://phorge.dev.yugabyte.com/D37146

Test Plan: Tested manually that YBA works

Reviewers: #yba-api-review!, nsingh, muthu, dshubin, sneelakantan

Reviewed By: nsingh, muthu

Subscribers: anijhawan, sneelakantan, yugaware

Tags: #jenkins-ready

Differential Revision: https://phorge.dev.yugabyte.com/D37194
  • Loading branch information
anmalysh-yb committed Aug 20, 2024
1 parent 13c5ca5 commit 9bd1140
Show file tree
Hide file tree
Showing 5 changed files with 77 additions and 64 deletions.
2 changes: 1 addition & 1 deletion managed/.sbtopts
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
-J-XX:MinHeapFreeRatio=10
-J-XX:MaxHeapFreeRatio=20
-J-Xmx2g
-J-Xmx3g
-J-XX:MaxMetaspaceSize=1g
-J-XX:MetaspaceSize=500m
-Dlogback.configurationFile=logback-sbt.xml
Expand Down
31 changes: 25 additions & 6 deletions managed/build.sbt
Original file line number Diff line number Diff line change
Expand Up @@ -177,7 +177,7 @@ libraryDependencies ++= Seq(
"org.pac4j" %% "play-pac4j" % "9.0.2",
"org.pac4j" % "pac4j-oauth" % "4.5.7" exclude("commons-io" , "commons-io"),
"org.pac4j" % "pac4j-oidc" % "4.5.7" exclude("commons-io" , "commons-io"),
"org.playframework" %% "play-json" % "3.0.1",
"org.playframework" %% "play-json" % "3.0.4",
"commons-validator" % "commons-validator" % "1.8.0",
"org.apache.velocity" % "velocity-engine-core" % "2.3",
"com.fasterxml.woodstox" % "woodstox-core" % "6.4.0",
Expand Down Expand Up @@ -463,7 +463,28 @@ dependencyOverrides += "com.google.guava" % "guava" % "32.1.1-jre"
dependencyOverrides += "com.nimbusds" % "oauth2-oidc-sdk" % "7.1.1"
dependencyOverrides += "org.reflections" % "reflections" % "0.10.2"

val jacksonVersion = "2.15.3"
// This is a custom version, built based on 1.0.3 with the following commit added on top:
// https://github.com/apache/pekko/commit/1e41829bf7abeec268b9a409f35051ed7f4e0090.
// This is required to fix TLS infinite loop issue, which causes high CPU usage.
// We can't use 1.1.0-M1 version yet, as it has the following issue:
// https://github.com/playframework/playframework/pull/12662
// Once the issue is fixed we should migrate back on stable version.
val pekkoVersion = "1.0.3-tls-loop-fix"

val pekkoLibs = Seq(
"org.apache.pekko" %% "pekko-actor-typed",
"org.apache.pekko" %% "pekko-actor",
"org.apache.pekko" %% "pekko-protobuf-v3",
"org.apache.pekko" %% "pekko-serialization-jackson",
"org.apache.pekko" %% "pekko-slf4j",
"org.apache.pekko" %% "pekko-stream",
)

val pekkoOverrides = pekkoLibs.map(_ % pekkoVersion)

dependencyOverrides ++= pekkoOverrides

val jacksonVersion = "2.17.1"

val jacksonLibs = Seq(
"com.fasterxml.jackson.core" % "jackson-core",
Expand Down Expand Up @@ -570,9 +591,6 @@ val swaggerGen: TaskKey[Unit] = taskKey[Unit](
"generate swagger.json"
)

val swaggerJacksonVersion = "2.11.1"
val swaggerJacksonOverrides = jacksonLibs.map(_ % swaggerJacksonVersion)

lazy val swagger = project
.dependsOn(root % "compile->compile;test->test")
.settings(commonSettings)
Expand All @@ -584,7 +602,8 @@ lazy val swagger = project
"com.github.dwickern" %% "swagger-play3.0" % "4.0.0"
),

dependencyOverrides ++= swaggerJacksonOverrides,
dependencyOverrides ++= pekkoOverrides,
dependencyOverrides ++= jacksonOverrides,
dependencyOverrides += "org.scala-lang.modules" %% "scala-xml" % "2.1.0",

swaggerGen := Def.taskDyn {
Expand Down
2 changes: 1 addition & 1 deletion managed/project/plugins.sbt
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ dependencyOverrides += "com.google.googlejavaformat" % "google-java-format" % "1
libraryDependencies += "ch.qos.logback" % "logback-classic" % "1.4.14"

// The Play plugin
addSbtPlugin("org.playframework" % "sbt-plugin" % "3.0.0")
addSbtPlugin("org.playframework" % "sbt-plugin" % "3.0.4")

// ORM
addSbtPlugin("org.playframework" % "sbt-play-ebean" % "8.0.0")
Expand Down
53 changes: 25 additions & 28 deletions managed/src/main/resources/swagger-strict.json
Original file line number Diff line number Diff line change
Expand Up @@ -2190,6 +2190,9 @@
"$ref" : "#/definitions/ExtraDependencies",
"description" : "Extra dependencies"
},
"fullBackup" : {
"type" : "boolean"
},
"fullChainSizeInBytes" : {
"description" : "Incremental backups chain size",
"format" : "int64",
Expand Down Expand Up @@ -2386,7 +2389,7 @@
"type" : "string"
}
},
"required" : [ "creatingUser", "platformUrl", "platformVersion", "sleepAfterMasterRestartMillis", "sleepAfterTServerRestartMillis", "storageConfigUUID", "timeTakenPartial" ],
"required" : [ "creatingUser", "fullBackup", "platformUrl", "platformVersion", "sleepAfterMasterRestartMillis", "sleepAfterTServerRestartMillis", "storageConfigUUID", "timeTakenPartial" ],
"type" : "object"
},
"Basic Auth information" : {
Expand Down Expand Up @@ -5750,10 +5753,6 @@
"format" : "int64",
"type" : "integer"
},
"isRecharts" : {
"description" : "Is Recharts",
"type" : "boolean"
},
"metrics" : {
"description" : "Metrics",
"items" : {
Expand All @@ -5779,6 +5778,9 @@
"description" : "Node prefix",
"type" : "string"
},
"recharts" : {
"type" : "boolean"
},
"regionCodes" : {
"description" : "Region code",
"items" : {
Expand Down Expand Up @@ -5813,7 +5815,7 @@
"type" : "string"
}
},
"required" : [ "start", "xclusterConfigUuid" ],
"required" : [ "recharts", "start", "xclusterConfigUuid" ],
"type" : "object"
},
"MetricSettings" : {
Expand Down Expand Up @@ -5980,6 +5982,9 @@
"$ref" : "#/definitions/ExtraDependencies",
"description" : "Extra dependencies"
},
"fullBackup" : {
"type" : "boolean"
},
"fullChainSizeInBytes" : {
"description" : "Incremental backups chain size",
"format" : "int64",
Expand Down Expand Up @@ -6176,7 +6181,7 @@
"type" : "string"
}
},
"required" : [ "creatingUser", "platformUrl", "platformVersion", "sleepAfterMasterRestartMillis", "sleepAfterTServerRestartMillis", "storageConfigUUID", "timeTakenPartial" ],
"required" : [ "creatingUser", "fullBackup", "platformUrl", "platformVersion", "sleepAfterMasterRestartMillis", "sleepAfterTServerRestartMillis", "storageConfigUUID", "timeTakenPartial" ],
"type" : "object"
},
"NamespaceInfoResp" : {
Expand Down Expand Up @@ -9143,12 +9148,6 @@
"description" : "Cron expression for the schedule",
"type" : "string"
},
"customerUUID" : {
"description" : "Customer UUID",
"format" : "uuid",
"readOnly" : true,
"type" : "string"
},
"failureCount" : {
"description" : "Number of failed backup attempts",
"format" : "int32",
Expand Down Expand Up @@ -12185,10 +12184,6 @@
"example" : "[email protected]",
"type" : "string"
},
"isPrimary" : {
"description" : "True if the user is the primary user",
"type" : "boolean"
},
"ldapSpecifiedRole" : {
"description" : "LDAP Specified Role",
"type" : "boolean"
Expand All @@ -12197,6 +12192,9 @@
"readOnly" : true,
"type" : "string"
},
"primary" : {
"type" : "boolean"
},
"role" : {
"description" : "User role",
"enum" : [ "ConnectOnly", "ReadOnly", "BackupAdmin", "Admin", "SuperAdmin" ],
Expand All @@ -12218,7 +12216,7 @@
"type" : "string"
}
},
"required" : [ "email" ],
"required" : [ "email", "primary" ],
"type" : "object"
},
"Users" : {
Expand Down Expand Up @@ -12249,10 +12247,6 @@
"example" : "[email protected]",
"type" : "string"
},
"isPrimary" : {
"description" : "True if the user is the primary user",
"type" : "boolean"
},
"ldapSpecifiedRole" : {
"description" : "LDAP Specified Role",
"type" : "boolean"
Expand All @@ -12261,6 +12255,9 @@
"readOnly" : true,
"type" : "string"
},
"primary" : {
"type" : "boolean"
},
"role" : {
"description" : "User role",
"enum" : [ "ConnectOnly", "ReadOnly", "BackupAdmin", "Admin", "SuperAdmin" ],
Expand All @@ -12282,7 +12279,7 @@
"type" : "string"
}
},
"required" : [ "email" ],
"required" : [ "email", "primary" ],
"type" : "object"
},
"VMImageUpgradeParams" : {
Expand Down Expand Up @@ -12519,21 +12516,21 @@
"description" : {
"type" : "string"
},
"isRequired" : {
"type" : "boolean"
},
"isValid" : {
"required" : {
"type" : "boolean"
},
"type" : {
"enum" : [ "NTP_SERVICE_STATUS", "PROMETHEUS_SPACE", "MOUNT_POINTS_WRITABLE", "USER", "USER_GROUP", "HOME_DIR_SPACE", "HOME_DIR_EXISTS", "RAM_SIZE", "INTERNET_CONNECTION", "CPU_CORES", "PROMETHEUS_NO_NODE_EXPORTER", "TMP_DIR_SPACE", "PAM_LIMITS_WRITABLE", "PYTHON_VERSION", "MOUNT_POINTS_VOLUME", "CHRONYD_RUNNING", "SSH_PORT", "SUDO_ACCESS", "OPENSSL", "POLICYCOREUTILS", "RSYNC", "XXHASH", "LIBATOMIC1", "LIBNCURSES6", "LIBATOMIC", "AZCOPY", "GSUTIL", "S3CMD", "NODE_EXPORTER_RUNNING", "NODE_EXPORTER_PORT", "SWAPPINESS", "ULIMIT_CORE", "ULIMIT_OPEN_FILES", "ULIMIT_USER_PROCESSES", "SYSTEMD_SUDOER_ENTRY", "SSH_ACCESS", "NODE_AGENT_ACCESS", "MASTER_HTTP_PORT", "MASTER_RPC_PORT", "TSERVER_HTTP_PORT", "TSERVER_RPC_PORT", "YB_CONTROLLER_HTTP_PORT", "YB_CONTROLLER_RPC_PORT", "REDIS_SERVER_HTTP_PORT", "REDIS_SERVER_RPC_PORT", "YCQL_SERVER_HTTP_PORT", "YCQL_SERVER_RPC_PORT", "YSQL_SERVER_HTTP_PORT", "YSQL_SERVER_RPC_PORT", "VM_MAX_MAP_COUNT" ],
"type" : "string"
},
"valid" : {
"type" : "boolean"
},
"value" : {
"type" : "string"
}
},
"required" : [ "description", "isRequired", "isValid", "type", "value" ],
"required" : [ "description", "required", "type", "valid", "value" ],
"type" : "object"
},
"VolumeDetails" : {
Expand Down
Loading

0 comments on commit 9bd1140

Please sign in to comment.