[BUG] Tests can't start Prometheus

[Yury-Fridlyand]

What is the bug?

Integration and doc tests often can't start Prometheus

How can one reproduce the bug?

$ ./gradlew ':integ-test:integTest'
=======================================
OpenSearch Build Hamster says Hello!
  Gradle Version        : 7.4.2
  OS Info               : Linux 5.15.68.1-microsoft-standard-WSL2 (amd64)
  JDK Version           : 11 (Ubuntu JDK)
  JAVA_HOME             : /usr/lib/jvm/java-11-openjdk-amd64
  Random Testing Seed   : 4C3DB35D2E65CC93
  In FIPS 140 mode      : false
=======================================

> Task :integ-test:compileJdbc
> Task :compileJava UP-TO-DATE
> Task :processResources UP-TO-DATE
> Task :classes UP-TO-DATE
> Task :shadowJar UP-TO-DATE

Deprecated Gradle features were used in this build, making it incompatible with Gradle 5.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/4.10.2/userguide/command_line_interface.html#sec:command_line_warnings

BUILD SUCCESSFUL in 1s
3 actionable tasks: 3 up-to-date

> Task :integ-test:startPrometheus
Download https://github.com/prometheus/prometheus/releases/download/v2.37.2/prometheus-2.37.2.linux-amd64.tar.gz
Renaming folder : prometheus-2.37.2.linux-amd64
true
Renaming folder : prometheus-2.39.1.linux-amd64
false

> Task :integ-test:startPrometheus FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':integ-test:startPrometheus'.
> Server already running!

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

See https://docs.gradle.org/7.4.2/userguide/command_line_interface.html#sec:command_line_warnings

BUILD FAILED in 12s
13 actionable tasks: 5 executed, 8 up-to-date

$ ps faxwwu | grep [p]rom
$
$ ls .prom.pid.lock
ls: cannot access '.prom.pid.lock': No such file or directory
$ 

What is the expected behavior?

Prometheus shouldn't be involved in main test processes. Tests flow should try running it only if Prometheus tests started. :integ-test: module should have another task for these tests.
At least, test flow should continue if start failed.

What is your host/environment?

662a938 @ 2.x

Do you have any additional context?

Tests also can't stop it properly if interrupted.
N/A

[MitchellGale]

Was able to reliably re-produce when doctests build failed due to port already being in use (e.g. port was used by opensearch instance being run somewhere else) then running build again.

Manually running stopPrometheus seems to properly stop prometheus server from running.

Possible fix is running stopPrometheus task as first step in startPrometheus.

[vmmusings]

@MitchellGale-BitQuill
doctest failure is stopping stopPrometheus task run which is eventually causing failure in starting Prometheus for integTests. [Is this the scenario?]

But @Yury-Fridlyand is only running integTests?[ Prior run for integ tests might have failed and didn't kill the prometheus server.]

I will try to come up with a more robust solution. [Either by separating out the tests or robust solution for starting and killing the process]

[Yury-Fridlyand]

But @Yury-Fridlyand is only running integTests?[ Prior run for integ tests might have failed and didn't kill the prometheus server.]

I checked that there is no process running or lock file left.
I faced this trouble before too, but it helped me to kill the process and delete the lock file.

Currently, I deactivated Prometheus tasks locally, because I don't have related tests nor tasks.

diff --git a/integ-test/build.gradle b/integ-test/build.gradle
index 454d67eca..957f26aa0 100644
--- a/integ-test/build.gradle
+++ b/integ-test/build.gradle
@@ -153,8 +154,9 @@ stopPrometheus.mustRunAfter startPrometheus
 // Run PPL ITs and new, legacy and comparison SQL ITs with new SQL engine enabled
 integTest {
     dependsOn ':opensearch-sql-plugin:bundlePlugin'
-    dependsOn startPrometheus
-    finalizedBy stopPrometheus
+
+    //dependsOn startPrometheus
+    //finalizedBy stopPrometheus

     systemProperty 'tests.security.manager', 'false'
     systemProperty('project.root', project.projectDir.absolutePath)