Integration tests broken reporting ChildProcess::LaunchError: Bad file descriptor -

[dliappis]

Since Jan 8 2024 all integration tests for pull requests are failing for main, 8.12, 8.11 branches (and possibly 7.17 but untested yet).

Example on Buildkite: https://buildkite.com/elastic/logstash-pull-request-pipeline/builds/640#018ce885-ad4c-4f0e-82f6-3b7f2072a524
Example on Jenkins: https://logstash-ci.elastic.co/job/elastic+logstash+main+multijob-integration-1/

Error output

    Failures:

      1) Beat Input Without TLS successfully send events
         Got 0 failures and 2 other errors:
         Shared Example Group: "send events" called from ./specs/beats_input_spec.rb:98

         1.1) Failure/Error: logstash_service.start_background(logstash_config)
              
              ChildProcess::LaunchError:
                Bad file descriptor - /opt/buildkite-agent/logstash/build/logstash-8.12.0-SNAPSHOT/bin/logstash
              # /opt/buildkite-agent/logstash/build/qa/integration/vendor/jruby/3.1.0/gems/childprocess-5.0.0/lib/childprocess/process_spawn_process.rb:88:in `launch_process'
              # /opt/buildkite-agent/logstash/build/qa/integration/vendor/jruby/3.1.0/gems/childprocess-5.0.0/lib/childprocess/abstract_process.rb:81:in `start'
              # ./services/logstash_service.rb:156:in `block in spawn_logstash'
              # ./services/logstash_service.rb:141:in `spawn_logstash'
              # ./services/logstash_service.rb:95:in `start_background'
              # ./specs/beats_input_spec.rb:63:in `block in <main>'
              # /opt/buildkite-agent/logstash/build/qa/integration/vendor/jruby/3.1.0/gems/rspec-wait-0.0.9/lib/rspec/wait.rb:46:in `block in <main>'
              # ./rspec.rb:36:in `<main>'
              # ------------------
              # --- Caused by: ---
              # Errno::EBADF:
              #   Bad file descriptor - /opt/buildkite-agent/logstash/build/logstash-8.12.0-SNAPSHOT/bin/logstash
              #   /opt/buildkite-agent/logstash/build/qa/integration/vendor/jruby/3.1.0/gems/childprocess-5.0.0/lib/childprocess/process_spawn_process.rb:86:in `launch_process'

Reproduction

On a fresh Ubuntu 22.04 VM, or via Docker using the instructions in https://docs.elastic.dev/ingest-dev-docs/logstash/logstash-pull-request-buildkite-ci#reproducing-failures:

1. Checkout to branch 8.12 of Logstash (or main, or 8.11):

   git clone https://github.com/elastic/logstash
   cd logstash
   git fetch --all
   git checkout 8.12
   

2. Run: ci/integration_tests.sh specs/beats_input_spec.rb

The suspicion falls on childprocess since yesterday (January 7 2024) brought the new release 5.0.0 after several years which changed backends for simplicity and portability (changelog).