Fixes for Elastic Agent v2 changes

[axw]

Motivation/summary

If fleet.hosts isn't set in ESS, assume Fleet Server is co-located and use "https://localhost:8220". This is a workaround, and may alternatively be fixed by elastic/elastic-agent#1802.

If the runner stops with an error, log the error immediately, rather than when reloading. Otherwise the error cannot be observed unless there is an input or output change.

Checklist

- [ ] Update CHANGELOG.asciidoc
- [ ] Update package changelog.yml (only if changes to apmpackage have been made)
- [ ] Documentation has been updated

How to test these changes

1. Create a deployment in ESS
2. Check that the APM endpoint is accessible
3. Make sure source maps still get applied to RUM errors

Related issues

None

[axw]

I'll backport this to 8.6 manually, as there will be conflicts.

Lies

[mergify]

This pull request does not have a backport label. Could you fix it @axw? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-7.x is the label to automatically backport to the 7.x branch.
• backport-7./d is the label to automatically backport to the 7./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

[axw]

@Mergifyio backport 8.6

[mergify]

backport 8.6

✅ Backports have been created

• #9682 Fixes for Elastic Agent v2 changes (backport #9681) has been created for branch 8.6

[apmmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-11-28T13:01:19.986+0000

• Duration: 26 min 4 sec

Test stats 🧪

Test	Results
Failed	0
Passed	153
Skipped	0
Total	153

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate and publish the docker images.

• /test windows : Build & tests on Windows.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[apmmachine]

📚 Go benchmark report

Diff with the main branch

name                                                                                              old time/op    new time/op     delta
pkg:github.com/elastic/apm-server/internal/agentcfg goos:linux goarch:amd64
FetchAndAdd/FetchFromCache-12                                                                       41.1ns ± 0%     46.1ns ± 0%   +12.16%  (p=0.008 n=5+5)
pkg:github.com/elastic/apm-server/internal/beater/request goos:linux goarch:amd64
ContextResetContentEncoding/empty-12                                                                 121ns ± 1%      135ns ± 1%   +11.50%  (p=0.008 n=5+5)
ContextResetContentEncoding/uncompressed-12                                                          144ns ± 1%      160ns ± 0%   +11.25%  (p=0.008 n=5+5)
ContextResetContentEncoding/deflate_sniff-12                                                        3.97µs ± 3%     3.81µs ± 8%    -3.94%  (p=0.032 n=5+5)
pkg:github.com/elastic/apm-server/internal/model/modelindexer goos:linux goarch:amd64
ModelIndexer/BestCompressionScaling-12                                                              5.94µs ± 3%     5.71µs ± 2%    -3.99%  (p=0.032 n=5+5)
pkg:github.com/elastic/apm-server/internal/processor/stream goos:linux goarch:amd64
BackendProcessor/invalid-metadata-2.ndjson-12                                                       8.03µs ± 8%     7.03µs ±14%   -12.40%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel2/metadata-null-values.ndjson-12          2.73µs ± 3%     2.86µs ± 5%    +4.86%  (p=0.032 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel2/transactions_spans.ndjson-12            30.7µs ±10%     36.5µs ±11%   +18.82%  (p=0.016 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel2/transactions_spans_rum_2.ndjson-12      5.09µs ± 1%     5.41µs ± 6%    +6.38%  (p=0.016 n=4+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel4/errors_transaction_id.ndjson-12         12.1µs ±10%     13.4µs ± 4%   +10.42%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel4/logs.ndjson-12                          14.1µs ± 3%     13.0µs ± 6%    -7.75%  (p=0.032 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel8/transactions-huge_traces.ndjson-12      5.60µs ± 2%     5.74µs ± 3%    +2.47%  (p=0.032 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel8/transactions.ndjson-12                  10.9µs ± 1%     11.2µs ± 1%    +2.96%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel8/transactions_spans.ndjson-12            10.8µs ± 2%     11.2µs ± 2%    +3.65%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel8/transactions_spans_rum.ndjson-12        2.01µs ± 1%     2.05µs ± 1%    +2.29%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel8/transactions_spans_rum_2.ndjson-12      1.93µs ± 1%     1.98µs ± 1%    +2.44%  (p=0.016 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel8/unknown-span-type.ndjson-12             7.27µs ± 2%     7.41µs ± 1%    +1.93%  (p=0.048 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/errors.ndjson-12                      6.86µs ± 2%     7.41µs ± 2%    +8.03%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/errors_2.ndjson-12                    6.57µs ± 5%     6.97µs ± 3%    +6.22%  (p=0.016 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/invalid-event-type.ndjson-12           800ns ± 1%      818ns ± 2%    +2.18%  (p=0.016 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/invalid-event.ndjson-12               3.31µs ± 0%     3.37µs ± 1%    +1.77%  (p=0.016 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/invalid-json-metadata.ndjson-12       1.87µs ± 0%     1.89µs ± 1%    +1.40%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/invalid-metadata-2.ndjson-12           466ns ± 1%      481ns ± 5%    +3.24%  (p=0.016 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/invalid-metadata.ndjson-12             472ns ± 2%      483ns ± 1%    +2.47%  (p=0.016 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/span-links.ndjson-12                  1.18µs ± 1%     1.21µs ± 2%    +2.29%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/transactions-huge_traces.ndjson-12    4.66µs ± 1%     4.75µs ± 2%    +1.75%  (p=0.016 n=5+5)
ReadBatch/minimal.ndjson-12                                                                         23.5µs ±11%     19.3µs ±23%   -17.74%  (p=0.032 n=5+5)
ReadBatch/transactions-huge_traces.ndjson-12                                                        72.7µs ±17%     56.4µs ±24%   -22.47%  (p=0.032 n=5+5)
pkg:github.com/elastic/apm-server/internal/publish goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/aggregation/spanmetrics goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/aggregation/txmetrics goos:linux goarch:amd64
AggregateTransaction-12                                                                             83.6ns ± 1%     84.4ns ± 1%    +0.94%  (p=0.032 n=5+5)
pkg:github.com/elastic/apm-server/x-pack/apm-server/sampling goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/sampling/eventstorage goos:linux goarch:amd64
WriteTransaction/json_codec-12                                                                      4.40µs ±15%    10.99µs ±46%  +149.95%  (p=0.008 n=5+5)
WriteTransaction/json_codec_big_tx-12                                                               5.07µs ± 1%    10.12µs ±41%   +99.55%  (p=0.008 n=5+5)
ReadEvents/json_codec/0_events-12                                                                    307ns ± 6%      341ns ±11%   +11.11%  (p=0.016 n=5+5)
ReadEvents/json_codec/1_events-12                                                                   9.15µs ± 4%    10.13µs ± 8%   +10.71%  (p=0.032 n=5+5)
ReadEvents/json_codec/10_events-12                                                                   127µs ±16%      158µs ±17%   +24.03%  (p=0.008 n=5+5)
ReadEvents/json_codec_big_tx/0_events-12                                                             303ns ± 4%      337ns ± 8%   +11.30%  (p=0.008 n=5+5)
ReadEvents/json_codec_big_tx/1000_events-12                                                         8.61ms ± 1%     8.76ms ± 1%    +1.69%  (p=0.016 n=5+4)
ReadEvents/nop_codec/0_events-12                                                                     309ns ± 7%      337ns ± 5%    +9.19%  (p=0.016 n=5+5)
IsTraceSampled/sampled-12                                                                           67.5ns ± 1%     74.8ns ± 1%   +10.82%  (p=0.008 n=5+5)
IsTraceSampled/unsampled-12                                                                         69.3ns ± 6%     77.3ns ± 1%   +11.64%  (p=0.008 n=5+5)
IsTraceSampled/unknown-12                                                                            373ns ± 2%      412ns ± 1%   +10.52%  (p=0.008 n=5+5)

name                                                                                              old alloc/op   new alloc/op    delta
pkg:github.com/elastic/apm-server/internal/agentcfg goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/internal/beater/request goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/internal/model/modelindexer goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/internal/processor/stream goos:linux goarch:amd64
BackendProcessor/minimal.ndjson-12                                                                  5.89kB ± 1%     5.79kB ± 1%    -1.73%  (p=0.032 n=5+5)
BackendProcessor/ratelimit.ndjson-12                                                                11.8kB ± 2%     11.6kB ± 0%    -1.55%  (p=0.032 n=5+5)
BackendProcessor/transactions-huge_traces.ndjson-12                                                 15.2kB ± 2%     14.9kB ± 1%    -1.52%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel2/invalid-event-type.ndjson-12            4.11kB ± 1%     4.18kB ± 1%    +1.65%  (p=0.032 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel2/invalid-metadata-2.ndjson-12            3.14kB ± 2%     3.20kB ± 1%    +1.85%  (p=0.032 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/invalid-json-metadata.ndjson-12       5.73kB ± 1%     5.67kB ± 1%    -1.02%  (p=0.016 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/transactions_spans.ndjson-12          25.0kB ± 1%     24.8kB ± 1%    -0.97%  (p=0.008 n=5+5)
ReadBatch/heavy.ndjson-12                                                                            831kB ± 0%      832kB ± 0%    +0.14%  (p=0.008 n=5+5)
pkg:github.com/elastic/apm-server/internal/publish goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/aggregation/spanmetrics goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/aggregation/txmetrics goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/sampling goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/sampling/eventstorage goos:linux goarch:amd64
WriteTransaction/json_codec-12                                                                      3.00kB ± 0%     3.00kB ± 0%    +0.05%  (p=0.016 n=5+4)
ReadEvents/nop_codec/100_events-12                                                                   250kB ± 0%      250kB ± 0%    -0.08%  (p=0.008 n=5+5)

name                                                                                              old allocs/op  new allocs/op   delta
pkg:github.com/elastic/apm-server/internal/agentcfg goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/internal/beater/request goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/internal/model/modelindexer goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/internal/processor/stream goos:linux goarch:amd64
ReadBatch/heavy.ndjson-12                                                                            22.1k ± 0%      22.1k ± 0%    +0.01%  (p=0.016 n=4+5)
pkg:github.com/elastic/apm-server/internal/publish goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/aggregation/spanmetrics goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/aggregation/txmetrics goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/sampling goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/sampling/eventstorage goos:linux goarch:amd64

name                                                                                              old speed      new speed       delta
pkg:github.com/elastic/apm-server/internal/model/modelindexer goos:linux goarch:amd64
ModelIndexer/BestCompressionScaling-12                                                             418MB/s ± 3%    435MB/s ± 2%    +4.12%  (p=0.032 n=5+5)
pkg:github.com/elastic/apm-server/internal/processor/stream goos:linux goarch:amd64
BackendProcessor/invalid-metadata-2.ndjson-12                                                     54.4MB/s ± 7%   62.3MB/s ±15%   +14.55%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel2/metadata-null-values.ndjson-12         193MB/s ± 3%    184MB/s ± 5%    -4.59%  (p=0.032 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel2/transactions_spans.ndjson-12           190MB/s ± 9%    160MB/s ±11%   -15.64%  (p=0.016 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel4/errors_transaction_id.ndjson-12        316MB/s ±10%    286MB/s ± 4%    -9.70%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel4/logs.ndjson-12                         249MB/s ± 3%    271MB/s ± 6%    +8.66%  (p=0.032 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel8/transactions-huge_traces.ndjson-12     566MB/s ± 2%    552MB/s ± 3%    -2.40%  (p=0.032 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel8/transactions.ndjson-12                 519MB/s ± 1%    504MB/s ± 1%    -2.87%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel8/transactions_spans.ndjson-12           541MB/s ± 2%    522MB/s ± 2%    -3.51%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel8/transactions_spans_rum.ndjson-12       576MB/s ± 1%    563MB/s ± 1%    -2.25%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel8/transactions_spans_rum_2.ndjson-12     578MB/s ± 1%    564MB/s ± 1%    -2.40%  (p=0.016 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/errors.ndjson-12                     925MB/s ± 2%    856MB/s ± 2%    -7.43%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/errors_2.ndjson-12                   718MB/s ± 4%    676MB/s ± 3%    -5.88%  (p=0.016 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/invalid-event-type.ndjson-12         489MB/s ± 1%    478MB/s ± 2%    -2.12%  (p=0.016 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/invalid-event.ndjson-12              231MB/s ± 0%    227MB/s ± 1%    -1.73%  (p=0.016 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/invalid-json-metadata.ndjson-12      239MB/s ± 0%    236MB/s ± 1%    -1.38%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/invalid-metadata-2.ndjson-12         935MB/s ± 1%    907MB/s ± 5%    -3.08%  (p=0.016 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/invalid-metadata.ndjson-12           945MB/s ± 2%    922MB/s ± 1%    -2.43%  (p=0.016 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/span-links.ndjson-12                 578MB/s ± 1%    565MB/s ± 2%    -2.23%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/transactions-huge_traces.ndjson-12   679MB/s ± 1%    668MB/s ± 2%    -1.71%  (p=0.016 n=5+5)
ReadBatch/minimal.ndjson-12                                                                       44.1MB/s ±10%   54.4MB/s ±27%   +23.50%  (p=0.032 n=5+5)
ReadBatch/transactions-huge_traces.ndjson-12                                                      44.1MB/s ±19%   57.5MB/s ±28%   +30.57%  (p=0.032 n=5+5)

report generated with https://pkg.go.dev/golang.org/x/perf/cmd/benchstat

[carsonip]

Tested on ESS 8.6. APM endpoint is accessible. Source maps are applied to RUM errors.