[updatecli] update elastic stack version for testing 9.0.0-b49e55df-SNAPSHOT

[github-actions]

Bump elastic-stack to latest snapshot version

Update snapshot.yml

1 file(s) updated with "$1=9.0.0-b49e55df-SNAPSHOT": * dev-tools/integration/.env

GitHub Action workflow link

---

Created automatically by Updatecli Options: Most of Updatecli configuration is done via its manifest(s). If you close this pull request, Updatecli will automatically reopen it, the next time it runs. If you close this pull request and delete the base branch, Updatecli will automatically recreate it, erasing all previous commits made. Feel free to report any issues at github.com/updatecli/updatecli. If you find this tool useful, do not hesitate to star our GitHub repository as a sign of appreciation, and/or to tell us directly on our chat!

[jlind23]

buildkite test it

[jlind23]

Looks like there is a recurring issue with APM server container never being ready. @michel-laterman does it ring a bell?

[michel-laterman]

We have two issues:

The first is with the integration tests where it looks like the utf8 invalid test has changed its return again:

=== RUN   TestBulkCreate/Invalid_utf-8
--
  | bulk_integration_test.go:103: expected error: elastic fail 500: json_parse_exception, got: elastic fail 400: parse_exception: could not parse bulk request body: json_parse_exception: Invalid UTF-8 start byte 0xfe
  | at [Source: (byte[])"{"create":{"_index":"����"}}"; line: 1, column: 23]

Is there someone one the elasticsearch team we can contact to see if this is expected? we've already changed to accepting a 400 response in the past in they have reverted it.

The second is the e2e test where the stand-alone apm server does not get ready; i can try to investigate this but we have not had any issues with it in the past

[ycombinator]

The first is with the integration tests where it looks like the utf8 invalid test has changed its return again:

=== RUN   TestBulkCreate/Invalid_utf-8
--
  | bulk_integration_test.go:103: expected error: elastic fail 500: json_parse_exception, got: elastic fail 400: parse_exception: could not parse bulk request body: json_parse_exception: Invalid UTF-8 start byte 0xfe
  | at [Source: (byte[])"{"create":{"_index":"����"}}"; line: 1, column: 23]

Is there someone one the elasticsearch team we can contact to see if this is expected? we've already changed to accepting a 400 response in the past in they have reverted it.

@mattc58 @dakrone could you please help us out here?

[mattc58]

@ycombinator certainly we'll help if we can! Can you first let me know what's failing here? I see bulk_integration_test.go in that error message. Point me to whatever Elasticsearch APIs it's calling and then we'll see if we can assist here.

[ycombinator]

Thanks @mattc! The test in question calls Elasticsearch's _bulk API with a create operation that deliberately uses an invalid index name.

This test was first created three years ago. At the time, the ES _bulk API returned a 400 response in such situations so the test asserted on this response code:

fleet-server/internal/pkg/bulk/bulk_integration_test.go

Lines 67 to 74 in 5f84562

	{
	Name: "Invalid utf-8",
	Index: string([]byte{0xfe, 0xfe, 0xff, 0xff}),
	Err: es.ErrElastic{
	Status: 400,
	Type: "json_parse_exception",
	},
	},

Then, a couple of years ago, ES changed it's behavior to run a 500 response instead so we updated the test to match: 63f20ce#diff-ae50dccb1778a130d116d0690e6c7e632e26b219bbfa3840798b233e889322d6L72

Then, a couple of weeks ago, ES changed its behavior to return a 400 response so we updated the test to match: efc1b7f#diff-ae50dccb1778a130d116d0690e6c7e632e26b219bbfa3840798b233e889322d6L72-L73

Then, a few days ago, it looks like ES changed its behavior to return a 500 response so we updated the test to match: https://github.com/elastic/fleet-server/pull/3962/files#diff-ae50dccb1778a130d116d0690e6c7e632e26b219bbfa3840798b233e889322d6L71-L73

And now, it looks like ES might've changed the behavior again so that the API returns 400, which is why the test is failing on this PR. Before we update it one more time, we'd like to know what to expect long-term so we don't have to keep updating this test. Thanks!

[mattc58]

Thanks for the really good context here @ycombinator.

@elastic/es-distributed is this possibly related to the incremental bulk work?

[Tim-Brooks]

is this possibly related to the incremental bulk work?

Maybe. Depending on how this error is created it is possible that the error code changed incidentally.

I created a JIRA to investigate: https://elasticco.atlassian.net/browse/ES-9711. We'll add a ES Rest IT to ensure that this does not change once we decide what the definitive status code should be.

[jlind23]

@Tim-Brooks I don't see any update on https://elasticco.atlassian.net/browse/ES-9711, do you know when we could expect this to be picked up?

[ycombinator]

Until https://elasticco.atlassian.net/browse/ES-9711 is addressed, I've created #4001 to serve as a temporary workaround.

[Tim-Brooks]

I think I should be able to take a look at 9711 tomorrow.

[jlind23]

Until https://elasticco.atlassian.net/browse/ES-9711 is addressed, I've created #4001 to serve as a temporary workaround.

Thanks @ycombinator just reviewed and merged it.

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarQube

[Tim-Brooks]

I think we will standardize on 400 unless something changes during the review cycle. (elastic/elasticsearch#114869)

[ycombinator]

Thanks @Tim-Brooks. @michel-laterman I've created #4012 to revert the AltErr changes in #4012.