E2E create-stack fails as of October 24, 2022

[aleksmaus]

Trying to spin up the stack for E2e, following the instructions, getting fleet server setup failure.

Observed:

1. Elasicsearch and kibana are set up. Kibana is accessible is on HTTP port (not HTTPS)
2. Fleet server setup failed, not running.

3. Navigating to the fleet-server setup page with 8.6.0 stack give UI validation error because the url can't be HTTP

Need to be able to test the agent install/uninstall with Endpoint service against 8.6.0 stack.
Please advice what is the best way to proceeed.

[cmacknz]

@AndersonQ have you seen this in other tests?

[AndersonQ]

yes, I've managed to reproduce it today.

However I haven't been able yet to pin point the problem. So far I found the agent docker container fails first because "License is not available", then it seems to be using wrong credentials: "unable to authenticate with provided credentials and anonymous access is not allowed for this request"

The last error unable to authenticate with provided credentials and anonymous access is not allowed for this request keeping being repeated until the container exits. I believe it keeps trying until a timeout is reached. However as I said, I haven't had the time to pinpoint the root cause yet.

Kibana Fleet setup failed: http POST request to http://3.142.220.142:5601/api/fleet/setup fails: fail to execute the HTTP POST request: Post "http://3.142.220.142:5601/api/fleet/setup": read tcp 172.18.0.4:45186->3.142.220.142:5601: read: connection reset by peer. Response: 
Kibana Fleet setup failed: http POST request to http://3.142.220.142:5601/api/fleet/setup fails: fail to execute the HTTP POST request: Post "http://3.142.220.142:5601/api/fleet/setup": dial tcp 3.142.220.142:5601: connect: connection refused. Response: 
Kibana Fleet setup failed: http POST request to http://3.142.220.142:5601/api/fleet/setup fails: License is not available.: %!w(<nil>). Response: {"statusCode":503,"error":"Service Unavailable","message":"License is not available."}
Kibana Fleet setup failed: http POST request to http://3.142.220.142:5601/api/fleet/setup fails: [security_exception: [security_exception] Reason: unable to authenticate with provided credentials and anonymous access is not allowed for this request]: unable to authenticate with provided credentials and anonymous access is not allowed for this request: %!w(<nil>). Response: {"statusCode":401,"error":"Unauthorized","message":"[security_exception: [security_exception] Reason: unable to authenticate with provided credentials and anonymous access is not allowed for this request]: unable to authenticate with provided credent... (truncated)

[cmacknz]

Thanks @AndersonQ, I assigned this to you to drive forward as part of the general effort to fix the E2E tests.

@cachedout is there anyone who understands the internals of how the E2E tests are provisioned that could help us investigate this?

[cachedout]

@cmacknz Yes of course. Please have @AndersonQ sync up with @pazone

[pazone]

We can start with changing the FLEET_URL here for 8.6 branch

[AndersonQ]

hi, @pazone,
Sorry, but I did not quite get what you're suggesting to change, you meant to change the FLEET_URL to use HTTPS instead?

[pazone]

@AndersonQ yes. Seems like it is specified there.

[AndersonQ]

I found out a workaround, I don't know yet the root cause, but recreating the stack fixes it.

Given the idea of recreating the stack is to fix a API token expired, I can only conclude the API token is expired/invalid since the beginning.

[kuisathaverat]

resolved at #3285