"Invalid Key" when attempting to stream logs over websocket #2129
Comments
|
@alex-mozejko we face the same issue. You should be able to rule out ingress/nginx issue by doing a port-forward to the service or pod and then try to call the job endpoint again. It doesn't change anything on our side though. I think it's coming from here(that's what the logs indicate), where it checks if the job exists with the given ID: For our setup I found the reason. The documentation says that the log stream doesn't work with custom apply / plan commands. We use a Makefile as a wrapper. I got the log stream working now by using the default init/plan/apply from Atlantis. |
|
Ah same here, we're using Terragrunt and encountering the same issue. Just tested on a plain Terraform project using the default workflow and it works perfectly. Custom workflow support would be amazing! |
|
👍 for custom workflow support |
|
I think there are two separate and unrelated issues being discussed in this issue:
FWIW, Atlantis really needs a configuration option to disable websockets. |
|
is this still happening with |
|
Still happend v0.19.8 |
jamengual
added
help wanted
|
it has been failed to get Environment details:source = "terraform-aws-modules/atlantis/aws" |
|
@hungran i observe the same behavior - logs reflected fine without ATLANTIS_TFE_TOKEN and i have blank screen with it |
|
as a workaround to solve this, instead of using the Create a secret from terraformrc file Then inject the secret as a volume in the helm chart |
|
We are having the same issue with This is what we see in the logs: {"level":"error","ts":"2022-11-10T11:40:31.070Z","caller":"logging/simple_logger.go:163","msg":"invalid key: 67b1873e-03cd-4574-a903-66a61805bdec","json":{},"stacktrace":"github.com/runatlantis/atlantis/server/logging.(*StructuredLogger).Log\n\tgithub.com/runatlantis/atlantis/server/logging/simple_logger.go:163\ngithub.com/runatlantis/atlantis/server/controllers.(*JobsController).respond\n\tgithub.com/runatlantis/atlantis/server/controllers/jobs_controller.go:92\ngithub.com/runatlantis/atlantis/server/controllers.(*JobsController).getProjectJobsWS\n\tgithub.com/runatlantis/atlantis/server/controllers/jobs_controller.go:70\ngithub.com/runatlantis/atlantis/server/controllers.(*JobsController).GetProjectJobsWS\n\tgithub.com/runatlantis/atlantis/server/controllers/jobs_controller.go:83\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2109\ngithub.com/gorilla/mux.(*Router).ServeHTTP\n\tgithub.com/gorilla/[email protected]/mux.go:210\ngithub.com/urfave/negroni.Wrap.func1\n\tgithub.com/urfave/[email protected]/negroni.go:46\ngithub.com/urfave/negroni.HandlerFunc.ServeHTTP\n\tgithub.com/urfave/[email protected]/negroni.go:29\ngithub.com/urfave/negroni.middleware.ServeHTTP\n\tgithub.com/urfave/[email protected]/negroni.go:38\ngithub.com/runatlantis/atlantis/server.(*RequestLogger).ServeHTTP\n\tgithub.com/runatlantis/atlantis/server/middleware.go:70\ngithub.com/urfave/negroni.middleware.ServeHTTP\n\tgithub.com/urfave/[email protected]/negroni.go:38\ngithub.com/urfave/negroni.(*Recovery).ServeHTTP\n\tgithub.com/urfave/[email protected]/recovery.go:193\ngithub.com/urfave/negroni.middleware.ServeHTTP\n\tgithub.com/urfave/[email protected]/negroni.go:38\ngithub.com/urfave/negroni.(*Negroni).ServeHTTP\n\tgithub.com/urfave/[email protected]/negroni.go:96\nnet/http.serverHandler.ServeHTTP\n\tnet/http/server.go:2947\nnet/http.(*conn).serve\n\tnet/http/server.go:1991"}{
"level": "error",
"ts": "2022-11-10T11:40:31.070Z",
"caller": "logging/simple_logger.go:163",
"msg": "invalid key: 67b1873e-03cd-4574-a903-66a61805bdec",
"json": {},
"stacktrace": "...see below..."
}$ echo -E $stacktrace | jq -r .stacktrace
github.com/runatlantis/atlantis/server/logging.(*StructuredLogger).Log
github.com/runatlantis/atlantis/server/logging/simple_logger.go:163
github.com/runatlantis/atlantis/server/controllers.(*JobsController).respond
github.com/runatlantis/atlantis/server/controllers/jobs_controller.go:92
github.com/runatlantis/atlantis/server/controllers.(*JobsController).getProjectJobsWS
github.com/runatlantis/atlantis/server/controllers/jobs_controller.go:70
github.com/runatlantis/atlantis/server/controllers.(*JobsController).GetProjectJobsWS
github.com/runatlantis/atlantis/server/controllers/jobs_controller.go:83
net/http.HandlerFunc.ServeHTTP
net/http/server.go:2109
github.com/gorilla/mux.(*Router).ServeHTTP
github.com/gorilla/[email protected]/mux.go:210
github.com/urfave/negroni.Wrap.func1
github.com/urfave/[email protected]/negroni.go:46
github.com/urfave/negroni.HandlerFunc.ServeHTTP
github.com/urfave/[email protected]/negroni.go:29
github.com/urfave/negroni.middleware.ServeHTTP
github.com/urfave/[email protected]/negroni.go:38
github.com/runatlantis/atlantis/server.(*RequestLogger).ServeHTTP
github.com/runatlantis/atlantis/server/middleware.go:70
github.com/urfave/negroni.middleware.ServeHTTP
github.com/urfave/[email protected]/negroni.go:38
github.com/urfave/negroni.(*Recovery).ServeHTTP
github.com/urfave/[email protected]/recovery.go:193
github.com/urfave/negroni.middleware.ServeHTTP
github.com/urfave/[email protected]/negroni.go:38
github.com/urfave/negroni.(*Negroni).ServeHTTP
github.com/urfave/[email protected]/negroni.go:96
net/http.serverHandler.ServeHTTP
net/http/server.go:2947
net/http.(*conn).serve
net/http/server.go:1991 |
|
Hmmm @amontalban seems like the issue thrown for you is in this block atlantis/server/controllers/jobs_controller.go Lines 66 to 88 in 4cacaeb |
|
we are using terragrunt for our workflows, and faced the exact same issue as well with the below msg and stacktrace turns out its because we had incorrect s3 permissions for this prompt wasn't shown on the UI's log stream likely because it hadn't been flushed yet as it waiting for user input and there weren't enough lines for the buffer to flush. we realised it only when we restarted our ECS services that forced the buffer to flush and then we saw the prompt on our ECS logs. once our s3 permissions was fixed, the websocket responded fine and we could see our log streams: @amontalban maybe this might help! repos.yml for reference repos:
- id: "/.*/"
workflow: terragrunt
apply_requirements: [ approved, mergeable ]
workflows:
terragrunt:
plan:
steps:
- env:
name: TERRAGRUNT_TFPATH
command: 'echo "terraform${ATLANTIS_TERRAFORM_VERSION}"'
- run: terragrunt plan -input=false -out=$PLANFILE
- run: terragrunt show -json $PLANFILE > $SHOWFILE
apply:
steps:
- env:
name: TERRAGRUNT_TFPATH
command: 'echo "terraform${ATLANTIS_TERRAFORM_VERSION}"'
- run: terragrunt apply -input=false $PLANFILE |
|
This may be related to #2026 Please try @houdinisparks method and use the latest version if possible. @amontalban @prastamaha @anton-livewyer @hungran @macropin @davidh-unmind @jonwtech @claas-fridtjof-lisowski |
|
This issue is still present on |
|
we solved the issue by this ATLANTIS_TFE_LOCAL_EXECUTION_MODE=true |
|
Same issue with ws connection failure and "invalid key", but we have never specified We're using v0.24.4 |
|
hello. please set local execution mode After applying ATLANTIS_TFE_LOCAL_EXECUTION_MODE=true we are able to see the logs. ref |
Community Note
Overview of the Issue
Cannot view streaming logs. The github actions check appears but the "details" button redirects to a page with the atlantis logo an a blank black text box.
The browser console shows:
but other, non-websocket requests (icons, css, js, etc) are successful.
Reproduction Steps
Logs
Environment details
Atlantis version: 0.19.1
helm chart values
Additional Context
I'm not sure if this is an Atlantis issue or maybe an nginx / ingress issue, i would be very thankful for any advice :)
The text was updated successfully, but these errors were encountered: