Worker no tofu token but not in correct session state

[acodemics]

Hi everyone,

I recently upgraded our systems to Boundary 0.10.1 and now our worker shows these error messages when trying to access proxied services.

It seems like the first connection to a target works but the subsequent connections (e.g. downloading website resources fails).

Trying to download resources individually works.

{"id":"LFmO3Z1vwy","source":"https://hashicorp.com/boundary/bastion/worker","specversion":"1.0","type":"system","data":{"version":"v0.1","op":"worker.(Worker).startAuthRotationTicking","data":{"msg":"using kms worker authentication; pki auth rotation ticking not running"}},"datacontentype":"application/cloudevents","time":"2022-08-18T15:31:09.793865506Z"}
{"id":"SoNiJ9jLUf","source":"https://hashicorp.com/boundary/bastion/worker","specversion":"1.0","type":"system","data":{"version":"v0.1","op":"worker.(Worker).controllerDialerFunc","data":{"msg":"worker has successfully authenticated"}},"datacontentype":"application/cloudevents","time":"2022-08-18T15:31:09.8039452Z"}
{"id":"ENtVNtpwn5","source":"https://hashicorp.com/boundary/bastion/worker","specversion":"1.0","type":"system","data":{"version":"v0.1","op":"worker.(Worker).sendWorkerStatus","data":{"msg":"Upstreams after first status set to: [127.0.0.1:9201]"}},"datacontentype":"application/cloudevents","time":"2022-08-18T15:31:09.819869155Z"}
{"id":"CrbzT0lt1j","source":"https://hashicorp.com/boundary/bastion/worker","specversion":"1.0","type":"system","data":{"version":"v0.1","op":"worker.(Worker).handleProxy","data":{"msg":"session successfully activated","session_id":"s_UtyZ81jhuB"}},"datacontentype":"application/cloudevents","time":"2022-08-18T15:32:04.557635761Z"}
{"id":"YYapu81WI9","source":"https://hashicorp.com/boundary/bastion/worker","specversion":"1.0","type":"system","data":{"version":"v0.1","op":"worker.(Worker).handleProxy","data":{"connection_id":"sc_FdsiXEbIzy","msg":"connection successfully authorized","session_id":"s_UtyZ81jhuB"}},"datacontentype":"application/cloudevents","time":"2022-08-18T15:32:04.566216109Z"}
{"id":"1QOuifbL2Y","source":"https://hashicorp.com/boundary/bastion/worker","specversion":"1.0","type":"error","data":{"error":"no tofu token but not in correct session state","error_fields":{},"id":"e_mbGbXC3574","version":"v0.1","op":"worker.(Worker).handleProxy","info":{"session_id":"s_UtyZ81jhuB"}},"datacontentype":"application/cloudevents","time":"2022-08-18T15:32:05.908585813Z"}
{"id":"1YaRSw19WJ","source":"https://hashicorp.com/boundary/bastion/worker","specversion":"1.0","type":"error","data":{"error":"no tofu token but not in correct session state","error_fields":{},"id":"e_EwCNr4zJkD","version":"v0.1","op":"worker.(Worker).handleProxy","info":{"session_id":"s_UtyZ81jhuB"}},"datacontentype":"application/cloudevents","time":"2022-08-18T15:32:06.132326783Z"}
{"id":"ZANC6Mepry","source":"https://hashicorp.com/boundary/bastion/worker","specversion":"1.0","type":"error","data":{"error":"no tofu token but not in correct session state","error_fields":{},"id":"e_ViTjNziTSQ","version":"v0.1","op":"worker.(Worker).handleProxy","info":{"session_id":"s_UtyZ81jhuB"}},"datacontentype":"application/cloudevents","time":"2022-08-18T15:32:06.355358114Z"}
{"id":"Sv7mwm9DsC","source":"https://hashicorp.com/boundary/bastion/worker","specversion":"1.0","type":"error","data":{"error":"no tofu token but not in correct session state","error_fields":{},"id":"e_KmVZE05OID","version":"v0.1","op":"worker.(Worker).handleProxy","info":{"session_id":"s_UtyZ81jhuB"}},"datacontentype":"application/cloudevents","time":"2022-08-18T15:32:06.593064996Z"}
{"id":"hHFXdmHmdk","source":"https://hashicorp.com/boundary/bastion/worker","specversion":"1.0","type":"error","data":{"error":"no tofu token but not in correct session state","error_fields":{},"id":"e_zLZq4xWtvc","version":"v0.1","op":"worker.(Worker).handleProxy","info":{"session_id":"s_UtyZ81jhuB"}},"datacontentype":"application/cloudevents","time":"2022-08-18T15:32:06.818712054Z"}

We didn't change anything in our server or worker hcl files, in fact they pretty much follow exactly the example configuration files from the tutorial pages.

We run a single controller and a single worker and reverting back to 0.9.1 works as expected.

[irenarindos]

@acodemics thank you so much for reporting this! This looks like a match to #2362 and we are actively working on a fix.

[acodemics]

Thanks for the quick response!

I saw that error and wasn't sure if it was the same as we are getting different logs. It's there if it helps at least :)

[jefferai]

Yep -- it's the same error, just what's written to observations vs. what's returned to the user!

Duplicate of #2362 -- closing as such!