failed to exec into task: malformed HTTP response

[scalp42]

Nomad version

Nomad v1.1.2

Operating system and Environment details

MacOS / Ubuntu

Issue

We just upgraded Nomad from 1.0.4 to 1.1.2 and we can't alloc exec into tasks anymore.

Nomad Server logs (if appropriate)

Can't find any error on server side.

Nomad Client logs (if appropriate)

failed to exec into task: malformed HTTP response "\x00\x00\x12\x04\x00\x00\x00\x00\x00\x00\x03\x00\x00\x00\x80\x00\x04\x00\x01\x00\x00\x00\x05\x00\xff\xff\xff\x00\x00\x04\b\x00\x00\x00\x00\x00\u007f\xff\x00\x00\x00\x00\b\a\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01"

[scalp42]

Just tested using client v1.1.1 and it's working fine.

$> which nomad; nomad version ; nomad alloc exec -job prod-platform-task-scheduler sh
/usr/local/bin/nomad
Nomad v1.1.2
failed to exec into task: malformed HTTP response "\x00\x00\x12\x04\x00\x00\x00\x00\x00\x00\x03\x00\x00\x00\x80\x00\x04\x00\x01\x00\x00\x00\x05\x00\xff\xff\xff\x00\x00\x04\b\x00\x00\x00\x00\x00\u007f\xff\x00\x00\x00\x00\b\a\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01"

With Nomad client v1.1.1:

$> ./nomad-1.1.1.sh alloc exec -job prod-platform-task-scheduler sh
#

[johnalotoski]

Confirmed -- works on Nomad 1.1.1, fails to exec on 1.1.2:

# Nomad v1.1.1
$ nomad --version
Nomad v1.1.1

$ nomad exec -task $TASK -namespace $NS $ALLOC /bin/bash
bash-4.4$

# Nomad v1.1.2
$ nomad --version
Nomad v1.1.2

$ nomad exec -task $TASK -namespace $NS $ALLOC /bin/bash
failed to exec into task: unexpected EOF

[notnoop]

Thanks for reporting the bug. I'll look into it this week.

[notnoop]

So I have tried to reproduce this without much luck sadly. I would appreciate more context here:

• The versions of the servers, clients, and CLI involved? Is there a deviance between the components?
• Whether there is a proxy or TLS involved? What configurations are used?

In my setup, I have tried a 3 server 2 client clusters, all running 1.1.2, with and without mutual TLS.

Thanks again!

[evandam]

Hi @notnoop, we're seeing this with Nomad v1.1.2 (60638a086ef9630e2a9ba1e237e8426192a44244) for both the server and client/CLI.

I see what you mean, it's hard to reproduce for me as well. Running Nomad locally or logging into a Nomad client and running nomad alloc exec ... works fine, but seemingly not from a remote client if it makes sense. We're seeing the issue only when accessing a Nomad server behind an AWS ALB.

CLI v1.1.2 (macOS) -> AWS ALB -> Nomad server v1.1.2 (Ubuntu 18.04)

I can confirm this does work when the CLI version is v1.1.1 and Nomad server is still 1.1.2.

I'll continue to poke around a bit to see if I can find a way to reliably reproduce the issue.

[notnoop]

@evandam That is very useful info! My hunch is that it's related attempting http/2 connections change in https://github.com/hashicorp/nomad/pull/10778/files#r653842473 .

Can you try compiling latest main branch but with the following diff:

diff --git a/api/api.go b/api/api.go
index d1f985dbe..a69c32ead 100644
--- a/api/api.go
+++ b/api/api.go
@@ -248,6 +248,7 @@ func defaultHttpClient() *http.Client {
        transport.TLSClientConfig = &tls.Config{
                MinVersion: tls.VersionTLS12,
        }
+       transport.ForceAttemptHTTP2 = false

        return httpClient
 }

As a matter of convenience, I have built binaries as CI artifacts:

• Linux and Windows: https://app.circleci.com/pipelines/github/hashicorp/nomad/17130/workflows/541dc951-b303-47e8-9eb9-563dbd0d984d/jobs/167516/artifacts
• macOS: https://app.circleci.com/pipelines/github/hashicorp/nomad/17130/workflows/541dc951-b303-47e8-9eb9-563dbd0d984d/jobs/167517/artifacts

I'm also going to test having Nomad behind AWS ALB as well. Thanks for the pointers.

[evandam]

@notnoop this works perfectly with the macOS binary you built 😄

[notnoop]

That's excellent to hear! I'll research the cause of http/2 being problematic here - and we will have a release with a fix very soon! Thank you so much for the quick validation!

[scalp42]

thanks a lot @notnoop 💛

[notnoop]

A quick update. I have reproduced the failure with ALB as well. Found out that the websocket libraries don't support http/2: gorilla/websocket#417 , and it doesn't seem to fallback to http/1 gracefully . I'm inclined to rollback to defaulting to http/1 after all. Thanks for your patience.

[surajthakur]

Thanks @notnoop for the update.
Linking my comments from here: #10657
I did find its not the ACL issue. In my two projects in two different cloud providers having similar setup (using HAProxy as load balancer), one worked with changes in ACL allowing nodeID permission, but the other didn't work, could not really get the causing issue.

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.