Vault Agent ignores VAULT_SKIP_VERIFY environment variable #14316

markafarrell · 2022-03-01T01:59:24Z

Describe the bug
When the VAULT_SKIP_VERIFY environment variable is set to FALSE, false, False or 0 TLS verification is still disabled.

This behavior is different from the regular vault cli client which respects the value in VAULT_SKIP_VERIFY. i.e. Skipping verification when VAULT_SKIP_VERIFY is TRUE and enabling when VAULT_SKIP_VERIFY is FALSE

To Reproduce
Steps to reproduce the behavior:

Start and initialise server with TLS enabled

mkdir tls
openssl req -x509 -nodes -days 9999 -newkey rsa:2048 \
-keyout tls/vault_server.key -out tls/vault_server.crt \
-subj "/CN=AU/ST=Some-State/L=Some-City/O=Internet Widgits Pty Ltd/OU=Something/\
CN=vault-server" \
-addext "subjectAltName = DNS:vault-server"

cat <<EOF > vault_main.hcl 
listener "tcp" {
  address = "[::]:8200"
  cluster_address = "[::]:8201"
  tls_cert_file = "/vault/tls/vault_server.crt"
  tls_key_file  = "/vault/tls/vault_server.key"
}

storage "file" {
  path = "/vault/data"
}
EOF

chmod g+r tls/*

docker network create vault

docker run --rm -v $PWD/tls/:/vault/tls/ -v $PWD/vault_main.hcl:/vault/config/vault_main.hcl --cap-add IPC_LOCK --network vault --name=vault-server vault:1.9.2 server

docker run --rm -it --cap-add IPC_LOCK -e VAULT_ADDR=https://vault-server:8200 -e NO_PROXY=vault-server -e VAULT_SKIP_VERIFY=TRUE --network vault vault:1.9.2 operator init

docker run --rm -it --cap-add IPC_LOCK -e VAULT_ADDR=https://vault-server:8200 -e NO_PROXY=vault-server -e VAULT_SKIP_VERIFY=TRUE --network vault vault:1.9.2 operator unseal
docker run --rm -it --cap-add IPC_LOCK -e VAULT_ADDR=https://vault-server:8200 -e NO_PROXY=vault-server -e VAULT_SKIP_VERIFY=TRUE --network vault vault:1.9.2 operator unseal
docker run --rm -it --cap-add IPC_LOCK -e VAULT_ADDR=https://vault-server:8200 -e NO_PROXY=vault-server -e VAULT_SKIP_VERIFY=TRUE --network vault vault:1.9.2 operator unseal

Create approle for vault agent

docker run --rm -it --cap-add IPC_LOCK -e VAULT_ADDR=https://vault-server:8200 -e VAULT_TOKEN=<ROOT TOKEN FROM initialize> -e NO_PROXY=vault-server -e VAULT_SKIP_VERIFY=TRUE --network vault vault:1.9.2 auth enable approle

docker run --rm -it --cap-add IPC_LOCK -e VAULT_ADDR=https://vault-server:8200 -e VAULT_TOKEN=<ROOT TOKEN FROM initialize> -e NO_PROXY=vault-server -e VAULT_SKIP_VERIFY=TRUE --network vault vault:1.9.2 write auth/approle/role/my-role secret_id_ttl=10m token_num_uses=10 token_ttl=20m token_max_ttl=30m secret_id_num_uses=40

Generate vault-agent config

cat <<EOF > vault_agent.hcl 
pid_file = "/tmp/pidfile"

vault {
  retry {
    num_retries = 5
  }
}

auto_auth {
  method {
    type      = "approle"
    config = {
      role_id_file_path = "/vault-agent/role-id"
      secret_id_file_path = "/vault-agent/secret-id"
    }
  }
  sink {
    type = "file"
    config = {
      path = "/vault/token"
    }
  }
}
EOF

Generate credentials for vault

mkdir vault-agent

chmod g+r vault-agent/*

docker run --rm --cap-add IPC_LOCK -e VAULT_ADDR=https://vault-server:8200 -e VAULT_TOKEN=<ROOT TOKEN FROM initialize> -e NO_PROXY=vault-server -e VAULT_SKIP_VERIFY=TRUE --network vault vault:1.9.2 read -field=role_id auth/approle/role/my-role/role-id > vault-agent/role-id
docker run --rm --cap-add IPC_LOCK -e VAULT_ADDR=https://vault-server:8200 -e VAULT_TOKEN=<ROOT TOKEN FROM initialize> -e NO_PROXY=vault-server -e VAULT_SKIP_VERIFY=TRUE --network vault vault:1.9.2 write -field=secret_id -f auth/approle/role/my-role/secret-id > vault-agent/secret-id

Run vault with VAULT_SKIP_VERIFY unset

docker run --rm -it --cap-add IPC_LOCK -e VAULT_ADDR=https://vault-server:8200 -e VAULT_LOG_LEVEL=debug -v $PWD/vault-agent/:/vault-agent/ -v $PWD/vault_agent.hcl:/vault/config/vault_agent.hcl -e NO_PROXY=vault-server --network vault vault:1.9.2 agent -config /vault/config/vault_agent.hcl

Vault agent correctly complains about certificate

==> Vault agent started! Log data will stream in below:

==> Vault agent configuration:

                     Cgo: disabled
               Log Level: debug
                 Version: Vault v1.9.2
             Version Sha: f4c6d873e2767c0d6853b5d9ffc77b0d297bfbdf

2022-03-01T01:53:23.227Z [INFO]  sink.file: creating file sink
2022-03-01T01:53:23.227Z [INFO]  sink.file: file sink configured: path=/vault/token mode=-rw-r-----
2022-03-01T01:53:23.227Z [INFO]  template.server: starting template server
2022-03-01T01:53:23.227Z [INFO]  template.server: no templates found
2022-03-01T01:53:23.227Z [INFO]  auth.handler: starting auth handler
2022-03-01T01:53:23.227Z [INFO]  sink.server: starting sink server
2022-03-01T01:53:23.228Z [INFO]  auth.handler: authenticating
2022-03-01T01:53:23.290Z [ERROR] auth.handler: error authenticating: error="Put \"https://vault-server:8200/v1/auth/approle/login\": x509: certificate signed by unknown authority" backoff=1s
2022-03-01T01:53:24.295Z [INFO]  auth.handler: authenticating
2022-03-01T01:53:24.302Z [ERROR] auth.handler: error authenticating: error="Put \"https://vault-server:8200/v1/auth/approle/login\": x509: certificate signed by unknown authority" backoff=1.56s
^C==> Vault agent shutdown triggered
2022-03-01T01:53:25.747Z [INFO]  template.server: template server stopped
2022-03-01T01:53:25.747Z [INFO]  sink.server: sink server stopped
2022-03-01T01:53:25.747Z [INFO]  sinks finished, exiting
2022-03-01T01:53:25.747Z [INFO]  auth.handler: auth handler stopped

Regenerate secret-id

docker run --rm --cap-add IPC_LOCK -e VAULT_ADDR=https://vault-server:8200 -e VAULT_TOKEN=<ROOT TOKEN from initialize> -e NO_PROXY=vault-server -e VAULT_SKIP_VERIFY=TRUE --network vault vault:1.9.2 write -field=secret_id -f auth/approle/role/my-role/secret-id > vault-agent/secret-id

Run vault with VAULT_SKIP_VERIFY=FALSE

docker run --rm -it --cap-add IPC_LOCK -e VAULT_ADDR=https://vault-server:8200 -e VAULT_LOG_LEVEL=debug -v $PWD/vault-agent/:/vault-agent/ -v $PWD/vault_agent.hcl:/vault/config/vault_agent.hcl -e NO_PROXY=vault-server -e VAULT_SKIP_VERIFY=FALSE --network vault vault:1.9.2 agent -config /vault/config/vault_agent.hcl

Vault agent skips tls verification (even though VAULT_SKIP_VERIFY=FALSE successfully acquires token

==> Vault agent started! Log data will stream in below:

==> Vault agent configuration:

                     Cgo: disabled
               Log Level: debug
                 Version: Vault v1.9.2
             Version Sha: f4c6d873e2767c0d6853b5d9ffc77b0d297bfbdf

2022-03-01T01:55:39.866Z [INFO]  sink.file: creating file sink
2022-03-01T01:55:39.866Z [INFO]  sink.file: file sink configured: path=/vault/token mode=-rw-r-----
2022-03-01T01:55:39.876Z [INFO]  template.server: starting template server
2022-03-01T01:55:39.876Z [INFO]  template.server: no templates found
2022-03-01T01:55:39.876Z [INFO]  sink.server: starting sink server
2022-03-01T01:55:39.876Z [INFO]  auth.handler: starting auth handler
2022-03-01T01:55:39.877Z [INFO]  auth.handler: authenticating
2022-03-01T01:55:39.899Z [INFO]  auth.handler: authentication successful, sending token to sinks
2022-03-01T01:55:39.899Z [INFO]  auth.handler: starting renewal process
2022-03-01T01:55:39.900Z [INFO]  sink.file: token written: path=/vault/token
2022-03-01T01:55:39.906Z [INFO]  auth.handler: renewed auth token
^C==> Vault agent shutdown triggered
2022-03-01T01:55:44.944Z [INFO]  template.server: template server stopped
2022-03-01T01:55:44.944Z [INFO]  auth.handler: shutdown triggered, stopping lifetime watcher
2022-03-01T01:55:44.944Z [INFO]  auth.handler: auth handler stopped
2022-03-01T01:55:44.944Z [INFO]  sink.server: sink server stopped
2022-03-01T01:55:44.944Z [INFO]  sinks finished, exiting

Expected behavior
A clear and concise description of what you expected to happen.

Environment:

Vault Server Version (retrieve with vault status): 1.9.2
Vault CLI Version (retrieve with vault version): 1.9.2
Server Operating System/Architecture: Docker

Vault server configuration file(s):

listener "tcp" {
  address = "[::]:8200"
  cluster_address = "[::]:8201"
  tls_cert_file = "/vault/tls/vault_server.crt"
  tls_key_file  = "/vault/tls/vault_server.key"
}

storage "file" {
  path = "/vault/data"
}

Vault agent configuration file(s):

pid_file = "/tmp/pidfile"

vault {
  retry {
    num_retries = 5
  }
}

auto_auth {
  method {
    type      = "approle"
    config = {
      role_id_file_path = "/vault-agent/role-id"
      secret_id_file_path = "/vault-agent/secret-id"
    }
  }
  sink {
    type = "file"
    config = {
      path = "/vault/token"
    }
  }
}

Additional context
Add any other context about the problem here.

The text was updated successfully, but these errors were encountered:

heatherezell · 2022-03-01T03:37:25Z

Thank you very much for this detailed report! I'll get an engineer's eyes on it ASAP.

reidlai · 2022-03-15T23:13:03Z

I am encountering the same case

nikhil-rajesh · 2023-11-24T07:16:31Z

Why is this one not resolved yet?

heatherezell added bug Used to indicate a potential bug tls agent ecosystem labels Mar 1, 2022

heatherezell removed the ecosystem label Aug 9, 2022

czembower mentioned this issue Aug 12, 2022

agent: set bool flag values correctly #16701

Closed

VioletHynes mentioned this issue Jan 10, 2024

Fix Agent and Proxy env var handling #24790

Merged

VioletHynes closed this as completed in #24790 Jan 11, 2024

gerardma77 assigned hc-github-team-secure-vault-core Jul 15, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Vault Agent ignores VAULT_SKIP_VERIFY environment variable #14316

Vault Agent ignores VAULT_SKIP_VERIFY environment variable #14316

markafarrell commented Mar 1, 2022

heatherezell commented Mar 1, 2022

reidlai commented Mar 15, 2022

nikhil-rajesh commented Nov 24, 2023

Vault Agent ignores VAULT_SKIP_VERIFY environment variable #14316

Vault Agent ignores VAULT_SKIP_VERIFY environment variable #14316

Comments

markafarrell commented Mar 1, 2022

heatherezell commented Mar 1, 2022

reidlai commented Mar 15, 2022

nikhil-rajesh commented Nov 24, 2023