Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

VAULT-15547 First pass at agent/proxy decoupling #20548

Merged
merged 15 commits into from
May 17, 2023
Merged
Show file tree
Hide file tree
Changes from 13 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 12 additions & 11 deletions .github/scripts/generate-test-package-lists.sh
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ fi

# Total time: 1009
test_packages[3]+=" $base/builtin/credential/approle"
test_packages[3]+=" $base/command/agent/sink/file"
test_packages[3]+=" $base/command/agentproxyshared/sink/file"
test_packages[3]+=" $base/command/agent/template"
test_packages[3]+=" $base/helper/random"
test_packages[3]+=" $base/helper/storagepacker"
Expand Down Expand Up @@ -87,16 +87,17 @@ test_packages[6]+=" $base/builtin/audit/file"
test_packages[6]+=" $base/builtin/credential/github"
test_packages[6]+=" $base/builtin/credential/okta"
test_packages[6]+=" $base/builtin/logical/database/dbplugin"
test_packages[6]+=" $base/command/agent/auth/cert"
test_packages[6]+=" $base/command/agent/auth/jwt"
test_packages[6]+=" $base/command/agent/auth/kerberos"
test_packages[6]+=" $base/command/agent/auth/kubernetes"
test_packages[6]+=" $base/command/agent/auth/token-file"
test_packages[6]+=" $base/command/agent/cache"
test_packages[6]+=" $base/command/agent/cache/cacheboltdb"
test_packages[6]+=" $base/command/agent/cache/cachememdb"
test_packages[6]+=" $base/command/agent/cache/keymanager"
test_packages[6]+=" $base/command/agentproxyshared/auth/cert"
test_packages[6]+=" $base/command/agentproxyshared/auth/jwt"
test_packages[6]+=" $base/command/agentproxyshared/auth/kerberos"
test_packages[6]+=" $base/command/agentproxyshared/auth/kubernetes"
test_packages[6]+=" $base/command/agentproxyshared/auth/token-file"
test_packages[6]+=" $base/command/agentproxyshared/cache"
test_packages[6]+=" $base/command/agentproxyshared/cache/cacheboltdb"
test_packages[6]+=" $base/command/agentproxyshared/cache/cachememdb"
test_packages[6]+=" $base/command/agentproxyshared/cache/keymanager"
test_packages[6]+=" $base/command/agent/config"
test_packages[6]+=" $base/command/proxy/config"
test_packages[6]+=" $base/command/config"
test_packages[6]+=" $base/command/token"
if [ "${ENTERPRISE:+x}" == "x" ] ; then
Expand Down Expand Up @@ -199,7 +200,7 @@ test_packages[7]+=" $base/vault/quotas"
# Total time: 779
test_packages[8]+=" $base/builtin/credential/aws/pkcs7"
test_packages[8]+=" $base/builtin/logical/totp"
test_packages[8]+=" $base/command/agent/auth"
test_packages[8]+=" $base/command/agentproxyshared/auth"
test_packages[8]+=" $base/physical/raft"
test_packages[8]+=" $base/sdk/framework"
test_packages[8]+=" $base/sdk/plugin"
Expand Down
3 changes: 3 additions & 0 deletions changelog/20548.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
```release-note:feature
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@VioletHynes could you please update this changelog file with the format for announcing a new feature? https://hashicorp.atlassian.net/wiki/spaces/VAULT/pages/1311244491/Changelog+Process#New-and-Major-Features

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will do! Good catch, sorry for missing this!

proxy: Introduced Vault Proxy, a new subcommand of the Vault binary that can be invoked using `vault proxy -config=config.hcl`. It currently has the same feature set as Vault Agent's API proxy, but the two may diverge in the future. We plan to deprecate the API proxy functionality of Vault Agent in a future release.
```
70 changes: 37 additions & 33 deletions command/agent.go

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Whilst a lot of the code isn't new, it might be nice to unify the casing for the import aliases while you're working in here (ctconfig, token_file, agentConfig). I think a few files have the varying casing issue.

Original file line number Diff line number Diff line change
Expand Up @@ -19,40 +19,37 @@ import (
"sync"
"time"

token_file "github.com/hashicorp/vault/command/agent/auth/token-file"

ctconfig "github.com/hashicorp/consul-template/config"
"github.com/hashicorp/go-multierror"

"github.com/hashicorp/vault/command/agent/sink/inmem"

systemd "github.com/coreos/go-systemd/daemon"
ctconfig "github.com/hashicorp/consul-template/config"
log "github.com/hashicorp/go-hclog"
"github.com/hashicorp/go-multierror"
"github.com/hashicorp/go-secure-stdlib/gatedwriter"
"github.com/hashicorp/go-secure-stdlib/parseutil"
"github.com/hashicorp/go-secure-stdlib/reloadutil"
"github.com/hashicorp/vault/api"
"github.com/hashicorp/vault/command/agent/auth"
"github.com/hashicorp/vault/command/agent/auth/alicloud"
"github.com/hashicorp/vault/command/agent/auth/approle"
"github.com/hashicorp/vault/command/agent/auth/aws"
"github.com/hashicorp/vault/command/agent/auth/azure"
"github.com/hashicorp/vault/command/agent/auth/cert"
"github.com/hashicorp/vault/command/agent/auth/cf"
"github.com/hashicorp/vault/command/agent/auth/gcp"
"github.com/hashicorp/vault/command/agent/auth/jwt"
"github.com/hashicorp/vault/command/agent/auth/kerberos"
"github.com/hashicorp/vault/command/agent/auth/kubernetes"
"github.com/hashicorp/vault/command/agent/auth/oci"
"github.com/hashicorp/vault/command/agent/cache"
"github.com/hashicorp/vault/command/agent/cache/cacheboltdb"
"github.com/hashicorp/vault/command/agent/cache/cachememdb"
"github.com/hashicorp/vault/command/agent/cache/keymanager"
agentConfig "github.com/hashicorp/vault/command/agent/config"
"github.com/hashicorp/vault/command/agent/sink"
"github.com/hashicorp/vault/command/agent/sink/file"
"github.com/hashicorp/vault/command/agent/template"
"github.com/hashicorp/vault/command/agent/winsvc"
"github.com/hashicorp/vault/command/agentproxyshared/auth"
"github.com/hashicorp/vault/command/agentproxyshared/auth/alicloud"
"github.com/hashicorp/vault/command/agentproxyshared/auth/approle"
"github.com/hashicorp/vault/command/agentproxyshared/auth/aws"
"github.com/hashicorp/vault/command/agentproxyshared/auth/azure"
"github.com/hashicorp/vault/command/agentproxyshared/auth/cert"
"github.com/hashicorp/vault/command/agentproxyshared/auth/cf"
"github.com/hashicorp/vault/command/agentproxyshared/auth/gcp"
"github.com/hashicorp/vault/command/agentproxyshared/auth/jwt"
"github.com/hashicorp/vault/command/agentproxyshared/auth/kerberos"
"github.com/hashicorp/vault/command/agentproxyshared/auth/kubernetes"
"github.com/hashicorp/vault/command/agentproxyshared/auth/oci"
token_file "github.com/hashicorp/vault/command/agentproxyshared/auth/token-file"
cache "github.com/hashicorp/vault/command/agentproxyshared/cache"
"github.com/hashicorp/vault/command/agentproxyshared/cache/cacheboltdb"
"github.com/hashicorp/vault/command/agentproxyshared/cache/cachememdb"
"github.com/hashicorp/vault/command/agentproxyshared/cache/keymanager"
"github.com/hashicorp/vault/command/agentproxyshared/sink"
"github.com/hashicorp/vault/command/agentproxyshared/sink/file"
"github.com/hashicorp/vault/command/agentproxyshared/sink/inmem"
"github.com/hashicorp/vault/command/agentproxyshared/winsvc"
"github.com/hashicorp/vault/helper/logging"
"github.com/hashicorp/vault/helper/metricsutil"
"github.com/hashicorp/vault/helper/useragent"
Expand All @@ -65,6 +62,8 @@ import (
"github.com/mitchellh/cli"
"github.com/oklog/run"
"github.com/posener/complete"
"golang.org/x/text/cases"
"golang.org/x/text/language"
"google.golang.org/grpc/test/bufconn"
)

Expand Down Expand Up @@ -258,7 +257,7 @@ func (c *AgentCommand) Run(args []string) int {

// Ignore any setting of Agent's address. This client is used by the Agent
// to reach out to Vault. This should never loop back to agent.
c.flagAgentAddress = ""
c.flagAgentProxyAddress = ""
client, err := c.Client()
if err != nil {
c.UI.Error(fmt.Sprintf(
Expand Down Expand Up @@ -505,10 +504,12 @@ func (c *AgentCommand) Run(args []string) int {

// The API proxy to be used, if listeners are configured
apiProxy, err := cache.NewAPIProxy(&cache.APIProxyConfig{
Client: proxyClient,
Logger: apiProxyLogger,
EnforceConsistency: enforceConsistency,
WhenInconsistentAction: whenInconsistent,
Client: proxyClient,
Logger: apiProxyLogger,
EnforceConsistency: enforceConsistency,
WhenInconsistentAction: whenInconsistent,
UserAgentStringFunction: useragent.AgentProxyStringWithProxiedUserAgent,
UserAgentString: useragent.AgentProxyString(),
})
if err != nil {
c.UI.Error(fmt.Sprintf("Error creating API proxy: %v", err))
Expand Down Expand Up @@ -873,6 +874,8 @@ func (c *AgentCommand) Run(args []string) int {
EnableTemplateTokenCh: enableTokenCh,
Token: previousToken,
ExitOnError: config.AutoAuth.Method.ExitOnError,
UserAgent: useragent.AgentAutoAuthString(),
MetricsSignifier: "agent",
})

ss := sink.NewSinkServer(&sink.SinkServerConfig{
Expand Down Expand Up @@ -949,12 +952,13 @@ func (c *AgentCommand) Run(args []string) int {
// Server configuration output
padding := 24
sort.Strings(infoKeys)
caser := cases.Title(language.English)
c.UI.Output("==> Vault Agent configuration:\n")
for _, k := range infoKeys {
c.UI.Output(fmt.Sprintf(
"%s%s: %s",
strings.Repeat(" ", padding-len(k)),
strings.Title(k),
caser.String(k),
info[k]))
}
c.UI.Output("")
Expand Down Expand Up @@ -1200,7 +1204,7 @@ func (c *AgentCommand) handleMetrics() http.Handler {
w.Header().Set("Content-Type", resp.Data[logical.HTTPContentType].(string))
switch v := resp.Data[logical.HTTPRawBody].(type) {
case string:
w.WriteHeader((status))
w.WriteHeader(status)
w.Write([]byte(v))
case []byte:
w.WriteHeader(status)
Expand Down
8 changes: 4 additions & 4 deletions command/agent/alicloud_end_to_end_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -19,10 +19,10 @@ import (
uuid "github.com/hashicorp/go-uuid"
vaultalicloud "github.com/hashicorp/vault-plugin-auth-alicloud"
"github.com/hashicorp/vault/api"
"github.com/hashicorp/vault/command/agent/auth"
agentalicloud "github.com/hashicorp/vault/command/agent/auth/alicloud"
"github.com/hashicorp/vault/command/agent/sink"
"github.com/hashicorp/vault/command/agent/sink/file"
"github.com/hashicorp/vault/command/agentproxyshared/auth"
agentalicloud "github.com/hashicorp/vault/command/agentproxyshared/auth/alicloud"
"github.com/hashicorp/vault/command/agentproxyshared/sink"
"github.com/hashicorp/vault/command/agentproxyshared/sink/file"
"github.com/hashicorp/vault/helper/testhelpers"
vaulthttp "github.com/hashicorp/vault/http"
"github.com/hashicorp/vault/sdk/helper/logging"
Expand Down
8 changes: 4 additions & 4 deletions command/agent/approle_end_to_end_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -16,10 +16,10 @@ import (
log "github.com/hashicorp/go-hclog"
"github.com/hashicorp/vault/api"
credAppRole "github.com/hashicorp/vault/builtin/credential/approle"
"github.com/hashicorp/vault/command/agent/auth"
agentapprole "github.com/hashicorp/vault/command/agent/auth/approle"
"github.com/hashicorp/vault/command/agent/sink"
"github.com/hashicorp/vault/command/agent/sink/file"
"github.com/hashicorp/vault/command/agentproxyshared/auth"
agentapprole "github.com/hashicorp/vault/command/agentproxyshared/auth/approle"
"github.com/hashicorp/vault/command/agentproxyshared/sink"
"github.com/hashicorp/vault/command/agentproxyshared/sink/file"
vaulthttp "github.com/hashicorp/vault/http"
"github.com/hashicorp/vault/sdk/helper/logging"
"github.com/hashicorp/vault/sdk/logical"
Expand Down
8 changes: 4 additions & 4 deletions command/agent/auto_auth_preload_token_end_to_end_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -13,10 +13,10 @@ import (
hclog "github.com/hashicorp/go-hclog"
"github.com/hashicorp/vault/api"
credAppRole "github.com/hashicorp/vault/builtin/credential/approle"
"github.com/hashicorp/vault/command/agent/auth"
agentAppRole "github.com/hashicorp/vault/command/agent/auth/approle"
"github.com/hashicorp/vault/command/agent/sink"
"github.com/hashicorp/vault/command/agent/sink/file"
"github.com/hashicorp/vault/command/agentproxyshared/auth"
agentAppRole "github.com/hashicorp/vault/command/agentproxyshared/auth/approle"
"github.com/hashicorp/vault/command/agentproxyshared/sink"
"github.com/hashicorp/vault/command/agentproxyshared/sink/file"
vaulthttp "github.com/hashicorp/vault/http"
"github.com/hashicorp/vault/sdk/helper/logging"
"github.com/hashicorp/vault/sdk/logical"
Expand Down
8 changes: 4 additions & 4 deletions command/agent/aws_end_to_end_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -18,10 +18,10 @@ import (
uuid "github.com/hashicorp/go-uuid"
"github.com/hashicorp/vault/api"
vaultaws "github.com/hashicorp/vault/builtin/credential/aws"
"github.com/hashicorp/vault/command/agent/auth"
agentaws "github.com/hashicorp/vault/command/agent/auth/aws"
"github.com/hashicorp/vault/command/agent/sink"
"github.com/hashicorp/vault/command/agent/sink/file"
"github.com/hashicorp/vault/command/agentproxyshared/auth"
agentaws "github.com/hashicorp/vault/command/agentproxyshared/auth/aws"
"github.com/hashicorp/vault/command/agentproxyshared/sink"
"github.com/hashicorp/vault/command/agentproxyshared/sink/file"
"github.com/hashicorp/vault/helper/testhelpers"
vaulthttp "github.com/hashicorp/vault/http"
"github.com/hashicorp/vault/sdk/helper/logging"
Expand Down
19 changes: 11 additions & 8 deletions command/agent/cache_end_to_end_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -17,12 +17,13 @@ import (
log "github.com/hashicorp/go-hclog"
"github.com/hashicorp/vault/api"
credAppRole "github.com/hashicorp/vault/builtin/credential/approle"
"github.com/hashicorp/vault/command/agent/auth"
agentapprole "github.com/hashicorp/vault/command/agent/auth/approle"
"github.com/hashicorp/vault/command/agent/cache"
"github.com/hashicorp/vault/command/agent/sink"
"github.com/hashicorp/vault/command/agent/sink/file"
"github.com/hashicorp/vault/command/agent/sink/inmem"
"github.com/hashicorp/vault/command/agentproxyshared/auth"
agentapprole "github.com/hashicorp/vault/command/agentproxyshared/auth/approle"
cache "github.com/hashicorp/vault/command/agentproxyshared/cache"
"github.com/hashicorp/vault/command/agentproxyshared/sink"
"github.com/hashicorp/vault/command/agentproxyshared/sink/file"
"github.com/hashicorp/vault/command/agentproxyshared/sink/inmem"
"github.com/hashicorp/vault/helper/useragent"
vaulthttp "github.com/hashicorp/vault/http"
"github.com/hashicorp/vault/sdk/helper/consts"
"github.com/hashicorp/vault/sdk/helper/logging"
Expand Down Expand Up @@ -166,8 +167,10 @@ func TestCache_UsingAutoAuthToken(t *testing.T) {

// Create the API proxier
apiProxy, err := cache.NewAPIProxy(&cache.APIProxyConfig{
Client: client,
Logger: cacheLogger.Named("apiproxy"),
Client: client,
Logger: cacheLogger.Named("apiproxy"),
UserAgentStringFunction: useragent.ProxyStringWithProxiedUserAgent,
UserAgentString: useragent.ProxyString(),
})
if err != nil {
t.Fatal(err)
Expand Down
12 changes: 5 additions & 7 deletions command/agent/cert_end_to_end_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -12,16 +12,14 @@ import (
"testing"
"time"

"github.com/hashicorp/vault/builtin/logical/pki"

hclog "github.com/hashicorp/go-hclog"

"github.com/hashicorp/vault/api"
vaultcert "github.com/hashicorp/vault/builtin/credential/cert"
"github.com/hashicorp/vault/command/agent/auth"
agentcert "github.com/hashicorp/vault/command/agent/auth/cert"
"github.com/hashicorp/vault/command/agent/sink"
"github.com/hashicorp/vault/command/agent/sink/file"
"github.com/hashicorp/vault/builtin/logical/pki"
"github.com/hashicorp/vault/command/agentproxyshared/auth"
agentcert "github.com/hashicorp/vault/command/agentproxyshared/auth/cert"
"github.com/hashicorp/vault/command/agentproxyshared/sink"
"github.com/hashicorp/vault/command/agentproxyshared/sink/file"
"github.com/hashicorp/vault/helper/dhutil"
vaulthttp "github.com/hashicorp/vault/http"
"github.com/hashicorp/vault/sdk/helper/jsonutil"
Expand Down
8 changes: 4 additions & 4 deletions command/agent/cf_end_to_end_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -15,10 +15,10 @@ import (
"github.com/hashicorp/vault-plugin-auth-cf/testing/certificates"
cfAPI "github.com/hashicorp/vault-plugin-auth-cf/testing/cf"
"github.com/hashicorp/vault/api"
"github.com/hashicorp/vault/command/agent/auth"
agentcf "github.com/hashicorp/vault/command/agent/auth/cf"
"github.com/hashicorp/vault/command/agent/sink"
"github.com/hashicorp/vault/command/agent/sink/file"
"github.com/hashicorp/vault/command/agentproxyshared/auth"
agentcf "github.com/hashicorp/vault/command/agentproxyshared/auth/cf"
"github.com/hashicorp/vault/command/agentproxyshared/sink"
"github.com/hashicorp/vault/command/agentproxyshared/sink/file"
vaulthttp "github.com/hashicorp/vault/http"
"github.com/hashicorp/vault/sdk/helper/logging"
"github.com/hashicorp/vault/sdk/logical"
Expand Down
8 changes: 4 additions & 4 deletions command/agent/jwt_end_to_end_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -14,10 +14,10 @@ import (
hclog "github.com/hashicorp/go-hclog"
vaultjwt "github.com/hashicorp/vault-plugin-auth-jwt"
"github.com/hashicorp/vault/api"
"github.com/hashicorp/vault/command/agent/auth"
agentjwt "github.com/hashicorp/vault/command/agent/auth/jwt"
"github.com/hashicorp/vault/command/agent/sink"
"github.com/hashicorp/vault/command/agent/sink/file"
"github.com/hashicorp/vault/command/agentproxyshared/auth"
agentjwt "github.com/hashicorp/vault/command/agentproxyshared/auth/jwt"
"github.com/hashicorp/vault/command/agentproxyshared/sink"
"github.com/hashicorp/vault/command/agentproxyshared/sink/file"
"github.com/hashicorp/vault/helper/dhutil"
vaulthttp "github.com/hashicorp/vault/http"
"github.com/hashicorp/vault/sdk/helper/jsonutil"
Expand Down
8 changes: 4 additions & 4 deletions command/agent/oci_end_to_end_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -13,10 +13,10 @@ import (
hclog "github.com/hashicorp/go-hclog"
vaultoci "github.com/hashicorp/vault-plugin-auth-oci"
"github.com/hashicorp/vault/api"
"github.com/hashicorp/vault/command/agent/auth"
agentoci "github.com/hashicorp/vault/command/agent/auth/oci"
"github.com/hashicorp/vault/command/agent/sink"
"github.com/hashicorp/vault/command/agent/sink/file"
"github.com/hashicorp/vault/command/agentproxyshared/auth"
agentoci "github.com/hashicorp/vault/command/agentproxyshared/auth/oci"
"github.com/hashicorp/vault/command/agentproxyshared/sink"
"github.com/hashicorp/vault/command/agentproxyshared/sink/file"
"github.com/hashicorp/vault/helper/testhelpers"
vaulthttp "github.com/hashicorp/vault/http"
"github.com/hashicorp/vault/sdk/helper/logging"
Expand Down
8 changes: 4 additions & 4 deletions command/agent/token_file_end_to_end_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -11,10 +11,10 @@ import (
"time"

log "github.com/hashicorp/go-hclog"
"github.com/hashicorp/vault/command/agent/auth"
token_file "github.com/hashicorp/vault/command/agent/auth/token-file"
"github.com/hashicorp/vault/command/agent/sink"
"github.com/hashicorp/vault/command/agent/sink/file"
"github.com/hashicorp/vault/command/agentproxyshared/auth"
token_file "github.com/hashicorp/vault/command/agentproxyshared/auth/token-file"
"github.com/hashicorp/vault/command/agentproxyshared/sink"
"github.com/hashicorp/vault/command/agentproxyshared/sink/file"
vaulthttp "github.com/hashicorp/vault/http"
"github.com/hashicorp/vault/sdk/helper/logging"
"github.com/hashicorp/vault/vault"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ import (
hclog "github.com/hashicorp/go-hclog"
"github.com/hashicorp/vault-plugin-auth-alicloud/tools"
"github.com/hashicorp/vault/api"
"github.com/hashicorp/vault/command/agent/auth"
"github.com/hashicorp/vault/command/agentproxyshared/auth"
)

/*
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ import (
hclog "github.com/hashicorp/go-hclog"
"github.com/hashicorp/go-secure-stdlib/parseutil"
"github.com/hashicorp/vault/api"
"github.com/hashicorp/vault/command/agent/auth"
"github.com/hashicorp/vault/command/agentproxyshared/auth"
)

type approleMethod struct {
Expand Down
Loading