Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AWS upgrade role entries #7025

Merged
merged 58 commits into from
Jul 5, 2019
Merged
Show file tree
Hide file tree
Changes from 54 commits
Commits
Show all changes
58 commits
Select commit Hold shift + click to select a range
981f708
upgrade aws roles
mjarmy Jun 30, 2019
1ead348
test upgrade aws roles
mjarmy Jun 30, 2019
cccb56e
Initialize aws credential backend at mount time
mjarmy Jun 30, 2019
ce1dbe0
add a TODO
mjarmy Jun 30, 2019
593048f
create end-to-end test for builtin/credential/aws
mjarmy Jun 30, 2019
7e900b6
fix bug in initializer
mjarmy Jun 30, 2019
add4b06
improve comments
mjarmy Jun 30, 2019
24a10f5
add Initialize() to logical.Backend
mjarmy Jun 30, 2019
6abe0ee
use Initialize() in Core.enableCredentialInternal()
mjarmy Jun 30, 2019
a3d10a5
use InitializeRequest to call Initialize()
mjarmy Jul 1, 2019
ae25817
improve unit testing for framework.Backend
mjarmy Jul 1, 2019
274ed9d
call logical.Backend.Initialize() from all of the places that it need…
mjarmy Jul 1, 2019
397dd6f
implement backend.proto changes for logical.Backend.Initialize()
mjarmy Jul 1, 2019
e38ffab
persist current role storage version when upgrading aws roles
mjarmy Jul 1, 2019
f68254b
format comments correctly
mjarmy Jul 1, 2019
07e4722
improve comments
mjarmy Jul 1, 2019
7bca471
use postUnseal funcs to initialize backends
mjarmy Jul 1, 2019
bffa3f5
simplify test suite
mjarmy Jul 1, 2019
851610c
improve test suite
mjarmy Jul 1, 2019
67ac41c
merge from master
mjarmy Jul 2, 2019
4f36e95
simplify logic in aws role upgrade
mjarmy Jul 2, 2019
ccb86b5
simplify aws credential initialization logic
mjarmy Jul 2, 2019
1f95d4b
simplify logic in aws role upgrade
mjarmy Jul 2, 2019
bdef021
use the core's activeContext for initialization
mjarmy Jul 2, 2019
65df479
refactor builtin/plugin/Backend
mjarmy Jul 2, 2019
ce5e3cf
use a goroutine to upgrade the aws roles
mjarmy Jul 2, 2019
5401e63
misc improvements and cleanup
mjarmy Jul 2, 2019
e67f828
do not run AWS role upgrade on DR Secondary
mjarmy Jul 2, 2019
e574a14
always call logical.Backend.Initialize() when loading a plugin.
mjarmy Jul 2, 2019
dc7db93
improve comments
mjarmy Jul 2, 2019
9e74fb3
on standbys and DR secondaries we do not want to run any kind of upgr…
mjarmy Jul 2, 2019
208eb8d
merge from master
mjarmy Jul 3, 2019
11dbf68
fix awsVersion struct
mjarmy Jul 3, 2019
034791b
merge from master
mjarmy Jul 3, 2019
f95bb32
merge fixes to aws version
mjarmy Jul 3, 2019
5187bd2
clarify aws version upgrade
mjarmy Jul 3, 2019
c5e63e4
make the upgrade logic for aws auth more explicit
mjarmy Jul 3, 2019
c652d55
aws upgrade is now called from a switch
mjarmy Jul 3, 2019
0344a87
fix fallthrough bug
mjarmy Jul 3, 2019
52dae95
simplify logic
mjarmy Jul 3, 2019
c3bdfac
simplify logic
mjarmy Jul 3, 2019
94e707b
rename things
mjarmy Jul 3, 2019
d5de690
introduce currentAwsVersion const to track aws version
mjarmy Jul 3, 2019
4aea057
improve comments
mjarmy Jul 3, 2019
e868585
rearrange things once more
mjarmy Jul 4, 2019
d38f38b
conglomerate things into one function
mjarmy Jul 4, 2019
0e1150a
stub out aws auth initialize e2e test
mjarmy Jul 3, 2019
e8da5e5
improve aws auth initialize e2e test
mjarmy Jul 3, 2019
8e3e3f9
finish aws auth initialize e2e test
mjarmy Jul 4, 2019
b1603a3
tinker with aws auth initialize e2e test
mjarmy Jul 4, 2019
9a13a44
tinker with aws auth initialize e2e test
mjarmy Jul 4, 2019
7a2044e
tinker with aws auth initialize e2e test
mjarmy Jul 5, 2019
840aa89
fix typo in test suite
mjarmy Jul 5, 2019
9d2275c
simplify logic a tad
mjarmy Jul 5, 2019
fcdae78
rearrange assignment
mjarmy Jul 5, 2019
a8afc74
Fix a few lifecycle related issues in #7025 (#7075)
briankassouf Jul 5, 2019
d59209f
Merge branch 'master' into aws-upgrade-role-entries-v2
briankassouf Jul 5, 2019
f50bf2f
Fix panic when plugin fails to load
Jul 5, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions builtin/credential/aws/backend.go
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ type backend struct {
configMutex sync.RWMutex

// Lock to make changes to role entries
roleMutex sync.RWMutex
roleMutex sync.Mutex

// Lock to make changes to the blacklist entries
blacklistMutex sync.RWMutex
Expand Down Expand Up @@ -134,8 +134,9 @@ func Backend(conf *logical.BackendConfig) (*backend, error) {
pathIdentityWhitelist(b),
pathTidyIdentityWhitelist(b),
},
Invalidate: b.invalidate,
BackendType: logical.TypeCredential,
Invalidate: b.invalidate,
InitializeFunc: b.initialize,
BackendType: logical.TypeCredential,
}

return b, nil
Expand Down
133 changes: 133 additions & 0 deletions builtin/credential/aws/backend_e2e_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,133 @@
package awsauth

import (
"context"
"testing"
"time"

hclog "github.com/hashicorp/go-hclog"
"github.com/hashicorp/vault/api"
vaulthttp "github.com/hashicorp/vault/http"
"github.com/hashicorp/vault/sdk/helper/logging"
"github.com/hashicorp/vault/sdk/logical"
"github.com/hashicorp/vault/vault"
)

func TestBackend_E2E_Initialize(t *testing.T) {

ctx := context.Background()

// Set up the cluster. This will trigger an Initialize(); we sleep briefly
// awaiting its completion.
cluster := setupAwsTestCluster(t, ctx)
defer cluster.Cleanup()
time.Sleep(time.Second)
core := cluster.Cores[0]

// Fetch the aws auth's path in storage. This is a uuid that is different
// every time we run the test
authUuids, err := core.UnderlyingStorage.List(ctx, "auth/")
if err != nil {
t.Fatal(err)
}
if len(authUuids) != 1 {
t.Fatalf("expected exactly one auth path")
}
awsPath := "auth/" + authUuids[0]

// Make sure that the upgrade happened, by fishing the 'config/version'
// entry out of storage. We can't use core.Client.Logical().Read() to do
// this, because 'config/version' hasn't been exposed as a path.
// TODO: should we expose 'config/version' as a path?
version, err := core.UnderlyingStorage.Get(ctx, awsPath+"config/version")
if err != nil {
t.Fatal(err)
}
if version == nil {
t.Fatalf("no config found")
}

// Nuke the version, so we can pretend that Initialize() has never been run
if err := core.UnderlyingStorage.Delete(ctx, awsPath+"config/version"); err != nil {
t.Fatal(err)
}
version, err = core.UnderlyingStorage.Get(ctx, awsPath+"config/version")
if err != nil {
t.Fatal(err)
}
if version != nil {
t.Fatalf("version found")
}

// Create a role
data := map[string]interface{}{
"auth_type": "ec2",
"policies": "default",
"bound_subnet_id": "subnet-abcdef"}
if _, err := core.Client.Logical().Write("auth/aws/role/test-role", data); err != nil {
t.Fatal(err)
}
role, err := core.Client.Logical().Read("auth/aws/role/test-role")
if err != nil {
t.Fatal(err)
}
if role == nil {
t.Fatalf("no role found")
}

// There should _still_ be no config version
version, err = core.UnderlyingStorage.Get(ctx, awsPath+"config/version")
if err != nil {
t.Fatal(err)
}
if version != nil {
t.Fatalf("version found")
}

// Seal, and then Unseal. This will once again trigger an Initialize(),
// only this time there will be a role present during the upgrade.
core.Seal(t)
cluster.UnsealCores(t)
time.Sleep(time.Second)

// Now the config version should be there again
version, err = core.UnderlyingStorage.Get(ctx, awsPath+"config/version")
if err != nil {
t.Fatal(err)
}
if version == nil {
t.Fatalf("no version found")
}
}

func setupAwsTestCluster(t *testing.T, ctx context.Context) *vault.TestCluster {

// create a cluster with the aws auth backend built-in
logger := logging.NewVaultLogger(hclog.Trace)
coreConfig := &vault.CoreConfig{
Logger: logger,
CredentialBackends: map[string]logical.Factory{
"aws": Factory,
},
}
cluster := vault.NewTestCluster(t, coreConfig, &vault.TestClusterOptions{
NumCores: 1,
HandlerFunc: vaulthttp.Handler,
})

cluster.Start()
if len(cluster.Cores) != 1 {
t.Fatalf("expected exactly one core")
}
core := cluster.Cores[0]
vault.TestWaitActive(t, core.Core)

// load the auth plugin
if err := core.Client.Sys().EnableAuthWithOptions("aws", &api.EnableAuthOptions{
Type: "aws",
}); err != nil {
t.Fatal(err)
}

return cluster
}
111 changes: 111 additions & 0 deletions builtin/credential/aws/path_role.go
Original file line number Diff line number Diff line change
Expand Up @@ -319,6 +319,117 @@ func (b *backend) setRole(ctx context.Context, s logical.Storage, roleName strin
return nil
}

// initialize is used to initialize the AWS roles
func (b *backend) initialize(ctx context.Context, req *logical.InitializationRequest) error {

// on standbys and DR secondaries we do not want to run any kind of upgrade logic
if b.System().ReplicationState().HasState(consts.ReplicationPerformanceStandby | consts.ReplicationDRSecondary) {
return nil
}

// Initialize only if we are either:
// (1) A local mount.
// (2) Are _NOT_ a replicated performance secondary
if b.System().LocalMount() || !b.System().ReplicationState().HasState(consts.ReplicationPerformanceSecondary) {

s := req.Storage

logger := b.Logger().Named("initialize")
logger.Info("starting initialization")

go func() {
mjarmy marked this conversation as resolved.
Show resolved Hide resolved
// The vault will become unsealed while this goroutine is running,
// so we could see some role requests block until the lock is
// released. However we'd rather see those requests block (and
// potentially start timing out) than allow a non-upgraded role to
// be fetched.
b.roleMutex.Lock()
defer b.roleMutex.Unlock()

upgraded, err := b.upgrade(ctx, s)
if err != nil {
logger.Error("error running initialization", "error", err)
return
}
if upgraded {
logger.Info("an upgrade was performed during initialization")
}
}()

}

return nil
}

// awsVersion stores info about the the latest aws version that we have
// upgraded to.
type awsVersion struct {
Version int `json:"version"`
}

// currentAwsVersion stores the latest version that we have upgraded to.
// Note that this is tracked independently from currentRoleStorageVersion.
const currentAwsVersion = 1

// upgrade does an upgrade, if necessary
func (b *backend) upgrade(ctx context.Context, s logical.Storage) (bool, error) {

entry, err := s.Get(ctx, "config/version")
if err != nil {
return false, err
}
var version awsVersion
if entry != nil {
err = entry.DecodeJSON(&version)
if err != nil {
return false, err
}
}

upgraded := version.Version < currentAwsVersion
switch version.Version {
case 0:
mjarmy marked this conversation as resolved.
Show resolved Hide resolved
// Read all the role names.
roleNames, err := s.List(ctx, "role/")
if err != nil {
return false, err
}

// Upgrade the roles as necessary.
for _, roleName := range roleNames {
// make sure the context hasn't been canceled
if ctx.Err() != nil {
return false, err
}
_, err := b.roleInternal(ctx, s, roleName)
if err != nil {
return false, err
}
}
mjarmy marked this conversation as resolved.
Show resolved Hide resolved
fallthrough

case currentAwsVersion:
mjarmy marked this conversation as resolved.
Show resolved Hide resolved
mjarmy marked this conversation as resolved.
Show resolved Hide resolved

default:
return false, fmt.Errorf("unrecognized role version: %d", version.Version)
}

// save the current version
if upgraded {
version.Version = currentAwsVersion
entry, err = logical.StorageEntryJSON("config/version", &version)
if err != nil {
return false, err
}
err = s.Put(ctx, entry)
if err != nil {
return false, err
}
}

return upgraded, nil
}

// If needed, updates the role entry and returns a bool indicating if it was updated
// (and thus needs to be persisted)
func (b *backend) upgradeRole(ctx context.Context, s logical.Storage, roleEntry *awsRoleEntry) (bool, error) {
Expand Down
Loading