Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dynamic Server Lists/Client Bootstrapping #1201

Merged
merged 166 commits into from
Jun 11, 2016
Merged
Show file tree
Hide file tree
Changes from 162 commits
Commits
Show all changes
166 commits
Select commit Hold shift + click to select a range
a1cb3f1
Use consul/lib's RateScaledInterval
sean- May 3, 2016
7db2eb0
Use consul/lib's RandomStagger
sean- May 3, 2016
2d23927
Use `rand.Int*n()` where appropriate
sean- May 3, 2016
2846614
Fix small typo
sean- May 3, 2016
3f9d385
Seed random once in main
sean- May 3, 2016
4030e38
Use the correctly typed `rand.Int*` variant
sean- May 3, 2016
0e1bdad
Rename consul.ConsulConfig to consul.AgentConfig
sean- May 21, 2016
4fed6b0
Rename client/config/config's ConsulConfig to ConsulAgentConfig
sean- May 21, 2016
1f7bfb3
Rename consulConfig to consulAgentConfig
sean- May 21, 2016
8e25647
Rename ConsulConfig to ConsulAgentConfig
sean- May 21, 2016
af72200
Distill config.Config.ConsulConfig down to config.Config.Consul
sean- May 22, 2016
7bf6af8
Fix copy pasta comment.
sean- May 22, 2016
092e07f
Create a `nomad/structs/config` to break an import cycle.
sean- May 22, 2016
d84d718
Rename client/consul/sync.ConsulService to client/consul/sync.Syncer
sean- May 22, 2016
f280c59
Rename client/consul/sync.PeriodicSync to Run
sean- May 23, 2016
7c6ad53
Rename NewConsulService to NewSyncer
sean- May 23, 2016
ffcd233
Rebalance Nomad client RPCs among different Nomad servers.
sean- May 23, 2016
f6e3587
Define a type for the PeriodicCallback handlers and ShutdownChannel
sean- May 24, 2016
d268dcb
Reduce all forms of ConsulConfig down to a single struct
sean- May 24, 2016
e07e77b
s/availble/runChecks/g
sean- May 24, 2016
8da18b6
Alpha sort config keys for Consul
sean- May 24, 2016
89d48c8
Revise Agent.syncAgentServicesWithConsul()'s interface
sean- May 24, 2016
c159e77
An Agent's consulSyncer is always not nil now.
sean- May 24, 2016
17927c8
Only poll Consul for servers when Nomad heartbeats begin to fail
sean- May 24, 2016
4e4f0a1
Remove unused code
sean- May 24, 2016
fc56585
index on f-dyn-server-list: bd38cb4 Remove unused code
sean- May 24, 2016
05bbbd2
Reconcile renamed structures during rebase
sean- May 24, 2016
90023c3
Rename RpcProxy's internal filename to match
sean- May 24, 2016
681a7d6
Remove testing for an API that doesn't exist any more.
sean- May 24, 2016
7125b85
Correct a mismerge
sean- May 26, 2016
71dde1b
Reconcile consul's address configuration section.
sean- May 26, 2016
bc94ce8
Rename `mergedNomadMap` to `mergedPrimaryMap`. Gratuitous, but more c…
sean- May 26, 2016
e6397da
Centralize the creation of a consul/api.Config struct.
sean- May 26, 2016
990a094
Use `rand.Int*n()` where appropriate
sean- May 3, 2016
07fa0c5
Use the correctly typed `rand.Int*` variant
sean- May 3, 2016
5e0d76f
Rename client/config/config's ConsulConfig to ConsulAgentConfig
sean- May 21, 2016
4d30c98
Rename consulConfig to consulAgentConfig
sean- May 21, 2016
6fae3db
Rename ConsulConfig to ConsulAgentConfig
sean- May 21, 2016
16b4e5c
Rebalance Nomad client RPCs among different Nomad servers.
sean- May 23, 2016
1fa43a0
Reduce all forms of ConsulConfig down to a single struct
sean- May 24, 2016
3728771
Remove unused code
sean- May 24, 2016
a703c8a
Rename RpcProxy's internal filename to match
sean- May 24, 2016
22bd2b5
Rename manager_test.go to rpc_proxy_test.go
sean- May 26, 2016
7cdf0ed
Rename the package from `client/rpc_proxy` to `client/rpcproxy`
sean- May 26, 2016
0a5fa55
Fix package name in comments
sean- May 27, 2016
cbed88a
Move struct member to reduce diff context
sean- May 27, 2016
1e3feae
Bump shuffle interval per suggestion from Alex
sean- May 27, 2016
1034e76
s/RpcVersion/RPCVersion/g
sean- May 27, 2016
916d93c
Bump the cluster test minimums to 10min.
sean- May 27, 2016
b293cf3
Fix typo in the comment to reflect the actual function name.
sean- May 27, 2016
ce8c03d
Advertise the server's RPC endpoint, not its HTTP endpoint.
sean- May 27, 2016
bc86e89
Register two services each for clients and servers, http and rpc.
sean- May 27, 2016
4d47eed
Teach Client to reuse an Agent's consulSyncer.
sean- May 27, 2016
9a93496
Consolidate all consul sync periodic go routines to handlers.
sean- May 27, 2016
bccf131
Invert check definition so the error is first
sean- May 27, 2016
2abd37c
Silence unused variable warning
sean- May 27, 2016
7034c50
Pass the datacenter name in the heartbeat
sean- May 27, 2016
57bb374
Fix unit tests
sean- May 27, 2016
3d22c22
Remove types.ShutdownChannel and replace with `chan struct{}`
sean- May 27, 2016
49266fa
Invert error handling logic
sean- May 27, 2016
9bd2882
Nuke trace-level logging in heartbeats
sean- May 27, 2016
ca0f2d9
Nuke a.consulConfig in favor of a.consul.Config
sean- May 27, 2016
9998573
Clear the backup server list when a Nomad heartbeat arives with servers
sean- May 27, 2016
bf4f031
Remove unused function.
sean- May 27, 2016
8a393d7
Improve language re: fingerprinting
sean- May 27, 2016
90f7eb4
When clearing the backup servers, set the length to zero.
sean- May 27, 2016
6cdc055
Change the constants used to match the struct definitions
sean- May 27, 2016
d6769f5
Create a weak decoder to parse time.Duration.
sean- May 27, 2016
6b8a9b4
Update the `nomad_server_service` default from `nomad-server` to just…
sean- May 27, 2016
b235759
Register the serf service with the Nomad server service.
sean- May 27, 2016
ab99e89
Reconcile, clean up, and centralize API version numbers (major and mi…
sean- May 28, 2016
6264a8e
Unused code wasn't as unused as I thought. Restore.
sean- May 28, 2016
c8bf53b
Fix a comment to be more correct
sean- May 28, 2016
b9f230b
Move client.DefaultConfig() to client/config.DefaultConfig()
sean- May 31, 2016
6785e76
Provide a default ConsulConfig for client/config.DefaultConfig()
sean- May 31, 2016
ed30876
Change the endpoint for `/v1/agent/servers` and fix tests.
sean- Jun 1, 2016
5b0a969
Fix the client/rpcproxy unit tests.
sean- Jun 1, 2016
a553956
Fix tests for client.TestAgent_ServerConfig
sean- Jun 1, 2016
37e7038
Fix config_parse_test to reflect that `consul.addr` does not exist.
sean- Jun 1, 2016
f037658
Add some trace-level logging for /v1/agent/servers when writing
sean- Jun 1, 2016
ac174db
Fix building tests that used `DefaultConfig()` but didn't pickup the …
sean- Jun 1, 2016
6908846
Add a quick set of client/rpcproxy.ServerEndpoint equality tests
sean- Jun 1, 2016
453a755
Pick the right `DefaultConfig` from the right package.
sean- Jun 1, 2016
f9862d4
Update godoc for newServer to reflect DNS and IP-based inputs
sean- Jun 1, 2016
329057e
Only actively test Consul when env `CONSUL_HTTP_ADDR` is set
sean- Jun 1, 2016
503c6a9
Fix test TestClientConfigCommand_UpdateServers()
sean- Jun 1, 2016
445386b
Remove unused constants
sean- Jun 1, 2016
41904f2
Clean up some docs and comments to be more accurate
sean- Jun 1, 2016
4300cb3
Remove unused variable
sean- Jun 1, 2016
ab434b4
Fix typo in comment
sean- Jun 1, 2016
66dc946
Don't clobber the default consul config in tests
sean- Jun 1, 2016
7538667
Push down the server list even on node registration and evaluation
sean- Jun 1, 2016
f15d84e
Use a monotonically incrementing number to create unique node names.
sean- Jun 1, 2016
73c1560
Rename `backupServerDeadline` to `consulPullHeartbeatDeadline`
sean- Jun 1, 2016
07799b6
Nuke the last of the explicit types in favor of using language idioms
sean- Jun 1, 2016
9de9634
Clean up various comments
sean- Jun 2, 2016
c8b2f7c
Flesh out the comment re: the client.rpcproxy.Run() task.
sean- Jun 2, 2016
8dd833f
Use the client configCopy and lock appropriately.
sean- Jun 2, 2016
8d478b9
Make the locking protocol more explicit in client.NewClient
sean- Jun 2, 2016
f0c6b70
Fix up the comments
sean- Jun 2, 2016
1ec5374
Rename rpcproxy.UpdateFromNodeUpdateResponse to RefreshServerLists
sean- Jun 2, 2016
cf052e5
Line wrap long line.
sean- Jun 2, 2016
8bbd763
Use client.getAllocRunners() where appropriate.
sean- Jun 2, 2016
998f285
Ensure that all accesses to Client.alloc are wrapped by allocLock.
sean- Jun 2, 2016
86b5d31
Move `const` block to the top of the file.
sean- Jun 2, 2016
bbf7348
Bump the default Consul client timeout from 500ms to 5s.
sean- Jun 2, 2016
273e8cf
Collapse rpcproxy_internal_test.go into rpcproxy_test.go
sean- Jun 2, 2016
68f7afc
Collapse server_endpoint_internal_test.go into server_endpoint_test.go
sean- Jun 2, 2016
4e03dc5
Remove named return parameters
sean- Jun 2, 2016
1352f7f
Change client/consul.NewSyncer() to accept a shutdown channel
sean- Jun 2, 2016
107fc1b
Rename createCheck() to createDelegatedCheck() for clarity
sean- Jun 7, 2016
acb3d58
Refine Nomad's Consul `port` handling.
sean- Jun 7, 2016
a208115
Rename structs.Services to structs.ConsulServices
sean- Jun 7, 2016
cf8beb7
Change the signature of the PeriodicCallback to return an error
sean- Jun 7, 2016
74e691c
Change the API signature of Syncer.SyncServices().
sean- Jun 7, 2016
e858928
Rename Syncer.SetServiceIdentifier to SetServiceRegPrefix()
sean- Jun 7, 2016
57c2c81
Move package client/consul/sync to command/agent/consul.
sean- Jun 8, 2016
b6a2ec2
Remove Syncer.registerService()
sean- Jun 8, 2016
54838b9
Rename structs.Task's `Service` attribute to `ConsulService`
sean- Jun 8, 2016
3052e74
Move the start of the UniversalExecutor's consulSyncer to initialize …
sean- Jun 8, 2016
7ad5cd5
Begin leveraging the Agent-level consul.Syncer
sean- Jun 8, 2016
802a8c4
Rename command/agent/consul/sync.go to syncer.go
sean- Jun 8, 2016
3e95ca6
Per-comment, remove structs.Allocation's Services attribute.
sean- Jun 9, 2016
a4f605a
Initialize Consul for the Nomad Agent in a more uniform way.
sean- Jun 10, 2016
8be79cf
Add "Service Groups" to the Syncer.
sean- Jun 10, 2016
d87c697
Update Syncer.Run() to call SyncServices().
sean- Jun 10, 2016
5913acf
Rename runChecks to consulAvailable
sean- Jun 10, 2016
d1ab21f
Generate and sync Consul ServiceIDs consistently
sean- Jun 10, 2016
d2dcf27
Populate the RPC Proxy's server list if heartbeat did not include a l…
sean- Jun 10, 2016
19f765e
Sync services with Consul by comparing the AgentServiceReg w/ ConsulS…
sean- Jun 10, 2016
7d060c2
Sync checks with Consul by comparing the AgentCheckReg w/ ConsulService
sean- Jun 10, 2016
9b984d0
Update the structure of ConsulService to match reality.
sean- Jun 10, 2016
27f6ffd
On Syncer Shutdown, remove all services that match a Syncer's prefix.
sean- Jun 10, 2016
d810590
Create a consulContext using a client's consul config.
sean- Jun 10, 2016
f7b2949
Properly cover Syncer attributes with the registryLock.
sean- Jun 10, 2016
05e713c
Skip nil check for agent's consulSyncer is always not nil
sean- Jun 10, 2016
69aa8e8
Don't spam the consul if Consul is not available.
sean- Jun 10, 2016
c426b85
Hand wave over the syncer tests atm, these will be fixed shortly.
sean- Jun 10, 2016
8f83c2e
Move RPCProxy.New() adjacent to its struct definition
sean- Jun 10, 2016
4b5310e
Properly guard consulPullHeartbeatDeadline behind heartbeatLock
sean- Jun 10, 2016
5757e50
Only return the Client's server addresses, never mix-in server peers
sean- Jun 10, 2016
a423f07
Stash client and server registration behind consul.auto_register
sean- Jun 10, 2016
a55d3f1
Rename `updateNodeUpdateResponse` to `constructNodeServerInfoResponse`
sean- Jun 10, 2016
8c8f33d
Always pass in a snapshot before calling constructNodeServerInfoRespo…
sean- Jun 10, 2016
b57f7c9
Style nit: remove `var` block
sean- Jun 10, 2016
709b6bd
Fold RaftPeers() into its only call site now
sean- Jun 10, 2016
a7047bb
Prefix all log entries in client/rpcproxy with client.rpcproxy
sean- Jun 10, 2016
f07c910
Formatting nit: remove brackets
sean- Jun 10, 2016
dbdebca
golint(1) police
sean- Jun 10, 2016
ecd84f4
Nomad does not use Serf at the client level. Use a hard lock.
sean- Jun 10, 2016
83e3df0
Rename listLock to activatedListLock
sean- Jun 10, 2016
2ac7ecb
Remove useless statement
sean- Jun 10, 2016
61a36a6
Commit miss, bump to 5s
sean- Jun 10, 2016
aff951c
Always create a consul.Syncer. Use a default Consul Config if necessary.
sean- Jun 10, 2016
91582dc
Always create a consul.Syncer. Use a default Consul Config if necessary.
sean- Jun 10, 2016
a0902c3
Prevent duplicate servers being added in AddPrimaryServer.
sean- Jun 10, 2016
4e543b6
Restore old behavior and have AddPrimaryServer() return a pointer to …
sean- Jun 10, 2016
4826728
Fix another unit test not expecting ServiceID
sean- Jun 10, 2016
5ffd970
Expose rpcproxy's `ServerEndpoint()` constructor, `newServer()` as `N…
sean- Jun 11, 2016
6819f2b
Query for the Nomad service across multiple Consul datacenters.
sean- Jun 11, 2016
5331ea7
goling(1) compliance pass (e.g. Rpc* -> RPC)
sean- Jun 11, 2016
2879c33
Perform a nil-check for Executor's consulServices.
sean- Jun 11, 2016
183b51d
Remove default values and use nil for the executor. Much better.
sean- Jun 11, 2016
a8b53b0
Prefer `%+q` over `%q` in log messages.
sean- Jun 11, 2016
8d0b798
Walk the DCs from nearest to most remote, no limit on the search.
sean- Jun 11, 2016
006d1a3
Walk the DCs from nearest to most remote.
sean- Jun 11, 2016
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 0 additions & 33 deletions api/agent_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -124,39 +124,6 @@ func TestAgent_ForceLeave(t *testing.T) {
// TODO: test force-leave on an existing node
}

func TestAgent_SetServers(t *testing.T) {
c, s := makeClient(t, nil, func(c *testutil.TestServerConfig) {
c.Client.Enabled = true
c.Server.BootstrapExpect = 0
})
defer s.Stop()
a := c.Agent()

// Attempting to set an empty list errors
err := a.SetServers([]string{})
if err == nil {
t.Fatalf("expected error, got nothing")
}

// Setting a valid list works
err = a.SetServers([]string{"foo", "bar"})
if err != nil {
t.Fatalf("err: %s", err)
}

// Returns the proper list of servers
out, err := a.Servers()
if err != nil {
t.Fatalf("err: %s", err)
}
if n := len(out); n != 2 {
t.Fatalf("expected 2 servers, got: %d", n)
}
if out[0] != "foo:4647" || out[1] != "bar:4647" {
t.Fatalf("bad server list: %v", out)
}
}

func (a *AgentMember) String() string {
return "{Name: " + a.Name + " Region: " + a.Tags["region"] + " DC: " + a.Tags["dc"] + "}"
}
Expand Down
3 changes: 1 addition & 2 deletions api/nodes_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ func TestNodes_PrefixList(t *testing.T) {
var err error

// Get the node ID
var nodeID, dc string
var nodeID string
testutil.WaitForResult(func() (bool, error) {
out, _, err := nodes.List(nil)
if err != nil {
Expand All @@ -60,7 +60,6 @@ func TestNodes_PrefixList(t *testing.T) {
return false, fmt.Errorf("expected 1 node, got: %d", n)
}
nodeID = out[0].ID
dc = out[0].Datacenter
return true, nil
}, func(err error) {
t.Fatalf("err: %s", err)
Expand Down
3 changes: 2 additions & 1 deletion client/alloc_runner_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ import (
"github.com/hashicorp/nomad/nomad/structs"
"github.com/hashicorp/nomad/testutil"

"github.com/hashicorp/nomad/client/config"
ctestutil "github.com/hashicorp/nomad/client/testutil"
)

Expand All @@ -25,7 +26,7 @@ func (m *MockAllocStateUpdater) Update(alloc *structs.Allocation) {

func testAllocRunner(restarts bool) (*MockAllocStateUpdater, *AllocRunner) {
logger := testLogger()
conf := DefaultConfig()
conf := config.DefaultConfig()
conf.StateDir = os.TempDir()
conf.AllocDir = os.TempDir()
upd := &MockAllocStateUpdater{}
Expand Down
Loading