Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor etcd config as discovery section with convenience types #2843

Merged
merged 51 commits into from
Nov 9, 2020
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
51 commits
Select commit Hold shift + click to select a range
238de0b
[dbnode] [coordinator] Refactor config to use defaults wherever possible
robskillington Nov 5, 2020
fdc1021
Refactor pooling policy
robskillington Nov 5, 2020
7051a16
By default remove the filesystem too
robskillington Nov 5, 2020
6b10865
default coordinator section in dbnode config
rallen090 Nov 5, 2020
10ee091
default coordinator section in dbnode config 2
rallen090 Nov 5, 2020
dcba7d9
merged in changes
rallen090 Nov 5, 2020
9252528
build fixes 1
rallen090 Nov 5, 2020
943afd2
build fixes 2
rallen090 Nov 5, 2020
4021b30
lint
rallen090 Nov 5, 2020
6382b2b
lint 2
rallen090 Nov 5, 2020
b38c754
lint
rallen090 Nov 5, 2020
496c7ce
lint removing exhaustivestruct
rallen090 Nov 5, 2020
89dc079
update writeNewSeriesLimitPerSecond to be under limits:
rallen090 Nov 5, 2020
4394e27
discovery environment config
rallen090 Nov 5, 2020
3bf7aa1
tests for discovery config
rallen090 Nov 6, 2020
b54de40
update configs w/ new discovery format
rallen090 Nov 6, 2020
519c53f
test fixing 1
rallen090 Nov 6, 2020
4116b1e
tidy
rallen090 Nov 6, 2020
ec2e96b
test fixing 2
rallen090 Nov 6, 2020
47442d3
use envcfg from discoverycfg
rallen090 Nov 6, 2020
048e5dd
test fixes 1
rallen090 Nov 6, 2020
4022019
test fixes 2
rallen090 Nov 6, 2020
3dc1969
test fixes 2
rallen090 Nov 6, 2020
e44cd1e
test fixes 3
rallen090 Nov 6, 2020
896cd3a
test fixes 3
rallen090 Nov 6, 2020
803b8b5
test fixes 4
rallen090 Nov 6, 2020
f9b8326
lint
rallen090 Nov 6, 2020
af93c26
lint again
rallen090 Nov 6, 2020
4cbf248
more test fixes 1
rallen090 Nov 6, 2020
623cf52
more test fixes 2
rallen090 Nov 6, 2020
5b934fe
lint more
rallen090 Nov 6, 2020
b6ca23a
more test fixing
rallen090 Nov 6, 2020
64bba79
remove lint we do not want
rallen090 Nov 6, 2020
38efe1d
more lint
rallen090 Nov 6, 2020
0ad5356
more test fix
rallen090 Nov 6, 2020
5294afd
more test fix 2
rallen090 Nov 6, 2020
26f21fd
more test fix 3
rallen090 Nov 6, 2020
901bfe3
more test fix 4
rallen090 Nov 6, 2020
83be2c5
mod tidy
rallen090 Nov 6, 2020
457c15d
Merge branch 'master' into ra/etcd-config-as-discovery-section
rallen090 Nov 6, 2020
4df8a78
Merge branch 'master' into refactor-config-use-defaults-wherever-poss…
rallen090 Nov 6, 2020
68f7f48
Merge branch 'master' into refactor-config-use-defaults-wherever-poss…
robskillington Nov 7, 2020
e97464f
Minor tweaks
robskillington Nov 7, 2020
95151d7
Merge branch 'refactor-config-use-defaults-wherever-possible' into ra…
robskillington Nov 7, 2020
692a3f2
Merge branch 'master' into ra/etcd-config-as-discovery-section
robskillington Nov 9, 2020
26759b1
Merge branch 'master' into ra/etcd-config-as-discovery-section
robskillington Nov 9, 2020
c2e832b
Update configs to match defaults being available
robskillington Nov 9, 2020
28e7f7d
Merge branch 'ra/etcd-config-as-discovery-section' of github.com:m3db…
robskillington Nov 9, 2020
2e0571c
Add nlreturn to disabled linters
robskillington Nov 9, 2020
65816d3
Merge branch 'master' into ra/etcd-config-as-discovery-section
robskillington Nov 9, 2020
7bfe97d
Merge branch 'master' into ra/etcd-config-as-discovery-section
robskillington Nov 9, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 13 additions & 12 deletions kube/bundle.yaml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

25 changes: 13 additions & 12 deletions kube/m3dbnode-configmap.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -66,15 +66,16 @@ data:
filesystem:
filePathPrefix: /var/lib/m3db

config:
service:
env: default_env
zone: embedded
service: m3db
cacheDir: /var/lib/m3kv
etcdClusters:
- zone: embedded
endpoints:
- http://etcd-0.etcd:2379
- http://etcd-1.etcd:2379
- http://etcd-2.etcd:2379
discovery:
config:
service:
env: default_env
zone: embedded
service: m3db
cacheDir: /var/lib/m3kv
etcdClusters:
- zone: embedded
endpoints:
- http://etcd-0.etcd:2379
- http://etcd-1.etcd:2379
- http://etcd-2.etcd:2379
29 changes: 15 additions & 14 deletions scripts/development/m3_stack/m3dbnode.yml
Original file line number Diff line number Diff line change
Expand Up @@ -54,20 +54,21 @@ db:
filesystem:
filePathPrefix: /var/lib/m3db

config:
service:
env: default_env
zone: embedded
service: m3db
cacheDir: /var/lib/m3kv
etcdClusters:
- zone: embedded
endpoints:
- m3db_seed:2379
seedNodes:
initialCluster:
- hostID: m3db_seed
endpoint: http://m3db_seed:2380
discovery:
config:
service:
env: default_env
zone: embedded
service: m3db
cacheDir: /var/lib/m3kv
etcdClusters:
- zone: embedded
endpoints:
- m3db_seed:2379
seedNodes:
initialCluster:
- hostID: m3db_seed
endpoint: http://m3db_seed:2380

# proto:
# schemaFilePath: /etc/m3dbnode/schema.proto
Original file line number Diff line number Diff line change
Expand Up @@ -62,13 +62,14 @@ db:
filesystem:
filePathPrefix: /var/lib/m3db

config:
service:
env: foo-namespace/foo-cluster
zone: bar-zone
service: m3db
cacheDir: /var/lib/m3kv
etcdClusters:
- zone: bar-zone
endpoints:
- etcd01:2379
discovery:
config:
service:
env: foo-namespace/foo-cluster
zone: bar-zone
service: m3db
cacheDir: /var/lib/m3kv
etcdClusters:
- zone: bar-zone
endpoints:
- etcd01:2379
Original file line number Diff line number Diff line change
Expand Up @@ -75,17 +75,18 @@ db:
filesystem:
filePathPrefix: /var/lib/m3db

config:
service:
env: default_env
zone: embedded
service: m3db
cacheDir: /var/lib/m3kv
etcdClusters:
- zone: embedded
endpoints:
- cluster_a_dbnode01:2379
seedNodes:
initialCluster:
- hostID: cluster_a_m3db_local_1
endpoint: http://cluster_a_dbnode01:2380
discovery:
config:
service:
env: default_env
zone: embedded
service: m3db
cacheDir: /var/lib/m3kv
etcdClusters:
- zone: embedded
endpoints:
- cluster_a_dbnode01:2379
seedNodes:
initialCluster:
- hostID: cluster_a_m3db_local_1
endpoint: http://cluster_a_dbnode01:2380
Original file line number Diff line number Diff line change
Expand Up @@ -54,17 +54,18 @@ db:
filesystem:
filePathPrefix: /var/lib/m3db

config:
service:
env: default_env
zone: embedded
service: m3db
cacheDir: /var/lib/m3kv
etcdClusters:
- zone: embedded
endpoints:
- cluster_b_dbnode01:2379
seedNodes:
initialCluster:
- hostID: cluster_b_m3db_local_1
endpoint: http://cluster_b_dbnode01:2380
discovery:
config:
service:
env: default_env
zone: embedded
service: m3db
cacheDir: /var/lib/m3kv
etcdClusters:
- zone: embedded
endpoints:
- cluster_b_dbnode01:2379
seedNodes:
initialCluster:
- hostID: cluster_b_m3db_local_1
endpoint: http://cluster_b_dbnode01:2380
29 changes: 15 additions & 14 deletions scripts/docker-integration-tests/repair/m3dbnode.yml
Original file line number Diff line number Diff line change
Expand Up @@ -54,20 +54,21 @@ db:
filesystem:
filePathPrefix: /var/lib/m3db

config:
service:
env: default_env
zone: embedded
service: m3db
cacheDir: /var/lib/m3kv
etcdClusters:
- zone: embedded
endpoints:
- dbnode01:2379
seedNodes:
initialCluster:
- hostID: m3db_local_1
endpoint: http://dbnode01:2380
discovery:
config:
service:
env: default_env
zone: embedded
service: m3db
cacheDir: /var/lib/m3kv
etcdClusters:
- zone: embedded
endpoints:
- dbnode01:2379
seedNodes:
initialCluster:
- hostID: m3db_local_1
endpoint: http://dbnode01:2380

# Enable repairs.
repair:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -54,20 +54,21 @@ db:
filesystem:
filePathPrefix: /var/lib/m3db

config:
service:
env: default_env
zone: embedded
service: m3db
cacheDir: /var/lib/m3kv
etcdClusters:
- zone: embedded
endpoints:
- cluster_a_dbnode01:2379
seedNodes:
initialCluster:
- hostID: cluster_a_m3db_local_1
endpoint: http://cluster_a_dbnode01:2380
discovery:
config:
service:
env: default_env
zone: embedded
service: m3db
cacheDir: /var/lib/m3kv
etcdClusters:
- zone: embedded
endpoints:
- cluster_a_dbnode01:2379
seedNodes:
initialCluster:
- hostID: cluster_a_m3db_local_1
endpoint: http://cluster_a_dbnode01:2380

# Enable repairs (within cluster a).
repair:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -54,20 +54,21 @@ db:
filesystem:
filePathPrefix: /var/lib/m3db

config:
service:
env: default_env
zone: embedded
service: m3db
cacheDir: /var/lib/m3kv
etcdClusters:
- zone: embedded
endpoints:
- cluster_b_dbnode01:2379
seedNodes:
initialCluster:
- hostID: cluster_b_m3db_local_1
endpoint: http://cluster_b_dbnode01:2380
discovery:
config:
service:
env: default_env
zone: embedded
service: m3db
cacheDir: /var/lib/m3kv
etcdClusters:
- zone: embedded
endpoints:
- cluster_b_dbnode01:2379
seedNodes:
initialCluster:
- hostID: cluster_b_m3db_local_1
endpoint: http://cluster_b_dbnode01:2380

# Enable repairs (within cluster b).
repair:
Expand Down
29 changes: 15 additions & 14 deletions scripts/docker-integration-tests/replication/m3dbnode-cluster-a.yml
Original file line number Diff line number Diff line change
Expand Up @@ -54,20 +54,21 @@ db:
filesystem:
filePathPrefix: /var/lib/m3db

config:
service:
env: default_env
zone: embedded
service: m3db
cacheDir: /var/lib/m3kv
etcdClusters:
- zone: embedded
endpoints:
- cluster_a_dbnode01:2379
seedNodes:
initialCluster:
- hostID: cluster_a_m3db_local_1
endpoint: http://cluster_a_dbnode01:2380
discovery:
config:
service:
env: default_env
zone: embedded
service: m3db
cacheDir: /var/lib/m3kv
etcdClusters:
- zone: embedded
endpoints:
- cluster_a_dbnode01:2379
seedNodes:
initialCluster:
- hostID: cluster_a_m3db_local_1
endpoint: http://cluster_a_dbnode01:2380

# Disable repairs (within cluster a).
repair:
Expand Down
29 changes: 15 additions & 14 deletions scripts/docker-integration-tests/replication/m3dbnode-cluster-b.yml
Original file line number Diff line number Diff line change
Expand Up @@ -54,20 +54,21 @@ db:
filesystem:
filePathPrefix: /var/lib/m3db

config:
service:
env: default_env
zone: embedded
service: m3db
cacheDir: /var/lib/m3kv
etcdClusters:
- zone: embedded
endpoints:
- cluster_b_dbnode01:2379
seedNodes:
initialCluster:
- hostID: cluster_b_m3db_local_1
endpoint: http://cluster_b_dbnode01:2380
discovery:
config:
service:
env: default_env
zone: embedded
service: m3db
cacheDir: /var/lib/m3kv
etcdClusters:
- zone: embedded
endpoints:
- cluster_b_dbnode01:2379
seedNodes:
initialCluster:
- hostID: cluster_b_m3db_local_1
endpoint: http://cluster_b_dbnode01:2380

# Disable repairs (within cluster b).
repair:
Expand Down
5 changes: 3 additions & 2 deletions src/cmd/services/m3dbnode/config/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ import (

coordinatorcfg "github.com/m3db/m3/src/cmd/services/m3query/config"
"github.com/m3db/m3/src/dbnode/client"
"github.com/m3db/m3/src/dbnode/discovery"
"github.com/m3db/m3/src/dbnode/environment"
"github.com/m3db/m3/src/dbnode/storage/series"
"github.com/m3db/m3/src/x/config/hostid"
Expand Down Expand Up @@ -140,8 +141,8 @@ type DBConfiguration struct {
// The pooling policy.
PoolingPolicy PoolingPolicy `yaml:"pooling"`

// The environment (static or dynamic) configuration.
EnvironmentConfig environment.Configuration `yaml:"config"`
// The discovery configuration.
DiscoveryConfig discovery.Configuration `yaml:"discovery"`

// The configuration for hashing
Hashing HashingConfiguration `yaml:"hashing"`
Expand Down
Loading