ETCD_CONFIG_FILE env variable is ignored #10747

weavage · 2019-05-21T17:42:04Z

etcd v3.3

etcd appears to recognize and then promptly ignore the ETCD_CONFIG_FILE environment variable. Passing the --config-file flag seems to work properly. As an example, create a super simplistic config file at /etc/etcd/etcd.conf.yaml containing only
data-dir: /var/lib/etcd

Setting ETCD_CONFIG_FILE=/etc/etcd/etcd.conf.yaml and trying to start etcd returns something similar to the following:

2019-05-21 17:32:51.695776 I | pkg/flags: recognized and used environment variable ETCD_CONFIG_FILE=/etc/etcd/etcd.conf.yaml  
2019-05-21 17:32:51.695920 I | etcdmain: etcd Version: 3.3.13  
2019-05-21 17:32:51.695929 I | etcdmain: Git SHA: 98d3084  
2019-05-21 17:32:51.695936 I | etcdmain: Go Version: go1.10.8  
2019-05-21 17:32:51.695942 I | etcdmain: Go OS/Arch: linux/amd64  
2019-05-21 17:32:51.695949 I | etcdmain: setting maximum number of CPUs to 40, total number of available CPUs is 40  
2019-05-21 17:32:51.695960 W | etcdmain: no data-dir provided, using default data-dir ./default.etcd  
2019-05-21 17:32:51.696439 I | embed: listening for peers on http://localhost:2380  
2019-05-21 17:32:51.696555 I | embed: listening for client requests on localhost:2379  
2019-05-21 17:32:51.696650 C | etcdmain: cannot access data directory: mkdir default.etcd: permission denied

However, starting etcd with the --config-file /etc/etcd/etcd.conf.yaml works properly:

2019-05-21 17:33:57.303742 I | etcdmain: Loading server configuration from "/etc/etcd/etcd.conf.yaml"
2019-05-21 17:33:57.305112 I | etcdmain: etcd Version: 3.3.13
2019-05-21 17:33:57.305124 I | etcdmain: Git SHA: 98d3084
2019-05-21 17:33:57.305128 I | etcdmain: Go Version: go1.10.8
2019-05-21 17:33:57.305131 I | etcdmain: Go OS/Arch: linux/amd64
2019-05-21 17:33:57.305136 I | etcdmain: setting maximum number of CPUs to 40, total number of available CPUs is 40
2019-05-21 17:33:57.306523 N | etcdmain: the server is already initialized as member before, starting as etcd member...
2019-05-21 17:33:57.306815 I | embed: listening for peers on http://localhost:2380
2019-05-21 17:33:57.306916 I | embed: listening for client requests on localhost:2379
2019-05-21 17:33:57.325706 I | etcdserver: name = default
2019-05-21 17:33:57.325736 I | etcdserver: data dir = /var/lib/etcd
2019-05-21 17:33:57.325751 I | etcdserver: member dir = /var/lib/etcd/member
2019-05-21 17:33:57.325760 I | etcdserver: heartbeat = 100ms
2019-05-21 17:33:57.325772 I | etcdserver: election = 1000ms
2019-05-21 17:33:57.325782 I | etcdserver: snapshot count = 100000
2019-05-21 17:33:57.325803 I | etcdserver: advertise client URLs = http://localhost:2379
2019-05-21 17:33:57.335837 I | etcdserver: restarting member 8e9e05c52164694d in cluster cdf818194e3a8c32 at commit index 4
2019-05-21 17:33:57.335910 I | raft: 8e9e05c52164694d became follower at term 2
2019-05-21 17:33:57.335955 I | raft: newRaft 8e9e05c52164694d [peers: [], term: 2, commit: 4, applied: 0, lastindex: 4, lastterm: 2]
2019-05-21 17:33:57.340926 W | auth: simple token is not cryptographically signed
2019-05-21 17:33:57.342299 I | etcdserver: starting server... [version: 3.3.13, cluster version: to_be_decided]
2019-05-21 17:33:57.344222 I | etcdserver/membership: added member 8e9e05c52164694d [http://localhost:2380] to cluster cdf818194e3a8c32
2019-05-21 17:33:57.344395 N | etcdserver/membership: set the initial cluster version to 3.3
2019-05-21 17:33:57.344492 I | etcdserver/api: enabled capabilities for version 3.3
2019-05-21 17:33:58.738818 I | raft: 8e9e05c52164694d is starting a new election at term 2
2019-05-21 17:33:58.738866 I | raft: 8e9e05c52164694d became candidate at term 3
2019-05-21 17:33:58.738916 I | raft: 8e9e05c52164694d received MsgVoteResp from 8e9e05c52164694d at term 3
2019-05-21 17:33:58.738945 I | raft: 8e9e05c52164694d became leader at term 3
2019-05-21 17:33:58.738960 I | raft: raft.node: 8e9e05c52164694d elected leader 8e9e05c52164694d at term 3
2019-05-21 17:33:58.741399 I | etcdserver: published {Name:default ClientURLs:[http://localhost:2379]} to cluster cdf818194e3a8c32
2019-05-21 17:33:58.743342 I | embed: ready to serve client requests
2019-05-21 17:33:58.744298 N | embed: serving insecure client requests on 127.0.0.1:2379, this is strongly discouraged!

The text was updated successfully, but these errors were encountered:

jingyih · 2019-05-21T18:07:25Z

I just did a quick look at the code, ETCD_CONFIG_FILE is not working as suggested. For now please use command line flag to provide config file.

weavage · 2019-05-21T18:49:40Z

Thanks for confirming! I tried hunting down what was causing it, but i don't speak 'Go' and I'm not really familiar with how all the bits interact.

jingyih · 2019-05-21T20:30:13Z

FYI, the current code will first check if there is config file provided via command line flag --config-file.

If there is a config file provided, use it and ignore the command line flags;
If there is not, use config from command line flags and also read environment variables.

So the current code does not support ETCD_CONFIG_FILE, as the code already decided to use command line flags (instead of using config file) before it recognizes ETCD_CONFIG_FILE.

Ref:

etcd/etcdmain/config.go

Lines 286 to 298 in e9f310a

    
           if cfg.configFile != "" { 
        
           	err = cfg.configFromFile(cfg.configFile) 
        
           	if lg := cfg.ec.GetLogger(); lg != nil { 
        
           		lg.Info( 
        
           			"loaded server configuration, other configuration command line flags and environment variables will be ignored if provided", 
        
           			zap.String("path", cfg.configFile), 
        
           		) 
        
           	} else { 
        
           		plog.Infof("Loading server configuration from %q. Other configuration command line flags and environment variables will be ignored if provided.", cfg.configFile) 
        
           	} 
        
           } else { 
        
           	err = cfg.configFromCmdLine() 
        
           }

etcdmain: fix ignoring of ETCD_CONFIG_FILE env variable

EugenKon · 2023-08-10T18:03:14Z

I have same issue. I can read from log files that ETCD_CONFIG_FILE was recognized and used, but none values are taken into account. This very confusing =( Here, in my case I did not provide any command line options and service was started via systemd

Below is my etcd version, log output and configuration file. Please note as heartbeat-interval value 500 is ignored:

$ etcd --version
etcd Version: 3.3.25
Git SHA: Not provided (use ./build instead of go build)
Go Version: go1.18.1
Go OS/Arch: linux/amd64

$ cat /lib/systemd/system/etcd.service 
[Unit]
Description=etcd - highly-available key value store
Documentation=https://etcd.io/docs
Documentation=man:etcd
After=network.target
Wants=network-online.target

[Service]
Environment=DAEMON_ARGS=
Environment=ETCD_NAME=%H
Environment=ETCD_DATA_DIR=/var/lib/etcd/default
EnvironmentFile=-/etc/default/%p
Type=notify
User=etcd
PermissionsStartOnly=true
#ExecStart=/bin/sh -c "GOMAXPROCS=$(nproc) /usr/bin/etcd $DAEMON_ARGS"
ExecStart=/usr/bin/etcd $DAEMON_ARGS
Restart=on-abnormal
#RestartSec=10s
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
Alias=etcd2.service

$ cat /etc/systemd/system/etcd.service.d/override.conf
[Service]
RestartSec=10s
Environment=ETCD_NAME=www
Environment=ETCD_CONFIG_FILE=/opt/project/etcd.yaml

$ journalctl -xeu etcd.service
░░ Subject: Unit succeeded
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ The unit etcd.service has successfully entered the 'dead' state.
Aug 10 17:52:24 ip-172-31-4-62 systemd[1]: Stopped etcd - highly-available key value store.
░░ Subject: A stop job for unit etcd.service has finished
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ A stop job for unit etcd.service has finished.
░░ 
░░ The job identifier is 3371975 and the job result is done.
Aug 10 17:52:24 ip-172-31-4-62 systemd[1]: Starting etcd - highly-available key value store...
░░ Subject: A start job for unit etcd.service has begun execution
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ A start job for unit etcd.service has begun execution.
░░ 
░░ The job identifier is 3371975.
Aug 10 17:52:24 ip-172-31-4-62 etcd[2304350]: recognized and used environment variable ETCD_CONFIG_FILE=/opt/project/etcd.yaml
Aug 10 17:52:24 ip-172-31-4-62 etcd[2304350]: recognized and used environment variable ETCD_DATA_DIR=/var/lib/etcd/default
Aug 10 17:52:24 ip-172-31-4-62 etcd[2304350]: recognized and used environment variable ETCD_NAME=www
Aug 10 17:52:24 ip-172-31-4-62 etcd[2304350]: etcd Version: 3.3.25
Aug 10 17:52:24 ip-172-31-4-62 etcd[2304350]: Git SHA: Not provided (use ./build instead of go build)
Aug 10 17:52:24 ip-172-31-4-62 etcd[2304350]: Go Version: go1.18.1
Aug 10 17:52:24 ip-172-31-4-62 etcd[2304350]: Go OS/Arch: linux/amd64
Aug 10 17:52:24 ip-172-31-4-62 etcd[2304350]: setting maximum number of CPUs to 4, total number of available CPUs is 4
Aug 10 17:52:24 ip-172-31-4-62 etcd[2304350]: the server is already initialized as member before, starting as etcd member...
Aug 10 17:52:24 ip-172-31-4-62 etcd[2304350]: listening for peers on http://localhost:2380
Aug 10 17:52:24 ip-172-31-4-62 etcd[2304350]: listening for client requests on localhost:2379
Aug 10 17:52:24 ip-172-31-4-62 etcd[2304350]: name = www
Aug 10 17:52:24 ip-172-31-4-62 etcd[2304350]: data dir = /var/lib/etcd/default
Aug 10 17:52:24 ip-172-31-4-62 etcd[2304350]: member dir = /var/lib/etcd/default/member
Aug 10 17:52:24 ip-172-31-4-62 etcd[2304350]: heartbeat = 100ms
Aug 10 17:52:24 ip-172-31-4-62 etcd[2304350]: election = 1000ms
Aug 10 17:52:24 ip-172-31-4-62 etcd[2304350]: snapshot count = 100000
Aug 10 17:52:24 ip-172-31-4-62 etcd[2304350]: advertise client URLs = http://localhost:2379
Aug 10 17:52:24 ip-172-31-4-62 etcd[2304350]: restarting member 8e9e05c52164694d in cluster cdf818194e3a8c32 at commit index 38000
Aug 10 17:52:24 ip-172-31-4-62 etcd[2304350]: 8e9e05c52164694d became follower at term 13
Aug 10 17:52:24 ip-172-31-4-62 etcd[2304350]: newRaft 8e9e05c52164694d [peers: [], term: 13, commit: 38000, applied: 0, lastindex: 38000, lastterm: 13]
Aug 10 17:52:24 ip-172-31-4-62 etcd[2304350]: simple token is not cryptographically signed
Aug 10 17:52:24 ip-172-31-4-62 etcd[2304350]: starting server... [version: 3.3.25, cluster version: to_be_decided]
Aug 10 17:52:24 ip-172-31-4-62 etcd[2304350]: added member 8e9e05c52164694d [http://localhost:2380] to cluster cdf818194e3a8c32
Aug 10 17:52:24 ip-172-31-4-62 etcd[2304350]: set the initial cluster version to 3.3
Aug 10 17:52:24 ip-172-31-4-62 etcd[2304350]: enabled capabilities for version 3.3
Aug 10 17:52:27 ip-172-31-4-62 etcd[2304350]: 8e9e05c52164694d is starting a new election at term 13
Aug 10 17:52:27 ip-172-31-4-62 etcd[2304350]: 8e9e05c52164694d became candidate at term 14
Aug 10 17:52:27 ip-172-31-4-62 etcd[2304350]: 8e9e05c52164694d received MsgVoteResp from 8e9e05c52164694d at term 14
Aug 10 17:52:27 ip-172-31-4-62 etcd[2304350]: 8e9e05c52164694d became leader at term 14
Aug 10 17:52:27 ip-172-31-4-62 etcd[2304350]: raft.node: 8e9e05c52164694d elected leader 8e9e05c52164694d at term 14
Aug 10 17:52:27 ip-172-31-4-62 etcd[2304350]: published {Name:www ClientURLs:[http://localhost:2379]} to cluster cdf818194e3a8c32
Aug 10 17:52:27 ip-172-31-4-62 etcd[2304350]: ready to serve client requests
Aug 10 17:52:27 ip-172-31-4-62 systemd[1]: Started etcd - highly-available key value store.
░░ Subject: A start job for unit etcd.service has finished successfully
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ A start job for unit etcd.service has finished successfully.
░░ 
░░ The job identifier is 3371975.
Aug 10 17:52:27 ip-172-31-4-62 etcd[2304350]: serving insecure client requests on 127.0.0.1:2379, this is strongly discouraged!

$ cat /opt/project/etcd.yaml
name: 'see env var ETCD_NAME'
heartbeat-interval: 500
election-timeout: 5000
initial-cluster: www=http://172.31.4.62:2380,www=http://172.31.4.62:7001
initial-cluster-state: new
initial-cluster-token: platform-exp
advertise-client-urls: http://172.31.4.62:2379,http://172.31.4.62:4001
initial-advertise-peer-urls: http://172.31.4.62:2380,http://172.31.4.62:7001
listen-client-urls: http://0.0.0.0:2379,http://0.0.0.0:4001
listen-peer-urls: http://172.31.4.62:2380,http://172.31.4.62:7001

jmhbnz · 2023-08-10T19:09:55Z

@EugenKon this issue was closed four years ago. The etcd version 3.3 is no longer supported by the project. There have been countless security, performance and other fixes made in etcd 3.4 and 3.5. Please update to a supported release and this issue should be resolved.

EugenKon · 2023-08-10T19:52:42Z

@jmhbnz I will try, but I did not found an easy way to install new version. apt-get installed this one.

jmhbnz · 2023-08-10T22:06:10Z

@jmhbnz I will try, but I did not found an easy way to install new version. apt-get installed this one.

The later releases are available as pre built binaries that just need to be downloaded and unpacked, please refer to https://etcd.io/docs/v3.5/install/#install-pre-built-binaries

jingyih added the type/bug label May 21, 2019

AstolfoKawaii mentioned this issue May 25, 2019

etcdmain: fix ignoring of ETCD_CONFIG_FILE env variable #10762

Merged

xiang90 closed this as completed in #10762 Jun 8, 2019

xiang90 referenced this issue Jun 8, 2019

Merge pull request #10762 from FrozenAndrey/fix#10747

ea70731

etcdmain: fix ignoring of ETCD_CONFIG_FILE env variable

kallisti5 mentioned this issue Jul 8, 2019

Merge "etcdmain: fix ignoring of ETCD_CONFIG_FILE env variable" into 3.3 branch? #10878

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ETCD_CONFIG_FILE env variable is ignored #10747

ETCD_CONFIG_FILE env variable is ignored #10747

weavage commented May 21, 2019

jingyih commented May 21, 2019

weavage commented May 21, 2019

jingyih commented May 21, 2019

EugenKon commented Aug 10, 2023 •

edited

Loading

jmhbnz commented Aug 10, 2023

EugenKon commented Aug 10, 2023

jmhbnz commented Aug 10, 2023

ETCD_CONFIG_FILE env variable is ignored #10747

ETCD_CONFIG_FILE env variable is ignored #10747

Comments

weavage commented May 21, 2019

jingyih commented May 21, 2019

weavage commented May 21, 2019

jingyih commented May 21, 2019

EugenKon commented Aug 10, 2023 • edited Loading

jmhbnz commented Aug 10, 2023

EugenKon commented Aug 10, 2023

jmhbnz commented Aug 10, 2023

EugenKon commented Aug 10, 2023 •

edited

Loading