etcd 3.4.0 won't start with name flag override when environment ETCD_NAME is available.

[pblaas]

I use CoreOS container linux and ignition files to boot a cluster.

A snip of a Jinja2 yaml which gets transpiled to Container linux config looks like this:

etcd:
  version:                     "{{ etcdver }}"
  advertise_client_urls:       "https://{{ ipaddress }}:2379"
  initial_advertise_peer_urls: "https://{{ ipaddress }}:2380"
  listen_client_urls:          "http://127.0.0.1:2379,https://{{ ipaddress }}:2379"
  listen_peer_urls:            "https://{{ ipaddress }}:2380"
  name:                        "infra{{ etcdid }}"
  initial_cluster:             "{{ initialclusterlist }}"
  initial_cluster_token:       "etcd-cluster-{{ etcdtoken }}"
  initial_cluster_state:       "new"
  cert_file:                   "/etc/ssl/certs/{{ ipaddress }}-etcd-node.pem"
  key_file:                    "/etc/ssl/certs/{{ ipaddress }}-etcd-node-key.pem"
  trusted_ca_file:             "/etc/ssl/certs/etcd-ca.pem"
  client_cert_auth:            true
  peer_cert_file:              "/etc/ssl/certs/{{ ipaddress }}-etcd-node.pem"
  peer_key_file:               "/etc/ssl/certs/{{ ipaddress }}-etcd-node-key.pem"
  peer_trusted_ca_file:        "/etc/ssl/certs/etcd-ca.pem"
  peer_client_cert_auth:       true
  metrics:                     "extensive" 

Since 3.4.0 it seems I can't override with the --name flag. Because somewhere on a fresh container linux system it defaults an ETCD_NAME environment variable which now prevents starting up the RKT etcd 3.4.0 process. I'm not sure where this ETCD_NAME environment variable comes from because I didn't set it anywhere. It seems the behaviour between 3.4.0 and 3.3.15 with name override has changed.

A small demo with docker explains the override issue I'm seeing with RKT too.
In etcd 3.3.15 it worked fine:

/usr/local/bin # ./etcd -version
etcd Version: 3.3.15
Git SHA: 94745a4ee
Go Version: go1.12.9
Go OS/Arch: linux/amd64

# cd ./usr/local/bin/
/usr/local/bin # export ETCD_NAME=test
/usr/local/bin # ./etcd --name=testtest
2019-09-03 21:01:47.642520 W | pkg/flags: recognized environment variable ETCD_NAME, but unused: shadowed by corresponding flag
2019-09-03 21:01:47.642567 I | etcdmain: etcd Version: 3.3.15

In etcd 3.4.0 the process stops.

# ./etcd --version
etcd Version: 3.4.0
Git SHA: 898bd1351
Go Version: go1.12.9
Go OS/Arch: linux/amd64

# export ETCD_NAME=test
# ./etcd --name=blabla 
2019-09-03 21:05:44.576119 C | pkg/flags: conflicting environment variable "ETCD_NAME" is shadowed by corresponding command-line flag (either unset environment variable or disable flag)
# 

It would be great if etcd 3.4.0 name override would behave just like 3.3.15 if that is still possible.
Other tips and tricks where I can find the conflicting ETCD_NAME environment variable would be great. I could add another dropin in the ignition files where I set the ETCD_NAME explicitly basicly overriding it again but I prefer not too.

Regards,
Patrick

[jingyih]

My understanding is that the new behavior is to prevent potential user error. It is highlighted in the changelog as breaking change: https://github.com/etcd-io/etcd/blob/master/CHANGELOG-3.4.md#breaking-changes

For more context: #9382, #8380

cc @gyuho
Please correct me if I am wrong.

[gyuho]

@jingyih is right.

I'm not sure where this ETCD_NAME environment variable comes from because I didn't set it anywhere.

etcd does not set any environmental variable. You have to find out where ETCD_NAME is being set in your environment. This was added to prevent configuration overrides.

[pblaas]

ETCD_NAME is being set to this on one instance 189cc9c436df446abc589ece1bc5bb01 by default when loaded through ignition spec and to f066cbe511bf425b9879de728cf7b28c on another. So it looks like some kind of random string generation.

It seems ETCD_NAME defaults to the machine id when using the RKT wrapper which is in the unit file generated by the ignition spec.
/etc/systemd/system/etcd-member.service.d/20-clct-etcd-member.conf
ExecStart= /usr/lib/coreos/etcd-wrapper $ETCD_OPTS \

cat /etc/machine-id 
189cc9c436df446abc589ece1bc5bb01

[bbotte]

In stand-alone mode, the configuration file has not changed. Upgrading from version 3.3.15 to 3.4.0 or 3.4.1 has failed, and can not start. I don't know how to handle it.

config systemd: Starting Etcd Server...
config etcd: recognized and used environment variable ETCD_ADVERTISE_CLIENT_URLS=http://localhost:2379
config etcd: recognized and used environment variable ETCD_INITIAL_CLUSTER_STATE=new
config etcd: recognized and used environment variable ETCD_LOG_OUTPUT=stderr
config etcd: conflicting environment variable "ETCD_NAME" is shadowed by corresponding command-line flag (either unset environment variable or disable flag)
config systemd: etcd.service: main process exited, code=exited, status=1/FAILURE
config systemd: Failed to start Etcd Server.
config systemd: Unit etcd.service entered failed state.
config systemd: etcd.service failed.
config systemd: etcd.service holdoff time over, scheduling restart.

It was my mistake, to refer to the document etcd-configuration The startup command changed

[sanqingshan]

i have the same problem with

etcd Version: 3.4.1
Git SHA: a14579fbf
Go Version: go1.12.9
Go OS/Arch: linux/amd64

10月 03 22:42:07 etcdinstaller-180401 systemd[1]: Starting Etcd Server...
-- Subject: Unit etcd.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit etcd.service has begun starting up.
10月 03 22:42:07 etcdinstaller-180401 etcd[1638]: conflicting environment variable "ETCD_NAME" is shadowed by corresponding command-line flag (either unset environment variable or disable flag)
10月 03 22:42:07 etcdinstaller-180401 systemd[1]: etcd.service: main process exited, code=exited, status=1/FAILURE
10月 03 22:42:07 etcdinstaller-180401 systemd[1]: Failed to start Etcd Server.
-- Subject: Unit etcd.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit etcd.service has failed.
-- 
-- The result is failed.
10月 03 22:42:07 etcdinstaller-180401 systemd[1]: Unit etcd.service entered failed state.
10月 03 22:42:07 etcdinstaller-180401 systemd[1]: etcd.service failed.
10月 03 22:42:07 etcdinstaller-180401 polkitd[586]: Unregistered Authentication Agent for unix-process:1632:30672692 (system bus name :1.1951, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale zh_C
10月 03 22:42:07 etcdinstaller-180401 polkitd[586]: Registered Authentication Agent for unix-process:1651:30672764 (system bus name :1.1952 [/usr/bin/pkttyagent --notify-fd 5 --fallback], object path /org/freedeskto
10月 03 22:42:07 etcdinstaller-180401 systemd[1]: start request repeated too quickly for etcd.service
10月 03 22:42:07 etcdinstaller-180401 systemd[1]: Failed to start Etcd Server.
-- Subject: Unit etcd.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit etcd.service has failed.
-- 
-- The result is failed.

i do not set any system enviroment variable

[sanqingshan]

i tryd the version 3.2,and it is ok

[dhawal55]

Any suggested workaround to use the hostname as etcd_name??

I tried setting ETCD_NAME environment variable but using one env var to set another doesn't seem to work:

ENVIRONMENT="ETCD_NAME=${COREOS_EC2_HOSTNAME}"

I tried using systemd var %H but its returning localhost instead, probably because hostname is not set yet
ENVIRONMENT="ETCD_NAME=%H"

[saltbo]

I encountered a similar problem and solved it.

etcd 3.4 will exit on shadowed environment variables

So, flag and environment cannot coexist.

error example:

[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
EnvironmentFile=-/etc/etcd/etcd.conf
User=etcd
ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /usr/bin/etcd \
  --name ${ETCD_NAME}

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.

[ayunwSky]

I have a similar problem:
when I use commad:

/usr/local/bin/etcdctl --endpoints=https://xxx:2379 snapshot save /data/backup_dir_etcd/etcd-snapshot-20210510.db

or

/usr/local/bin/etcdctl snapshot save /data/backup_dir_etcd/etcd-snapshot-20210510.db

It show the following error:
2021-05-10 11:38:22.791775 C | pkg/flags: conflicting environment variable "ETCDCTL_ENDPOINTS" is shadowed by corresponding command-line flag (either unset environment variable or disable flag)

and i found a variables in my /etc/profile files aoubt "ETCDCTL_ENDPOINTS",when i unset variables in "/etc/profile" file, It becomes normal。

Is there can't use flag in command to overwrite variables ? Or there have other way to resolv this question?

[ayunwSky]

@saltbo Hello, saltbo。 you have ensure that flag and environment cannot coexist Right?
when I test and have a results is similar like when i unset the system varibales,and use etcdctl xx snapshot save command,it chage to normal。when /etc/profile file's have varibales, it turn wrong。

[saltbo]

Yes. @allenjol

[ayunwSky]

@saltbo Thank you very much!