Failed to start consul server in bootstrap mode using systemctl

[rickyma]

Overview of the Issue

I followed this documentation: https://learn.hashicorp.com/consul/datacenter-deploy/deployment-guide

I set
bootstrap = true
in server.hcl. (In our scenario, we have to use bootstrap=true at the initialization of our installation. Because we need to start only one server ahead of everything.)

After all the steps, when I ran systemctl start consul, it stuck for a few seconds and then failed.

When I use journalctl -xe to check the error, there is no ERROR logs but only WARN and INFO logs.
I guess it is the WARN log that stopped consul from being startup??

I ran the startup command manually, and it started successfully also with WARN logs. And everything works fine. But it didn't work with systemctl command.

This is my consul.service file:

[Unit]
Description="HashiCorp Consul - A service mesh solution"
Documentation=https://www.consul.io/
Requires=network-online.target
After=network-online.target
ConditionFileNotEmpty=/etc/consul.d/consul.hcl

[Service]
Type=notify
User=consul
Group=consul
ExecStart=/usr/local/bin/consul agent -config-dir=/etc/consul.d/ -bind "127.0.0.1" -client "127.0.0.1" -ui-content-path "/tdw/consul/"
ExecReload=/usr/local/bin/consul reload
KillMode=process
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

Reproduction Steps

1. set bootstrap = true in server.hcl, and follow steps in https://learn.hashicorp.com/consul/datacenter-deploy/deployment-guide
2. run systemctl start consul

Consul info for Server

Server info

consul info
agent:
	check_monitors = 0
	check_ttls = 0
	checks = 0
	services = 0
build:
	prerelease =
	revision = 9ea1a204
	version = 1.7.2
consul:
	acl = disabled
	bootstrap = true
	known_datacenters = 1
	leader = true
	leader_addr = 127.0.0.1:8300
	server = true
raft:
	applied_index = 237
	commit_index = 237
	fsm_pending = 0
	last_contact = 0
	last_log_index = 237
	last_log_term = 35
	last_snapshot_index = 0
	last_snapshot_term = 0
	latest_configuration = [{Suffrage:Voter ID:7237a5df-4590-cb41-ba8c-aae1603d4e6d Address:127.0.0.1:8300}]
	latest_configuration_index = 0
	num_peers = 0
	protocol_version = 3
	protocol_version_max = 3
	protocol_version_min = 0
	snapshot_version_max = 1
	snapshot_version_min = 0
	state = Leader
	term = 35
runtime:
	arch = amd64
	cpu_count = 16
	goroutines = 81
	max_procs = 16
	os = linux
	version = go1.13.7
serf_lan:
	coordinate_resets = 0
	encrypted = true
	event_queue = 1
	event_time = 35
	failed = 0
	health_score = 0
	intent_queue = 0
	left = 0
	member_time = 1
	members = 1
	query_queue = 0
	query_time = 1
serf_wan:
	coordinate_resets = 0
	encrypted = true
	event_queue = 0
	event_time = 1
	failed = 0
	health_score = 0
	intent_queue = 0
	left = 0
	member_time = 1
	members = 1
	query_queue = 0
	query_time = 1

Operating system and Environment details

CentOS Linux release 7.6.1810 (Core)
Linux tbds-10-1-0-106 3.10.0-1062.9.1.el7.x86_64 #1 SMP Fri Dec 6 15:49:49 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

[rickyma]

Same issue with: #7137

[jsosulska]

Closing in favor of #7137