[BUG]: Consul deletes K/V if ACL policy is key with value being a path ending in /

[danlsgiga]

Overview of the Issue

In consul v1.4.4 if an ACL token has the following policy attached to it, it will cause the key/value to be deleted once the owner of the ACL token navigates to it in the UI

key "app/" { policy = "write" }

Note that our intention was to give key_prefix but we by mistake created it as above and that was the cause of the issue. Once we updated the policy to key_prefix the issue could not be reproduced anymore.

Reproduction Steps

1. Create a cluster with 3 server nodes with ACL enabled and set to default = deny
2. Create a policy with the contents as posted above and then an ACL token with the policy assigned to it
3. Via the UI, try to open the k/v referenced in the ACL policy.
4. The k/v is entirely deleted from Consul

Consul info for both Client and Server

Client info

agent:
	check_monitors = 0
	check_ttls = 0
	checks = 2
	services = 1
build:
	prerelease =
	revision = ea5210a3
	version = 1.4.4
consul:
	acl = enabled
	known_servers = 3
	server = false
runtime:
	arch = amd64
	cpu_count = 2
	goroutines = 53
	max_procs = 2
	os = linux
	version = go1.11.4
serf_lan:
	coordinate_resets = 0
	encrypted = true
	event_queue = 0
	event_time = 338
	failed = 0
	health_score = 0
	intent_queue = 0
	left = 0
	member_time = 4134
	members = 41
	query_queue = 0
	query_time = 1

Server info

agent:
	check_monitors = 0
	check_ttls = 1
	checks = 4
	services = 4
build:
	prerelease =
	revision = ea5210a3
	version = 1.4.4
consul:
	acl = enabled
	bootstrap = false
	known_datacenters = 1
	leader = false
	leader_addr = x.x.x.x:8300
	server = true
raft:
	applied_index = 15731666
	commit_index = 15731666
	fsm_pending = 0
	last_contact = 4.69369ms
	last_log_index = 15731666
	last_log_term = 234
	last_snapshot_index = 15720337
	last_snapshot_term = 234
	latest_configuration = [{Suffrage:Voter ID:bd9ab4bc-763f-5083-425e-480334ea6dd0 Address:x.x.x.x:8300} {Suffrage:Voter ID:3ba10a8f-d837-d740-6160-5bef18aceb83 Address:y.y.y.y:8300} {Suffrage:Voter ID:c21048dc-ef54-909f-a475-5d6bd5a68d49 Address:z.z.z.z:8300}]
	latest_configuration_index = 14894601
	num_peers = 2
	protocol_version = 3
	protocol_version_max = 3
	protocol_version_min = 0
	snapshot_version_max = 1
	snapshot_version_min = 0
	state = Follower
	term = 234
runtime:
	arch = amd64
	cpu_count = 2
	goroutines = 239
	max_procs = 2
	os = linux
	version = go1.11.4
serf_lan:
	coordinate_resets = 0
	encrypted = true
	event_queue = 0
	event_time = 79
	failed = 0
	health_score = 0
	intent_queue = 0
	left = 0
	member_time = 1445
	members = 40
	query_queue = 0
	query_time = 1
serf_wan:
	coordinate_resets = 0
	encrypted = true
	event_queue = 0
	event_time = 1
	failed = 0
	health_score = 0
	intent_queue = 0
	left = 0
	member_time = 134
	members = 3
	query_queue = 0
	query_time = 1

Operating system and Environment details

CentOS Linux release 7.6.1810 (Core)
3.10.0-957.1.3.el7.x86_64

[johncowen]

Hi @danlsgiga

Thanks for the bug report!

Right now I am unable to reproduce your bug, so it would be great if you could help me exactly recreate the steps you've gone through to produce the bug.

I've tried the following in both 1.4.4 and 1.5.0, hopefully between us we can sync up steps to reproduce what you are seeing:

1. I started a single server of consul with ACLs enabled and and default_policy of 'deny'
2. I logged in to the ACLs area of the UI using my master token.
3. I created a KV 'folder' called app/ at the root of the KV store using the UI
4. I went back to the ACLs area and created a new policy called 'app-policy' with the exact rules you mention above.
5. I created a new token with the 'app-policy' policy assigned to it.
6. I re-logged in using the new token (via the 'Use' action in the UI)
7. I then navigated straight to the previously created app/ KV by pasting the URL for it into my browser location bar.

Everything seemed to work as expected.

Some further notes incase it helps:

1. I also tried navigating to the KV via the UI instead of entering the URL in the browser bar.
2. I also tried re-logging in using the master token and navigating to the KV again, both via entering the URL in the location bar and navigating to it via the UI.
3. I also tried putting deeper values into the KV and then navigating to it.

Once we updated the policy to key_prefix the issue could not be reproduced anymore.

Can I ask if you see the KV actually deleted in Consul itself, or is it just not visible via the UI?

Thanks again for the report!

John

[danlsgiga]

Hi @johncowen, this was a very weird ocurrence for us and we could reproduce it at least 3 times until we changed the policy to key_prefix.

I was sitting beside a colleague showing him some procedures for our deployment.. I was logged with a management token and he with a client token with that policy attached. At the moment he clicked in the root k/v path app/ in the UI, the k/v disappeared for him and also for myself.

I'll try to reproduce it again switching the policy to key again.

[danlsgiga]

K... I was able to reproduce the issue once I set key_prefix back to key in the policy.

I asked my colleague to try the same thing in Firefox, Chrome and Safari and we can reproduce it all the time.

Note: the app/ path I gave was just an example... in our real case it is salt/git and as soon as he tries to open salt, the k/v is deleted and I have to manually recreate it. There's nothing being logged when that happens!

A gif with the steps reproducing the issue... at the end we can see that the UI is sending a DELETE to that endpoint.

[johncowen]

This is great (well not, but you know what I mean!) @danlsgiga thankyou so much. I'll see if I can recreate again here and report back.

[danlsgiga]

Well, at least we know the "fix" for now which is to use key_prefix, but that issue can do some nasty things. Thanks for looking into it right away!!

As a side note, I'm not sure this would have any impact but the Description of that token is Daniel's token... I know that single things like an apostrophe can be very destructive, so... just as a heads up if that ends up being the root cause. :D