"etcdctl txn" does not properly parse newline characters

[jutley]

We are using etcdctl txn to make sure servers running in parallel do not both write a new value for a specific key. To do this, we are running a command from a bash script that looks something like this:

etcdctl txn <<< 'create("my_key") = "0"

put "my_key" "my
multiline
value"



'

Essentially: If the value does not already exist, write it. Otherwise, do nothing.

The problem is that etcdctl does not parse this as intended. Instead, we get this error:

Error: invalid txn compare request: multiline"

Looking at the source code, it's pretty clear that the command is strictly parsing for newline characters. I've verified that using my\nmultiline\nvalue as the value gives the desired effect, so we can get by. However, this functionality is inconsistent with etcdctl put, which DOES handle the newline characters.

These commands should parse consistently.

We are currently using etcd 3.3.2, though the changelog shows no changes to etcdctl txn in newer versions, so I would assume this is still an issue.

[hexfusion]

@jutley this isn't really an etcd issue in my opinion. How about something like this with a Heredoc?

etcdctl txn <<< 'create("my_key") = "0" 

put "my_key" << EOM
my
multiline
value
EOM
'

[hexfusion]

But to your point, the put would work on its own so this probably needs more investigation.

etcdctl put my_key "my
multiline
value"

[jutley]

I've tried your literal example and a few alterations to try to get this to work, with no success. The only thing that I've gotten to work is escaping newlines in my value using a command like sed -e ':a;N;$!ba;s/\n/\\n/g'.

I think this is an issue with etcdctl because it makes the txn command really difficult to use in anything except the most simple of examples. We can work around the limitations, but it adds a lot of complexity on our end.

[hexfusion]

@jutley I will dig into this for you to better understand what is going on. The error message is very specific so we might not support it for a reason but I will explore further and at least document. Thanks for the report.

[jutley]

I think the fundamental difference between the put and the txn is the when using put, bash is parsing the multiline string, and passing the result to etcdctl. With txn, etcdctl is handling the string parsing. So I can understand how they ended up showing inconsistent behavior.

If nothing else, it would be a big improvement to update the documentation to call out that values must have escaped newlines.

[hexfusion]

@jutley I don't want to discourage you from exploring this as well. You seem to have a solid grasp of the situation and we would love the help :). If not I can review over the next week or so.

[wenjiaswe]

/cc @jingyih

[jingyih]

@jutley Your observation is correct. Currently the Txn command relies on '\n' to separate CMP, THEN, ELSE. It also relies on '\n' to separate multiple ops/requests. Thanks for pointing this out.

[jutley]

I'd be happy to at least make a PR to improve the documentation, but it'll be at least a week before I can get started.

[jingyih]

@jutley Sounds good. Thanks!