modules/kvs/kvstxn: Fix mem-leak in kvs merge

[chu11]

In kvstxn_create_empty(), the 'keys' array was created that
could be overwritten without being destroyed. This was a corner
case in PR #1859, when key generation was altered.

Fixes #1944

[chu11]

In one run hit

FAIL: t0001-basic.t 55 - instance can stop cleanly with subscribers (#1025)
             t0001-basic.t:  FAIL: N=62  PASS=61  FAIL=1 SKIP=0 XPASS=0 XFAIL=0

which appears to be #1077.

But have never seen

FAIL: t2000-wreck.t 15 - wreckrun: --input different for each task works
             t2000-wreck.t:  FAIL: N=73  PASS=71  FAIL=1 SKIP=1 XPASS=0 XFAIL=0

debug shows

expecting success: 
        for i in `seq 0 3`; do echo "hi $i" > i.$i; done &&
        flux wreckrun --input="i.0:0;i.1:1;i.2:2;i.3:3" -l -n4 cat |
		sort -n > output.multi &&
        cat >expected.multi <<-EOF &&
	0: hi 0
	1: hi 1
	2: hi 2
	3: hi 3
	EOF
        test_cmp expected.multi output.multi

--- expected.multi	2019-01-22 21:31:31.304176846 +0000
+++ output.multi	2019-01-22 21:31:31.304176846 +0000
@@ -1,4 +1,3 @@
 0: hi 0
 1: hi 1
-2: hi 2
-3: hi 3
+2: hi 3
not ok 15 - wreckrun: --input different for each task works

which seems disturbing. But I can't imagine my stupid mem-leak fix here could be the cause of this. If my mem-leak fix were truly horrible, there'd be way more bugs than this one. @grondo, got any idea on this one?

[grondo]

@grondo, got any idea on this one?

No and I'm not sure I've seen that failure before. Disturbingly, task 2 is getting the output from task 3 and task 3 is missing output. We should open a separate issue and I'll try to see if it can be reproduced.

[garlick]

Actually, looks like the above wreckrun --input test failure was seen once before here...

[grondo]

Well, that's embarrassing. I completely forgot about that.

[chu11]

rebased

[codecov-io]

Codecov Report

Merging #1946 into master will increase coverage by <.01%.
The diff coverage is 75%.

@@            Coverage Diff             @@
##           master    #1946      +/-   ##
==========================================
+ Coverage   80.03%   80.03%   +<.01%     
==========================================
  Files         195      195              
  Lines       34945    34931      -14     
==========================================
- Hits        27968    27957      -11     
+ Misses       6977     6974       -3

Impacted Files	Coverage Δ
src/modules/kvs/kvstxn.c	77.15% <75%> (+0.32%)	⬆️
src/common/libflux/mrpc.c	87.74% <0%> (-1.19%)	⬇️
src/cmd/flux-module.c	83.72% <0%> (-0.48%)	⬇️
src/modules/kvs/kvs.c	66.35% <0%> (+0.14%)	⬆️
src/common/libutil/veb.c	98.85% <0%> (+0.57%)	⬆️

[chu11]

actually hold off on a merge, I realized I could refactor this a tiny bit to prevent unexpected errors from coming up in the future. I can't remember why I decided to make separate kvstxn_create() and kvstxn_create_empty() functions in the past, but it's not necessary.

[chu11]

re-pushed with a few refactoring fixes

[chu11]

rebased

[chu11]

rebased

[garlick]

Thanks!