ZOOKEEPER-2994 Tool required to recover log and snapshot entries with CRC errors

[anmolnar]

https://issues.apache.org/jira/browse/ZOOKEEPER-2994

In the event of ZooKeeper transaction log becomes corrupted and fail CRC checks (preventing startup) we should have a mechanism to get the cluster running again.

Previously we achieved this by loading the broken transaction log with a modified version of ZK with disabled CRC check and forced it to write new txn log files.

It has proven that once you end up with the corrupt txn log there is no way to recover except manually modifying the crc check. That's basically why the tool is needed.

It's called TxnLogToolkit, a new console application similar to LogFormatter and SnapshotFormatter, but it's intentionally separated to keep backward compatibility in the existing tools.

This PR contains TXN log tool only.

You probably also notice a refactoring to extract file padding logic from FileTxnLog to reuse in the new tool. Related code changes can be reviewed alone in a separate commit if preferred.

[phunt]

Looks promising - doesn't seem very useful (and potentially dangerous) without docs - perhaps add a troubleshooting or recovery section here?
http://zookeeper.apache.org/doc/r3.4.11/zookeeperAdmin.html#sc_dataFileManagement
The jira was original for 3.5+, I think this would be great to get into 3.4+.

[nkalmar]

Useful addition, +1

As @phunt pointed out, docs in zookeeperAdmin.xml could be updated.

[nkalmar]

I used your updated documentation, and managed to recover a corrupted log file:

bin/zkTxnLogToolkit.sh -d ~/workspace/zookeeper/standalone/version-2/log.1
ZooKeeper Transactional Log File with dbid 0 txnlog format version 2
4/9/18 3:13:19 PM CEST session 0x10000ebe13a0000 cxid 0x0 zxid 0x1 createSession 30000
4/9/18 3:15:21 PM CEST session 0x10000ebe13a0000 cxid 0x0 zxid 0x2 closeSession null
4/9/18 3:17:41 PM CEST session 0x10000ebe13a0001 cxid 0x0 zxid 0x3 createSession 30000
4/9/18 3:18:13 PM CEST session 0x10000ebe13a0001 cxid 0x0 zxid 0x4 closeSession null
EOF reached after 4 txns.

Corrupted log.1 file

bin/zkTxnLogToolkit.sh -d ~/workspace/zookeeper/standalone/version-2/log.1
ZooKeeper Transactional Log File with dbid 0 txnlog format version 2
CRC ERROR - 4/10/18 5:12:11 AM CEST session 0x10000ebe13a0000 cxid 0x0 zxid 0x1 createSession 30000
4/10/18 5:12:11 AM CEST session 0x10000ebe13a0000 cxid 0x0 zxid 0x1 createSession 30000
4/9/18 3:15:21 PM CEST session 0x10000ebe13a0000 cxid 0x0 zxid 0x2 closeSession null
CRC ERROR - 4/9/18 3:17:41 PM CEST session 0x10044aa44aaaaaa cxid 0x0 zxid 0x3 createSession 30000
4/9/18 3:17:41 PM CEST session 0x10044aa44aaaaaa cxid 0x0 zxid 0x3 createSession 30000
4/9/18 3:18:13 PM CEST session 0x10000ebe13a0001 cxid 0x0 zxid 0x4 closeSession null
EOF reached after 4 txns.

bin/zkTxnLogToolkit.sh -r ~/workspace/zookeeper/standalone/version-2/log.1
ZooKeeper Transactional Log File with dbid 0 txnlog format version 2
CRC ERROR - 4/10/18 5:12:11 AM CEST session 0x10000ebe13a0000 cxid 0x0 zxid 0x1 createSession 30000
Would you like to fix it (Yes/No/Abort) ? Y
EOF reached after 4 txns.
Recovery file /Users/.../zookeeper/standalone/version-2/log.1.fixed has been written with 1 fixed CRC error(s)

bin/zkTxnLogToolkit.sh -d ~/workspace/zookeeper/standalone/version-2/log.1.fixed
ZooKeeper Transactional Log File with dbid 0 txnlog format version 2
4/9/18 3:13:19 PM CEST session 0x10000ebe13a0000 cxid 0x0 zxid 0x1 createSession 30000
4/9/18 3:15:21 PM CEST session 0x10000ebe13a0000 cxid 0x0 zxid 0x2 closeSession null
4/9/18 3:17:41 PM CEST session 0x10000ebe13a0001 cxid 0x0 zxid 0x3 createSession 30000
4/9/18 3:18:13 PM CEST session 0x10000ebe13a0001 cxid 0x0 zxid 0x4 closeSession null
EOF reached after 4 txns.

LGTM!

[anmolnar]

@phunt I added documentation to ZookeeperAdmin docs.
@nkalmar thanks!

[nkalmar]

LGTM, tested, works fine!

[phunt]

+1 Thanks @anmolnar this looks good. Please consider backporting to 3.4 (separate jira).

Also in future please don't include any changed files from the toplevel docs directory (html/pdf files) as these are regenerated during commit.