Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Properly wait for resolve snapshots work to finish instead of failing and retrying #1750

Closed
MonsieurNicolas opened this issue Aug 8, 2018 · 9 comments
Assignees

Comments

@MonsieurNicolas
Copy link
Contributor

Issue type

bug

Version found

10.0.0rc1

Issue description

When a snapshot gets published, it fails to properly take the snapshot, causing a delay in publishing.

Actual result

Delay is systematic, logs show that it marks the work as failed (and retries) without trying to do anything (at least no processes are started).

Expected result

No error when publishing.

Supporting files


2018-08-08T21:51:06.466 GCUCJ [Ledger INFO] Got consensus: [seq=10447167, prev=f72285, tx_count=0, sv: [  txH: 999d4a, ct: 1533765060, upgrades: [ ] ]]
2018-08-08T21:51:06.466 GCUCJ [Ledger DEBUG] starting closeLedger() on ledgerSeq=10447167
2018-08-08T21:51:06.466 GCUCJ [Ledger DEBUG] processing fees and sequence numbers
2018-08-08T21:51:06.467 GCUCJ [Bucket DEBUG] Deleting empty bucket file /data/buckets/tmp/bucket-f2a2103581f3f33e/tmp-bucket-172037e8169e019a.xdr
2018-08-08T21:51:06.467 GCUCJ [Bucket DEBUG] Deleting empty bucket file /data/buckets/tmp/bucket-f2a2103581f3f33e/tmp-bucket-4221b75c417980bf.xdr
2018-08-08T21:51:06.467 GCUCJ [Bucket DEBUG] Deleting empty bucket file /data/buckets/tmp/bucket-f2a2103581f3f33e/tmp-bucket-fcde48c0acac7d6f.xdr
2018-08-08T21:51:06.467 GCUCJ [Bucket DEBUG] Deleting bucket file /data/buckets/tmp/bucket-f2a2103581f3f33e/tmp-bucket-2c3c5e7a947b14be.xdr that is redundant with existing bucket
2018-08-08T21:51:06.468 GCUCJ [Ledger DEBUG] Advancing LCL: [seq=10447166, hash=f72285] -> [seq=10447167, hash=aabb48]
2018-08-08T21:51:06.468 GCUCJ [Ledger DEBUG] New current ledger: seq=10447168
2018-08-08T21:51:06.469 GCUCJ [History DEBUG] Queueing publish state for ledger 10447167
2018-08-08T21:51:06.470 GCUCJ [History DEBUG] Activating publish for ledger 10447167
2018-08-08T21:51:06.470 GCUCJ [Fs DEBUG] created dir /data/buckets/tmp/snapshot-ec421db5b63f34bc
2018-08-08T21:51:06.470 GCUCJ [Work DEBUG] resetting publish-009f693f
2018-08-08T21:51:06.470 GCUCJ [Work DEBUG] advancing publish-009f693f
2018-08-08T21:51:06.470 GCUCJ [Work DEBUG] all 0 children of publish-009f693f successful, scheduling run
2018-08-08T21:51:06.470 GCUCJ [Work DEBUG] scheduling run of publish-009f693f
2018-08-08T21:51:06.473 GCUCJ [History INFO] Publishing 1 queued checkpoints [10447167-10447167]: Awaiting 0/0 prerequisites of: publish-009f693f
2018-08-08T21:51:06.473 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 0000000000000000000000000000000000000000000000000000000000000000 referenced by publish queue
2018-08-08T21:51:06.473 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 02c358035b6470656fb425862de1d2eba1f2fdf52254b4f4756062ffefe89b4b referenced by publish queue
2018-08-08T21:51:06.473 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 0ed0b722ee76db5634e32a4e8ad087d7a4ddcdb3fa9a8939533a9dada5d82a10 referenced by publish queue
2018-08-08T21:51:06.473 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 116fe640258b03d0015b20ab2225e83379436bdc9b3306a7affb7abff20f33d5 referenced by publish queue
2018-08-08T21:51:06.473 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 16161d7ddfea5ece936f0c330eadeb60210b9b7348f9e306d6e6806ef9f9b4d6 referenced by publish queue
2018-08-08T21:51:06.473 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 1724d77be46fa62a0ed46b106a3e47af1c0c8d42deac4b8bc98d505755e0437a referenced by publish queue
2018-08-08T21:51:06.473 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 1dd2ec206ea293d1072611d6771673e534f3fa47119189ed43cbdbdf473ae4d6 referenced by publish queue
2018-08-08T21:51:06.473 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 1e1cdc270817cdd95246632fd78d5b482ec559ecae390e2180096f7b58bc77c5 referenced by publish queue
2018-08-08T21:51:06.473 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 205506f667460e78f0b9f7758b3c20fd4a232120ef5b51a44d0f8e44fb842177 referenced by publish queue
2018-08-08T21:51:06.473 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 25975bcc1ea7ee891e91f7a0936c17bf407e9f7c58ba5087fccbebb45c94cfbb referenced by publish queue
2018-08-08T21:51:06.474 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 28f84eace1e9911618dc61739842e598235d90e75f6ea2f0a4cdf907f2bd3128 referenced by publish queue
2018-08-08T21:51:06.474 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 2a6caeeb25bcffd5022e35c16a7d03b541e3a89bf5f48f18e8af614884aa6a90 referenced by publish queue
2018-08-08T21:51:06.474 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 2c6f2ee1ffb097d6cce2cef39538b53d508be370d74f4626d8cf32d2758e1c8f referenced by publish queue
2018-08-08T21:51:06.474 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 2d656019fa60c6a2e45e5df66107ff75ccb82e01d40e5433901476e02168aa08 referenced by publish queue
2018-08-08T21:51:06.474 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 2d931869ff7803e86877a848a083bd7324b59b928411b5d298e839462aeae126 referenced by publish queue
2018-08-08T21:51:06.474 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 327ab7b0a3691f9bc7bde7b46b0b9842ae2c9a5ef15e7992ee11c88f50e08b9d referenced by publish queue
2018-08-08T21:51:06.474 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 38dd5f08b431213a1374b9a4f36ee3ee126b9f62194a7dd196ae88a81560224e referenced by publish queue
2018-08-08T21:51:06.474 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 3c600072de517bc4868d7432182ab3f62920dad3c58e7ba8b196c09810cca4c2 referenced by publish queue
2018-08-08T21:51:06.474 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 3c87b3dfddadceec084ccbd22efb76aefc48425870388ca416777f766e12baec referenced by publish queue
2018-08-08T21:51:06.474 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 3dde3499fd0d8a4297bbbe957759657d98c7d7ca0bc5b4581dd9f079c54e7392 referenced by publish queue
2018-08-08T21:51:06.474 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 46a854adf19cadd698d6c36ccaed78f645889d85ddd076e2e107799fec0172a1 referenced by publish queue
2018-08-08T21:51:06.475 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 4a303763c10813263c0d78d314642544b64fb529e5f4514122edf9986738c7a0 referenced by publish queue
2018-08-08T21:51:06.475 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 4c5fdd8ac9daaaf5a6b4ef0b69da27fa39b7d4e35f503798af54c27b070db671 referenced by publish queue
2018-08-08T21:51:06.475 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 4f09c585b0e314f87ff64f0fca5346ed36d47fb12b802d273740043bf72a51e6 referenced by publish queue
2018-08-08T21:51:06.475 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 53298af6ee9c77541610994a8af96f60f26fe2ca8e995302a3da9911f85092ea referenced by publish queue
2018-08-08T21:51:06.475 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 57257b83f9c5ccb7dd24c56aa7a5101bf9ba0d350e313d985b6ebe5535846a12 referenced by publish queue
2018-08-08T21:51:06.475 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 5794c7922c078248a1f497fbeaa0cde0708b3d6ece98ac59ddb6ecdadb7c9010 referenced by publish queue
2018-08-08T21:51:06.475 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 5e2cc0a2fc6dd474a2aa3baef80cf8fc425723465be6f21489ce730c12ce0548 referenced by publish queue
2018-08-08T21:51:06.475 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 600af102fb6bce6a07470d9a6f476a1b48444e4b41336a518247fba8e609a18d referenced by publish queue
2018-08-08T21:51:06.475 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 6c56152fd52a33e6c1589ca225f5107fd18ecad19c036c23f6f3a7bfa46e42d1 referenced by publish queue
2018-08-08T21:51:06.476 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 6fbb2e480cd258caa7f13a2d74c9306a0a7c12e9c74d926f249d0e78fa8e04a5 referenced by publish queue
2018-08-08T21:51:06.476 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 727e26fc2b8407ea155ef61365eebcb5377e90299f6805a04eba105209511f47 referenced by publish queue
2018-08-08T21:51:06.476 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 85d140cb1c1eb24faa0b10f44d17b9954d24fd4c5d6db3c91adfe7e72d7b4df2 referenced by publish queue
2018-08-08T21:51:06.476 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 8830f702c32ce5e8d493031453fba0ec6be2c1b07e3fc5d0123f9895ecbc5523 referenced by publish queue
2018-08-08T21:51:06.476 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 8ba45f03a99923112657726bb94c01e78a31449997d7cd0b275d999209568ae1 referenced by publish queue
2018-08-08T21:51:06.476 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 8c29ec8ced10027ebce0533e1e852841f94dfb9a9adf5f2ec4aeb44650371332 referenced by publish queue
2018-08-08T21:51:06.476 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 8df1e17d986608744a66130af542f00f8766717a4e551eeed01a5aac0bc19217 referenced by publish queue
2018-08-08T21:51:06.476 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 97a13bc1fefa10aabba48c581bcb78e8f04320245a5f1bec2a027a6cc29603ee referenced by publish queue
2018-08-08T21:51:06.476 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 97ff09d76d73abe10327d030cebcfa79ff61f7a83829b078fe50ec0a5f53b325 referenced by publish queue
2018-08-08T21:51:06.476 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 99740e79206c6969abd62dfa163c6c33df69ca22e6ff4cc95dfb477a600fbfd6 referenced by publish queue
2018-08-08T21:51:06.477 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 9a9066488205fc66520871dfdcdea1edba04ad6fc97734cd5a8aba68cf1ba9cc referenced by publish queue
2018-08-08T21:51:06.477 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: a3066a7178325e7fa220571f54627052ebcee1effce3bad5e5d9c8dfc9fe191e referenced by publish queue
2018-08-08T21:51:06.477 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: a3408247d60479710009e4b9a845fb39d13be9368084a76a270bd4c846652c04 referenced by publish queue
2018-08-08T21:51:06.477 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: a6dc7f0ecc115c107d64b83734c683735a017e22c5ab494a530417ccf34ca97e referenced by publish queue
2018-08-08T21:51:06.477 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: b1d8c6951e16f1a74e1a204753762f9702fa0678e51c863d0b72c9ad70490df5 referenced by publish queue
2018-08-08T21:51:06.477 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: b2fca86b8c2be3f0ef49e7eaba08621f6696c00f2a881caee72f616cbba25099 referenced by publish queue
2018-08-08T21:51:06.477 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: b66a52f7759066253f3f14eafc4ae5fc3f41acadf1890500fdaaa4eda6dd4917 referenced by publish queue
2018-08-08T21:51:06.477 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: c06f26ce5d478ddf502899df9ce622c715591a7d102ec38983e2e42865df080a referenced by publish queue
2018-08-08T21:51:06.477 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: c286e9fab913bca9f93038f2312b6566fc7f4bb56798817c9234e0ce6a06380d referenced by publish queue
2018-08-08T21:51:06.477 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: cdb27b9bb3ff34873b5432c6b5202045a7a1a0b40c74ab27100315933bed4f06 referenced by publish queue
2018-08-08T21:51:06.477 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: d8511a783335d7a8841ae48b609b3d3992c67abe3a32b04a43c5fc6e55ffed1c referenced by publish queue
2018-08-08T21:51:06.477 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: d9ee859667458d7c1fe40bcab7d70a5ef51e71bc93d0018c784cd351f48cc086 referenced by publish queue
2018-08-08T21:51:06.477 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: e10cbeba9c8b9e94620436c819cb60f2fd56910f6c158e6e1700013662b7bb06 referenced by publish queue
2018-08-08T21:51:06.477 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: e1e15b7b94e3523f2ec7e122761d2c6861804978e428db4ee9912e1a6be488d1 referenced by publish queue
2018-08-08T21:51:06.478 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: ea200e569383562739e58ccc653d80fb52b8ccfaae04b57d7d2da589088f00f7 referenced by publish queue
2018-08-08T21:51:06.478 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: ec98a9992a6ba53b76d0075d658ae40c90a476417acb13f0e84a6d60be6d368e referenced by publish queue
2018-08-08T21:51:06.478 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: f5d738d89601828e209a3117d5b88067ea451264f085ebae8cc2a09a1c6fd613 referenced by publish queue
2018-08-08T21:51:06.478 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: f64da3c35f82d9672a7253907e60371c675253f8c54bcc858d51681f9b746185 referenced by publish queue
2018-08-08T21:51:06.478 GCUCJ [Ledger INFO] Closed ledger: [seq=10447167, hash=aabb48]
2018-08-08T21:51:06.492 GCUCJ [Work DEBUG] starting publish-009f693f
2018-08-08T21:51:06.493 GCUCJ [Work DEBUG] running publish-009f693f
2018-08-08T21:51:06.493 GCUCJ [Work DEBUG] work publish-009f693f : WORK_PENDING -> WORK_RUNNING
2018-08-08T21:51:06.493 GCUCJ [Work DEBUG] scheduling completion of publish-009f693f
2018-08-08T21:51:06.500 GCUCJ [Work DEBUG] completed publish-009f693f
2018-08-08T21:51:06.500 GCUCJ [Work DEBUG] resetting prepare-snapshot
2018-08-08T21:51:06.500 GCUCJ [Work DEBUG] work publish-009f693f : WORK_RUNNING -> WORK_PENDING
2018-08-08T21:51:06.500 GCUCJ [Work DEBUG] advancing publish-009f693f
2018-08-08T21:51:06.500 GCUCJ [Work DEBUG] advancing prepare-snapshot
2018-08-08T21:51:06.500 GCUCJ [Work DEBUG] all 0 children of prepare-snapshot successful, scheduling run
2018-08-08T21:51:06.500 GCUCJ [Work DEBUG] scheduling run of prepare-snapshot
2018-08-08T21:51:06.514 GCUCJ [Work DEBUG] starting prepare-snapshot
2018-08-08T21:51:06.514 GCUCJ [Work DEBUG] running prepare-snapshot
2018-08-08T21:51:06.515 GCUCJ [Work DEBUG] work prepare-snapshot : WORK_PENDING -> WORK_RUNNING
2018-08-08T21:51:06.515 GCUCJ [Work DEBUG] scheduling completion of prepare-snapshot
2018-08-08T21:51:06.522 GCUCJ [Work DEBUG] completed prepare-snapshot
2018-08-08T21:51:06.522 GCUCJ [Work DEBUG] work prepare-snapshot : WORK_RUNNING -> WORK_FAILURE_RETRY
2018-08-08T21:51:06.522 GCUCJ [Work WARNING] Scheduling retry #1/4294967295 in 2 sec, for prepare-snapshot
2018-08-08T21:51:08.522 GCUCJ [Work DEBUG] resetting prepare-snapshot
2018-08-08T21:51:08.522 GCUCJ [Work DEBUG] work prepare-snapshot : WORK_FAILURE_RETRY -> WORK_PENDING
2018-08-08T21:51:08.522 GCUCJ [Work DEBUG] advancing prepare-snapshot
2018-08-08T21:51:08.522 GCUCJ [Work DEBUG] all 0 children of prepare-snapshot successful, scheduling run
2018-08-08T21:51:08.522 GCUCJ [Work DEBUG] scheduling run of prepare-snapshot
2018-08-08T21:51:08.523 GCUCJ [Work DEBUG] starting prepare-snapshot
2018-08-08T21:51:08.523 GCUCJ [Work DEBUG] running prepare-snapshot
2018-08-08T21:51:08.523 GCUCJ [Work DEBUG] work prepare-snapshot : WORK_PENDING -> WORK_RUNNING
2018-08-08T21:51:08.523 GCUCJ [Work DEBUG] scheduling completion of prepare-snapshot
2018-08-08T21:51:08.523 GCUCJ [Work DEBUG] completed prepare-snapshot
2018-08-08T21:51:08.523 GCUCJ [Work DEBUG] work prepare-snapshot : WORK_RUNNING -> WORK_FAILURE_RETRY
2018-08-08T21:51:08.523 GCUCJ [Work WARNING] Scheduling retry #2/4294967295 in 3 sec, for prepare-snapshot

2018-08-08T21:51:11.424 GCUCJ [Ledger INFO] Got consensus: [seq=10447168, prev=aabb48, tx_count=3, sv: [  txH: a7bde5, ct: 1533765071, upgrades: [ ] ]]
2018-08-08T21:51:11.425 GCUCJ [Ledger DEBUG] starting closeLedger() on ledgerSeq=10447168
2018-08-08T21:51:11.425 GCUCJ [Ledger DEBUG] processing fees and sequence numbers
2018-08-08T21:51:11.523 GCUCJ [Bucket DEBUG] Adopting bucket file /data/buckets/tmp/bucket-f2a2103581f3f33e/tmp-bucket-79abd01c0797dea1.xdr as /data/buckets/bucket-0e13021de1cd2e970bca44ad243823aff65a4ccc3f3a11e77aadfb31e5582046.xdr
2018-08-08T21:51:11.523 GCUCJ [Bucket DEBUG] Deleting empty bucket file /data/buckets/tmp/bucket-f2a2103581f3f33e/tmp-bucket-2d8a211c33f69e6b.xdr
2018-08-08T21:51:11.523 GCUCJ [Bucket DEBUG] Deleting bucket file /data/buckets/tmp/bucket-f2a2103581f3f33e/tmp-bucket-4e2c0ae14b1dbcc4.xdr that is redundant with existing bucket
2018-08-08T21:51:11.524 GCUCJ [Bucket DEBUG] Adopting bucket file /data/buckets/tmp/bucket-f2a2103581f3f33e/tmp-bucket-294a5f4e19ab114e.xdr as /data/buckets/bucket-7621949361b09ff690f9cf4d799c711df6e84aa8ac18c97105276cc269cf9bb8.xdr
2018-08-08T21:51:11.524 GCUCJ [Bucket DEBUG] Adopting bucket file /data/buckets/tmp/bucket-f2a2103581f3f33e/tmp-bucket-e79881f43069a401.xdr as /data/buckets/bucket-284d8126525ed874363bed8a2e2793e83e8f263b0f3c51dfb7bcd7a167b39101.xdr
2018-08-08T21:51:11.525 GCUCJ [Bucket DEBUG] Deleting bucket file /data/buckets/tmp/bucket-f2a2103581f3f33e/tmp-bucket-52843c0c9eb8304b.xdr that is redundant with existing bucket
2018-08-08T21:51:11.525 GCUCJ [Bucket DEBUG] Adopting bucket file /data/buckets/tmp/bucket-f2a2103581f3f33e/tmp-bucket-ee8157a3a3511e0d.xdr as /data/buckets/bucket-67e750eed914f3766095c66fb140509abeb10b3ec42882515c2491a69ab99044.xdr
2018-08-08T21:51:11.527 GCUCJ [Ledger DEBUG] Advancing LCL: [seq=10447167, hash=aabb48] -> [seq=10447168, hash=bb7569]
2018-08-08T21:51:11.527 GCUCJ [Ledger DEBUG] New current ledger: seq=10447169
2018-08-08T21:51:11.530 GCUCJ [History INFO] Publishing 1 queued checkpoints [10447167-10447167]: Retrying in 0 sec: prepare-snapshot
2018-08-08T21:51:11.530 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 0000000000000000000000000000000000000000000000000000000000000000 referenced by publish queue
2018-08-08T21:51:11.531 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 02c358035b6470656fb425862de1d2eba1f2fdf52254b4f4756062ffefe89b4b referenced by publish queue
2018-08-08T21:51:11.531 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 0ed0b722ee76db5634e32a4e8ad087d7a4ddcdb3fa9a8939533a9dada5d82a10 referenced by publish queue
2018-08-08T21:51:11.531 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 116fe640258b03d0015b20ab2225e83379436bdc9b3306a7affb7abff20f33d5 referenced by publish queue
2018-08-08T21:51:11.531 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 16161d7ddfea5ece936f0c330eadeb60210b9b7348f9e306d6e6806ef9f9b4d6 referenced by publish queue
2018-08-08T21:51:11.531 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 1724d77be46fa62a0ed46b106a3e47af1c0c8d42deac4b8bc98d505755e0437a referenced by publish queue
2018-08-08T21:51:11.531 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 1dd2ec206ea293d1072611d6771673e534f3fa47119189ed43cbdbdf473ae4d6 referenced by publish queue
2018-08-08T21:51:11.531 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 1e1cdc270817cdd95246632fd78d5b482ec559ecae390e2180096f7b58bc77c5 referenced by publish queue
2018-08-08T21:51:11.531 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 205506f667460e78f0b9f7758b3c20fd4a232120ef5b51a44d0f8e44fb842177 referenced by publish queue
2018-08-08T21:51:11.531 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 25975bcc1ea7ee891e91f7a0936c17bf407e9f7c58ba5087fccbebb45c94cfbb referenced by publish queue
2018-08-08T21:51:11.531 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 28f84eace1e9911618dc61739842e598235d90e75f6ea2f0a4cdf907f2bd3128 referenced by publish queue
2018-08-08T21:51:11.531 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 2a6caeeb25bcffd5022e35c16a7d03b541e3a89bf5f48f18e8af614884aa6a90 referenced by publish queue
2018-08-08T21:51:11.531 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 2c6f2ee1ffb097d6cce2cef39538b53d508be370d74f4626d8cf32d2758e1c8f referenced by publish queue
2018-08-08T21:51:11.531 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 2d656019fa60c6a2e45e5df66107ff75ccb82e01d40e5433901476e02168aa08 referenced by publish queue
2018-08-08T21:51:11.531 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 2d931869ff7803e86877a848a083bd7324b59b928411b5d298e839462aeae126 referenced by publish queue
2018-08-08T21:51:11.531 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 327ab7b0a3691f9bc7bde7b46b0b9842ae2c9a5ef15e7992ee11c88f50e08b9d referenced by publish queue
2018-08-08T21:51:11.531 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 38dd5f08b431213a1374b9a4f36ee3ee126b9f62194a7dd196ae88a81560224e referenced by publish queue
2018-08-08T21:51:11.531 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 3c600072de517bc4868d7432182ab3f62920dad3c58e7ba8b196c09810cca4c2 referenced by publish queue
2018-08-08T21:51:11.531 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 3c87b3dfddadceec084ccbd22efb76aefc48425870388ca416777f766e12baec referenced by publish queue
2018-08-08T21:51:11.531 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 3dde3499fd0d8a4297bbbe957759657d98c7d7ca0bc5b4581dd9f079c54e7392 referenced by publish queue
2018-08-08T21:51:11.531 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 46a854adf19cadd698d6c36ccaed78f645889d85ddd076e2e107799fec0172a1 referenced by publish queue
2018-08-08T21:51:11.531 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 4a303763c10813263c0d78d314642544b64fb529e5f4514122edf9986738c7a0 referenced by publish queue
2018-08-08T21:51:11.531 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 4c5fdd8ac9daaaf5a6b4ef0b69da27fa39b7d4e35f503798af54c27b070db671 referenced by publish queue
2018-08-08T21:51:11.531 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 4f09c585b0e314f87ff64f0fca5346ed36d47fb12b802d273740043bf72a51e6 referenced by publish queue
2018-08-08T21:51:11.531 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 53298af6ee9c77541610994a8af96f60f26fe2ca8e995302a3da9911f85092ea referenced by publish queue
2018-08-08T21:51:11.531 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 57257b83f9c5ccb7dd24c56aa7a5101bf9ba0d350e313d985b6ebe5535846a12 referenced by publish queue
2018-08-08T21:51:11.531 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 5794c7922c078248a1f497fbeaa0cde0708b3d6ece98ac59ddb6ecdadb7c9010 referenced by publish queue
2018-08-08T21:51:11.531 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 5e2cc0a2fc6dd474a2aa3baef80cf8fc425723465be6f21489ce730c12ce0548 referenced by publish queue
2018-08-08T21:51:11.531 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 600af102fb6bce6a07470d9a6f476a1b48444e4b41336a518247fba8e609a18d referenced by publish queue
2018-08-08T21:51:11.531 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 6c56152fd52a33e6c1589ca225f5107fd18ecad19c036c23f6f3a7bfa46e42d1 referenced by publish queue
2018-08-08T21:51:11.531 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 6fbb2e480cd258caa7f13a2d74c9306a0a7c12e9c74d926f249d0e78fa8e04a5 referenced by publish queue
2018-08-08T21:51:11.531 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 727e26fc2b8407ea155ef61365eebcb5377e90299f6805a04eba105209511f47 referenced by publish queue
2018-08-08T21:51:11.531 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 85d140cb1c1eb24faa0b10f44d17b9954d24fd4c5d6db3c91adfe7e72d7b4df2 referenced by publish queue
2018-08-08T21:51:11.531 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 8830f702c32ce5e8d493031453fba0ec6be2c1b07e3fc5d0123f9895ecbc5523 referenced by publish queue
2018-08-08T21:51:11.532 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 8ba45f03a99923112657726bb94c01e78a31449997d7cd0b275d999209568ae1 referenced by publish queue
2018-08-08T21:51:11.532 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 8c29ec8ced10027ebce0533e1e852841f94dfb9a9adf5f2ec4aeb44650371332 referenced by publish queue
2018-08-08T21:51:11.532 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 8df1e17d986608744a66130af542f00f8766717a4e551eeed01a5aac0bc19217 referenced by publish queue
2018-08-08T21:51:11.532 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 97a13bc1fefa10aabba48c581bcb78e8f04320245a5f1bec2a027a6cc29603ee referenced by publish queue
2018-08-08T21:51:11.532 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 97ff09d76d73abe10327d030cebcfa79ff61f7a83829b078fe50ec0a5f53b325 referenced by publish queue
2018-08-08T21:51:11.532 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 99740e79206c6969abd62dfa163c6c33df69ca22e6ff4cc95dfb477a600fbfd6 referenced by publish queue
2018-08-08T21:51:11.532 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 9a9066488205fc66520871dfdcdea1edba04ad6fc97734cd5a8aba68cf1ba9cc referenced by publish queue
2018-08-08T21:51:11.532 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: a3066a7178325e7fa220571f54627052ebcee1effce3bad5e5d9c8dfc9fe191e referenced by publish queue
2018-08-08T21:51:11.532 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: a3408247d60479710009e4b9a845fb39d13be9368084a76a270bd4c846652c04 referenced by publish queue
2018-08-08T21:51:11.532 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: a6dc7f0ecc115c107d64b83734c683735a017e22c5ab494a530417ccf34ca97e referenced by publish queue
2018-08-08T21:51:11.532 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: b1d8c6951e16f1a74e1a204753762f9702fa0678e51c863d0b72c9ad70490df5 referenced by publish queue
2018-08-08T21:51:11.532 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: b2fca86b8c2be3f0ef49e7eaba08621f6696c00f2a881caee72f616cbba25099 referenced by publish queue
2018-08-08T21:51:11.532 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: b66a52f7759066253f3f14eafc4ae5fc3f41acadf1890500fdaaa4eda6dd4917 referenced by publish queue
2018-08-08T21:51:11.532 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: c06f26ce5d478ddf502899df9ce622c715591a7d102ec38983e2e42865df080a referenced by publish queue
2018-08-08T21:51:11.532 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: c286e9fab913bca9f93038f2312b6566fc7f4bb56798817c9234e0ce6a06380d referenced by publish queue
2018-08-08T21:51:11.532 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: cdb27b9bb3ff34873b5432c6b5202045a7a1a0b40c74ab27100315933bed4f06 referenced by publish queue
2018-08-08T21:51:11.532 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: d8511a783335d7a8841ae48b609b3d3992c67abe3a32b04a43c5fc6e55ffed1c referenced by publish queue
2018-08-08T21:51:11.532 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: d9ee859667458d7c1fe40bcab7d70a5ef51e71bc93d0018c784cd351f48cc086 referenced by publish queue
2018-08-08T21:51:11.532 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: e10cbeba9c8b9e94620436c819cb60f2fd56910f6c158e6e1700013662b7bb06 referenced by publish queue
2018-08-08T21:51:11.532 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: e1e15b7b94e3523f2ec7e122761d2c6861804978e428db4ee9912e1a6be488d1 referenced by publish queue
2018-08-08T21:51:11.532 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: ea200e569383562739e58ccc653d80fb52b8ccfaae04b57d7d2da589088f00f7 referenced by publish queue
2018-08-08T21:51:11.532 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: ec98a9992a6ba53b76d0075d658ae40c90a476417acb13f0e84a6d60be6d368e referenced by publish queue
2018-08-08T21:51:11.532 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: f5d738d89601828e209a3117d5b88067ea451264f085ebae8cc2a09a1c6fd613 referenced by publish queue
2018-08-08T21:51:11.532 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: f64da3c35f82d9672a7253907e60371c675253f8c54bcc858d51681f9b746185 referenced by publish queue
2018-08-08T21:51:11.533 GCUCJ [Ledger INFO] Closed ledger: [seq=10447168, hash=bb7569]
2018-08-08T21:51:11.536 GCUCJ [Work DEBUG] resetting prepare-snapshot
2018-08-08T21:51:11.536 GCUCJ [Work DEBUG] work prepare-snapshot : WORK_FAILURE_RETRY -> WORK_PENDING
2018-08-08T21:51:11.537 GCUCJ [Work DEBUG] advancing prepare-snapshot
2018-08-08T21:51:11.537 GCUCJ [Work DEBUG] all 0 children of prepare-snapshot successful, scheduling run
2018-08-08T21:51:11.537 GCUCJ [Work DEBUG] scheduling run of prepare-snapshot
2018-08-08T21:51:11.540 GCUCJ [Work DEBUG] starting prepare-snapshot
2018-08-08T21:51:11.540 GCUCJ [Work DEBUG] running prepare-snapshot
2018-08-08T21:51:11.540 GCUCJ [Work DEBUG] work prepare-snapshot : WORK_PENDING -> WORK_RUNNING
2018-08-08T21:51:11.540 GCUCJ [Work DEBUG] scheduling completion of prepare-snapshot
2018-08-08T21:51:11.558 GCUCJ [Work DEBUG] completed prepare-snapshot
2018-08-08T21:51:11.558 GCUCJ [Work DEBUG] work prepare-snapshot : WORK_RUNNING -> WORK_SUCCESS
2018-08-08T21:51:11.558 GCUCJ [Work DEBUG] notifying parent of successful prepare-snapshot
2018-08-08T21:51:11.558 GCUCJ [Work DEBUG] notified publish-009f693f of completed child prepare-snapshot
2018-08-08T21:51:11.558 GCUCJ [Work DEBUG] advancing publish-009f693f
2018-08-08T21:51:11.558 GCUCJ [Work DEBUG] all 1 children of publish-009f693f successful, scheduling run
2018-08-08T21:51:11.558 GCUCJ [Work DEBUG] scheduling run of publish-009f693f
2018-08-08T21:51:11.562 GCUCJ [Work DEBUG] starting publish-009f693f
2018-08-08T21:51:11.562 GCUCJ [Work DEBUG] running publish-009f693f
2018-08-08T21:51:11.562 GCUCJ [Work DEBUG] work publish-009f693f : WORK_PENDING -> WORK_RUNNING
2018-08-08T21:51:11.562 GCUCJ [Work DEBUG] scheduling completion of publish-009f693f
2018-08-08T21:51:11.578 GCUCJ [Work DEBUG] completed publish-009f693f
2018-08-08T21:51:11.578 GCUCJ [Work DEBUG] resetting write-snapshot
2018-08-08T21:51:11.578 GCUCJ [Work DEBUG] work publish-009f693f : WORK_RUNNING -> WORK_PENDING
2018-08-08T21:51:11.578 GCUCJ [Work DEBUG] advancing publish-009f693f
2018-08-08T21:51:11.578 GCUCJ [Work DEBUG] advancing write-snapshot
2018-08-08T21:51:11.578 GCUCJ [Work DEBUG] all 0 children of write-snapshot successful, scheduling run
2018-08-08T21:51:11.578 GCUCJ [Work DEBUG] scheduling run of write-snapshot
2018-08-08T21:51:11.583 GCUCJ [Work DEBUG] starting write-snapshot
2018-08-08T21:51:11.584 GCUCJ [History DEBUG] Streaming 64 ledgers worth of history, from 10447104
2018-08-08T21:51:11.584 GCUCJ [Work DEBUG] running write-snapshot
2018-08-08T21:51:11.584 GCUCJ [Work DEBUG] work write-snapshot : WORK_PENDING -> WORK_RUNNING
2018-08-08T21:51:11.585 GCUCJ [Ledger DEBUG] Streaming ledger-header 10447104
2018-08-08T21:51:11.585 GCUCJ [Ledger DEBUG] Streaming ledger-header 10447105
2018-08-08T21:51:11.585 GCUCJ [Ledger DEBUG] Streaming ledger-header 10447106
2018-08-08T21:51:11.585 GCUCJ [Ledger DEBUG] Streaming ledger-header 10447107
2018-08-08T21:51:11.586 GCUCJ [Ledger DEBUG] Streaming ledger-header 10447108
2018-08-08T21:51:11.586 GCUCJ [Ledger DEBUG] Streaming ledger-header 10447109
2018-08-08T21:51:11.586 GCUCJ [Ledger DEBUG] Streaming ledger-header 10447110
2018-08-08T21:51:11.586 GCUCJ [Ledger DEBUG] Streaming ledger-header 10447111
2018-08-08T21:51:11.586 GCUCJ [Ledger DEBUG] Streaming ledger-header 10447112
2018-08-08T21:51:11.586 GCUCJ [Ledger DEBUG] Streaming ledger-header 10447113
2018-08-08T21:51:11.586 GCUCJ [Ledger DEBUG] Streaming ledger-header 10447114
2018-08-08T21:51:11.586 GCUCJ [Ledger DEBUG] Streaming ledger-header 10447115
2018-08-08T21:51:11.587 GCUCJ [Ledger DEBUG] Streaming ledger-header 10447116
2018-08-08T21:51:11.587 GCUCJ [Ledger DEBUG] Streaming ledger-header 10447117
2018-08-08T21:51:11.587 GCUCJ [Ledger DEBUG] Streaming ledger-header 10447118
2018-08-08T21:51:11.587 GCUCJ [Ledger DEBUG] Streaming ledger-header 10447119
2018-08-08T21:51:11.587 GCUCJ [Ledger DEBUG] Streaming ledger-header 10447120
2018-08-08T21:51:11.587 GCUCJ [Ledger DEBUG] Streaming ledger-header 10447121
2018-08-08T21:51:11.588 GCUCJ [Ledger DEBUG] Streaming ledger-header 10447122
2018-08-08T21:51:11.588 GCUCJ [Ledger DEBUG] Streaming ledger-header 10447123
2018-08-08T21:51:11.588 GCUCJ [Ledger DEBUG] Streaming ledger-header 10447124
2018-08-08T21:51:11.588 GCUCJ [Ledger DEBUG] Streaming ledger-header 10447125
2018-08-08T21:51:11.588 GCUCJ [Ledger DEBUG] Streaming ledger-header 10447126
2018-08-08T21:51:11.588 GCUCJ [Ledger DEBUG] Streaming ledger-header 10447127
2018-08-08T21:51:11.588 GCUCJ [Ledger DEBUG] Streaming ledger-header 10447128
2018-08-08T21:51:11.589 GCUCJ [Ledger DEBUG] Streaming ledger-header 10447129
2018-08-08T21:51:11.589 GCUCJ [Ledger DEBUG] Streaming ledger-header 10447130
2018-08-08T21:51:11.589 GCUCJ [Ledger DEBUG] Streaming ledger-header 10447131
2018-08-08T21:51:11.589 GCUCJ [Ledger DEBUG] Streaming ledger-header 10447132
2018-08-08T21:51:11.589 GCUCJ [Ledger DEBUG] Streaming ledger-header 10447133
2018-08-08T21:51:11.589 GCUCJ [Ledger DEBUG] Streaming ledger-header 10447134
2018-08-08T21:51:11.589 GCUCJ [Ledger DEBUG] Streaming ledger-header 10447135
2018-08-08T21:51:11.589 GCUCJ [Ledger DEBUG] Streaming ledger-header 10447136
2018-08-08T21:51:11.589 GCUCJ [Ledger DEBUG] Streaming ledger-header 10447137
2018-08-08T21:51:11.589 GCUCJ [Ledger DEBUG] Streaming ledger-header 10447138
2018-08-08T21:51:11.589 GCUCJ [Ledger DEBUG] Streaming ledger-header 10447139
2018-08-08T21:51:11.590 GCUCJ [Ledger DEBUG] Streaming ledger-header 10447140
2018-08-08T21:51:11.590 GCUCJ [Ledger DEBUG] Streaming ledger-header 10447141
2018-08-08T21:51:11.590 GCUCJ [Ledger DEBUG] Streaming ledger-header 10447142
2018-08-08T21:51:11.590 GCUCJ [Ledger DEBUG] Streaming ledger-header 10447143
2018-08-08T21:51:11.590 GCUCJ [Ledger DEBUG] Streaming ledger-header 10447144
2018-08-08T21:51:11.590 GCUCJ [Ledger DEBUG] Streaming ledger-header 10447145
2018-08-08T21:51:11.590 GCUCJ [Ledger DEBUG] Streaming ledger-header 10447146
2018-08-08T21:51:11.590 GCUCJ [Ledger DEBUG] Streaming ledger-header 10447147
2018-08-08T21:51:11.591 GCUCJ [Ledger DEBUG] Streaming ledger-header 10447148
2018-08-08T21:51:11.591 GCUCJ [Ledger DEBUG] Streaming ledger-header 10447149
2018-08-08T21:51:11.591 GCUCJ [Ledger DEBUG] Streaming ledger-header 10447150
2018-08-08T21:51:11.591 GCUCJ [Ledger DEBUG] Streaming ledger-header 10447151
2018-08-08T21:51:11.591 GCUCJ [Ledger DEBUG] Streaming ledger-header 10447152
2018-08-08T21:51:11.591 GCUCJ [Ledger DEBUG] Streaming ledger-header 10447153
2018-08-08T21:51:11.591 GCUCJ [Ledger DEBUG] Streaming ledger-header 10447154
2018-08-08T21:51:11.591 GCUCJ [Ledger DEBUG] Streaming ledger-header 10447155
2018-08-08T21:51:11.591 GCUCJ [Ledger DEBUG] Streaming ledger-header 10447156
2018-08-08T21:51:11.592 GCUCJ [Ledger DEBUG] Streaming ledger-header 10447157
2018-08-08T21:51:11.592 GCUCJ [Ledger DEBUG] Streaming ledger-header 10447158
2018-08-08T21:51:11.592 GCUCJ [Ledger DEBUG] Streaming ledger-header 10447159
2018-08-08T21:51:11.592 GCUCJ [Ledger DEBUG] Streaming ledger-header 10447160
2018-08-08T21:51:11.592 GCUCJ [Ledger DEBUG] Streaming ledger-header 10447161
2018-08-08T21:51:11.592 GCUCJ [Ledger DEBUG] Streaming ledger-header 10447162
2018-08-08T21:51:11.592 GCUCJ [Ledger DEBUG] Streaming ledger-header 10447163
2018-08-08T21:51:11.592 GCUCJ [Ledger DEBUG] Streaming ledger-header 10447164
2018-08-08T21:51:11.592 GCUCJ [Ledger DEBUG] Streaming ledger-header 10447165
2018-08-08T21:51:11.593 GCUCJ [Ledger DEBUG] Streaming ledger-header 10447166
2018-08-08T21:51:11.593 GCUCJ [Ledger DEBUG] Streaming ledger-header 10447167
2018-08-08T21:51:11.626 GCUCJ [History DEBUG] Wrote 64 ledger headers to /data/buckets/tmp/snapshot-ec421db5b63f34bc/ledger-009f693f.xdr
2018-08-08T21:51:11.626 GCUCJ [History DEBUG] Wrote 86 transactions to /data/buckets/tmp/snapshot-ec421db5b63f34bc/transactions-009f693f.xdr and /data/buckets/tmp/snapshot-ec421db5b63f34bc/results-009f693f.xdr
2018-08-08T21:51:11.761 GCUCJ [Work DEBUG] completed write-snapshot
2018-08-08T21:51:11.761 GCUCJ [Work DEBUG] work write-snapshot : WORK_RUNNING -> WORK_SUCCESS
2018-08-08T21:51:11.761 GCUCJ [Work DEBUG] notifying parent of successful write-snapshot
2018-08-08T21:51:11.761 GCUCJ [Work DEBUG] notified publish-009f693f of completed child write-snapshot
2018-08-08T21:51:11.761 GCUCJ [Work DEBUG] advancing publish-009f693f
2018-08-08T21:51:11.761 GCUCJ [Work DEBUG] all 2 children of publish-009f693f successful, scheduling run
2018-08-08T21:51:11.762 GCUCJ [Work DEBUG] scheduling run of publish-009f693f
2018-08-08T21:51:11.762 GCUCJ [Work DEBUG] starting publish-009f693f
2018-08-08T21:51:11.762 GCUCJ [Work DEBUG] running publish-009f693f
2018-08-08T21:51:11.762 GCUCJ [Work DEBUG] work publish-009f693f : WORK_PENDING -> WORK_RUNNING
2018-08-08T21:51:11.762 GCUCJ [Work DEBUG] scheduling completion of publish-009f693f
2018-08-08T21:51:11.762 GCUCJ [Work DEBUG] completed publish-009f693f
2018-08-08T21:51:11.762 GCUCJ [Work DEBUG] resetting update-archives
2018-08-08T21:51:11.763 GCUCJ [Work DEBUG] resetting put-snapshot-files-core_testnet_002
2018-08-08T21:51:11.763 GCUCJ [Work DEBUG] work publish-009f693f : WORK_RUNNING -> WORK_PENDING
2018-08-08T21:51:11.763 GCUCJ [Work DEBUG] advancing publish-009f693f
2018-08-08T21:51:11.763 GCUCJ [Work DEBUG] advancing update-archives
2018-08-08T21:51:11.763 GCUCJ [Work DEBUG] advancing put-snapshot-files-core_testnet_002
2018-08-08T21:51:11.763 GCUCJ [Work DEBUG] all 0 children of put-snapshot-files-core_testnet_002 successful, scheduling run
2018-08-08T21:51:11.763 GCUCJ [Work DEBUG] scheduling run of put-snapshot-files-core_testnet_002
2018-08-08T21:51:11.763 GCUCJ [Work DEBUG] starting put-snapshot-files-core_testnet_002
2018-08-08T21:51:11.763 GCUCJ [Work DEBUG] running put-snapshot-files-core_testnet_002
2018-08-08T21:51:11.763 GCUCJ [Work DEBUG] work put-snapshot-files-core_testnet_002 : WORK_PENDING -> WORK_RUNNING
2018-08-08T21:51:11.763 GCUCJ [Work DEBUG] scheduling completion of put-snapshot-files-core_testnet_002
2018-08-08T21:51:11.763 GCUCJ [Work DEBUG] completed put-snapshot-files-core_testnet_002
2018-08-08T21:51:11.764 GCUCJ [Work DEBUG] resetting get-history-archive-state
2018-08-08T21:51:11.764 GCUCJ [Work DEBUG] resetting get-remote-file .well-known/stellar-history.json
2018-08-08T21:51:11.764 GCUCJ [Work DEBUG] work put-snapshot-files-core_testnet_002 : WORK_RUNNING -> WORK_PENDING
2018-08-08T21:51:11.764 GCUCJ [Work DEBUG] advancing put-snapshot-files-core_testnet_002
2018-08-08T21:51:11.764 GCUCJ [Work DEBUG] advancing get-history-archive-state
2018-08-08T21:51:11.764 GCUCJ [Work DEBUG] advancing get-remote-file .well-known/stellar-history.json
2018-08-08T21:51:11.764 GCUCJ [Work DEBUG] all 0 children of get-remote-file .well-known/stellar-history.json successful, scheduling run
2018-08-08T21:51:11.764 GCUCJ [Work DEBUG] scheduling run of get-remote-file .well-known/stellar-history.json
2018-08-08T21:51:11.764 GCUCJ [Work DEBUG] starting get-remote-file .well-known/stellar-history.json
2018-08-08T21:51:11.764 GCUCJ [Process DEBUG] Running: aws s3 cp --region eu-west-1 s3://history.stellar.org/prd/core-testnet/core_testnet_002/.well-known/stellar-history.json /data/buckets/tmp/history-6464b9088928f47c/core_testnet_002-stellar-history.json
2018-08-08T21:51:11.958 GCUCJ [Work DEBUG] running get-remote-file .well-known/stellar-history.json
2018-08-08T21:51:11.958 GCUCJ [Work DEBUG] work get-remote-file .well-known/stellar-history.json : WORK_PENDING -> WORK_RUNNING
Completed 3.8 KiB/3.8 KiB (10.8 KiB/s) with 1 file(s) remaining
download: s3://history.stellar.org/prd/core-testnet/core_testnet_002/.well-known/stellar-history.json to data/buckets/tmp/history-6464b9088928f47c/core_testnet_002-stellar-history.json
2018-08-08T21:51:13.186 GCUCJ [Process DEBUG] found 1 child processes that terminated
2018-08-08T21:51:13.186 GCUCJ [Process DEBUG] process 5244 exited 0: aws s3 cp --region eu-west-1 s3://history.stellar.org/prd/core-testnet/core_testnet_002/.well-known/stellar-history.json /data/buckets/tmp/history-6464b9088928f47c/core_testnet_002-stellar-history.json
2018-08-08T21:51:13.186 GCUCJ [Work DEBUG] completed get-remote-file .well-known/stellar-history.json
2018-08-08T21:51:13.186 GCUCJ [Work DEBUG] work get-remote-file .well-known/stellar-history.json : WORK_RUNNING -> WORK_SUCCESS
2018-08-08T21:51:13.186 GCUCJ [Work DEBUG] notifying parent of successful get-remote-file .well-known/stellar-history.json
2018-08-08T21:51:13.186 GCUCJ [Work DEBUG] notified get-history-archive-state of completed child get-remote-file .well-known/stellar-history.json
2018-08-08T21:51:13.186 GCUCJ [Work DEBUG] advancing get-history-archive-state
2018-08-08T21:51:13.186 GCUCJ [Work DEBUG] all 1 children of get-history-archive-state successful, scheduling run
2018-08-08T21:51:13.186 GCUCJ [Work DEBUG] scheduling run of get-history-archive-state
2018-08-08T21:51:13.186 GCUCJ [Work DEBUG] starting get-history-archive-state
2018-08-08T21:51:13.186 GCUCJ [Work DEBUG] running get-history-archive-state
2018-08-08T21:51:13.186 GCUCJ [Work DEBUG] work get-history-archive-state : WORK_PENDING -> WORK_RUNNING
2018-08-08T21:51:13.187 GCUCJ [Work DEBUG] scheduling completion of get-history-archive-state
2018-08-08T21:51:13.187 GCUCJ [Work DEBUG] completed get-history-archive-state
2018-08-08T21:51:13.187 GCUCJ [Work DEBUG] work get-history-archive-state : WORK_RUNNING -> WORK_SUCCESS
2018-08-08T21:51:13.187 GCUCJ [Work DEBUG] notifying parent of successful get-history-archive-state
2018-08-08T21:51:13.187 GCUCJ [Work DEBUG] notified put-snapshot-files-core_testnet_002 of completed child get-history-archive-state
2018-08-08T21:51:13.187 GCUCJ [Work DEBUG] advancing put-snapshot-files-core_testnet_002
2018-08-08T21:51:13.187 GCUCJ [Work DEBUG] all 1 children of put-snapshot-files-core_testnet_002 successful, scheduling run
2018-08-08T21:51:13.187 GCUCJ [Work DEBUG] scheduling run of put-snapshot-files-core_testnet_002
2018-08-08T21:51:13.187 GCUCJ [Work DEBUG] starting put-snapshot-files-core_testnet_002
2018-08-08T21:51:13.187 GCUCJ [Work DEBUG] running put-snapshot-files-core_testnet_002
2018-08-08T21:51:13.188 GCUCJ [Work DEBUG] work put-snapshot-files-core_testnet_002 : WORK_PENDING -> WORK_RUNNING
2018-08-08T21:51:13.188 GCUCJ [Work DEBUG] scheduling completion of put-snapshot-files-core_testnet_002
2018-08-08T21:51:13.188 GCUCJ [Work DEBUG] completed put-snapshot-files-core_testnet_002
2018-08-08T21:51:13.188 GCUCJ [Work DEBUG] resetting put-files
2018-08-08T21:51:13.189 GCUCJ [Work DEBUG] resetting put-remote-file ledger/00/9f/69/ledger-009f693f.xdr.gz
2018-08-08T21:51:13.189 GCUCJ [Work DEBUG] resetting make-remote-dir ledger/00/9f/69
2018-08-08T21:51:13.189 GCUCJ [Work DEBUG] resetting gzip-file /data/buckets/tmp/snapshot-ec421db5b63f34bc/ledger-009f693f.xdr
2018-08-08T21:51:13.190 GCUCJ [Work DEBUG] resetting put-remote-file transactions/00/9f/69/transactions-009f693f.xdr.gz
2018-08-08T21:51:13.190 GCUCJ [Work DEBUG] resetting make-remote-dir transactions/00/9f/69
2018-08-08T21:51:13.190 GCUCJ [Work DEBUG] resetting gzip-file /data/buckets/tmp/snapshot-ec421db5b63f34bc/transactions-009f693f.xdr
2018-08-08T21:51:13.191 GCUCJ [Work DEBUG] resetting put-remote-file results/00/9f/69/results-009f693f.xdr.gz
2018-08-08T21:51:13.191 GCUCJ [Work DEBUG] resetting make-remote-dir results/00/9f/69
2018-08-08T21:51:13.191 GCUCJ [Work DEBUG] resetting gzip-file /data/buckets/tmp/snapshot-ec421db5b63f34bc/results-009f693f.xdr
2018-08-08T21:51:13.192 GCUCJ [Work DEBUG] resetting put-remote-file scp/00/9f/69/scp-009f693f.xdr.gz
2018-08-08T21:51:13.192 GCUCJ [Work DEBUG] resetting make-remote-dir scp/00/9f/69
2018-08-08T21:51:13.192 GCUCJ [Work DEBUG] resetting gzip-file /data/buckets/tmp/snapshot-ec421db5b63f34bc/scp-009f693f.xdr
2018-08-08T21:51:13.193 GCUCJ [Work DEBUG] resetting put-remote-file bucket/0c/f7/26/bucket-0cf726fafa2ba7b3da004fc0c3fbe723adcfc204f7fd01854cf97d86d2b3ea11.xdr.gz
2018-08-08T21:51:13.193 GCUCJ [Work DEBUG] resetting make-remote-dir bucket/0c/f7/26
2018-08-08T21:51:13.193 GCUCJ [Work DEBUG] resetting gzip-file /data/buckets/bucket-0cf726fafa2ba7b3da004fc0c3fbe723adcfc204f7fd01854cf97d86d2b3ea11.xdr
2018-08-08T21:51:13.194 GCUCJ [Work DEBUG] resetting put-remote-file bucket/36/90/5c/bucket-36905c16c0d345923c5fa8d0adc28fe03c1e47050e51579075064f159e7f16bd.xdr.gz
2018-08-08T21:51:13.195 GCUCJ [Work DEBUG] resetting make-remote-dir bucket/36/90/5c
2018-08-08T21:51:13.195 GCUCJ [Work DEBUG] resetting gzip-file /data/buckets/bucket-36905c16c0d345923c5fa8d0adc28fe03c1e47050e51579075064f159e7f16bd.xdr
2018-08-08T21:51:13.195 GCUCJ [Work DEBUG] resetting put-remote-file bucket/72/7e/26/bucket-727e26fc2b8407ea155ef61365eebcb5377e90299f6805a04eba105209511f47.xdr.gz
2018-08-08T21:51:13.196 GCUCJ [Work DEBUG] resetting make-remote-dir bucket/72/7e/26
2018-08-08T21:51:13.196 GCUCJ [Work DEBUG] resetting gzip-file /data/buckets/bucket-727e26fc2b8407ea155ef61365eebcb5377e90299f6805a04eba105209511f47.xdr
2018-08-08T21:51:13.196 GCUCJ [Work DEBUG] resetting put-remote-file bucket/8d/f1/e1/bucket-8df1e17d986608744a66130af542f00f8766717a4e551eeed01a5aac0bc19217.xdr.gz
2018-08-08T21:51:13.197 GCUCJ [Work DEBUG] resetting make-remote-dir bucket/8d/f1/e1
2018-08-08T21:51:13.197 GCUCJ [Work DEBUG] resetting gzip-file /data/buckets/bucket-8df1e17d986608744a66130af542f00f8766717a4e551eeed01a5aac0bc19217.xdr
2018-08-08T21:51:13.197 GCUCJ [Work DEBUG] resetting put-remote-file bucket/19/01/ee/bucket-1901eea7636ae2363e6989325da37a84c2f038ce9b94f6484a86cff37ca82217.xdr.gz
2018-08-08T21:51:13.198 GCUCJ [Work DEBUG] resetting make-remote-dir bucket/19/01/ee
2018-08-08T21:51:13.198 GCUCJ [Work DEBUG] resetting gzip-file /data/buckets/bucket-1901eea7636ae2363e6989325da37a84c2f038ce9b94f6484a86cff37ca82217.xdr
2018-08-08T21:51:13.198 GCUCJ [Work DEBUG] resetting put-remote-file bucket/3c/87/b3/bucket-3c87b3dfddadceec084ccbd22efb76aefc48425870388ca416777f766e12baec.xdr.gz
2018-08-08T21:51:13.199 GCUCJ [Work DEBUG] resetting make-remote-dir bucket/3c/87/b3
2018-08-08T21:51:13.199 GCUCJ [Work DEBUG] resetting gzip-file /data/buckets/bucket-3c87b3dfddadceec084ccbd22efb76aefc48425870388ca416777f766e12baec.xdr
2018-08-08T21:51:13.199 GCUCJ [Work DEBUG] resetting put-remote-file bucket/a6/dc/7f/bucket-a6dc7f0ecc115c107d64b83734c683735a017e22c5ab494a530417ccf34ca97e.xdr.gz
2018-08-08T21:51:13.200 GCUCJ [Work DEBUG] resetting make-remote-dir bucket/a6/dc/7f
2018-08-08T21:51:13.200 GCUCJ [Work DEBUG] resetting gzip-file /data/buckets/bucket-a6dc7f0ecc115c107d64b83734c683735a017e22c5ab494a530417ccf34ca97e.xdr
2018-08-08T21:51:13.200 GCUCJ [Work DEBUG] resetting put-remote-file bucket/88/30/f7/bucket-8830f702c32ce5e8d493031453fba0ec6be2c1b07e3fc5d0123f9895ecbc5523.xdr.gz
2018-08-08T21:51:13.201 GCUCJ [Work DEBUG] resetting make-remote-dir bucket/88/30/f7
2018-08-08T21:51:13.201 GCUCJ [Work DEBUG] resetting gzip-file /data/buckets/bucket-8830f702c32ce5e8d493031453fba0ec6be2c1b07e3fc5d0123f9895ecbc5523.xdr
2018-08-08T21:51:13.201 GCUCJ [Work DEBUG] resetting put-remote-file bucket/17/24/d7/bucket-1724d77be46fa62a0ed46b106a3e47af1c0c8d42deac4b8bc98d505755e0437a.xdr.gz
2018-08-08T21:51:13.202 GCUCJ [Work DEBUG] resetting make-remote-dir bucket/17/24/d7
2018-08-08T21:51:13.202 GCUCJ [Work DEBUG] resetting gzip-file /data/buckets/bucket-1724d77be46fa62a0ed46b106a3e47af1c0c8d42deac4b8bc98d505755e0437a.xdr
2018-08-08T21:51:13.203 GCUCJ [Work DEBUG] resetting put-remote-file bucket/a3/06/6a/bucket-a3066a7178325e7fa220571f54627052ebcee1effce3bad5e5d9c8dfc9fe191e.xdr.gz
2018-08-08T21:51:13.203 GCUCJ [Work DEBUG] resetting make-remote-dir bucket/a3/06/6a
2018-08-08T21:51:13.203 GCUCJ [Work DEBUG] resetting gzip-file /data/buckets/bucket-a3066a7178325e7fa220571f54627052ebcee1effce3bad5e5d9c8dfc9fe191e.xdr
2018-08-08T21:51:13.203 GCUCJ [Work DEBUG] work put-snapshot-files-core_testnet_002 : WORK_RUNNING -> WORK_PENDING
2018-08-08T21:51:13.203 GCUCJ [Work DEBUG] advancing put-snapshot-files-core_testnet_002
2018-08-08T21:51:13.203 GCUCJ [Work DEBUG] advancing put-files
2018-08-08T21:51:13.203 GCUCJ [Work DEBUG] advancing put-remote-file bucket/0c/f7/26/bucket-0cf726fafa2ba7b3da004fc0c3fbe723adcfc204f7fd01854cf97d86d2b3ea11.xdr.gz
2018-08-08T21:51:13.203 GCUCJ [Work DEBUG] advancing make-remote-dir bucket/0c/f7/26
2018-08-08T21:51:13.203 GCUCJ [Work DEBUG] advancing gzip-file /data/buckets/bucket-0cf726fafa2ba7b3da004fc0c3fbe723adcfc204f7fd01854cf97d86d2b3ea11.xdr
2018-08-08T21:51:13.203 GCUCJ [Work DEBUG] all 0 children of gzip-file /data/buckets/bucket-0cf726fafa2ba7b3da004fc0c3fbe723adcfc204f7fd01854cf97d86d2b3ea11.xdr successful, scheduling run
2018-08-08T21:51:13.203 GCUCJ [Work DEBUG] scheduling run of gzip-file /data/buckets/bucket-0cf726fafa2ba7b3da004fc0c3fbe723adcfc204f7fd01854cf97d86d2b3ea11.xdr
2018-08-08T21:51:13.203 GCUCJ [Work DEBUG] advancing put-remote-file bucket/17/24/d7/bucket-1724d77be46fa62a0ed46b106a3e47af1c0c8d42deac4b8bc98d505755e0437a.xdr.gz
2018-08-08T21:51:13.203 GCUCJ [Work DEBUG] advancing make-remote-dir bucket/17/24/d7
2018-08-08T21:51:13.203 GCUCJ [Work DEBUG] advancing gzip-file /data/buckets/bucket-1724d77be46fa62a0ed46b106a3e47af1c0c8d42deac4b8bc98d505755e0437a.xdr
2018-08-08T21:51:13.203 GCUCJ [Work DEBUG] all 0 children of gzip-file /data/buckets/bucket-1724d77be46fa62a0ed46b106a3e47af1c0c8d42deac4b8bc98d505755e0437a.xdr successful, scheduling run
2018-08-08T21:51:13.203 GCUCJ [Work DEBUG] scheduling run of gzip-file /data/buckets/bucket-1724d77be46fa62a0ed46b106a3e47af1c0c8d42deac4b8bc98d505755e0437a.xdr
2018-08-08T21:51:13.203 GCUCJ [Work DEBUG] advancing put-remote-file bucket/19/01/ee/bucket-1901eea7636ae2363e6989325da37a84c2f038ce9b94f6484a86cff37ca82217.xdr.gz
2018-08-08T21:51:13.203 GCUCJ [Work DEBUG] advancing make-remote-dir bucket/19/01/ee
2018-08-08T21:51:13.203 GCUCJ [Work DEBUG] advancing gzip-file /data/buckets/bucket-1901eea7636ae2363e6989325da37a84c2f038ce9b94f6484a86cff37ca82217.xdr
2018-08-08T21:51:13.203 GCUCJ [Work DEBUG] all 0 children of gzip-file /data/buckets/bucket-1901eea7636ae2363e6989325da37a84c2f038ce9b94f6484a86cff37ca82217.xdr successful, scheduling run
2018-08-08T21:51:13.203 GCUCJ [Work DEBUG] scheduling run of gzip-file /data/buckets/bucket-1901eea7636ae2363e6989325da37a84c2f038ce9b94f6484a86cff37ca82217.xdr
2018-08-08T21:51:13.203 GCUCJ [Work DEBUG] advancing put-remote-file bucket/36/90/5c/bucket-36905c16c0d345923c5fa8d0adc28fe03c1e47050e51579075064f159e7f16bd.xdr.gz
2018-08-08T21:51:13.203 GCUCJ [Work DEBUG] advancing make-remote-dir bucket/36/90/5c
2018-08-08T21:51:13.203 GCUCJ [Work DEBUG] advancing gzip-file /data/buckets/bucket-36905c16c0d345923c5fa8d0adc28fe03c1e47050e51579075064f159e7f16bd.xdr
2018-08-08T21:51:13.203 GCUCJ [Work DEBUG] all 0 children of gzip-file /data/buckets/bucket-36905c16c0d345923c5fa8d0adc28fe03c1e47050e51579075064f159e7f16bd.xdr successful, scheduling run
2018-08-08T21:51:13.203 GCUCJ [Work DEBUG] scheduling run of gzip-file /data/buckets/bucket-36905c16c0d345923c5fa8d0adc28fe03c1e47050e51579075064f159e7f16bd.xdr
2018-08-08T21:51:13.203 GCUCJ [Work DEBUG] advancing put-remote-file bucket/3c/87/b3/bucket-3c87b3dfddadceec084ccbd22efb76aefc48425870388ca416777f766e12baec.xdr.gz
2018-08-08T21:51:13.203 GCUCJ [Work DEBUG] advancing make-remote-dir bucket/3c/87/b3
2018-08-08T21:51:13.203 GCUCJ [Work DEBUG] advancing gzip-file /data/buckets/bucket-3c87b3dfddadceec084ccbd22efb76aefc48425870388ca416777f766e12baec.xdr
2018-08-08T21:51:13.203 GCUCJ [Work DEBUG] all 0 children of gzip-file /data/buckets/bucket-3c87b3dfddadceec084ccbd22efb76aefc48425870388ca416777f766e12baec.xdr successful, scheduling run
2018-08-08T21:51:13.203 GCUCJ [Work DEBUG] scheduling run of gzip-file /data/buckets/bucket-3c87b3dfddadceec084ccbd22efb76aefc48425870388ca416777f766e12baec.xdr
2018-08-08T21:51:13.203 GCUCJ [Work DEBUG] advancing put-remote-file bucket/72/7e/26/bucket-727e26fc2b8407ea155ef61365eebcb5377e90299f6805a04eba105209511f47.xdr.gz
2018-08-08T21:51:13.203 GCUCJ [Work DEBUG] advancing make-remote-dir bucket/72/7e/26
2018-08-08T21:51:13.204 GCUCJ [Work DEBUG] advancing gzip-file /data/buckets/bucket-727e26fc2b8407ea155ef61365eebcb5377e90299f6805a04eba105209511f47.xdr
2018-08-08T21:51:13.204 GCUCJ [Work DEBUG] all 0 children of gzip-file /data/buckets/bucket-727e26fc2b8407ea155ef61365eebcb5377e90299f6805a04eba105209511f47.xdr successful, scheduling run
2018-08-08T21:51:13.204 GCUCJ [Work DEBUG] scheduling run of gzip-file /data/buckets/bucket-727e26fc2b8407ea155ef61365eebcb5377e90299f6805a04eba105209511f47.xdr
2018-08-08T21:51:13.204 GCUCJ [Work DEBUG] advancing put-remote-file bucket/88/30/f7/bucket-8830f702c32ce5e8d493031453fba0ec6be2c1b07e3fc5d0123f9895ecbc5523.xdr.gz
2018-08-08T21:51:13.204 GCUCJ [Work DEBUG] advancing make-remote-dir bucket/88/30/f7
2018-08-08T21:51:13.204 GCUCJ [Work DEBUG] advancing gzip-file /data/buckets/bucket-8830f702c32ce5e8d493031453fba0ec6be2c1b07e3fc5d0123f9895ecbc5523.xdr
2018-08-08T21:51:13.204 GCUCJ [Work DEBUG] all 0 children of gzip-file /data/buckets/bucket-8830f702c32ce5e8d493031453fba0ec6be2c1b07e3fc5d0123f9895ecbc5523.xdr successful, scheduling run
2018-08-08T21:51:13.204 GCUCJ [Work DEBUG] scheduling run of gzip-file /data/buckets/bucket-8830f702c32ce5e8d493031453fba0ec6be2c1b07e3fc5d0123f9895ecbc5523.xdr
2018-08-08T21:51:13.204 GCUCJ [Work DEBUG] advancing put-remote-file bucket/8d/f1/e1/bucket-8df1e17d986608744a66130af542f00f8766717a4e551eeed01a5aac0bc19217.xdr.gz
2018-08-08T21:51:13.204 GCUCJ [Work DEBUG] advancing make-remote-dir bucket/8d/f1/e1
2018-08-08T21:51:13.204 GCUCJ [Work DEBUG] advancing gzip-file /data/buckets/bucket-8df1e17d986608744a66130af542f00f8766717a4e551eeed01a5aac0bc19217.xdr
2018-08-08T21:51:13.204 GCUCJ [Work DEBUG] all 0 children of gzip-file /data/buckets/bucket-8df1e17d986608744a66130af542f00f8766717a4e551eeed01a5aac0bc19217.xdr successful, scheduling run
2018-08-08T21:51:13.204 GCUCJ [Work DEBUG] scheduling run of gzip-file /data/buckets/bucket-8df1e17d986608744a66130af542f00f8766717a4e551eeed01a5aac0bc19217.xdr
2018-08-08T21:51:13.204 GCUCJ [Work DEBUG] advancing put-remote-file bucket/a3/06/6a/bucket-a3066a7178325e7fa220571f54627052ebcee1effce3bad5e5d9c8dfc9fe191e.xdr.gz
2018-08-08T21:51:13.204 GCUCJ [Work DEBUG] advancing make-remote-dir bucket/a3/06/6a
2018-08-08T21:51:13.204 GCUCJ [Work DEBUG] advancing gzip-file /data/buckets/bucket-a3066a7178325e7fa220571f54627052ebcee1effce3bad5e5d9c8dfc9fe191e.xdr
2018-08-08T21:51:13.204 GCUCJ [Work DEBUG] all 0 children of gzip-file /data/buckets/bucket-a3066a7178325e7fa220571f54627052ebcee1effce3bad5e5d9c8dfc9fe191e.xdr successful, scheduling run
2018-08-08T21:51:13.204 GCUCJ [Work DEBUG] scheduling run of gzip-file /data/buckets/bucket-a3066a7178325e7fa220571f54627052ebcee1effce3bad5e5d9c8dfc9fe191e.xdr
2018-08-08T21:51:13.204 GCUCJ [Work DEBUG] advancing put-remote-file bucket/a6/dc/7f/bucket-a6dc7f0ecc115c107d64b83734c683735a017e22c5ab494a530417ccf34ca97e.xdr.gz
2018-08-08T21:51:13.204 GCUCJ [Work DEBUG] advancing make-remote-dir bucket/a6/dc/7f
2018-08-08T21:51:13.204 GCUCJ [Work DEBUG] advancing gzip-file /data/buckets/bucket-a6dc7f0ecc115c107d64b83734c683735a017e22c5ab494a530417ccf34ca97e.xdr
2018-08-08T21:51:13.204 GCUCJ [Work DEBUG] all 0 children of gzip-file /data/buckets/bucket-a6dc7f0ecc115c107d64b83734c683735a017e22c5ab494a530417ccf34ca97e.xdr successful, scheduling run
2018-08-08T21:51:13.204 GCUCJ [Work DEBUG] scheduling run of gzip-file /data/buckets/bucket-a6dc7f0ecc115c107d64b83734c683735a017e22c5ab494a530417ccf34ca97e.xdr
2018-08-08T21:51:13.204 GCUCJ [Work DEBUG] advancing put-remote-file ledger/00/9f/69/ledger-009f693f.xdr.gz
2018-08-08T21:51:13.204 GCUCJ [Work DEBUG] advancing make-remote-dir ledger/00/9f/69
2018-08-08T21:51:13.204 GCUCJ [Work DEBUG] advancing gzip-file /data/buckets/tmp/snapshot-ec421db5b63f34bc/ledger-009f693f.xdr
2018-08-08T21:51:13.204 GCUCJ [Work DEBUG] all 0 children of gzip-file /data/buckets/tmp/snapshot-ec421db5b63f34bc/ledger-009f693f.xdr successful, scheduling run
2018-08-08T21:51:13.204 GCUCJ [Work DEBUG] scheduling run of gzip-file /data/buckets/tmp/snapshot-ec421db5b63f34bc/ledger-009f693f.xdr
2018-08-08T21:51:13.204 GCUCJ [Work DEBUG] advancing put-remote-file results/00/9f/69/results-009f693f.xdr.gz
2018-08-08T21:51:13.204 GCUCJ [Work DEBUG] advancing make-remote-dir results/00/9f/69
2018-08-08T21:51:13.204 GCUCJ [Work DEBUG] advancing gzip-file /data/buckets/tmp/snapshot-ec421db5b63f34bc/results-009f693f.xdr
2018-08-08T21:51:13.204 GCUCJ [Work DEBUG] all 0 children of gzip-file /data/buckets/tmp/snapshot-ec421db5b63f34bc/results-009f693f.xdr successful, scheduling run
2018-08-08T21:51:13.204 GCUCJ [Work DEBUG] scheduling run of gzip-file /data/buckets/tmp/snapshot-ec421db5b63f34bc/results-009f693f.xdr
2018-08-08T21:51:13.204 GCUCJ [Work DEBUG] advancing put-remote-file scp/00/9f/69/scp-009f693f.xdr.gz
2018-08-08T21:51:13.204 GCUCJ [Work DEBUG] advancing make-remote-dir scp/00/9f/69
2018-08-08T21:51:13.204 GCUCJ [Work DEBUG] advancing gzip-file /data/buckets/tmp/snapshot-ec421db5b63f34bc/scp-009f693f.xdr
2018-08-08T21:51:13.204 GCUCJ [Work DEBUG] all 0 children of gzip-file /data/buckets/tmp/snapshot-ec421db5b63f34bc/scp-009f693f.xdr successful, scheduling run
2018-08-08T21:51:13.204 GCUCJ [Work DEBUG] scheduling run of gzip-file /data/buckets/tmp/snapshot-ec421db5b63f34bc/scp-009f693f.xdr
2018-08-08T21:51:13.204 GCUCJ [Work DEBUG] advancing put-remote-file transactions/00/9f/69/transactions-009f693f.xdr.gz
2018-08-08T21:51:13.204 GCUCJ [Work DEBUG] advancing make-remote-dir transactions/00/9f/69
2018-08-08T21:51:13.204 GCUCJ [Work DEBUG] advancing gzip-file /data/buckets/tmp/snapshot-ec421db5b63f34bc/transactions-009f693f.xdr
2018-08-08T21:51:13.204 GCUCJ [Work DEBUG] all 0 children of gzip-file /data/buckets/tmp/snapshot-ec421db5b63f34bc/transactions-009f693f.xdr successful, scheduling run
2018-08-08T21:51:13.204 GCUCJ [Work DEBUG] scheduling run of gzip-file /data/buckets/tmp/snapshot-ec421db5b63f34bc/transactions-009f693f.xdr
2018-08-08T21:51:13.204 GCUCJ [Work DEBUG] starting gzip-file /data/buckets/bucket-0cf726fafa2ba7b3da004fc0c3fbe723adcfc204f7fd01854cf97d86d2b3ea11.xdr
2018-08-08T21:51:13.204 GCUCJ [Process DEBUG] Running: gzip -c /data/buckets/bucket-0cf726fafa2ba7b3da004fc0c3fbe723adcfc204f7fd01854cf97d86d2b3ea11.xdr
2018-08-08T21:51:13.393 GCUCJ [Work DEBUG] running gzip-file /data/buckets/bucket-0cf726fafa2ba7b3da004fc0c3fbe723adcfc204f7fd01854cf97d86d2b3ea11.xdr
2018-08-08T21:51:13.393 GCUCJ [Work DEBUG] work gzip-file /data/buckets/bucket-0cf726fafa2ba7b3da004fc0c3fbe723adcfc204f7fd01854cf97d86d2b3ea11.xdr : WORK_PENDING -> WORK_RUNNING
2018-08-08T21:51:13.394 GCUCJ [Work DEBUG] starting gzip-file /data/buckets/bucket-1724d77be46fa62a0ed46b106a3e47af1c0c8d42deac4b8bc98d505755e0437a.xdr
2018-08-08T21:51:13.394 GCUCJ [Process DEBUG] Running: gzip -c /data/buckets/bucket-1724d77be46fa62a0ed46b106a3e47af1c0c8d42deac4b8bc98d505755e0437a.xdr
2018-08-08T21:51:13.584 GCUCJ [Work DEBUG] running gzip-file /data/buckets/bucket-1724d77be46fa62a0ed46b106a3e47af1c0c8d42deac4b8bc98d505755e0437a.xdr
2018-08-08T21:51:13.584 GCUCJ [Work DEBUG] work gzip-file /data/buckets/bucket-1724d77be46fa62a0ed46b106a3e47af1c0c8d42deac4b8bc98d505755e0437a.xdr : WORK_PENDING -> WORK_RUNNING
2018-08-08T21:51:13.584 GCUCJ [Work DEBUG] starting gzip-file /data/buckets/bucket-1901eea7636ae2363e6989325da37a84c2f038ce9b94f6484a86cff37ca82217.xdr
2018-08-08T21:51:13.585 GCUCJ [Process DEBUG] Running: gzip -c /data/buckets/bucket-1901eea7636ae2363e6989325da37a84c2f038ce9b94f6484a86cff37ca82217.xdr
2018-08-08T21:51:13.773 GCUCJ [Work DEBUG] running gzip-file /data/buckets/bucket-1901eea7636ae2363e6989325da37a84c2f038ce9b94f6484a86cff37ca82217.xdr
2018-08-08T21:51:13.773 GCUCJ [Work DEBUG] work gzip-file /data/buckets/bucket-1901eea7636ae2363e6989325da37a84c2f038ce9b94f6484a86cff37ca82217.xdr : WORK_PENDING -> WORK_RUNNING
2018-08-08T21:51:13.774 GCUCJ [Work DEBUG] starting gzip-file /data/buckets/bucket-36905c16c0d345923c5fa8d0adc28fe03c1e47050e51579075064f159e7f16bd.xdr
2018-08-08T21:51:13.774 GCUCJ [Process DEBUG] Running: gzip -c /data/buckets/bucket-36905c16c0d345923c5fa8d0adc28fe03c1e47050e51579075064f159e7f16bd.xdr
2018-08-08T21:51:13.963 GCUCJ [Work DEBUG] running gzip-file /data/buckets/bucket-36905c16c0d345923c5fa8d0adc28fe03c1e47050e51579075064f159e7f16bd.xdr
2018-08-08T21:51:13.963 GCUCJ [Work DEBUG] work gzip-file /data/buckets/bucket-36905c16c0d345923c5fa8d0adc28fe03c1e47050e51579075064f159e7f16bd.xdr : WORK_PENDING -> WORK_RUNNING
2018-08-08T21:51:13.963 GCUCJ [Work DEBUG] starting gzip-file /data/buckets/bucket-3c87b3dfddadceec084ccbd22efb76aefc48425870388ca416777f766e12baec.xdr
2018-08-08T21:51:13.964 GCUCJ [Process DEBUG] Running: gzip -c /data/buckets/bucket-3c87b3dfddadceec084ccbd22efb76aefc48425870388ca416777f766e12baec.xdr
2018-08-08T21:51:14.153 GCUCJ [Work DEBUG] running gzip-file /data/buckets/bucket-3c87b3dfddadceec084ccbd22efb76aefc48425870388ca416777f766e12baec.xdr
2018-08-08T21:51:14.153 GCUCJ [Work DEBUG] work gzip-file /data/buckets/bucket-3c87b3dfddadceec084ccbd22efb76aefc48425870388ca416777f766e12baec.xdr : WORK_PENDING -> WORK_RUNNING
2018-08-08T21:51:14.153 GCUCJ [Work DEBUG] starting gzip-file /data/buckets/bucket-727e26fc2b8407ea155ef61365eebcb5377e90299f6805a04eba105209511f47.xdr
2018-08-08T21:51:14.153 GCUCJ [Process DEBUG] Running: gzip -c /data/buckets/bucket-727e26fc2b8407ea155ef61365eebcb5377e90299f6805a04eba105209511f47.xdr
2018-08-08T21:51:14.342 GCUCJ [Work DEBUG] running gzip-file /data/buckets/bucket-727e26fc2b8407ea155ef61365eebcb5377e90299f6805a04eba105209511f47.xdr
2018-08-08T21:51:14.343 GCUCJ [Work DEBUG] work gzip-file /data/buckets/bucket-727e26fc2b8407ea155ef61365eebcb5377e90299f6805a04eba105209511f47.xdr : WORK_PENDING -> WORK_RUNNING
2018-08-08T21:51:14.343 GCUCJ [Work DEBUG] starting gzip-file /data/buckets/bucket-8830f702c32ce5e8d493031453fba0ec6be2c1b07e3fc5d0123f9895ecbc5523.xdr
2018-08-08T21:51:14.343 GCUCJ [Process DEBUG] Running: gzip -c /data/buckets/bucket-8830f702c32ce5e8d493031453fba0ec6be2c1b07e3fc5d0123f9895ecbc5523.xdr
2018-08-08T21:51:14.532 GCUCJ [Work DEBUG] running gzip-file /data/buckets/bucket-8830f702c32ce5e8d493031453fba0ec6be2c1b07e3fc5d0123f9895ecbc5523.xdr
2018-08-08T21:51:14.532 GCUCJ [Work DEBUG] work gzip-file /data/buckets/bucket-8830f702c32ce5e8d493031453fba0ec6be2c1b07e3fc5d0123f9895ecbc5523.xdr : WORK_PENDING -> WORK_RUNNING
2018-08-08T21:51:14.532 GCUCJ [Work DEBUG] starting gzip-file /data/buckets/bucket-8df1e17d986608744a66130af542f00f8766717a4e551eeed01a5aac0bc19217.xdr
2018-08-08T21:51:14.533 GCUCJ [Process DEBUG] Running: gzip -c /data/buckets/bucket-8df1e17d986608744a66130af542f00f8766717a4e551eeed01a5aac0bc19217.xdr
2018-08-08T21:51:14.722 GCUCJ [Work DEBUG] running gzip-file /data/buckets/bucket-8df1e17d986608744a66130af542f00f8766717a4e551eeed01a5aac0bc19217.xdr
2018-08-08T21:51:14.722 GCUCJ [Work DEBUG] work gzip-file /data/buckets/bucket-8df1e17d986608744a66130af542f00f8766717a4e551eeed01a5aac0bc19217.xdr : WORK_PENDING -> WORK_RUNNING
2018-08-08T21:51:14.722 GCUCJ [Work DEBUG] starting gzip-file /data/buckets/bucket-a3066a7178325e7fa220571f54627052ebcee1effce3bad5e5d9c8dfc9fe191e.xdr
2018-08-08T21:51:14.723 GCUCJ [Process DEBUG] Running: gzip -c /data/buckets/bucket-a3066a7178325e7fa220571f54627052ebcee1effce3bad5e5d9c8dfc9fe191e.xdr
2018-08-08T21:51:14.911 GCUCJ [Work DEBUG] running gzip-file /data/buckets/bucket-a3066a7178325e7fa220571f54627052ebcee1effce3bad5e5d9c8dfc9fe191e.xdr
2018-08-08T21:51:14.911 GCUCJ [Work DEBUG] work gzip-file /data/buckets/bucket-a3066a7178325e7fa220571f54627052ebcee1effce3bad5e5d9c8dfc9fe191e.xdr : WORK_PENDING -> WORK_RUNNING
2018-08-08T21:51:14.911 GCUCJ [Work DEBUG] starting gzip-file /data/buckets/bucket-a6dc7f0ecc115c107d64b83734c683735a017e22c5ab494a530417ccf34ca97e.xdr
2018-08-08T21:51:14.911 GCUCJ [Process DEBUG] Running: gzip -c /data/buckets/bucket-a6dc7f0ecc115c107d64b83734c683735a017e22c5ab494a530417ccf34ca97e.xdr
2018-08-08T21:51:15.100 GCUCJ [Work DEBUG] running gzip-file /data/buckets/bucket-a6dc7f0ecc115c107d64b83734c683735a017e22c5ab494a530417ccf34ca97e.xdr
2018-08-08T21:51:15.100 GCUCJ [Work DEBUG] work gzip-file /data/buckets/bucket-a6dc7f0ecc115c107d64b83734c683735a017e22c5ab494a530417ccf34ca97e.xdr : WORK_PENDING -> WORK_RUNNING
2018-08-08T21:51:15.100 GCUCJ [Work DEBUG] starting gzip-file /data/buckets/tmp/snapshot-ec421db5b63f34bc/ledger-009f693f.xdr
2018-08-08T21:51:15.100 GCUCJ [Process DEBUG] Running: gzip -c /data/buckets/tmp/snapshot-ec421db5b63f34bc/ledger-009f693f.xdr
2018-08-08T21:51:15.289 GCUCJ [Work DEBUG] running gzip-file /data/buckets/tmp/snapshot-ec421db5b63f34bc/ledger-009f693f.xdr
2018-08-08T21:51:15.289 GCUCJ [Work DEBUG] work gzip-file /data/buckets/tmp/snapshot-ec421db5b63f34bc/ledger-009f693f.xdr : WORK_PENDING -> WORK_RUNNING
2018-08-08T21:51:15.290 GCUCJ [Work DEBUG] starting gzip-file /data/buckets/tmp/snapshot-ec421db5b63f34bc/results-009f693f.xdr
2018-08-08T21:51:15.290 GCUCJ [Process DEBUG] Running: gzip -c /data/buckets/tmp/snapshot-ec421db5b63f34bc/results-009f693f.xdr
2018-08-08T21:51:15.478 GCUCJ [Work DEBUG] running gzip-file /data/buckets/tmp/snapshot-ec421db5b63f34bc/results-009f693f.xdr
2018-08-08T21:51:15.478 GCUCJ [Work DEBUG] work gzip-file /data/buckets/tmp/snapshot-ec421db5b63f34bc/results-009f693f.xdr : WORK_PENDING -> WORK_RUNNING
2018-08-08T21:51:15.478 GCUCJ [Work DEBUG] starting gzip-file /data/buckets/tmp/snapshot-ec421db5b63f34bc/scp-009f693f.xdr
2018-08-08T21:51:15.478 GCUCJ [Process DEBUG] Running: gzip -c /data/buckets/tmp/snapshot-ec421db5b63f34bc/scp-009f693f.xdr
2018-08-08T21:51:15.667 GCUCJ [Work DEBUG] running gzip-file /data/buckets/tmp/snapshot-ec421db5b63f34bc/scp-009f693f.xdr
2018-08-08T21:51:15.668 GCUCJ [Work DEBUG] work gzip-file /data/buckets/tmp/snapshot-ec421db5b63f34bc/scp-009f693f.xdr : WORK_PENDING -> WORK_RUNNING
2018-08-08T21:51:15.668 GCUCJ [Work DEBUG] starting gzip-file /data/buckets/tmp/snapshot-ec421db5b63f34bc/transactions-009f693f.xdr
2018-08-08T21:51:15.668 GCUCJ [Process DEBUG] Running: gzip -c /data/buckets/tmp/snapshot-ec421db5b63f34bc/transactions-009f693f.xdr
2018-08-08T21:51:15.857 GCUCJ [Work DEBUG] running gzip-file /data/buckets/tmp/snapshot-ec421db5b63f34bc/transactions-009f693f.xdr
2018-08-08T21:51:15.857 GCUCJ [Work DEBUG] work gzip-file /data/buckets/tmp/snapshot-ec421db5b63f34bc/transactions-009f693f.xdr : WORK_PENDING -> WORK_RUNNING
2018-08-08T21:51:15.862 GCUCJ [Process DEBUG] found 14 child processes that terminated
2018-08-08T21:51:15.862 GCUCJ [Process DEBUG] process 5251 exited 0: gzip -c /data/buckets/bucket-0cf726fafa2ba7b3da004fc0c3fbe723adcfc204f7fd01854cf97d86d2b3ea11.xdr
2018-08-08T21:51:15.862 GCUCJ [Process DEBUG] process 5252 exited 0: gzip -c /data/buckets/bucket-1724d77be46fa62a0ed46b106a3e47af1c0c8d42deac4b8bc98d505755e0437a.xdr
2018-08-08T21:51:15.862 GCUCJ [Process DEBUG] process 5253 exited 0: gzip -c /data/buckets/bucket-1901eea7636ae2363e6989325da37a84c2f038ce9b94f6484a86cff37ca82217.xdr
2018-08-08T21:51:15.862 GCUCJ [Process DEBUG] process 5254 exited 0: gzip -c /data/buckets/bucket-36905c16c0d345923c5fa8d0adc28fe03c1e47050e51579075064f159e7f16bd.xdr
2018-08-08T21:51:15.862 GCUCJ [Process DEBUG] process 5255 exited 0: gzip -c /data/buckets/bucket-3c87b3dfddadceec084ccbd22efb76aefc48425870388ca416777f766e12baec.xdr
2018-08-08T21:51:15.862 GCUCJ [Process DEBUG] process 5256 exited 0: gzip -c /data/buckets/bucket-727e26fc2b8407ea155ef61365eebcb5377e90299f6805a04eba105209511f47.xdr
2018-08-08T21:51:15.862 GCUCJ [Process DEBUG] process 5257 exited 0: gzip -c /data/buckets/bucket-8830f702c32ce5e8d493031453fba0ec6be2c1b07e3fc5d0123f9895ecbc5523.xdr
2018-08-08T21:51:15.862 GCUCJ [Process DEBUG] process 5258 exited 0: gzip -c /data/buckets/bucket-8df1e17d986608744a66130af542f00f8766717a4e551eeed01a5aac0bc19217.xdr
2018-08-08T21:51:15.862 GCUCJ [Process DEBUG] process 5259 exited 0: gzip -c /data/buckets/bucket-a3066a7178325e7fa220571f54627052ebcee1effce3bad5e5d9c8dfc9fe191e.xdr
2018-08-08T21:51:15.862 GCUCJ [Process DEBUG] process 5260 exited 0: gzip -c /data/buckets/bucket-a6dc7f0ecc115c107d64b83734c683735a017e22c5ab494a530417ccf34ca97e.xdr
2018-08-08T21:51:15.862 GCUCJ [Process DEBUG] process 5261 exited 0: gzip -c /data/buckets/tmp/snapshot-ec421db5b63f34bc/ledger-009f693f.xdr
2018-08-08T21:51:15.862 GCUCJ [Process DEBUG] process 5262 exited 0: gzip -c /data/buckets/tmp/snapshot-ec421db5b63f34bc/results-009f693f.xdr
2018-08-08T21:51:15.862 GCUCJ [Process DEBUG] process 5263 exited 0: gzip -c /data/buckets/tmp/snapshot-ec421db5b63f34bc/scp-009f693f.xdr
2018-08-08T21:51:15.862 GCUCJ [Process DEBUG] process 5264 exited 0: gzip -c /data/buckets/tmp/snapshot-ec421db5b63f34bc/transactions-009f693f.xdr
2018-08-08T21:51:15.893 GCUCJ [Work DEBUG] completed gzip-file /data/buckets/bucket-0cf726fafa2ba7b3da004fc0c3fbe723adcfc204f7fd01854cf97d86d2b3ea11.xdr
2018-08-08T21:51:15.893 GCUCJ [Work DEBUG] work gzip-file /data/buckets/bucket-0cf726fafa2ba7b3da004fc0c3fbe723adcfc204f7fd01854cf97d86d2b3ea11.xdr : WORK_RUNNING -> WORK_SUCCESS
2018-08-08T21:51:15.893 GCUCJ [Work DEBUG] notifying parent of successful gzip-file /data/buckets/bucket-0cf726fafa2ba7b3da004fc0c3fbe723adcfc204f7fd01854cf97d86d2b3ea11.xdr
2018-08-08T21:51:15.894 GCUCJ [Work DEBUG] notified make-remote-dir bucket/0c/f7/26 of completed child gzip-file /data/buckets/bucket-0cf726fafa2ba7b3da004fc0c3fbe723adcfc204f7fd01854cf97d86d2b3ea11.xdr
2018-08-08T21:51:15.894 GCUCJ [Work DEBUG] advancing make-remote-dir bucket/0c/f7/26
2018-08-08T21:51:15.894 GCUCJ [Work DEBUG] all 1 children of make-remote-dir bucket/0c/f7/26 successful, scheduling run
2018-08-08T21:51:15.894 GCUCJ [Work DEBUG] scheduling run of make-remote-dir bucket/0c/f7/26
2018-08-08T21:51:15.894 GCUCJ [Work DEBUG] completed gzip-file /data/buckets/bucket-1724d77be46fa62a0ed46b106a3e47af1c0c8d42deac4b8bc98d505755e0437a.xdr
2018-08-08T21:51:15.894 GCUCJ [Work DEBUG] work gzip-file /data/buckets/bucket-1724d77be46fa62a0ed46b106a3e47af1c0c8d42deac4b8bc98d505755e0437a.xdr : WORK_RUNNING -> WORK_SUCCESS
2018-08-08T21:51:15.894 GCUCJ [Work DEBUG] notifying parent of successful gzip-file /data/buckets/bucket-1724d77be46fa62a0ed46b106a3e47af1c0c8d42deac4b8bc98d505755e0437a.xdr
2018-08-08T21:51:15.894 GCUCJ [Work DEBUG] notified make-remote-dir bucket/17/24/d7 of completed child gzip-file /data/buckets/bucket-1724d77be46fa62a0ed46b106a3e47af1c0c8d42deac4b8bc98d505755e0437a.xdr
2018-08-08T21:51:15.894 GCUCJ [Work DEBUG] advancing make-remote-dir bucket/17/24/d7
2018-08-08T21:51:15.894 GCUCJ [Work DEBUG] all 1 children of make-remote-dir bucket/17/24/d7 successful, scheduling run
2018-08-08T21:51:15.894 GCUCJ [Work DEBUG] scheduling run of make-remote-dir bucket/17/24/d7
2018-08-08T21:51:15.894 GCUCJ [Work DEBUG] completed gzip-file /data/buckets/bucket-1901eea7636ae2363e6989325da37a84c2f038ce9b94f6484a86cff37ca82217.xdr
2018-08-08T21:51:15.894 GCUCJ [Work DEBUG] work gzip-file /data/buckets/bucket-1901eea7636ae2363e6989325da37a84c2f038ce9b94f6484a86cff37ca82217.xdr : WORK_RUNNING -> WORK_SUCCESS
2018-08-08T21:51:15.894 GCUCJ [Work DEBUG] notifying parent of successful gzip-file /data/buckets/bucket-1901eea7636ae2363e6989325da37a84c2f038ce9b94f6484a86cff37ca82217.xdr
2018-08-08T21:51:15.894 GCUCJ [Work DEBUG] notified make-remote-dir bucket/19/01/ee of completed child gzip-file /data/buckets/bucket-1901eea7636ae2363e6989325da37a84c2f038ce9b94f6484a86cff37ca82217.xdr
2018-08-08T21:51:15.894 GCUCJ [Work DEBUG] advancing make-remote-dir bucket/19/01/ee
2018-08-08T21:51:15.894 GCUCJ [Work DEBUG] all 1 children of make-remote-dir bucket/19/01/ee successful, scheduling run
2018-08-08T21:51:15.894 GCUCJ [Work DEBUG] scheduling run of make-remote-dir bucket/19/01/ee
2018-08-08T21:51:15.894 GCUCJ [Work DEBUG] completed gzip-file /data/buckets/bucket-36905c16c0d345923c5fa8d0adc28fe03c1e47050e51579075064f159e7f16bd.xdr
2018-08-08T21:51:15.894 GCUCJ [Work DEBUG] work gzip-file /data/buckets/bucket-36905c16c0d345923c5fa8d0adc28fe03c1e47050e51579075064f159e7f16bd.xdr : WORK_RUNNING -> WORK_SUCCESS
2018-08-08T21:51:15.894 GCUCJ [Work DEBUG] notifying parent of successful gzip-file /data/buckets/bucket-36905c16c0d345923c5fa8d0adc28fe03c1e47050e51579075064f159e7f16bd.xdr
2018-08-08T21:51:15.894 GCUCJ [Work DEBUG] notified make-remote-dir bucket/36/90/5c of completed child gzip-file /data/buckets/bucket-36905c16c0d345923c5fa8d0adc28fe03c1e47050e51579075064f159e7f16bd.xdr
2018-08-08T21:51:15.894 GCUCJ [Work DEBUG] advancing make-remote-dir bucket/36/90/5c
2018-08-08T21:51:15.894 GCUCJ [Work DEBUG] all 1 children of make-remote-dir bucket/36/90/5c successful, scheduling run
2018-08-08T21:51:15.894 GCUCJ [Work DEBUG] scheduling run of make-remote-dir bucket/36/90/5c
2018-08-08T21:51:15.894 GCUCJ [Work DEBUG] completed gzip-file /data/buckets/bucket-3c87b3dfddadceec084ccbd22efb76aefc48425870388ca416777f766e12baec.xdr
2018-08-08T21:51:15.894 GCUCJ [Work DEBUG] work gzip-file /data/buckets/bucket-3c87b3dfddadceec084ccbd22efb76aefc48425870388ca416777f766e12baec.xdr : WORK_RUNNING -> WORK_SUCCESS
2018-08-08T21:51:15.894 GCUCJ [Work DEBUG] notifying parent of successful gzip-file /data/buckets/bucket-3c87b3dfddadceec084ccbd22efb76aefc48425870388ca416777f766e12baec.xdr
2018-08-08T21:51:15.894 GCUCJ [Work DEBUG] notified make-remote-dir bucket/3c/87/b3 of completed child gzip-file /data/buckets/bucket-3c87b3dfddadceec084ccbd22efb76aefc48425870388ca416777f766e12baec.xdr
2018-08-08T21:51:15.894 GCUCJ [Work DEBUG] advancing make-remote-dir bucket/3c/87/b3
2018-08-08T21:51:15.894 GCUCJ [Work DEBUG] all 1 children of make-remote-dir bucket/3c/87/b3 successful, scheduling run
2018-08-08T21:51:15.894 GCUCJ [Work DEBUG] scheduling run of make-remote-dir bucket/3c/87/b3
2018-08-08T21:51:15.894 GCUCJ [Work DEBUG] completed gzip-file /data/buckets/bucket-727e26fc2b8407ea155ef61365eebcb5377e90299f6805a04eba105209511f47.xdr
2018-08-08T21:51:15.894 GCUCJ [Work DEBUG] work gzip-file /data/buckets/bucket-727e26fc2b8407ea155ef61365eebcb5377e90299f6805a04eba105209511f47.xdr : WORK_RUNNING -> WORK_SUCCESS
2018-08-08T21:51:15.894 GCUCJ [Work DEBUG] notifying parent of successful gzip-file /data/buckets/bucket-727e26fc2b8407ea155ef61365eebcb5377e90299f6805a04eba105209511f47.xdr
2018-08-08T21:51:15.894 GCUCJ [Work DEBUG] notified make-remote-dir bucket/72/7e/26 of completed child gzip-file /data/buckets/bucket-727e26fc2b8407ea155ef61365eebcb5377e90299f6805a04eba105209511f47.xdr
2018-08-08T21:51:15.894 GCUCJ [Work DEBUG] advancing make-remote-dir bucket/72/7e/26
2018-08-08T21:51:15.894 GCUCJ [Work DEBUG] all 1 children of make-remote-dir bucket/72/7e/26 successful, scheduling run
2018-08-08T21:51:15.894 GCUCJ [Work DEBUG] scheduling run of make-remote-dir bucket/72/7e/26
2018-08-08T21:51:15.894 GCUCJ [Work DEBUG] completed gzip-file /data/buckets/bucket-8830f702c32ce5e8d493031453fba0ec6be2c1b07e3fc5d0123f9895ecbc5523.xdr
2018-08-08T21:51:15.894 GCUCJ [Work DEBUG] work gzip-file /data/buckets/bucket-8830f702c32ce5e8d493031453fba0ec6be2c1b07e3fc5d0123f9895ecbc5523.xdr : WORK_RUNNING -> WORK_SUCCESS
2018-08-08T21:51:15.894 GCUCJ [Work DEBUG] notifying parent of successful gzip-file /data/buckets/bucket-8830f702c32ce5e8d493031453fba0ec6be2c1b07e3fc5d0123f9895ecbc5523.xdr
2018-08-08T21:51:15.894 GCUCJ [Work DEBUG] notified make-remote-dir bucket/88/30/f7 of completed child gzip-file /data/buckets/bucket-8830f702c32ce5e8d493031453fba0ec6be2c1b07e3fc5d0123f9895ecbc5523.xdr
2018-08-08T21:51:15.895 GCUCJ [Work DEBUG] advancing make-remote-dir bucket/88/30/f7
2018-08-08T21:51:15.895 GCUCJ [Work DEBUG] all 1 children of make-remote-dir bucket/88/30/f7 successful, scheduling run
2018-08-08T21:51:15.895 GCUCJ [Work DEBUG] scheduling run of make-remote-dir bucket/88/30/f7
2018-08-08T21:51:15.895 GCUCJ [Work DEBUG] completed gzip-file /data/buckets/bucket-8df1e17d986608744a66130af542f00f8766717a4e551eeed01a5aac0bc19217.xdr
2018-08-08T21:51:15.895 GCUCJ [Work DEBUG] work gzip-file /data/buckets/bucket-8df1e17d986608744a66130af542f00f8766717a4e551eeed01a5aac0bc19217.xdr : WORK_RUNNING -> WORK_SUCCESS
2018-08-08T21:51:15.895 GCUCJ [Work DEBUG] notifying parent of successful gzip-file /data/buckets/bucket-8df1e17d986608744a66130af542f00f8766717a4e551eeed01a5aac0bc19217.xdr
2018-08-08T21:51:15.895 GCUCJ [Work DEBUG] notified make-remote-dir bucket/8d/f1/e1 of completed child gzip-file /data/buckets/bucket-8df1e17d986608744a66130af542f00f8766717a4e551eeed01a5aac0bc19217.xdr
2018-08-08T21:51:15.895 GCUCJ [Work DEBUG] advancing make-remote-dir bucket/8d/f1/e1
2018-08-08T21:51:15.895 GCUCJ [Work DEBUG] all 1 children of make-remote-dir bucket/8d/f1/e1 successful, scheduling run
2018-08-08T21:51:15.895 GCUCJ [Work DEBUG] scheduling run of make-remote-dir bucket/8d/f1/e1
2018-08-08T21:51:15.895 GCUCJ [Work DEBUG] completed gzip-file /data/buckets/bucket-a3066a7178325e7fa220571f54627052ebcee1effce3bad5e5d9c8dfc9fe191e.xdr
2018-08-08T21:51:15.895 GCUCJ [Work DEBUG] work gzip-file /data/buckets/bucket-a3066a7178325e7fa220571f54627052ebcee1effce3bad5e5d9c8dfc9fe191e.xdr : WORK_RUNNING -> WORK_SUCCESS
2018-08-08T21:51:15.895 GCUCJ [Work DEBUG] notifying parent of successful gzip-file /data/buckets/bucket-a3066a7178325e7fa220571f54627052ebcee1effce3bad5e5d9c8dfc9fe191e.xdr
2018-08-08T21:51:15.895 GCUCJ [Work DEBUG] notified make-remote-dir bucket/a3/06/6a of completed child gzip-file /data/buckets/bucket-a3066a7178325e7fa220571f54627052ebcee1effce3bad5e5d9c8dfc9fe191e.xdr
2018-08-08T21:51:15.895 GCUCJ [Work DEBUG] advancing make-remote-dir bucket/a3/06/6a
2018-08-08T21:51:15.895 GCUCJ [Work DEBUG] all 1 children of make-remote-dir bucket/a3/06/6a successful, scheduling run
2018-08-08T21:51:15.895 GCUCJ [Work DEBUG] scheduling run of make-remote-dir bucket/a3/06/6a
2018-08-08T21:51:15.895 GCUCJ [Work DEBUG] completed gzip-file /data/buckets/bucket-a6dc7f0ecc115c107d64b83734c683735a017e22c5ab494a530417ccf34ca97e.xdr
2018-08-08T21:51:15.895 GCUCJ [Work DEBUG] work gzip-file /data/buckets/bucket-a6dc7f0ecc115c107d64b83734c683735a017e22c5ab494a530417ccf34ca97e.xdr : WORK_RUNNING -> WORK_SUCCESS
2018-08-08T21:51:15.895 GCUCJ [Work DEBUG] notifying parent of successful gzip-file /data/buckets/bucket-a6dc7f0ecc115c107d64b83734c683735a017e22c5ab494a530417ccf34ca97e.xdr
2018-08-08T21:51:15.895 GCUCJ [Work DEBUG] notified make-remote-dir bucket/a6/dc/7f of completed child gzip-file /data/buckets/bucket-a6dc7f0ecc115c107d64b83734c683735a017e22c5ab494a530417ccf34ca97e.xdr
2018-08-08T21:51:15.895 GCUCJ [Work DEBUG] advancing make-remote-dir bucket/a6/dc/7f
2018-08-08T21:51:15.895 GCUCJ [Work DEBUG] all 1 children of make-remote-dir bucket/a6/dc/7f successful, scheduling run
2018-08-08T21:51:15.895 GCUCJ [Work DEBUG] scheduling run of make-remote-dir bucket/a6/dc/7f
2018-08-08T21:51:15.896 GCUCJ [Work DEBUG] completed gzip-file /data/buckets/tmp/snapshot-ec421db5b63f34bc/ledger-009f693f.xdr
2018-08-08T21:51:15.896 GCUCJ [Work DEBUG] work gzip-file /data/buckets/tmp/snapshot-ec421db5b63f34bc/ledger-009f693f.xdr : WORK_RUNNING -> WORK_SUCCESS
2018-08-08T21:51:15.896 GCUCJ [Work DEBUG] notifying parent of successful gzip-file /data/buckets/tmp/snapshot-ec421db5b63f34bc/ledger-009f693f.xdr
2018-08-08T21:51:15.896 GCUCJ [Work DEBUG] notified make-remote-dir ledger/00/9f/69 of completed child gzip-file /data/buckets/tmp/snapshot-ec421db5b63f34bc/ledger-009f693f.xdr
2018-08-08T21:51:15.896 GCUCJ [Work DEBUG] advancing make-remote-dir ledger/00/9f/69
2018-08-08T21:51:15.896 GCUCJ [Work DEBUG] all 1 children of make-remote-dir ledger/00/9f/69 successful, scheduling run
2018-08-08T21:51:15.896 GCUCJ [Work DEBUG] scheduling run of make-remote-dir ledger/00/9f/69
2018-08-08T21:51:15.896 GCUCJ [Work DEBUG] completed gzip-file /data/buckets/tmp/snapshot-ec421db5b63f34bc/results-009f693f.xdr
2018-08-08T21:51:15.897 GCUCJ [Work DEBUG] work gzip-file /data/buckets/tmp/snapshot-ec421db5b63f34bc/results-009f693f.xdr : WORK_RUNNING -> WORK_SUCCESS
2018-08-08T21:51:15.897 GCUCJ [Work DEBUG] notifying parent of successful gzip-file /data/buckets/tmp/snapshot-ec421db5b63f34bc/results-009f693f.xdr
2018-08-08T21:51:15.897 GCUCJ [Work DEBUG] notified make-remote-dir results/00/9f/69 of completed child gzip-file /data/buckets/tmp/snapshot-ec421db5b63f34bc/results-009f693f.xdr
2018-08-08T21:51:15.897 GCUCJ [Work DEBUG] advancing make-remote-dir results/00/9f/69
2018-08-08T21:51:15.897 GCUCJ [Work DEBUG] all 1 children of make-remote-dir results/00/9f/69 successful, scheduling run
2018-08-08T21:51:15.897 GCUCJ [Work DEBUG] scheduling run of make-remote-dir results/00/9f/69
2018-08-08T21:51:15.897 GCUCJ [Work DEBUG] completed gzip-file /data/buckets/tmp/snapshot-ec421db5b63f34bc/scp-009f693f.xdr
2018-08-08T21:51:15.897 GCUCJ [Work DEBUG] work gzip-file /data/buckets/tmp/snapshot-ec421db5b63f34bc/scp-009f693f.xdr : WORK_RUNNING -> WORK_SUCCESS
2018-08-08T21:51:15.897 GCUCJ [Work DEBUG] notifying parent of successful gzip-file /data/buckets/tmp/snapshot-ec421db5b63f34bc/scp-009f693f.xdr
2018-08-08T21:51:15.897 GCUCJ [Work DEBUG] notified make-remote-dir scp/00/9f/69 of completed child gzip-file /data/buckets/tmp/snapshot-ec421db5b63f34bc/scp-009f693f.xdr
2018-08-08T21:51:15.897 GCUCJ [Work DEBUG] advancing make-remote-dir scp/00/9f/69
2018-08-08T21:51:15.897 GCUCJ [Work DEBUG] all 1 children of make-remote-dir scp/00/9f/69 successful, scheduling run
2018-08-08T21:51:15.897 GCUCJ [Work DEBUG] scheduling run of make-remote-dir scp/00/9f/69
2018-08-08T21:51:15.897 GCUCJ [Work DEBUG] completed gzip-file /data/buckets/tmp/snapshot-ec421db5b63f34bc/transactions-009f693f.xdr
2018-08-08T21:51:15.897 GCUCJ [Work DEBUG] work gzip-file /data/buckets/tmp/snapshot-ec421db5b63f34bc/transactions-009f693f.xdr : WORK_RUNNING -> WORK_SUCCESS
2018-08-08T21:51:15.897 GCUCJ [Work DEBUG] notifying parent of successful gzip-file /data/buckets/tmp/snapshot-ec421db5b63f34bc/transactions-009f693f.xdr
2018-08-08T21:51:15.897 GCUCJ [Work DEBUG] notified make-remote-dir transactions/00/9f/69 of completed child gzip-file /data/buckets/tmp/snapshot-ec421db5b63f34bc/transactions-009f693f.xdr
2018-08-08T21:51:15.897 GCUCJ [Work DEBUG] advancing make-remote-dir transactions/00/9f/69
2018-08-08T21:51:15.897 GCUCJ [Work DEBUG] all 1 children of make-remote-dir transactions/00/9f/69 successful, scheduling run
2018-08-08T21:51:15.898 GCUCJ [Work DEBUG] scheduling run of make-remote-dir transactions/00/9f/69
2018-08-08T21:51:15.903 GCUCJ [Work DEBUG] starting make-remote-dir bucket/0c/f7/26
2018-08-08T21:51:15.903 GCUCJ [Work DEBUG] scheduling completion of make-remote-dir bucket/0c/f7/26
2018-08-08T21:51:15.903 GCUCJ [Work DEBUG] running make-remote-dir bucket/0c/f7/26
2018-08-08T21:51:15.903 GCUCJ [Work DEBUG] work make-remote-dir bucket/0c/f7/26 : WORK_PENDING -> WORK_RUNNING
2018-08-08T21:51:15.903 GCUCJ [Work DEBUG] starting make-remote-dir bucket/17/24/d7
2018-08-08T21:51:15.903 GCUCJ [Work DEBUG] scheduling completion of make-remote-dir bucket/17/24/d7
2018-08-08T21:51:15.903 GCUCJ [Work DEBUG] running make-remote-dir bucket/17/24/d7
2018-08-08T21:51:15.903 GCUCJ [Work DEBUG] work make-remote-dir bucket/17/24/d7 : WORK_PENDING -> WORK_RUNNING
2018-08-08T21:51:15.903 GCUCJ [Work DEBUG] starting make-remote-dir bucket/19/01/ee
2018-08-08T21:51:15.903 GCUCJ [Work DEBUG] scheduling completion of make-remote-dir bucket/19/01/ee
2018-08-08T21:51:15.903 GCUCJ [Work DEBUG] running make-remote-dir bucket/19/01/ee
2018-08-08T21:51:15.903 GCUCJ [Work DEBUG] work make-remote-dir bucket/19/01/ee : WORK_PENDING -> WORK_RUNNING
2018-08-08T21:51:15.903 GCUCJ [Work DEBUG] starting make-remote-dir bucket/36/90/5c
2018-08-08T21:51:15.903 GCUCJ [Work DEBUG] scheduling completion of make-remote-dir bucket/36/90/5c
2018-08-08T21:51:15.903 GCUCJ [Work DEBUG] running make-remote-dir bucket/36/90/5c
2018-08-08T21:51:15.903 GCUCJ [Work DEBUG] work make-remote-dir bucket/36/90/5c : WORK_PENDING -> WORK_RUNNING
2018-08-08T21:51:15.903 GCUCJ [Work DEBUG] starting make-remote-dir bucket/3c/87/b3
2018-08-08T21:51:15.903 GCUCJ [Work DEBUG] scheduling completion of make-remote-dir bucket/3c/87/b3
2018-08-08T21:51:15.903 GCUCJ [Work DEBUG] running make-remote-dir bucket/3c/87/b3
2018-08-08T21:51:15.903 GCUCJ [Work DEBUG] work make-remote-dir bucket/3c/87/b3 : WORK_PENDING -> WORK_RUNNING
2018-08-08T21:51:15.903 GCUCJ [Work DEBUG] starting make-remote-dir bucket/72/7e/26
2018-08-08T21:51:15.903 GCUCJ [Work DEBUG] scheduling completion of make-remote-dir bucket/72/7e/26
2018-08-08T21:51:15.903 GCUCJ [Work DEBUG] running make-remote-dir bucket/72/7e/26
2018-08-08T21:51:15.903 GCUCJ [Work DEBUG] work make-remote-dir bucket/72/7e/26 : WORK_PENDING -> WORK_RUNNING
2018-08-08T21:51:15.903 GCUCJ [Work DEBUG] starting make-remote-dir bucket/88/30/f7
2018-08-08T21:51:15.903 GCUCJ [Work DEBUG] scheduling completion of make-remote-dir bucket/88/30/f7
2018-08-08T21:51:15.903 GCUCJ [Work DEBUG] running make-remote-dir bucket/88/30/f7
2018-08-08T21:51:15.903 GCUCJ [Work DEBUG] work make-remote-dir bucket/88/30/f7 : WORK_PENDING -> WORK_RUNNING
2018-08-08T21:51:15.903 GCUCJ [Work DEBUG] starting make-remote-dir bucket/8d/f1/e1
2018-08-08T21:51:15.903 GCUCJ [Work DEBUG] scheduling completion of make-remote-dir bucket/8d/f1/e1
2018-08-08T21:51:15.903 GCUCJ [Work DEBUG] running make-remote-dir bucket/8d/f1/e1
2018-08-08T21:51:15.903 GCUCJ [Work DEBUG] work make-remote-dir bucket/8d/f1/e1 : WORK_PENDING -> WORK_RUNNING
2018-08-08T21:51:15.903 GCUCJ [Work DEBUG] starting make-remote-dir bucket/a3/06/6a
2018-08-08T21:51:15.903 GCUCJ [Work DEBUG] scheduling completion of make-remote-dir bucket/a3/06/6a
2018-08-08T21:51:15.903 GCUCJ [Work DEBUG] running make-remote-dir bucket/a3/06/6a
2018-08-08T21:51:15.903 GCUCJ [Work DEBUG] work make-remote-dir bucket/a3/06/6a : WORK_PENDING -> WORK_RUNNING
2018-08-08T21:51:15.904 GCUCJ [Work DEBUG] starting make-remote-dir bucket/a6/dc/7f
2018-08-08T21:51:15.904 GCUCJ [Work DEBUG] scheduling completion of make-remote-dir bucket/a6/dc/7f
2018-08-08T21:51:15.904 GCUCJ [Work DEBUG] running make-remote-dir bucket/a6/dc/7f
2018-08-08T21:51:15.904 GCUCJ [Work DEBUG] work make-remote-dir bucket/a6/dc/7f : WORK_PENDING -> WORK_RUNNING
2018-08-08T21:51:15.904 GCUCJ [Work DEBUG] starting make-remote-dir ledger/00/9f/69
2018-08-08T21:51:15.904 GCUCJ [Work DEBUG] scheduling completion of make-remote-dir ledger/00/9f/69
2018-08-08T21:51:15.904 GCUCJ [Work DEBUG] running make-remote-dir ledger/00/9f/69
2018-08-08T21:51:15.904 GCUCJ [Work DEBUG] work make-remote-dir ledger/00/9f/69 : WORK_PENDING -> WORK_RUNNING
2018-08-08T21:51:15.904 GCUCJ [Work DEBUG] starting make-remote-dir results/00/9f/69
2018-08-08T21:51:15.904 GCUCJ [Work DEBUG] scheduling completion of make-remote-dir results/00/9f/69
2018-08-08T21:51:15.904 GCUCJ [Work DEBUG] running make-remote-dir results/00/9f/69
2018-08-08T21:51:15.904 GCUCJ [Work DEBUG] work make-remote-dir results/00/9f/69 : WORK_PENDING -> WORK_RUNNING
2018-08-08T21:51:15.904 GCUCJ [Work DEBUG] starting make-remote-dir scp/00/9f/69
2018-08-08T21:51:15.904 GCUCJ [Work DEBUG] scheduling completion of make-remote-dir scp/00/9f/69
2018-08-08T21:51:15.904 GCUCJ [Work DEBUG] running make-remote-dir scp/00/9f/69
2018-08-08T21:51:15.904 GCUCJ [Work DEBUG] work make-remote-dir scp/00/9f/69 : WORK_PENDING -> WORK_RUNNING
2018-08-08T21:51:15.904 GCUCJ [Work DEBUG] starting make-remote-dir transactions/00/9f/69
2018-08-08T21:51:15.904 GCUCJ [Work DEBUG] scheduling completion of make-remote-dir transactions/00/9f/69
2018-08-08T21:51:15.904 GCUCJ [Work DEBUG] running make-remote-dir transactions/00/9f/69
2018-08-08T21:51:15.904 GCUCJ [Work DEBUG] work make-remote-dir transactions/00/9f/69 : WORK_PENDING -> WORK_RUNNING
2018-08-08T21:51:15.905 GCUCJ [Work DEBUG] completed make-remote-dir bucket/0c/f7/26
2018-08-08T21:51:15.905 GCUCJ [Work DEBUG] work make-remote-dir bucket/0c/f7/26 : WORK_RUNNING -> WORK_SUCCESS
2018-08-08T21:51:15.905 GCUCJ [Work DEBUG] notifying parent of successful make-remote-dir bucket/0c/f7/26
2018-08-08T21:51:15.905 GCUCJ [Work DEBUG] notified put-remote-file bucket/0c/f7/26/bucket-0cf726fafa2ba7b3da004fc0c3fbe723adcfc204f7fd01854cf97d86d2b3ea11.xdr.gz of completed child make-remote-dir bucket/0c/f7/26
2018-08-08T21:51:15.905 GCUCJ [Work DEBUG] advancing put-remote-file bucket/0c/f7/26/bucket-0cf726fafa2ba7b3da004fc0c3fbe723adcfc204f7fd01854cf97d86d2b3ea11.xdr.gz
2018-08-08T21:51:15.905 GCUCJ [Work DEBUG] all 1 children of put-remote-file bucket/0c/f7/26/bucket-0cf726fafa2ba7b3da004fc0c3fbe723adcfc204f7fd01854cf97d86d2b3ea11.xdr.gz successful, scheduling run
2018-08-08T21:51:15.905 GCUCJ [Work DEBUG] scheduling run of put-remote-file bucket/0c/f7/26/bucket-0cf726fafa2ba7b3da004fc0c3fbe723adcfc204f7fd01854cf97d86d2b3ea11.xdr.gz
2018-08-08T21:51:15.905 GCUCJ [Work DEBUG] completed make-remote-dir bucket/17/24/d7
2018-08-08T21:51:15.905 GCUCJ [Work DEBUG] work make-remote-dir bucket/17/24/d7 : WORK_RUNNING -> WORK_SUCCESS
2018-08-08T21:51:15.905 GCUCJ [Work DEBUG] notifying parent of successful make-remote-dir bucket/17/24/d7
2018-08-08T21:51:15.905 GCUCJ [Work DEBUG] notified put-remote-file bucket/17/24/d7/bucket-1724d77be46fa62a0ed46b106a3e47af1c0c8d42deac4b8bc98d505755e0437a.xdr.gz of completed child make-remote-dir bucket/17/24/d7
2018-08-08T21:51:15.905 GCUCJ [Work DEBUG] advancing put-remote-file bucket/17/24/d7/bucket-1724d77be46fa62a0ed46b106a3e47af1c0c8d42deac4b8bc98d505755e0437a.xdr.gz
2018-08-08T21:51:15.905 GCUCJ [Work DEBUG] all 1 children of put-remote-file bucket/17/24/d7/bucket-1724d77be46fa62a0ed46b106a3e47af1c0c8d42deac4b8bc98d505755e0437a.xdr.gz successful, scheduling run
2018-08-08T21:51:15.905 GCUCJ [Work DEBUG] scheduling run of put-remote-file bucket/17/24/d7/bucket-1724d77be46fa62a0ed46b106a3e47af1c0c8d42deac4b8bc98d505755e0437a.xdr.gz
2018-08-08T21:51:15.905 GCUCJ [Work DEBUG] completed make-remote-dir bucket/19/01/ee
2018-08-08T21:51:15.905 GCUCJ [Work DEBUG] work make-remote-dir bucket/19/01/ee : WORK_RUNNING -> WORK_SUCCESS
2018-08-08T21:51:15.905 GCUCJ [Work DEBUG] notifying parent of successful make-remote-dir bucket/19/01/ee
2018-08-08T21:51:15.905 GCUCJ [Work DEBUG] notified put-remote-file bucket/19/01/ee/bucket-1901eea7636ae2363e6989325da37a84c2f038ce9b94f6484a86cff37ca82217.xdr.gz of completed child make-remote-dir bucket/19/01/ee
2018-08-08T21:51:15.905 GCUCJ [Work DEBUG] advancing put-remote-file bucket/19/01/ee/bucket-1901eea7636ae2363e6989325da37a84c2f038ce9b94f6484a86cff37ca82217.xdr.gz
2018-08-08T21:51:15.905 GCUCJ [Work DEBUG] all 1 children of put-remote-file bucket/19/01/ee/bucket-1901eea7636ae2363e6989325da37a84c2f038ce9b94f6484a86cff37ca82217.xdr.gz successful, scheduling run
2018-08-08T21:51:15.905 GCUCJ [Work DEBUG] scheduling run of put-remote-file bucket/19/01/ee/bucket-1901eea7636ae2363e6989325da37a84c2f038ce9b94f6484a86cff37ca82217.xdr.gz
2018-08-08T21:51:15.905 GCUCJ [Work DEBUG] completed make-remote-dir bucket/36/90/5c
2018-08-08T21:51:15.905 GCUCJ [Work DEBUG] work make-remote-dir bucket/36/90/5c : WORK_RUNNING -> WORK_SUCCESS
2018-08-08T21:51:15.905 GCUCJ [Work DEBUG] notifying parent of successful make-remote-dir bucket/36/90/5c
2018-08-08T21:51:15.905 GCUCJ [Work DEBUG] notified put-remote-file bucket/36/90/5c/bucket-36905c16c0d345923c5fa8d0adc28fe03c1e47050e51579075064f159e7f16bd.xdr.gz of completed child make-remote-dir bucket/36/90/5c
2018-08-08T21:51:15.905 GCUCJ [Work DEBUG] advancing put-remote-file bucket/36/90/5c/bucket-36905c16c0d345923c5fa8d0adc28fe03c1e47050e51579075064f159e7f16bd.xdr.gz
2018-08-08T21:51:15.905 GCUCJ [Work DEBUG] all 1 children of put-remote-file bucket/36/90/5c/bucket-36905c16c0d345923c5fa8d0adc28fe03c1e47050e51579075064f159e7f16bd.xdr.gz successful, scheduling run
2018-08-08T21:51:15.905 GCUCJ [Work DEBUG] scheduling run of put-remote-file bucket/36/90/5c/bucket-36905c16c0d345923c5fa8d0adc28fe03c1e47050e51579075064f159e7f16bd.xdr.gz
2018-08-08T21:51:15.905 GCUCJ [Work DEBUG] completed make-remote-dir bucket/3c/87/b3
2018-08-08T21:51:15.905 GCUCJ [Work DEBUG] work make-remote-dir bucket/3c/87/b3 : WORK_RUNNING -> WORK_SUCCESS
2018-08-08T21:51:15.905 GCUCJ [Work DEBUG] notifying parent of successful make-remote-dir bucket/3c/87/b3
2018-08-08T21:51:15.905 GCUCJ [Work DEBUG] notified put-remote-file bucket/3c/87/b3/bucket-3c87b3dfddadceec084ccbd22efb76aefc48425870388ca416777f766e12baec.xdr.gz of completed child make-remote-dir bucket/3c/87/b3
2018-08-08T21:51:15.905 GCUCJ [Work DEBUG] advancing put-remote-file bucket/3c/87/b3/bucket-3c87b3dfddadceec084ccbd22efb76aefc48425870388ca416777f766e12baec.xdr.gz
2018-08-08T21:51:15.905 GCUCJ [Work DEBUG] all 1 children of put-remote-file bucket/3c/87/b3/bucket-3c87b3dfddadceec084ccbd22efb76aefc48425870388ca416777f766e12baec.xdr.gz successful, scheduling run
2018-08-08T21:51:15.905 GCUCJ [Work DEBUG] scheduling run of put-remote-file bucket/3c/87/b3/bucket-3c87b3dfddadceec084ccbd22efb76aefc48425870388ca416777f766e12baec.xdr.gz
2018-08-08T21:51:15.905 GCUCJ [Work DEBUG] completed make-remote-dir bucket/72/7e/26
2018-08-08T21:51:15.905 GCUCJ [Work DEBUG] work make-remote-dir bucket/72/7e/26 : WORK_RUNNING -> WORK_SUCCESS
2018-08-08T21:51:15.905 GCUCJ [Work DEBUG] notifying parent of successful make-remote-dir bucket/72/7e/26
2018-08-08T21:51:15.905 GCUCJ [Work DEBUG] notified put-remote-file bucket/72/7e/26/bucket-727e26fc2b8407ea155ef61365eebcb5377e90299f6805a04eba105209511f47.xdr.gz of completed child make-remote-dir bucket/72/7e/26
2018-08-08T21:51:15.905 GCUCJ [Work DEBUG] advancing put-remote-file bucket/72/7e/26/bucket-727e26fc2b8407ea155ef61365eebcb5377e90299f6805a04eba105209511f47.xdr.gz
2018-08-08T21:51:15.905 GCUCJ [Work DEBUG] all 1 children of put-remote-file bucket/72/7e/26/bucket-727e26fc2b8407ea155ef61365eebcb5377e90299f6805a04eba105209511f47.xdr.gz successful, scheduling run
2018-08-08T21:51:15.905 GCUCJ [Work DEBUG] scheduling run of put-remote-file bucket/72/7e/26/bucket-727e26fc2b8407ea155ef61365eebcb5377e90299f6805a04eba105209511f47.xdr.gz
2018-08-08T21:51:15.905 GCUCJ [Work DEBUG] completed make-remote-dir bucket/88/30/f7
2018-08-08T21:51:15.905 GCUCJ [Work DEBUG] work make-remote-dir bucket/88/30/f7 : WORK_RUNNING -> WORK_SUCCESS
2018-08-08T21:51:15.905 GCUCJ [Work DEBUG] notifying parent of successful make-remote-dir bucket/88/30/f7
2018-08-08T21:51:15.905 GCUCJ [Work DEBUG] notified put-remote-file bucket/88/30/f7/bucket-8830f702c32ce5e8d493031453fba0ec6be2c1b07e3fc5d0123f9895ecbc5523.xdr.gz of completed child make-remote-dir bucket/88/30/f7
2018-08-08T21:51:15.905 GCUCJ [Work DEBUG] advancing put-remote-file bucket/88/30/f7/bucket-8830f702c32ce5e8d493031453fba0ec6be2c1b07e3fc5d0123f9895ecbc5523.xdr.gz
2018-08-08T21:51:15.905 GCUCJ [Work DEBUG] all 1 children of put-remote-file bucket/88/30/f7/bucket-8830f702c32ce5e8d493031453fba0ec6be2c1b07e3fc5d0123f9895ecbc5523.xdr.gz successful, scheduling run
2018-08-08T21:51:15.905 GCUCJ [Work DEBUG] scheduling run of put-remote-file bucket/88/30/f7/bucket-8830f702c32ce5e8d493031453fba0ec6be2c1b07e3fc5d0123f9895ecbc5523.xdr.gz
2018-08-08T21:51:15.905 GCUCJ [Work DEBUG] completed make-remote-dir bucket/8d/f1/e1
2018-08-08T21:51:15.905 GCUCJ [Work DEBUG] work make-remote-dir bucket/8d/f1/e1 : WORK_RUNNING -> WORK_SUCCESS
2018-08-08T21:51:15.906 GCUCJ [Work DEBUG] notifying parent of successful make-remote-dir bucket/8d/f1/e1
2018-08-08T21:51:15.906 GCUCJ [Work DEBUG] notified put-remote-file bucket/8d/f1/e1/bucket-8df1e17d986608744a66130af542f00f8766717a4e551eeed01a5aac0bc19217.xdr.gz of completed child make-remote-dir bucket/8d/f1/e1
2018-08-08T21:51:15.906 GCUCJ [Work DEBUG] advancing put-remote-file bucket/8d/f1/e1/bucket-8df1e17d986608744a66130af542f00f8766717a4e551eeed01a5aac0bc19217.xdr.gz
2018-08-08T21:51:15.906 GCUCJ [Work DEBUG] all 1 children of put-remote-file bucket/8d/f1/e1/bucket-8df1e17d986608744a66130af542f00f8766717a4e551eeed01a5aac0bc19217.xdr.gz successful, scheduling run
2018-08-08T21:51:15.906 GCUCJ [Work DEBUG] scheduling run of put-remote-file bucket/8d/f1/e1/bucket-8df1e17d986608744a66130af542f00f8766717a4e551eeed01a5aac0bc19217.xdr.gz
2018-08-08T21:51:15.906 GCUCJ [Work DEBUG] completed make-remote-dir bucket/a3/06/6a
2018-08-08T21:51:15.906 GCUCJ [Work DEBUG] work make-remote-dir bucket/a3/06/6a : WORK_RUNNING -> WORK_SUCCESS
2018-08-08T21:51:15.906 GCUCJ [Work DEBUG] notifying parent of successful make-remote-dir bucket/a3/06/6a
2018-08-08T21:51:15.906 GCUCJ [Work DEBUG] notified put-remote-file bucket/a3/06/6a/bucket-a3066a7178325e7fa220571f54627052ebcee1effce3bad5e5d9c8dfc9fe191e.xdr.gz of completed child make-remote-dir bucket/a3/06/6a
2018-08-08T21:51:15.906 GCUCJ [Work DEBUG] advancing put-remote-file bucket/a3/06/6a/bucket-a3066a7178325e7fa220571f54627052ebcee1effce3bad5e5d9c8dfc9fe191e.xdr.gz
2018-08-08T21:51:15.906 GCUCJ [Work DEBUG] all 1 children of put-remote-file bucket/a3/06/6a/bucket-a3066a7178325e7fa220571f54627052ebcee1effce3bad5e5d9c8dfc9fe191e.xdr.gz successful, scheduling run
2018-08-08T21:51:15.906 GCUCJ [Work DEBUG] scheduling run of put-remote-file bucket/a3/06/6a/bucket-a3066a7178325e7fa220571f54627052ebcee1effce3bad5e5d9c8dfc9fe191e.xdr.gz
2018-08-08T21:51:15.906 GCUCJ [Work DEBUG] completed make-remote-dir bucket/a6/dc/7f
2018-08-08T21:51:15.906 GCUCJ [Work DEBUG] work make-remote-dir bucket/a6/dc/7f : WORK_RUNNING -> WORK_SUCCESS
2018-08-08T21:51:15.906 GCUCJ [Work DEBUG] notifying parent of successful make-remote-dir bucket/a6/dc/7f
2018-08-08T21:51:15.906 GCUCJ [Work DEBUG] notified put-remote-file bucket/a6/dc/7f/bucket-a6dc7f0ecc115c107d64b83734c683735a017e22c5ab494a530417ccf34ca97e.xdr.gz of completed child make-remote-dir bucket/a6/dc/7f
2018-08-08T21:51:15.906 GCUCJ [Work DEBUG] advancing put-remote-file bucket/a6/dc/7f/bucket-a6dc7f0ecc115c107d64b83734c683735a017e22c5ab494a530417ccf34ca97e.xdr.gz
2018-08-08T21:51:15.906 GCUCJ [Work DEBUG] all 1 children of put-remote-file bucket/a6/dc/7f/bucket-a6dc7f0ecc115c107d64b83734c683735a017e22c5ab494a530417ccf34ca97e.xdr.gz successful, scheduling run
2018-08-08T21:51:15.906 GCUCJ [Work DEBUG] scheduling run of put-remote-file bucket/a6/dc/7f/bucket-a6dc7f0ecc115c107d64b83734c683735a017e22c5ab494a530417ccf34ca97e.xdr.gz
2018-08-08T21:51:15.906 GCUCJ [Work DEBUG] completed make-remote-dir ledger/00/9f/69
2018-08-08T21:51:15.906 GCUCJ [Work DEBUG] work make-remote-dir ledger/00/9f/69 : WORK_RUNNING -> WORK_SUCCESS
2018-08-08T21:51:15.906 GCUCJ [Work DEBUG] notifying parent of successful make-remote-dir ledger/00/9f/69
2018-08-08T21:51:15.906 GCUCJ [Work DEBUG] notified put-remote-file ledger/00/9f/69/ledger-009f693f.xdr.gz of completed child make-remote-dir ledger/00/9f/69
2018-08-08T21:51:15.906 GCUCJ [Work DEBUG] advancing put-remote-file ledger/00/9f/69/ledger-009f693f.xdr.gz
2018-08-08T21:51:15.906 GCUCJ [Work DEBUG] all 1 children of put-remote-file ledger/00/9f/69/ledger-009f693f.xdr.gz successful, scheduling run
2018-08-08T21:51:15.906 GCUCJ [Work DEBUG] scheduling run of put-remote-file ledger/00/9f/69/ledger-009f693f.xdr.gz
2018-08-08T21:51:15.906 GCUCJ [Work DEBUG] completed make-remote-dir results/00/9f/69
2018-08-08T21:51:15.906 GCUCJ [Work DEBUG] work make-remote-dir results/00/9f/69 : WORK_RUNNING -> WORK_SUCCESS
2018-08-08T21:51:15.906 GCUCJ [Work DEBUG] notifying parent of successful make-remote-dir results/00/9f/69
2018-08-08T21:51:15.906 GCUCJ [Work DEBUG] notified put-remote-file results/00/9f/69/results-009f693f.xdr.gz of completed child make-remote-dir results/00/9f/69
2018-08-08T21:51:15.906 GCUCJ [Work DEBUG] advancing put-remote-file results/00/9f/69/results-009f693f.xdr.gz
2018-08-08T21:51:15.906 GCUCJ [Work DEBUG] all 1 children of put-remote-file results/00/9f/69/results-009f693f.xdr.gz successful, scheduling run
2018-08-08T21:51:15.906 GCUCJ [Work DEBUG] scheduling run of put-remote-file results/00/9f/69/results-009f693f.xdr.gz
2018-08-08T21:51:15.906 GCUCJ [Work DEBUG] completed make-remote-dir scp/00/9f/69
2018-08-08T21:51:15.906 GCUCJ [Work DEBUG] work make-remote-dir scp/00/9f/69 : WORK_RUNNING -> WORK_SUCCESS
2018-08-08T21:51:15.906 GCUCJ [Work DEBUG] notifying parent of successful make-remote-dir scp/00/9f/69
2018-08-08T21:51:15.906 GCUCJ [Work DEBUG] notified put-remote-file scp/00/9f/69/scp-009f693f.xdr.gz of completed child make-remote-dir scp/00/9f/69
2018-08-08T21:51:15.906 GCUCJ [Work DEBUG] advancing put-remote-file scp/00/9f/69/scp-009f693f.xdr.gz
2018-08-08T21:51:15.906 GCUCJ [Work DEBUG] all 1 children of put-remote-file scp/00/9f/69/scp-009f693f.xdr.gz successful, scheduling run
2018-08-08T21:51:15.906 GCUCJ [Work DEBUG] scheduling run of put-remote-file scp/00/9f/69/scp-009f693f.xdr.gz
2018-08-08T21:51:15.906 GCUCJ [Work DEBUG] completed make-remote-dir transactions/00/9f/69
2018-08-08T21:51:15.906 GCUCJ [Work DEBUG] work make-remote-dir transactions/00/9f/69 : WORK_RUNNING -> WORK_SUCCESS
2018-08-08T21:51:15.906 GCUCJ [Work DEBUG] notifying parent of successful make-remote-dir transactions/00/9f/69
2018-08-08T21:51:15.906 GCUCJ [Work DEBUG] notified put-remote-file transactions/00/9f/69/transactions-009f693f.xdr.gz of completed child make-remote-dir transactions/00/9f/69
2018-08-08T21:51:15.906 GCUCJ [Work DEBUG] advancing put-remote-file transactions/00/9f/69/transactions-009f693f.xdr.gz
2018-08-08T21:51:15.906 GCUCJ [Work DEBUG] all 1 children of put-remote-file transactions/00/9f/69/transactions-009f693f.xdr.gz successful, scheduling run
2018-08-08T21:51:15.906 GCUCJ [Work DEBUG] scheduling run of put-remote-file transactions/00/9f/69/transactions-009f693f.xdr.gz
2018-08-08T21:51:15.907 GCUCJ [Work DEBUG] starting put-remote-file bucket/0c/f7/26/bucket-0cf726fafa2ba7b3da004fc0c3fbe723adcfc204f7fd01854cf97d86d2b3ea11.xdr.gz
2018-08-08T21:51:15.907 GCUCJ [Process DEBUG] Running: aws s3 cp --region eu-west-1 /data/buckets/bucket-0cf726fafa2ba7b3da004fc0c3fbe723adcfc204f7fd01854cf97d86d2b3ea11.xdr.gz s3://history.stellar.org/prd/core-testnet/core_testnet_002/bucket/0c/f7/26/bucket-0cf726fafa2ba7b3da004fc0c3fbe723adcfc204f7fd01854cf97d86d2b3ea11.xdr.gz
2018-08-08T21:51:16.098 GCUCJ [Work DEBUG] running put-remote-file bucket/0c/f7/26/bucket-0cf726fafa2ba7b3da004fc0c3fbe723adcfc204f7fd01854cf97d86d2b3ea11.xdr.gz
2018-08-08T21:51:16.098 GCUCJ [Work DEBUG] work put-remote-file bucket/0c/f7/26/bucket-0cf726fafa2ba7b3da004fc0c3fbe723adcfc204f7fd01854cf97d86d2b3ea11.xdr.gz : WORK_PENDING -> WORK_RUNNING
2018-08-08T21:51:16.098 GCUCJ [Work DEBUG] starting put-remote-file bucket/17/24/d7/bucket-1724d77be46fa62a0ed46b106a3e47af1c0c8d42deac4b8bc98d505755e0437a.xdr.gz
2018-08-08T21:51:16.098 GCUCJ [Process DEBUG] Running: aws s3 cp --region eu-west-1 /data/buckets/bucket-1724d77be46fa62a0ed46b106a3e47af1c0c8d42deac4b8bc98d505755e0437a.xdr.gz s3://history.stellar.org/prd/core-testnet/core_testnet_002/bucket/17/24/d7/bucket-1724d77be46fa62a0ed46b106a3e47af1c0c8d42deac4b8bc98d505755e0437a.xdr.gz
2018-08-08T21:51:16.319 GCUCJ [Work DEBUG] running put-remote-file bucket/17/24/d7/bucket-1724d77be46fa62a0ed46b106a3e47af1c0c8d42deac4b8bc98d505755e0437a.xdr.gz
2018-08-08T21:51:16.319 GCUCJ [Work DEBUG] work put-remote-file bucket/17/24/d7/bucket-1724d77be46fa62a0ed46b106a3e47af1c0c8d42deac4b8bc98d505755e0437a.xdr.gz : WORK_PENDING -> WORK_RUNNING
2018-08-08T21:51:16.319 GCUCJ [Work DEBUG] starting put-remote-file bucket/19/01/ee/bucket-1901eea7636ae2363e6989325da37a84c2f038ce9b94f6484a86cff37ca82217.xdr.gz
2018-08-08T21:51:16.319 GCUCJ [Process DEBUG] Running: aws s3 cp --region eu-west-1 /data/buckets/bucket-1901eea7636ae2363e6989325da37a84c2f038ce9b94f6484a86cff37ca82217.xdr.gz s3://history.stellar.org/prd/core-testnet/core_testnet_002/bucket/19/01/ee/bucket-1901eea7636ae2363e6989325da37a84c2f038ce9b94f6484a86cff37ca82217.xdr.gz
2018-08-08T21:51:16.657 GCUCJ [Work DEBUG] running put-remote-file bucket/19/01/ee/bucket-1901eea7636ae2363e6989325da37a84c2f038ce9b94f6484a86cff37ca82217.xdr.gz
2018-08-08T21:51:16.657 GCUCJ [Work DEBUG] work put-remote-file bucket/19/01/ee/bucket-1901eea7636ae2363e6989325da37a84c2f038ce9b94f6484a86cff37ca82217.xdr.gz : WORK_PENDING -> WORK_RUNNING
2018-08-08T21:51:16.657 GCUCJ [Work DEBUG] starting put-remote-file bucket/36/90/5c/bucket-36905c16c0d345923c5fa8d0adc28fe03c1e47050e51579075064f159e7f16bd.xdr.gz
2018-08-08T21:51:16.657 GCUCJ [Process DEBUG] Running: aws s3 cp --region eu-west-1 /data/buckets/bucket-36905c16c0d345923c5fa8d0adc28fe03c1e47050e51579075064f159e7f16bd.xdr.gz s3://history.stellar.org/prd/core-testnet/core_testnet_002/bucket/36/90/5c/bucket-36905c16c0d345923c5fa8d0adc28fe03c1e47050e51579075064f159e7f16bd.xdr.gz
2018-08-08T21:51:16.916 GCUCJ [Work DEBUG] running put-remote-file bucket/36/90/5c/bucket-36905c16c0d345923c5fa8d0adc28fe03c1e47050e51579075064f159e7f16bd.xdr.gz
2018-08-08T21:51:16.916 GCUCJ [Work DEBUG] work put-remote-file bucket/36/90/5c/bucket-36905c16c0d345923c5fa8d0adc28fe03c1e47050e51579075064f159e7f16bd.xdr.gz : WORK_PENDING -> WORK_RUNNING
2018-08-08T21:51:16.916 GCUCJ [Work DEBUG] starting put-remote-file bucket/3c/87/b3/bucket-3c87b3dfddadceec084ccbd22efb76aefc48425870388ca416777f766e12baec.xdr.gz
2018-08-08T21:51:16.916 GCUCJ [Process DEBUG] Running: aws s3 cp --region eu-west-1 /data/buckets/bucket-3c87b3dfddadceec084ccbd22efb76aefc48425870388ca416777f766e12baec.xdr.gz s3://history.stellar.org/prd/core-testnet/core_testnet_002/bucket/3c/87/b3/bucket-3c87b3dfddadceec084ccbd22efb76aefc48425870388ca416777f766e12baec.xdr.gz
2018-08-08T21:51:17.453 GCUCJ [Work DEBUG] running put-remote-file bucket/3c/87/b3/bucket-3c87b3dfddadceec084ccbd22efb76aefc48425870388ca416777f766e12baec.xdr.gz
2018-08-08T21:51:17.454 GCUCJ [Work DEBUG] work put-remote-file bucket/3c/87/b3/bucket-3c87b3dfddadceec084ccbd22efb76aefc48425870388ca416777f766e12baec.xdr.gz : WORK_PENDING -> WORK_RUNNING
2018-08-08T21:51:17.455 GCUCJ [Work DEBUG] starting put-remote-file bucket/72/7e/26/bucket-727e26fc2b8407ea155ef61365eebcb5377e90299f6805a04eba105209511f47.xdr.gz
2018-08-08T21:51:17.455 GCUCJ [Process DEBUG] Running: aws s3 cp --region eu-west-1 /data/buckets/bucket-727e26fc2b8407ea155ef61365eebcb5377e90299f6805a04eba105209511f47.xdr.gz s3://history.stellar.org/prd/core-testnet/core_testnet_002/bucket/72/7e/26/bucket-727e26fc2b8407ea155ef61365eebcb5377e90299f6805a04eba105209511f47.xdr.gz
Completed 13.8 KiB/13.8 KiB (25.0 KiB/s) with 1 file(s) remaining
upload: data/buckets/bucket-0cf726fafa2ba7b3da004fc0c3fbe723adcfc204f7fd01854cf97d86d2b3ea11.xdr.gz to s3://history.stellar.org/prd/core-testnet/core_testnet_002/bucket/0c/f7/26/bucket-0cf726fafa2ba7b3da004fc0c3fbe723adcfc204f7fd01854cf97d86d2b3ea11.xdr.gz
2018-08-08T21:51:17.873 GCUCJ [Work DEBUG] running put-remote-file bucket/72/7e/26/bucket-727e26fc2b8407ea155ef61365eebcb5377e90299f6805a04eba105209511f47.xdr.gz
2018-08-08T21:51:17.874 GCUCJ [Work DEBUG] work put-remote-file bucket/72/7e/26/bucket-727e26fc2b8407ea155ef61365eebcb5377e90299f6805a04eba105209511f47.xdr.gz : WORK_PENDING -> WORK_RUNNING
2018-08-08T21:51:17.875 GCUCJ [Work DEBUG] starting put-remote-file bucket/88/30/f7/bucket-8830f702c32ce5e8d493031453fba0ec6be2c1b07e3fc5d0123f9895ecbc5523.xdr.gz
2018-08-08T21:51:17.875 GCUCJ [Process DEBUG] Running: aws s3 cp --region eu-west-1 /data/buckets/bucket-8830f702c32ce5e8d493031453fba0ec6be2c1b07e3fc5d0123f9895ecbc5523.xdr.gz s3://history.stellar.org/prd/core-testnet/core_testnet_002/bucket/88/30/f7/bucket-8830f702c32ce5e8d493031453fba0ec6be2c1b07e3fc5d0123f9895ecbc5523.xdr.gz
Completed 695 Bytes/695 Bytes (1.5 KiB/s) with 1 file(s) remaining
upload: data/buckets/bucket-1724d77be46fa62a0ed46b106a3e47af1c0c8d42deac4b8bc98d505755e0437a.xdr.gz to s3://history.stellar.org/prd/core-testnet/core_testnet_002/bucket/17/24/d7/bucket-1724d77be46fa62a0ed46b106a3e47af1c0c8d42deac4b8bc98d505755e0437a.xdr.gz
2018-08-08T21:51:18.458 GCUCJ [Work DEBUG] running put-remote-file bucket/88/30/f7/bucket-8830f702c32ce5e8d493031453fba0ec6be2c1b07e3fc5d0123f9895ecbc5523.xdr.gz
2018-08-08T21:51:18.458 GCUCJ [Work DEBUG] work put-remote-file bucket/88/30/f7/bucket-8830f702c32ce5e8d493031453fba0ec6be2c1b07e3fc5d0123f9895ecbc5523.xdr.gz : WORK_PENDING -> WORK_RUNNING
2018-08-08T21:51:18.466 GCUCJ [Work DEBUG] starting put-remote-file bucket/8d/f1/e1/bucket-8df1e17d986608744a66130af542f00f8766717a4e551eeed01a5aac0bc19217.xdr.gz
2018-08-08T21:51:18.466 GCUCJ [Process DEBUG] Running: aws s3 cp --region eu-west-1 /data/buckets/bucket-8df1e17d986608744a66130af542f00f8766717a4e551eeed01a5aac0bc19217.xdr.gz s3://history.stellar.org/prd/core-testnet/core_testnet_002/bucket/8d/f1/e1/bucket-8df1e17d986608744a66130af542f00f8766717a4e551eeed01a5aac0bc19217.xdr.gz
Completed 561 Bytes/561 Bytes (1.2 KiB/s) with 1 file(s) remaining
upload: data/buckets/bucket-1901eea7636ae2363e6989325da37a84c2f038ce9b94f6484a86cff37ca82217.xdr.gz to s3://history.stellar.org/prd/core-testnet/core_testnet_002/bucket/19/01/ee/bucket-1901eea7636ae2363e6989325da37a84c2f038ce9b94f6484a86cff37ca82217.xdr.gz
2018-08-08T21:51:18.950 GCUCJ [Work DEBUG] running put-remote-file bucket/8d/f1/e1/bucket-8df1e17d986608744a66130af542f00f8766717a4e551eeed01a5aac0bc19217.xdr.gz
2018-08-08T21:51:18.950 GCUCJ [Work DEBUG] work put-remote-file bucket/8d/f1/e1/bucket-8df1e17d986608744a66130af542f00f8766717a4e551eeed01a5aac0bc19217.xdr.gz : WORK_PENDING -> WORK_RUNNING
2018-08-08T21:51:18.951 GCUCJ [Work DEBUG] starting put-remote-file bucket/a3/06/6a/bucket-a3066a7178325e7fa220571f54627052ebcee1effce3bad5e5d9c8dfc9fe191e.xdr.gz
2018-08-08T21:51:18.951 GCUCJ [Process DEBUG] Running: aws s3 cp --region eu-west-1 /data/buckets/bucket-a3066a7178325e7fa220571f54627052ebcee1effce3bad5e5d9c8dfc9fe191e.xdr.gz s3://history.stellar.org/prd/core-testnet/core_testnet_002/bucket/a3/06/6a/bucket-a3066a7178325e7fa220571f54627052ebcee1effce3bad5e5d9c8dfc9fe191e.xdr.gz
Completed 4.2 KiB/4.2 KiB (9.3 KiB/s) with 1 file(s) remaining
upload: data/buckets/bucket-36905c16c0d345923c5fa8d0adc28fe03c1e47050e51579075064f159e7f16bd.xdr.gz to s3://history.stellar.org/prd/core-testnet/core_testnet_002/bucket/36/90/5c/bucket-36905c16c0d345923c5fa8d0adc28fe03c1e47050e51579075064f159e7f16bd.xdr.gz
2018-08-08T21:51:19.443 GCUCJ [Work DEBUG] running put-remote-file bucket/a3/06/6a/bucket-a3066a7178325e7fa220571f54627052ebcee1effce3bad5e5d9c8dfc9fe191e.xdr.gz
2018-08-08T21:51:19.443 GCUCJ [Work DEBUG] work put-remote-file bucket/a3/06/6a/bucket-a3066a7178325e7fa220571f54627052ebcee1effce3bad5e5d9c8dfc9fe191e.xdr.gz : WORK_PENDING -> WORK_RUNNING
2018-08-08T21:51:19.443 GCUCJ [Work DEBUG] starting put-remote-file bucket/a6/dc/7f/bucket-a6dc7f0ecc115c107d64b83734c683735a017e22c5ab494a530417ccf34ca97e.xdr.gz
2018-08-08T21:51:19.443 GCUCJ [Process DEBUG] Running: aws s3 cp --region eu-west-1 /data/buckets/bucket-a6dc7f0ecc115c107d64b83734c683735a017e22c5ab494a530417ccf34ca97e.xdr.gz s3://history.stellar.org/prd/core-testnet/core_testnet_002/bucket/a6/dc/7f/bucket-a6dc7f0ecc115c107d64b83734c683735a017e22c5ab494a530417ccf34ca97e.xdr.gz
Completed 3.2 KiB/3.2 KiB (7.2 KiB/s) with 1 file(s) remaining
upload: data/buckets/bucket-3c87b3dfddadceec084ccbd22efb76aefc48425870388ca416777f766e12baec.xdr.gz to s3://history.stellar.org/prd/core-testnet/core_testnet_002/bucket/3c/87/b3/bucket-3c87b3dfddadceec084ccbd22efb76aefc48425870388ca416777f766e12baec.xdr.gz
Completed 3.5 KiB/3.5 KiB (8.8 KiB/s) with 1 file(s) remaining
upload: data/buckets/bucket-727e26fc2b8407ea155ef61365eebcb5377e90299f6805a04eba105209511f47.xdr.gz to s3://history.stellar.org/prd/core-testnet/core_testnet_002/bucket/72/7e/26/bucket-727e26fc2b8407ea155ef61365eebcb5377e90299f6805a04eba105209511f47.xdr.gz
2018-08-08T21:51:19.865 GCUCJ [Work DEBUG] running put-remote-file bucket/a6/dc/7f/bucket-a6dc7f0ecc115c107d64b83734c683735a017e22c5ab494a530417ccf34ca97e.xdr.gz
2018-08-08T21:51:19.865 GCUCJ [Work DEBUG] work put-remote-file bucket/a6/dc/7f/bucket-a6dc7f0ecc115c107d64b83734c683735a017e22c5ab494a530417ccf34ca97e.xdr.gz : WORK_PENDING -> WORK_RUNNING
2018-08-08T21:51:19.865 GCUCJ [Work DEBUG] starting put-remote-file ledger/00/9f/69/ledger-009f693f.xdr.gz
2018-08-08T21:51:19.865 GCUCJ [Process DEBUG] Running: aws s3 cp --region eu-west-1 /data/buckets/tmp/snapshot-ec421db5b63f34bc/ledger-009f693f.xdr.gz s3://history.stellar.org/prd/core-testnet/core_testnet_002/ledger/00/9f/69/ledger-009f693f.xdr.gz
2018-08-08T21:51:20.354 GCUCJ [Work DEBUG] running put-remote-file ledger/00/9f/69/ledger-009f693f.xdr.gz
2018-08-08T21:51:20.354 GCUCJ [Work DEBUG] work put-remote-file ledger/00/9f/69/ledger-009f693f.xdr.gz : WORK_PENDING -> WORK_RUNNING
2018-08-08T21:51:20.354 GCUCJ [Work DEBUG] starting put-remote-file results/00/9f/69/results-009f693f.xdr.gz
2018-08-08T21:51:20.354 GCUCJ [Process DEBUG] Running: aws s3 cp --region eu-west-1 /data/buckets/tmp/snapshot-ec421db5b63f34bc/results-009f693f.xdr.gz s3://history.stellar.org/prd/core-testnet/core_testnet_002/results/00/9f/69/results-009f693f.xdr.gz
Completed 541 Bytes/541 Bytes (1.2 KiB/s) with 1 file(s) remaining
upload: data/buckets/bucket-8830f702c32ce5e8d493031453fba0ec6be2c1b07e3fc5d0123f9895ecbc5523.xdr.gz to s3://history.stellar.org/prd/core-testnet/core_testnet_002/bucket/88/30/f7/bucket-8830f702c32ce5e8d493031453fba0ec6be2c1b07e3fc5d0123f9895ecbc5523.xdr.gz
2018-08-08T21:51:20.905 GCUCJ [Work DEBUG] running put-remote-file results/00/9f/69/results-009f693f.xdr.gz
2018-08-08T21:51:20.906 GCUCJ [Work DEBUG] work put-remote-file results/00/9f/69/results-009f693f.xdr.gz : WORK_PENDING -> WORK_RUNNING
2018-08-08T21:51:20.906 GCUCJ [Work DEBUG] starting put-remote-file scp/00/9f/69/scp-009f693f.xdr.gz
2018-08-08T21:51:20.907 GCUCJ [Process DEBUG] Running: aws s3 cp --region eu-west-1 /data/buckets/tmp/snapshot-ec421db5b63f34bc/scp-009f693f.xdr.gz s3://history.stellar.org/prd/core-testnet/core_testnet_002/scp/00/9f/69/scp-009f693f.xdr.gz
Completed 2.6 KiB/2.6 KiB (5.6 KiB/s) with 1 file(s) remaining
upload: data/buckets/bucket-8df1e17d986608744a66130af542f00f8766717a4e551eeed01a5aac0bc19217.xdr.gz to s3://history.stellar.org/prd/core-testnet/core_testnet_002/bucket/8d/f1/e1/bucket-8df1e17d986608744a66130af542f00f8766717a4e551eeed01a5aac0bc19217.xdr.gz
2018-08-08T21:51:21.269 GCUCJ [Work DEBUG] running put-remote-file scp/00/9f/69/scp-009f693f.xdr.gz
2018-08-08T21:51:21.270 GCUCJ [Work DEBUG] work put-remote-file scp/00/9f/69/scp-009f693f.xdr.gz : WORK_PENDING -> WORK_RUNNING
2018-08-08T21:51:21.270 GCUCJ [Work DEBUG] starting put-remote-file transactions/00/9f/69/transactions-009f693f.xdr.gz
2018-08-08T21:51:21.271 GCUCJ [Process DEBUG] Running: aws s3 cp --region eu-west-1 /data/buckets/tmp/snapshot-ec421db5b63f34bc/transactions-009f693f.xdr.gz s3://history.stellar.org/prd/core-testnet/core_testnet_002/transactions/00/9f/69/transactions-009f693f.xdr.gz
Completed 336 Bytes/336 Bytes (769 Bytes/s) with 1 file(s) remaining
upload: data/buckets/bucket-a3066a7178325e7fa220571f54627052ebcee1effce3bad5e5d9c8dfc9fe191e.xdr.gz to s3://history.stellar.org/prd/core-testnet/core_testnet_002/bucket/a3/06/6a/bucket-a3066a7178325e7fa220571f54627052ebcee1effce3bad5e5d9c8dfc9fe191e.xdr.gz
2018-08-08T21:51:21.738 GCUCJ [Work DEBUG] running put-remote-file transactions/00/9f/69/transactions-009f693f.xdr.gz
2018-08-08T21:51:21.738 GCUCJ [Work DEBUG] work put-remote-file transactions/00/9f/69/transactions-009f693f.xdr.gz : WORK_PENDING -> WORK_RUNNING
2018-08-08T21:51:21.739 GCUCJ [Process DEBUG] found 9 child processes that terminated
2018-08-08T21:51:21.739 GCUCJ [Process DEBUG] process 5265 exited 0: aws s3 cp --region eu-west-1 /data/buckets/bucket-0cf726fafa2ba7b3da004fc0c3fbe723adcfc204f7fd01854cf97d86d2b3ea11.xdr.gz s3://history.stellar.org/prd/core-testnet/core_testnet_002/bucket/0c/f7/26/bucket-0cf726fafa2ba7b3da004fc0c3fbe723adcfc204f7fd01854cf97d86d2b3ea11.xdr.gz
2018-08-08T21:51:21.740 GCUCJ [Process DEBUG] process 5267 exited 0: aws s3 cp --region eu-west-1 /data/buckets/bucket-1724d77be46fa62a0ed46b106a3e47af1c0c8d42deac4b8bc98d505755e0437a.xdr.gz s3://history.stellar.org/prd/core-testnet/core_testnet_002/bucket/17/24/d7/bucket-1724d77be46fa62a0ed46b106a3e47af1c0c8d42deac4b8bc98d505755e0437a.xdr.gz
2018-08-08T21:51:21.740 GCUCJ [Process DEBUG] process 5269 exited 0: aws s3 cp --region eu-west-1 /data/buckets/bucket-1901eea7636ae2363e6989325da37a84c2f038ce9b94f6484a86cff37ca82217.xdr.gz s3://history.stellar.org/prd/core-testnet/core_testnet_002/bucket/19/01/ee/bucket-1901eea7636ae2363e6989325da37a84c2f038ce9b94f6484a86cff37ca82217.xdr.gz
2018-08-08T21:51:21.741 GCUCJ [Process DEBUG] process 5273 exited 0: aws s3 cp --region eu-west-1 /data/buckets/bucket-36905c16c0d345923c5fa8d0adc28fe03c1e47050e51579075064f159e7f16bd.xdr.gz s3://history.stellar.org/prd/core-testnet/core_testnet_002/bucket/36/90/5c/bucket-36905c16c0d345923c5fa8d0adc28fe03c1e47050e51579075064f159e7f16bd.xdr.gz
2018-08-08T21:51:21.741 GCUCJ [Process DEBUG] process 5280 exited 0: aws s3 cp --region eu-west-1 /data/buckets/bucket-3c87b3dfddadceec084ccbd22efb76aefc48425870388ca416777f766e12baec.xdr.gz s3://history.stellar.org/prd/core-testnet/core_testnet_002/bucket/3c/87/b3/bucket-3c87b3dfddadceec084ccbd22efb76aefc48425870388ca416777f766e12baec.xdr.gz
2018-08-08T21:51:21.742 GCUCJ [Process DEBUG] process 5287 exited 0: aws s3 cp --region eu-west-1 /data/buckets/bucket-727e26fc2b8407ea155ef61365eebcb5377e90299f6805a04eba105209511f47.xdr.gz s3://history.stellar.org/prd/core-testnet/core_testnet_002/bucket/72/7e/26/bucket-727e26fc2b8407ea155ef61365eebcb5377e90299f6805a04eba105209511f47.xdr.gz
2018-08-08T21:51:21.742 GCUCJ [Process DEBUG] process 5294 exited 0: aws s3 cp --region eu-west-1 /data/buckets/bucket-8830f702c32ce5e8d493031453fba0ec6be2c1b07e3fc5d0123f9895ecbc5523.xdr.gz s3://history.stellar.org/prd/core-testnet/core_testnet_002/bucket/88/30/f7/bucket-8830f702c32ce5e8d493031453fba0ec6be2c1b07e3fc5d0123f9895ecbc5523.xdr.gz
2018-08-08T21:51:21.743 GCUCJ [Process DEBUG] process 5301 exited 0: aws s3 cp --region eu-west-1 /data/buckets/bucket-8df1e17d986608744a66130af542f00f8766717a4e551eeed01a5aac0bc19217.xdr.gz s3://history.stellar.org/prd/core-testnet/core_testnet_002/bucket/8d/f1/e1/bucket-8df1e17d986608744a66130af542f00f8766717a4e551eeed01a5aac0bc19217.xdr.gz
2018-08-08T21:51:21.743 GCUCJ [Process DEBUG] process 5311 exited 0: aws s3 cp --region eu-west-1 /data/buckets/bucket-a3066a7178325e7fa220571f54627052ebcee1effce3bad5e5d9c8dfc9fe191e.xdr.gz s3://history.stellar.org/prd/core-testnet/core_testnet_002/bucket/a3/06/6a/bucket-a3066a7178325e7fa220571f54627052ebcee1effce3bad5e5d9c8dfc9fe191e.xdr.gz
2018-08-08T21:51:21.752 GCUCJ [Work DEBUG] completed put-remote-file bucket/0c/f7/26/bucket-0cf726fafa2ba7b3da004fc0c3fbe723adcfc204f7fd01854cf97d86d2b3ea11.xdr.gz
2018-08-08T21:51:21.752 GCUCJ [Work DEBUG] work put-remote-file bucket/0c/f7/26/bucket-0cf726fafa2ba7b3da004fc0c3fbe723adcfc204f7fd01854cf97d86d2b3ea11.xdr.gz : WORK_RUNNING -> WORK_SUCCESS
2018-08-08T21:51:21.752 GCUCJ [Work DEBUG] notifying parent of successful put-remote-file bucket/0c/f7/26/bucket-0cf726fafa2ba7b3da004fc0c3fbe723adcfc204f7fd01854cf97d86d2b3ea11.xdr.gz
2018-08-08T21:51:21.752 GCUCJ [Work DEBUG] notified put-files of completed child put-remote-file bucket/0c/f7/26/bucket-0cf726fafa2ba7b3da004fc0c3fbe723adcfc204f7fd01854cf97d86d2b3ea11.xdr.gz
2018-08-08T21:51:21.752 GCUCJ [Work DEBUG] advancing put-files
2018-08-08T21:51:21.752 GCUCJ [Work DEBUG] completed put-remote-file bucket/17/24/d7/bucket-1724d77be46fa62a0ed46b106a3e47af1c0c8d42deac4b8bc98d505755e0437a.xdr.gz
2018-08-08T21:51:21.752 GCUCJ [Work DEBUG] work put-remote-file bucket/17/24/d7/bucket-1724d77be46fa62a0ed46b106a3e47af1c0c8d42deac4b8bc98d505755e0437a.xdr.gz : WORK_RUNNING -> WORK_SUCCESS
2018-08-08T21:51:21.752 GCUCJ [Work DEBUG] notifying parent of successful put-remote-file bucket/17/24/d7/bucket-1724d77be46fa62a0ed46b106a3e47af1c0c8d42deac4b8bc98d505755e0437a.xdr.gz
2018-08-08T21:51:21.752 GCUCJ [Work DEBUG] notified put-files of completed child put-remote-file bucket/17/24/d7/bucket-1724d77be46fa62a0ed46b106a3e47af1c0c8d42deac4b8bc98d505755e0437a.xdr.gz
2018-08-08T21:51:21.752 GCUCJ [Work DEBUG] advancing put-files
2018-08-08T21:51:21.752 GCUCJ [Work DEBUG] completed put-remote-file bucket/19/01/ee/bucket-1901eea7636ae2363e6989325da37a84c2f038ce9b94f6484a86cff37ca82217.xdr.gz
2018-08-08T21:51:21.752 GCUCJ [Work DEBUG] work put-remote-file bucket/19/01/ee/bucket-1901eea7636ae2363e6989325da37a84c2f038ce9b94f6484a86cff37ca82217.xdr.gz : WORK_RUNNING -> WORK_SUCCESS
2018-08-08T21:51:21.752 GCUCJ [Work DEBUG] notifying parent of successful put-remote-file bucket/19/01/ee/bucket-1901eea7636ae2363e6989325da37a84c2f038ce9b94f6484a86cff37ca82217.xdr.gz
2018-08-08T21:51:21.752 GCUCJ [Work DEBUG] notified put-files of completed child put-remote-file bucket/19/01/ee/bucket-1901eea7636ae2363e6989325da37a84c2f038ce9b94f6484a86cff37ca82217.xdr.gz
2018-08-08T21:51:21.752 GCUCJ [Work DEBUG] advancing put-files
2018-08-08T21:51:21.752 GCUCJ [Work DEBUG] completed put-remote-file bucket/36/90/5c/bucket-36905c16c0d345923c5fa8d0adc28fe03c1e47050e51579075064f159e7f16bd.xdr.gz
2018-08-08T21:51:21.752 GCUCJ [Work DEBUG] work put-remote-file bucket/36/90/5c/bucket-36905c16c0d345923c5fa8d0adc28fe03c1e47050e51579075064f159e7f16bd.xdr.gz : WORK_RUNNING -> WORK_SUCCESS
2018-08-08T21:51:21.752 GCUCJ [Work DEBUG] notifying parent of successful put-remote-file bucket/36/90/5c/bucket-36905c16c0d345923c5fa8d0adc28fe03c1e47050e51579075064f159e7f16bd.xdr.gz
2018-08-08T21:51:21.752 GCUCJ [Work DEBUG] notified put-files of completed child put-remote-file bucket/36/90/5c/bucket-36905c16c0d345923c5fa8d0adc28fe03c1e47050e51579075064f159e7f16bd.xdr.gz
2018-08-08T21:51:21.752 GCUCJ [Work DEBUG] advancing put-files
2018-08-08T21:51:21.752 GCUCJ [Work DEBUG] completed put-remote-file bucket/3c/87/b3/bucket-3c87b3dfddadceec084ccbd22efb76aefc48425870388ca416777f766e12baec.xdr.gz
2018-08-08T21:51:21.752 GCUCJ [Work DEBUG] work put-remote-file bucket/3c/87/b3/bucket-3c87b3dfddadceec084ccbd22efb76aefc48425870388ca416777f766e12baec.xdr.gz : WORK_RUNNING -> WORK_SUCCESS
2018-08-08T21:51:21.752 GCUCJ [Work DEBUG] notifying parent of successful put-remote-file bucket/3c/87/b3/bucket-3c87b3dfddadceec084ccbd22efb76aefc48425870388ca416777f766e12baec.xdr.gz
2018-08-08T21:51:21.752 GCUCJ [Work DEBUG] notified put-files of completed child put-remote-file bucket/3c/87/b3/bucket-3c87b3dfddadceec084ccbd22efb76aefc48425870388ca416777f766e12baec.xdr.gz
2018-08-08T21:51:21.752 GCUCJ [Work DEBUG] advancing put-files
2018-08-08T21:51:21.752 GCUCJ [Work DEBUG] completed put-remote-file bucket/72/7e/26/bucket-727e26fc2b8407ea155ef61365eebcb5377e90299f6805a04eba105209511f47.xdr.gz
2018-08-08T21:51:21.752 GCUCJ [Work DEBUG] work put-remote-file bucket/72/7e/26/bucket-727e26fc2b8407ea155ef61365eebcb5377e90299f6805a04eba105209511f47.xdr.gz : WORK_RUNNING -> WORK_SUCCESS
2018-08-08T21:51:21.752 GCUCJ [Work DEBUG] notifying parent of successful put-remote-file bucket/72/7e/26/bucket-727e26fc2b8407ea155ef61365eebcb5377e90299f6805a04eba105209511f47.xdr.gz
2018-08-08T21:51:21.752 GCUCJ [Work DEBUG] notified put-files of completed child put-remote-file bucket/72/7e/26/bucket-727e26fc2b8407ea155ef61365eebcb5377e90299f6805a04eba105209511f47.xdr.gz
2018-08-08T21:51:21.752 GCUCJ [Work DEBUG] advancing put-files
2018-08-08T21:51:21.752 GCUCJ [Work DEBUG] completed put-remote-file bucket/88/30/f7/bucket-8830f702c32ce5e8d493031453fba0ec6be2c1b07e3fc5d0123f9895ecbc5523.xdr.gz
2018-08-08T21:51:21.752 GCUCJ [Work DEBUG] work put-remote-file bucket/88/30/f7/bucket-8830f702c32ce5e8d493031453fba0ec6be2c1b07e3fc5d0123f9895ecbc5523.xdr.gz : WORK_RUNNING -> WORK_SUCCESS
2018-08-08T21:51:21.752 GCUCJ [Work DEBUG] notifying parent of successful put-remote-file bucket/88/30/f7/bucket-8830f702c32ce5e8d493031453fba0ec6be2c1b07e3fc5d0123f9895ecbc5523.xdr.gz
2018-08-08T21:51:21.752 GCUCJ [Work DEBUG] notified put-files of completed child put-remote-file bucket/88/30/f7/bucket-8830f702c32ce5e8d493031453fba0ec6be2c1b07e3fc5d0123f9895ecbc5523.xdr.gz
2018-08-08T21:51:21.752 GCUCJ [Work DEBUG] advancing put-files
2018-08-08T21:51:21.752 GCUCJ [Work DEBUG] completed put-remote-file bucket/8d/f1/e1/bucket-8df1e17d986608744a66130af542f00f8766717a4e551eeed01a5aac0bc19217.xdr.gz
2018-08-08T21:51:21.752 GCUCJ [Work DEBUG] work put-remote-file bucket/8d/f1/e1/bucket-8df1e17d986608744a66130af542f00f8766717a4e551eeed01a5aac0bc19217.xdr.gz : WORK_RUNNING -> WORK_SUCCESS
2018-08-08T21:51:21.752 GCUCJ [Work DEBUG] notifying parent of successful put-remote-file bucket/8d/f1/e1/bucket-8df1e17d986608744a66130af542f00f8766717a4e551eeed01a5aac0bc19217.xdr.gz
2018-08-08T21:51:21.752 GCUCJ [Work DEBUG] notified put-files of completed child put-remote-file bucket/8d/f1/e1/bucket-8df1e17d986608744a66130af542f00f8766717a4e551eeed01a5aac0bc19217.xdr.gz
2018-08-08T21:51:21.752 GCUCJ [Work DEBUG] advancing put-files
2018-08-08T21:51:21.752 GCUCJ [Work DEBUG] completed put-remote-file bucket/a3/06/6a/bucket-a3066a7178325e7fa220571f54627052ebcee1effce3bad5e5d9c8dfc9fe191e.xdr.gz
2018-08-08T21:51:21.752 GCUCJ [Work DEBUG] work put-remote-file bucket/a3/06/6a/bucket-a3066a7178325e7fa220571f54627052ebcee1effce3bad5e5d9c8dfc9fe191e.xdr.gz : WORK_RUNNING -> WORK_SUCCESS
2018-08-08T21:51:21.752 GCUCJ [Work DEBUG] notifying parent of successful put-remote-file bucket/a3/06/6a/bucket-a3066a7178325e7fa220571f54627052ebcee1effce3bad5e5d9c8dfc9fe191e.xdr.gz
2018-08-08T21:51:21.752 GCUCJ [Work DEBUG] notified put-files of completed child put-remote-file bucket/a3/06/6a/bucket-a3066a7178325e7fa220571f54627052ebcee1effce3bad5e5d9c8dfc9fe191e.xdr.gz
2018-08-08T21:51:21.752 GCUCJ [Work DEBUG] advancing put-files

2018-08-08T21:51:22.044 GCUCJ [Ledger INFO] Got consensus: [seq=10447169, prev=bb7569, tx_count=0, sv: [  txH: 1158df, ct: 1533765072, upgrades: [ ] ]]
2018-08-08T21:51:22.044 GCUCJ [Ledger DEBUG] starting closeLedger() on ledgerSeq=10447169
2018-08-08T21:51:22.044 GCUCJ [Ledger DEBUG] processing fees and sequence numbers
2018-08-08T21:51:22.045 GCUCJ [Bucket DEBUG] Deleting empty bucket file /data/buckets/tmp/bucket-f2a2103581f3f33e/tmp-bucket-1945165086b64866.xdr
2018-08-08T21:51:22.045 GCUCJ [Bucket DEBUG] Deleting empty bucket file /data/buckets/tmp/bucket-f2a2103581f3f33e/tmp-bucket-5100055b014225de.xdr
2018-08-08T21:51:22.046 GCUCJ [Bucket DEBUG] Deleting empty bucket file /data/buckets/tmp/bucket-f2a2103581f3f33e/tmp-bucket-f35c5f025ee59e96.xdr
2018-08-08T21:51:22.046 GCUCJ [Bucket DEBUG] Deleting bucket file /data/buckets/tmp/bucket-f2a2103581f3f33e/tmp-bucket-37d1b02198f6059e.xdr that is redundant with existing bucket
2018-08-08T21:51:22.050 GCUCJ [Ledger DEBUG] Advancing LCL: [seq=10447168, hash=bb7569] -> [seq=10447169, hash=c6f5f0]
2018-08-08T21:51:22.050 GCUCJ [Ledger DEBUG] New current ledger: seq=10447170
2018-08-08T21:51:22.056 GCUCJ [History INFO] Publishing 1 queued checkpoints [10447167-10447167]: Succeded: prepare-snapshot
2018-08-08T21:51:22.056 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 0000000000000000000000000000000000000000000000000000000000000000 referenced by publish queue
2018-08-08T21:51:22.056 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 02c358035b6470656fb425862de1d2eba1f2fdf52254b4f4756062ffefe89b4b referenced by publish queue
2018-08-08T21:51:22.056 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 0ed0b722ee76db5634e32a4e8ad087d7a4ddcdb3fa9a8939533a9dada5d82a10 referenced by publish queue
2018-08-08T21:51:22.056 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 116fe640258b03d0015b20ab2225e83379436bdc9b3306a7affb7abff20f33d5 referenced by publish queue
2018-08-08T21:51:22.056 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 16161d7ddfea5ece936f0c330eadeb60210b9b7348f9e306d6e6806ef9f9b4d6 referenced by publish queue
2018-08-08T21:51:22.056 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 1724d77be46fa62a0ed46b106a3e47af1c0c8d42deac4b8bc98d505755e0437a referenced by publish queue
2018-08-08T21:51:22.056 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 1dd2ec206ea293d1072611d6771673e534f3fa47119189ed43cbdbdf473ae4d6 referenced by publish queue
2018-08-08T21:51:22.056 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 1e1cdc270817cdd95246632fd78d5b482ec559ecae390e2180096f7b58bc77c5 referenced by publish queue
2018-08-08T21:51:22.056 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 205506f667460e78f0b9f7758b3c20fd4a232120ef5b51a44d0f8e44fb842177 referenced by publish queue
2018-08-08T21:51:22.056 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 25975bcc1ea7ee891e91f7a0936c17bf407e9f7c58ba5087fccbebb45c94cfbb referenced by publish queue
2018-08-08T21:51:22.056 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 28f84eace1e9911618dc61739842e598235d90e75f6ea2f0a4cdf907f2bd3128 referenced by publish queue
2018-08-08T21:51:22.056 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 2a6caeeb25bcffd5022e35c16a7d03b541e3a89bf5f48f18e8af614884aa6a90 referenced by publish queue
2018-08-08T21:51:22.056 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 2c6f2ee1ffb097d6cce2cef39538b53d508be370d74f4626d8cf32d2758e1c8f referenced by publish queue
2018-08-08T21:51:22.056 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 2d656019fa60c6a2e45e5df66107ff75ccb82e01d40e5433901476e02168aa08 referenced by publish queue
2018-08-08T21:51:22.056 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 2d931869ff7803e86877a848a083bd7324b59b928411b5d298e839462aeae126 referenced by publish queue
2018-08-08T21:51:22.056 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 327ab7b0a3691f9bc7bde7b46b0b9842ae2c9a5ef15e7992ee11c88f50e08b9d referenced by publish queue
2018-08-08T21:51:22.056 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 38dd5f08b431213a1374b9a4f36ee3ee126b9f62194a7dd196ae88a81560224e referenced by publish queue
2018-08-08T21:51:22.056 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 3c600072de517bc4868d7432182ab3f62920dad3c58e7ba8b196c09810cca4c2 referenced by publish queue
2018-08-08T21:51:22.056 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 3c87b3dfddadceec084ccbd22efb76aefc48425870388ca416777f766e12baec referenced by publish queue
2018-08-08T21:51:22.056 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 3dde3499fd0d8a4297bbbe957759657d98c7d7ca0bc5b4581dd9f079c54e7392 referenced by publish queue
2018-08-08T21:51:22.056 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 46a854adf19cadd698d6c36ccaed78f645889d85ddd076e2e107799fec0172a1 referenced by publish queue
2018-08-08T21:51:22.056 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 4a303763c10813263c0d78d314642544b64fb529e5f4514122edf9986738c7a0 referenced by publish queue
2018-08-08T21:51:22.056 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 4c5fdd8ac9daaaf5a6b4ef0b69da27fa39b7d4e35f503798af54c27b070db671 referenced by publish queue
2018-08-08T21:51:22.057 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 4f09c585b0e314f87ff64f0fca5346ed36d47fb12b802d273740043bf72a51e6 referenced by publish queue
2018-08-08T21:51:22.057 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 53298af6ee9c77541610994a8af96f60f26fe2ca8e995302a3da9911f85092ea referenced by publish queue
2018-08-08T21:51:22.057 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 57257b83f9c5ccb7dd24c56aa7a5101bf9ba0d350e313d985b6ebe5535846a12 referenced by publish queue
2018-08-08T21:51:22.057 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 5794c7922c078248a1f497fbeaa0cde0708b3d6ece98ac59ddb6ecdadb7c9010 referenced by publish queue
2018-08-08T21:51:22.057 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 5e2cc0a2fc6dd474a2aa3baef80cf8fc425723465be6f21489ce730c12ce0548 referenced by publish queue
2018-08-08T21:51:22.057 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 600af102fb6bce6a07470d9a6f476a1b48444e4b41336a518247fba8e609a18d referenced by publish queue
2018-08-08T21:51:22.057 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 6c56152fd52a33e6c1589ca225f5107fd18ecad19c036c23f6f3a7bfa46e42d1 referenced by publish queue
2018-08-08T21:51:22.057 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 6fbb2e480cd258caa7f13a2d74c9306a0a7c12e9c74d926f249d0e78fa8e04a5 referenced by publish queue
2018-08-08T21:51:22.057 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 727e26fc2b8407ea155ef61365eebcb5377e90299f6805a04eba105209511f47 referenced by publish queue
2018-08-08T21:51:22.057 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 85d140cb1c1eb24faa0b10f44d17b9954d24fd4c5d6db3c91adfe7e72d7b4df2 referenced by publish queue
2018-08-08T21:51:22.057 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 8830f702c32ce5e8d493031453fba0ec6be2c1b07e3fc5d0123f9895ecbc5523 referenced by publish queue
2018-08-08T21:51:22.057 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 8ba45f03a99923112657726bb94c01e78a31449997d7cd0b275d999209568ae1 referenced by publish queue
2018-08-08T21:51:22.057 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 8c29ec8ced10027ebce0533e1e852841f94dfb9a9adf5f2ec4aeb44650371332 referenced by publish queue
2018-08-08T21:51:22.057 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 8df1e17d986608744a66130af542f00f8766717a4e551eeed01a5aac0bc19217 referenced by publish queue
2018-08-08T21:51:22.057 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 97a13bc1fefa10aabba48c581bcb78e8f04320245a5f1bec2a027a6cc29603ee referenced by publish queue
2018-08-08T21:51:22.057 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 97ff09d76d73abe10327d030cebcfa79ff61f7a83829b078fe50ec0a5f53b325 referenced by publish queue
2018-08-08T21:51:22.057 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 99740e79206c6969abd62dfa163c6c33df69ca22e6ff4cc95dfb477a600fbfd6 referenced by publish queue
2018-08-08T21:51:22.057 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: 9a9066488205fc66520871dfdcdea1edba04ad6fc97734cd5a8aba68cf1ba9cc referenced by publish queue
2018-08-08T21:51:22.057 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: a3066a7178325e7fa220571f54627052ebcee1effce3bad5e5d9c8dfc9fe191e referenced by publish queue
2018-08-08T21:51:22.057 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: a3408247d60479710009e4b9a845fb39d13be9368084a76a270bd4c846652c04 referenced by publish queue
2018-08-08T21:51:22.057 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: a6dc7f0ecc115c107d64b83734c683735a017e22c5ab494a530417ccf34ca97e referenced by publish queue
2018-08-08T21:51:22.057 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: b1d8c6951e16f1a74e1a204753762f9702fa0678e51c863d0b72c9ad70490df5 referenced by publish queue
2018-08-08T21:51:22.057 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: b2fca86b8c2be3f0ef49e7eaba08621f6696c00f2a881caee72f616cbba25099 referenced by publish queue
2018-08-08T21:51:22.057 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: b66a52f7759066253f3f14eafc4ae5fc3f41acadf1890500fdaaa4eda6dd4917 referenced by publish queue
2018-08-08T21:51:22.057 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: c06f26ce5d478ddf502899df9ce622c715591a7d102ec38983e2e42865df080a referenced by publish queue
2018-08-08T21:51:22.057 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: c286e9fab913bca9f93038f2312b6566fc7f4bb56798817c9234e0ce6a06380d referenced by publish queue
2018-08-08T21:51:22.057 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: cdb27b9bb3ff34873b5432c6b5202045a7a1a0b40c74ab27100315933bed4f06 referenced by publish queue
2018-08-08T21:51:22.057 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: d8511a783335d7a8841ae48b609b3d3992c67abe3a32b04a43c5fc6e55ffed1c referenced by publish queue
2018-08-08T21:51:22.057 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: d9ee859667458d7c1fe40bcab7d70a5ef51e71bc93d0018c784cd351f48cc086 referenced by publish queue
2018-08-08T21:51:22.057 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: e10cbeba9c8b9e94620436c819cb60f2fd56910f6c158e6e1700013662b7bb06 referenced by publish queue
2018-08-08T21:51:22.057 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: e1e15b7b94e3523f2ec7e122761d2c6861804978e428db4ee9912e1a6be488d1 referenced by publish queue
2018-08-08T21:51:22.057 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: ea200e569383562739e58ccc653d80fb52b8ccfaae04b57d7d2da589088f00f7 referenced by publish queue
2018-08-08T21:51:22.057 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: ec98a9992a6ba53b76d0075d658ae40c90a476417acb13f0e84a6d60be6d368e referenced by publish queue
2018-08-08T21:51:22.057 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: f5d738d89601828e209a3117d5b88067ea451264f085ebae8cc2a09a1c6fd613 referenced by publish queue
2018-08-08T21:51:22.057 GCUCJ [Bucket DEBUG] BucketManager::forgetUnreferencedBuckets: f64da3c35f82d9672a7253907e60371c675253f8c54bcc858d51681f9b746185 referenced by publish queue
2018-08-08T21:51:22.057 GCUCJ [Ledger INFO] Closed ledger: [seq=10447169, hash=c6f5f0]
Completed 589 Bytes/589 Bytes (1.2 KiB/s) with 1 file(s) remaining
upload: data/buckets/bucket-a6dc7f0ecc115c107d64b83734c683735a017e22c5ab494a530417ccf34ca97e.xdr.gz to s3://history.stellar.org/prd/core-testnet/core_testnet_002/bucket/a6/dc/7f/bucket-a6dc7f0ecc115c107d64b83734c683735a017e22c5ab494a530417ccf34ca97e.xdr.gz
2018-08-08T21:51:22.335 GCUCJ [Process DEBUG] found 1 child processes that terminated
2018-08-08T21:51:22.335 GCUCJ [Process DEBUG] process 5315 exited 0: aws s3 cp --region eu-west-1 /data/buckets/bucket-a6dc7f0ecc115c107d64b83734c683735a017e22c5ab494a530417ccf34ca97e.xdr.gz s3://history.stellar.org/prd/core-testnet/core_testnet_002/bucket/a6/dc/7f/bucket-a6dc7f0ecc115c107d64b83734c683735a017e22c5ab494a530417ccf34ca97e.xdr.gz
2018-08-08T21:51:22.337 GCUCJ [Work DEBUG] completed put-remote-file bucket/a6/dc/7f/bucket-a6dc7f0ecc115c107d64b83734c683735a017e22c5ab494a530417ccf34ca97e.xdr.gz
2018-08-08T21:51:22.337 GCUCJ [Work DEBUG] work put-remote-file bucket/a6/dc/7f/bucket-a6dc7f0ecc115c107d64b83734c683735a017e22c5ab494a530417ccf34ca97e.xdr.gz : WORK_RUNNING -> WORK_SUCCESS
2018-08-08T21:51:22.337 GCUCJ [Work DEBUG] notifying parent of successful put-remote-file bucket/a6/dc/7f/bucket-a6dc7f0ecc115c107d64b83734c683735a017e22c5ab494a530417ccf34ca97e.xdr.gz
2018-08-08T21:51:22.337 GCUCJ [Work DEBUG] notified put-files of completed child put-remote-file bucket/a6/dc/7f/bucket-a6dc7f0ecc115c107d64b83734c683735a017e22c5ab494a530417ccf34ca97e.xdr.gz
2018-08-08T21:51:22.337 GCUCJ [Work DEBUG] advancing put-files
Completed 9.0 KiB/9.0 KiB (17.8 KiB/s) with 1 file(s) remaining
upload: data/buckets/tmp/snapshot-ec421db5b63f34bc/ledger-009f693f.xdr.gz to s3://history.stellar.org/prd/core-testnet/core_testnet_002/ledger/00/9f/69/ledger-009f693f.xdr.gz
2018-08-08T21:51:23.043 GCUCJ [Process DEBUG] found 1 child processes that terminated
2018-08-08T21:51:23.044 GCUCJ [Process DEBUG] process 5322 exited 0: aws s3 cp --region eu-west-1 /data/buckets/tmp/snapshot-ec421db5b63f34bc/ledger-009f693f.xdr.gz s3://history.stellar.org/prd/core-testnet/core_testnet_002/ledger/00/9f/69/ledger-009f693f.xdr.gz
2018-08-08T21:51:23.044 GCUCJ [Work DEBUG] completed put-remote-file ledger/00/9f/69/ledger-009f693f.xdr.gz
2018-08-08T21:51:23.044 GCUCJ [Work DEBUG] work put-remote-file ledger/00/9f/69/ledger-009f693f.xdr.gz : WORK_RUNNING -> WORK_SUCCESS
2018-08-08T21:51:23.045 GCUCJ [Work DEBUG] notifying parent of successful put-remote-file ledger/00/9f/69/ledger-009f693f.xdr.gz
2018-08-08T21:51:23.045 GCUCJ [Work DEBUG] notified put-files of completed child put-remote-file ledger/00/9f/69/ledger-009f693f.xdr.gz
2018-08-08T21:51:23.046 GCUCJ [Work DEBUG] advancing put-files
Completed 6.7 KiB/6.7 KiB (16.0 KiB/s) with 1 file(s) remaining
upload: data/buckets/tmp/snapshot-ec421db5b63f34bc/results-009f693f.xdr.gz to s3://history.stellar.org/prd/core-testnet/core_testnet_002/results/00/9f/69/results-009f693f.xdr.gz
2018-08-08T21:51:23.313 GCUCJ [Process DEBUG] found 1 child processes that terminated
2018-08-08T21:51:23.313 GCUCJ [Process DEBUG] process 5332 exited 0: aws s3 cp --region eu-west-1 /data/buckets/tmp/snapshot-ec421db5b63f34bc/results-009f693f.xdr.gz s3://history.stellar.org/prd/core-testnet/core_testnet_002/results/00/9f/69/results-009f693f.xdr.gz
2018-08-08T21:51:23.313 GCUCJ [Work DEBUG] completed put-remote-file results/00/9f/69/results-009f693f.xdr.gz
2018-08-08T21:51:23.313 GCUCJ [Work DEBUG] work put-remote-file results/00/9f/69/results-009f693f.xdr.gz : WORK_RUNNING -> WORK_SUCCESS
2018-08-08T21:51:23.313 GCUCJ [Work DEBUG] notifying parent of successful put-remote-file results/00/9f/69/results-009f693f.xdr.gz
2018-08-08T21:51:23.313 GCUCJ [Work DEBUG] notified put-files of completed child put-remote-file results/00/9f/69/results-009f693f.xdr.gz
2018-08-08T21:51:23.313 GCUCJ [Work DEBUG] advancing put-files
Completed 16.8 KiB/16.8 KiB (36.8 KiB/s) with 1 file(s) remaining
upload: data/buckets/tmp/snapshot-ec421db5b63f34bc/scp-009f693f.xdr.gz to s3://history.stellar.org/prd/core-testnet/core_testnet_002/scp/00/9f/69/scp-009f693f.xdr.gz
2018-08-08T21:51:23.583 GCUCJ [Process DEBUG] found 1 child processes that terminated
2018-08-08T21:51:23.583 GCUCJ [Process DEBUG] process 5336 exited 0: aws s3 cp --region eu-west-1 /data/buckets/tmp/snapshot-ec421db5b63f34bc/scp-009f693f.xdr.gz s3://history.stellar.org/prd/core-testnet/core_testnet_002/scp/00/9f/69/scp-009f693f.xdr.gz
2018-08-08T21:51:23.583 GCUCJ [Work DEBUG] completed put-remote-file scp/00/9f/69/scp-009f693f.xdr.gz
2018-08-08T21:51:23.584 GCUCJ [Work DEBUG] work put-remote-file scp/00/9f/69/scp-009f693f.xdr.gz : WORK_RUNNING -> WORK_SUCCESS
2018-08-08T21:51:23.584 GCUCJ [Work DEBUG] notifying parent of successful put-remote-file scp/00/9f/69/scp-009f693f.xdr.gz
2018-08-08T21:51:23.584 GCUCJ [Work DEBUG] notified put-files of completed child put-remote-file scp/00/9f/69/scp-009f693f.xdr.gz
2018-08-08T21:51:23.584 GCUCJ [Work DEBUG] advancing put-files
Completed 13.1 KiB/13.1 KiB (32.5 KiB/s) with 1 file(s) remaining
upload: data/buckets/tmp/snapshot-ec421db5b63f34bc/transactions-009f693f.xdr.gz to s3://history.stellar.org/prd/core-testnet/core_testnet_002/transactions/00/9f/69/transactions-009f693f.xdr.gz
2018-08-08T21:51:23.702 GCUCJ [Process DEBUG] found 1 child processes that terminated
2018-08-08T21:51:23.702 GCUCJ [Process DEBUG] process 5343 exited 0: aws s3 cp --region eu-west-1 /data/buckets/tmp/snapshot-ec421db5b63f34bc/transactions-009f693f.xdr.gz s3://history.stellar.org/prd/core-testnet/core_testnet_002/transactions/00/9f/69/transactions-009f693f.xdr.gz
2018-08-08T21:51:23.702 GCUCJ [Work DEBUG] completed put-remote-file transactions/00/9f/69/transactions-009f693f.xdr.gz
2018-08-08T21:51:23.702 GCUCJ [Work DEBUG] work put-remote-file transactions/00/9f/69/transactions-009f693f.xdr.gz : WORK_RUNNING -> WORK_SUCCESS
2018-08-08T21:51:23.702 GCUCJ [Work DEBUG] notifying parent of successful put-remote-file transactions/00/9f/69/transactions-009f693f.xdr.gz
2018-08-08T21:51:23.702 GCUCJ [Work DEBUG] notified put-files of completed child put-remote-file transactions/00/9f/69/transactions-009f693f.xdr.gz
2018-08-08T21:51:23.702 GCUCJ [Work DEBUG] advancing put-files
2018-08-08T21:51:23.702 GCUCJ [Work DEBUG] all 14 children of put-files successful, scheduling run
2018-08-08T21:51:23.702 GCUCJ [Work DEBUG] scheduling run of put-files
2018-08-08T21:51:23.702 GCUCJ [Work DEBUG] starting put-files
2018-08-08T21:51:23.702 GCUCJ [Work DEBUG] running put-files
2018-08-08T21:51:23.702 GCUCJ [Work DEBUG] work put-files : WORK_PENDING -> WORK_RUNNING
2018-08-08T21:51:23.702 GCUCJ [Work DEBUG] scheduling completion of put-files
2018-08-08T21:51:23.702 GCUCJ [Work DEBUG] completed put-files
2018-08-08T21:51:23.702 GCUCJ [Work DEBUG] work put-files : WORK_RUNNING -> WORK_SUCCESS
2018-08-08T21:51:23.702 GCUCJ [Work DEBUG] notifying parent of successful put-files
2018-08-08T21:51:23.703 GCUCJ [Work DEBUG] notified put-snapshot-files-core_testnet_002 of completed child put-files
2018-08-08T21:51:23.703 GCUCJ [Work DEBUG] advancing put-snapshot-files-core_testnet_002
2018-08-08T21:51:23.703 GCUCJ [Work DEBUG] all 2 children of put-snapshot-files-core_testnet_002 successful, scheduling run
2018-08-08T21:51:23.703 GCUCJ [Work DEBUG] scheduling run of put-snapshot-files-core_testnet_002
2018-08-08T21:51:23.703 GCUCJ [Work DEBUG] starting put-snapshot-files-core_testnet_002
2018-08-08T21:51:23.703 GCUCJ [Work DEBUG] running put-snapshot-files-core_testnet_002
2018-08-08T21:51:23.703 GCUCJ [Work DEBUG] work put-snapshot-files-core_testnet_002 : WORK_PENDING -> WORK_RUNNING
2018-08-08T21:51:23.703 GCUCJ [Work DEBUG] scheduling completion of put-snapshot-files-core_testnet_002
2018-08-08T21:51:23.703 GCUCJ [Work DEBUG] completed put-snapshot-files-core_testnet_002
2018-08-08T21:51:23.704 GCUCJ [Work DEBUG] resetting put-history-archive-state
2018-08-08T21:51:23.704 GCUCJ [Work DEBUG] work put-snapshot-files-core_testnet_002 : WORK_RUNNING -> WORK_PENDING
2018-08-08T21:51:23.704 GCUCJ [Work DEBUG] advancing put-snapshot-files-core_testnet_002
2018-08-08T21:51:23.704 GCUCJ [Work DEBUG] advancing put-history-archive-state
2018-08-08T21:51:23.704 GCUCJ [Work DEBUG] all 0 children of put-history-archive-state successful, scheduling run
2018-08-08T21:51:23.705 GCUCJ [Work DEBUG] scheduling run of put-history-archive-state
2018-08-08T21:51:23.705 GCUCJ [Work DEBUG] starting put-history-archive-state
2018-08-08T21:51:23.705 GCUCJ [Work DEBUG] running put-history-archive-state
2018-08-08T21:51:23.705 GCUCJ [Work DEBUG] work put-history-archive-state : WORK_PENDING -> WORK_RUNNING
2018-08-08T21:51:23.705 GCUCJ [Work DEBUG] scheduling completion of put-history-archive-state
2018-08-08T21:51:23.706 GCUCJ [Work DEBUG] completed put-history-archive-state
2018-08-08T21:51:23.706 GCUCJ [Work DEBUG] resetting put-remote-file history/00/9f/69/history-009f693f.json
2018-08-08T21:51:23.706 GCUCJ [Work DEBUG] resetting make-remote-dir history/00/9f/69
2018-08-08T21:51:23.707 GCUCJ [Work DEBUG] resetting put-remote-file .well-known/stellar-history.json
2018-08-08T21:51:23.707 GCUCJ [Work DEBUG] resetting make-remote-dir .well-known
2018-08-08T21:51:23.707 GCUCJ [Work DEBUG] work put-history-archive-state : WORK_RUNNING -> WORK_PENDING
2018-08-08T21:51:23.707 GCUCJ [Work DEBUG] advancing put-history-archive-state
2018-08-08T21:51:23.707 GCUCJ [Work DEBUG] advancing put-remote-file .well-known/stellar-history.json
2018-08-08T21:51:23.707 GCUCJ [Work DEBUG] advancing make-remote-dir .well-known
2018-08-08T21:51:23.707 GCUCJ [Work DEBUG] all 0 children of make-remote-dir .well-known successful, scheduling run
2018-08-08T21:51:23.707 GCUCJ [Work DEBUG] scheduling run of make-remote-dir .well-known
2018-08-08T21:51:23.707 GCUCJ [Work DEBUG] advancing put-remote-file history/00/9f/69/history-009f693f.json
2018-08-08T21:51:23.707 GCUCJ [Work DEBUG] advancing make-remote-dir history/00/9f/69
2018-08-08T21:51:23.707 GCUCJ [Work DEBUG] all 0 children of make-remote-dir history/00/9f/69 successful, scheduling run
2018-08-08T21:51:23.707 GCUCJ [Work DEBUG] scheduling run of make-remote-dir history/00/9f/69
2018-08-08T21:51:23.707 GCUCJ [Work DEBUG] starting make-remote-dir .well-known
2018-08-08T21:51:23.707 GCUCJ [Work DEBUG] scheduling completion of make-remote-dir .well-known
2018-08-08T21:51:23.707 GCUCJ [Work DEBUG] running make-remote-dir .well-known
2018-08-08T21:51:23.707 GCUCJ [Work DEBUG] work make-remote-dir .well-known : WORK_PENDING -> WORK_RUNNING
2018-08-08T21:51:23.707 GCUCJ [Work DEBUG] starting make-remote-dir history/00/9f/69
2018-08-08T21:51:23.707 GCUCJ [Work DEBUG] scheduling completion of make-remote-dir history/00/9f/69
2018-08-08T21:51:23.707 GCUCJ [Work DEBUG] running make-remote-dir history/00/9f/69
2018-08-08T21:51:23.707 GCUCJ [Work DEBUG] work make-remote-dir history/00/9f/69 : WORK_PENDING -> WORK_RUNNING
2018-08-08T21:51:23.707 GCUCJ [Work DEBUG] completed make-remote-dir .well-known
2018-08-08T21:51:23.707 GCUCJ [Work DEBUG] work make-remote-dir .well-known : WORK_RUNNING -> WORK_SUCCESS
2018-08-08T21:51:23.707 GCUCJ [Work DEBUG] notifying parent of successful make-remote-dir .well-known
2018-08-08T21:51:23.707 GCUCJ [Work DEBUG] notified put-remote-file .well-known/stellar-history.json of completed child make-remote-dir .well-known
2018-08-08T21:51:23.707 GCUCJ [Work DEBUG] advancing put-remote-file .well-known/stellar-history.json
2018-08-08T21:51:23.707 GCUCJ [Work DEBUG] all 1 children of put-remote-file .well-known/stellar-history.json successful, scheduling run
2018-08-08T21:51:23.707 GCUCJ [Work DEBUG] scheduling run of put-remote-file .well-known/stellar-history.json
2018-08-08T21:51:23.707 GCUCJ [Work DEBUG] completed make-remote-dir history/00/9f/69
2018-08-08T21:51:23.707 GCUCJ [Work DEBUG] work make-remote-dir history/00/9f/69 : WORK_RUNNING -> WORK_SUCCESS
2018-08-08T21:51:23.708 GCUCJ [Work DEBUG] notifying parent of successful make-remote-dir history/00/9f/69
2018-08-08T21:51:23.708 GCUCJ [Work DEBUG] notified put-remote-file history/00/9f/69/history-009f693f.json of completed child make-remote-dir history/00/9f/69
2018-08-08T21:51:23.708 GCUCJ [Work DEBUG] advancing put-remote-file history/00/9f/69/history-009f693f.json
2018-08-08T21:51:23.708 GCUCJ [Work DEBUG] all 1 children of put-remote-file history/00/9f/69/history-009f693f.json successful, scheduling run
2018-08-08T21:51:23.708 GCUCJ [Work DEBUG] scheduling run of put-remote-file history/00/9f/69/history-009f693f.json
2018-08-08T21:51:23.708 GCUCJ [Work DEBUG] starting put-remote-file .well-known/stellar-history.json
2018-08-08T21:51:23.708 GCUCJ [Process DEBUG] Running: aws s3 cp --region eu-west-1 /data/buckets/tmp/history-6464b9088928f47c/core_testnet_002-stellar-history.json s3://history.stellar.org/prd/core-testnet/core_testnet_002/.well-known/stellar-history.json
2018-08-08T21:51:23.897 GCUCJ [Work DEBUG] running put-remote-file .well-known/stellar-history.json
2018-08-08T21:51:23.897 GCUCJ [Work DEBUG] work put-remote-file .well-known/stellar-history.json : WORK_PENDING -> WORK_RUNNING
2018-08-08T21:51:23.897 GCUCJ [Work DEBUG] starting put-remote-file history/00/9f/69/history-009f693f.json
2018-08-08T21:51:23.898 GCUCJ [Process DEBUG] Running: aws s3 cp --region eu-west-1 /data/buckets/tmp/history-6464b9088928f47c/core_testnet_002-stellar-history.json s3://history.stellar.org/prd/core-testnet/core_testnet_002/history/00/9f/69/history-009f693f.json
2018-08-08T21:51:24.120 GCUCJ [Work DEBUG] running put-remote-file history/00/9f/69/history-009f693f.json
2018-08-08T21:51:24.121 GCUCJ [Work DEBUG] work put-remote-file history/00/9f/69/history-009f693f.json : WORK_PENDING -> WORK_RUNNING
Completed 3.8 KiB/3.8 KiB (9.8 KiB/s) with 1 file(s) remaining
upload: data/buckets/tmp/history-6464b9088928f47c/core_testnet_002-stellar-history.json to s3://history.stellar.org/prd/core-testnet/core_testnet_002/.well-known/stellar-history.json
2018-08-08T21:51:24.995 GCUCJ [Process DEBUG] found 1 child processes that terminated
2018-08-08T21:51:24.995 GCUCJ [Process DEBUG] process 5363 exited 0: aws s3 cp --region eu-west-1 /data/buckets/tmp/history-6464b9088928f47c/core_testnet_002-stellar-history.json s3://history.stellar.org/prd/core-testnet/core_testnet_002/.well-known/stellar-history.json
2018-08-08T21:51:24.995 GCUCJ [Work DEBUG] completed put-remote-file .well-known/stellar-history.json
2018-08-08T21:51:24.995 GCUCJ [Work DEBUG] work put-remote-file .well-known/stellar-history.json : WORK_RUNNING -> WORK_SUCCESS
2018-08-08T21:51:24.995 GCUCJ [Work DEBUG] notifying parent of successful put-remote-file .well-known/stellar-history.json
2018-08-08T21:51:24.995 GCUCJ [Work DEBUG] notified put-history-archive-state of completed child put-remote-file .well-known/stellar-history.json
2018-08-08T21:51:24.995 GCUCJ [Work DEBUG] advancing put-history-archive-state
Completed 3.8 KiB/3.8 KiB (9.6 KiB/s) with 1 file(s) remaining
upload: data/buckets/tmp/history-6464b9088928f47c/core_testnet_002-stellar-history.json to s3://history.stellar.org/prd/core-testnet/core_testnet_002/history/00/9f/69/history-009f693f.json
2018-08-08T21:51:25.163 GCUCJ [Process DEBUG] found 1 child processes that terminated
2018-08-08T21:51:25.163 GCUCJ [Process DEBUG] process 5365 exited 0: aws s3 cp --region eu-west-1 /data/buckets/tmp/history-6464b9088928f47c/core_testnet_002-stellar-history.json s3://history.stellar.org/prd/core-testnet/core_testnet_002/history/00/9f/69/history-009f693f.json
2018-08-08T21:51:25.163 GCUCJ [Work DEBUG] completed put-remote-file history/00/9f/69/history-009f693f.json
2018-08-08T21:51:25.163 GCUCJ [Work DEBUG] work put-remote-file history/00/9f/69/history-009f693f.json : WORK_RUNNING -> WORK_SUCCESS
2018-08-08T21:51:25.163 GCUCJ [Work DEBUG] notifying parent of successful put-remote-file history/00/9f/69/history-009f693f.json
2018-08-08T21:51:25.163 GCUCJ [Work DEBUG] notified put-history-archive-state of completed child put-remote-file history/00/9f/69/history-009f693f.json
2018-08-08T21:51:25.163 GCUCJ [Work DEBUG] advancing put-history-archive-state
2018-08-08T21:51:25.163 GCUCJ [Work DEBUG] all 2 children of put-history-archive-state successful, scheduling run
2018-08-08T21:51:25.163 GCUCJ [Work DEBUG] scheduling run of put-history-archive-state
2018-08-08T21:51:25.163 GCUCJ [Work DEBUG] starting put-history-archive-state
2018-08-08T21:51:25.163 GCUCJ [Work DEBUG] running put-history-archive-state
2018-08-08T21:51:25.163 GCUCJ [Work DEBUG] work put-history-archive-state : WORK_PENDING -> WORK_RUNNING
2018-08-08T21:51:25.164 GCUCJ [Work DEBUG] scheduling completion of put-history-archive-state
2018-08-08T21:51:25.164 GCUCJ [Work DEBUG] completed put-history-archive-state
2018-08-08T21:51:25.164 GCUCJ [Work DEBUG] work put-history-archive-state : WORK_RUNNING -> WORK_SUCCESS
2018-08-08T21:51:25.164 GCUCJ [Work DEBUG] notifying parent of successful put-history-archive-state
2018-08-08T21:51:25.164 GCUCJ [Work DEBUG] notified put-snapshot-files-core_testnet_002 of completed child put-history-archive-state
2018-08-08T21:51:25.164 GCUCJ [Work DEBUG] advancing put-snapshot-files-core_testnet_002
2018-08-08T21:51:25.164 GCUCJ [Work DEBUG] all 3 children of put-snapshot-files-core_testnet_002 successful, scheduling run
2018-08-08T21:51:25.165 GCUCJ [Work DEBUG] scheduling run of put-snapshot-files-core_testnet_002
2018-08-08T21:51:25.165 GCUCJ [Work DEBUG] starting put-snapshot-files-core_testnet_002
2018-08-08T21:51:25.165 GCUCJ [Work DEBUG] running put-snapshot-files-core_testnet_002
2018-08-08T21:51:25.165 GCUCJ [Work DEBUG] work put-snapshot-files-core_testnet_002 : WORK_PENDING -> WORK_RUNNING
2018-08-08T21:51:25.165 GCUCJ [Work DEBUG] scheduling completion of put-snapshot-files-core_testnet_002
2018-08-08T21:51:25.165 GCUCJ [Work DEBUG] completed put-snapshot-files-core_testnet_002
2018-08-08T21:51:25.165 GCUCJ [Work DEBUG] work put-snapshot-files-core_testnet_002 : WORK_RUNNING -> WORK_SUCCESS
2018-08-08T21:51:25.165 GCUCJ [Work DEBUG] notifying parent of successful put-snapshot-files-core_testnet_002
2018-08-08T21:51:25.165 GCUCJ [Work DEBUG] notified update-archives of completed child put-snapshot-files-core_testnet_002
2018-08-08T21:51:25.165 GCUCJ [Work DEBUG] advancing update-archives
2018-08-08T21:51:25.165 GCUCJ [Work DEBUG] all 1 children of update-archives successful, scheduling run
2018-08-08T21:51:25.165 GCUCJ [Work DEBUG] scheduling run of update-archives
2018-08-08T21:51:25.165 GCUCJ [Work DEBUG] starting update-archives
2018-08-08T21:51:25.165 GCUCJ [Work DEBUG] running update-archives
2018-08-08T21:51:25.165 GCUCJ [Work DEBUG] work update-archives : WORK_PENDING -> WORK_RUNNING
2018-08-08T21:51:25.165 GCUCJ [Work DEBUG] scheduling completion of update-archives
2018-08-08T21:51:25.165 GCUCJ [Work DEBUG] completed update-archives
2018-08-08T21:51:25.165 GCUCJ [Work DEBUG] work update-archives : WORK_RUNNING -> WORK_SUCCESS
2018-08-08T21:51:25.165 GCUCJ [Work DEBUG] notifying parent of successful update-archives
2018-08-08T21:51:25.165 GCUCJ [Work DEBUG] notified publish-009f693f of completed child update-archives
2018-08-08T21:51:25.165 GCUCJ [Work DEBUG] advancing publish-009f693f
2018-08-08T21:51:25.165 GCUCJ [Work DEBUG] all 3 children of publish-009f693f successful, scheduling run
2018-08-08T21:51:25.165 GCUCJ [Work DEBUG] scheduling run of publish-009f693f
2018-08-08T21:51:25.165 GCUCJ [Work DEBUG] starting publish-009f693f
2018-08-08T21:51:25.165 GCUCJ [Work DEBUG] running publish-009f693f
2018-08-08T21:51:25.165 GCUCJ [Work DEBUG] work publish-009f693f : WORK_PENDING -> WORK_RUNNING
2018-08-08T21:51:25.166 GCUCJ [Work DEBUG] scheduling completion of publish-009f693f
2018-08-08T21:51:25.166 GCUCJ [Work DEBUG] completed publish-009f693f
2018-08-08T21:51:25.167 GCUCJ [Work DEBUG] work publish-009f693f : WORK_RUNNING -> WORK_SUCCESS
2018-08-08T21:51:25.167 GCUCJ [Work DEBUG] notifying parent of successful publish-009f693f
2018-08-08T21:51:25.167 GCUCJ [Work INFO] WorkManager got SUCCESS from publish-009f693f
2018-08-08T21:51:25.167 GCUCJ [Fs DEBUG] deleting: /data/buckets/tmp/snapshot-ec421db5b63f34bc/scp-009f693f.xdr
2018-08-08T21:51:25.168 GCUCJ [Fs DEBUG] deleting: /data/buckets/tmp/snapshot-ec421db5b63f34bc/ledger-009f693f.xdr.gz
2018-08-08T21:51:25.168 GCUCJ [Fs DEBUG] deleting: /data/buckets/tmp/snapshot-ec421db5b63f34bc/scp-009f693f.xdr.gz
2018-08-08T21:51:25.168 GCUCJ [Fs DEBUG] deleting: /data/buckets/tmp/snapshot-ec421db5b63f34bc/transactions-009f693f.xdr.gz
2018-08-08T21:51:25.168 GCUCJ [Fs DEBUG] deleting: /data/buckets/tmp/snapshot-ec421db5b63f34bc/transactions-009f693f.xdr
2018-08-08T21:51:25.168 GCUCJ [Fs DEBUG] deleting: /data/buckets/tmp/snapshot-ec421db5b63f34bc/ledger-009f693f.xdr
2018-08-08T21:51:25.168 GCUCJ [Fs DEBUG] deleting: /data/buckets/tmp/snapshot-ec421db5b63f34bc/results-009f693f.xdr.gz
2018-08-08T21:51:25.168 GCUCJ [Fs DEBUG] deleting: /data/buckets/tmp/snapshot-ec421db5b63f34bc/results-009f693f.xdr
2018-08-08T21:51:25.169 GCUCJ [Fs DEBUG] deleting: /data/buckets/tmp/snapshot-ec421db5b63f34bc
@marta-lokhova
Copy link
Contributor

This seems to be the "expected" behavior. Isn't the whole point of ResolveSnapshotWork to wait for all futures to be resolved? At least this code and the fact that it retries forever seem to suggest so. Maybe it should be implemented differently, and schedule re-run instead of a failure.

@MonsieurNicolas
Copy link
Contributor Author

Oh I see... in this case I suspect it's just that the condition mApp.getLedgerManager().getLastClosedLedgerNum() > mSnapshot->mLocalState.currentLedger) is always false.

It's a bit of an abuse of failures (failures imply somebody should be looking at a problem), couldn't it just schedule to retry a little later with standard callbacks like we do elsewhere?

@marta-lokhova
Copy link
Contributor

I don't know the rationale behind implementing it this way, but yes, having it fail is quite confusing. I do think that scheduling it to run more (i.e. return WORK_RUNNING) is more clear.

@MonsieurNicolas
Copy link
Contributor Author

Actually the problem is worst than that:
I just noticed that the gating (and its failures) is triggered randomly even when there is a backlog of snapshots to publish.

Glancing at the code, I think there is a race condition:

  • mPublishWork is either set directly (when closing a ledger) or by publishQueuedHistory
  • when a publish is complete, it resets mPublishWork and schedules publishQueuedHistory
    • at this point, if a ledger closes before publishQueuedHistory, we will publish the latest ledger instead of the one in the queue (causing failures)

So we need to fix 2 things as far as I can tell:

  • remove the race condition by always pushing to the publish queue and always triggering publishQueuedHistory (that will do the right thing)
  • change the delay to use a proper sleep (using a timer) instead of triggering a failure

@marta-lokhova
Copy link
Contributor

Regarding race condition, it seems like a fix would be removing redundant takeSnapshotAndPublish call in queueCurrentHistory, since during ledger close publishQueuedHistory is called anyway.

@MonsieurNicolas
Copy link
Contributor Author

yes, queueCurrentHistory should only queue like the name says

@marta-lokhova
Copy link
Contributor

ok, i can look into this one.

@marta-lokhova
Copy link
Contributor

I think the delay issue will be handled with the new design in #1755

@marta-lokhova marta-lokhova self-assigned this Nov 27, 2018
@marta-lokhova marta-lokhova changed the title bogus failure on publish causes retries Properly wait for resolve snapshots work to finish instead of failing and retrying Feb 13, 2019
@marta-lokhova
Copy link
Contributor

Resolved by #2063

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants