multi: add BgBlkTmplGenerator. #1424
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dnldd
force-pushed
the
add_bg_blk_gen
branch
5 times, most recently
from
d9d6b1d
to
e4528f9
Compare
dnldd
force-pushed
the
add_bg_blk_gen
branch
from
e4528f9
to
486589c
Compare
davecgh
requested changes
Sep 13, 2018
dnldd
force-pushed
the
add_bg_blk_gen
branch
2 times, most recently
from
deef824
to
2a86b2d
Compare
davecgh
requested changes
Sep 14, 2018
dnldd
force-pushed
the
add_bg_blk_gen
branch
4 times, most recently
from
2de6a5c
to
b265805
Compare
davecgh
reviewed
Sep 14, 2018
dnldd
force-pushed
the
add_bg_blk_gen
branch
4 times, most recently
from
df79836
to
0dac815
Compare
dnldd
force-pushed
the
add_bg_blk_gen
branch
2 times, most recently
from
33947d4
to
5aa05f7
Compare
davecgh
requested changes
Sep 24, 2018
dnldd
force-pushed
the
add_bg_blk_gen
branch
3 times, most recently
from
fee72cc
to
74bf30e
Compare
davecgh
requested changes
Sep 25, 2018
davecgh
requested changes
Sep 25, 2018
There was a problem hiding this comment.
I suggest firing up simnet and thoroughly testing this. You're going to have to work through several issues that I saw when testing:
- It doesn't update immediately when connecting a new block before SVH
- It doesn't handle waiting to generate a new template until there are at least 3 votes properly
- It doesn't prune before SVH-2, and even that doesn't seem to be working properly since a lot of old templates are being left in the pool
- It calls the template regen function 5 times on every block without any further testing if the regen is even needed such as if the vote is for an alternate block at the tip which has less votes then the current candidate
- It doesn't handle block disconnects at all and leave the current and parent template incorrect.
dnldd
force-pushed
the
add_bg_blk_gen
branch
3 times, most recently
from
619a79d
to
1ec6a31
Compare
dnldd
force-pushed
the
add_bg_blk_gen
branch
4 times, most recently
from
9c454fc
to
77762f9
Compare
dnldd
force-pushed
the
add_bg_blk_gen
branch
5 times, most recently
from
b2d2a8e
to
eb426b2
Compare
dajohi
requested changes
Feb 10, 2019
dnldd
force-pushed
the
add_bg_blk_gen
branch
2 times, most recently
from
a647f48
to
bf86676
Compare
dnldd
force-pushed
the
add_bg_blk_gen
branch
2 times, most recently
from
77eabd9
to
4e326c9
Compare
davecgh
requested changes
Mar 29, 2019
davecgh
requested changes
Mar 29, 2019
BgBlkTmplGenerator represents the background process that generates block templates and notifies all subscribed clients on template regeneration. It generates new templates based on mempool activity for vote and non-vote transactions and the time elapsed since last template regeneration. This also adds a template pool to the background block generator for recreating submitted blocks.
davecgh
approved these changes
Mar 30, 2019
There was a problem hiding this comment.
I'm going to go ahead and approve this and merge it since it is working correctly in all important aspects. There are a couple of remaining efficiency and less than ideal behaviors, however, we can address those in future PRs with a reduced scope. Those are detailed after the test results.
Here are the testing results with latest code and some added debug to register for template updates and print the details as well as template pool details.
Initial startup before at genesis block generates a template immediately as expected:
2019-03-30 14:39:04.741 [INF] CHAN: Chain state: height 0, hash 5bec7567af40504e0994db3b573c186fffcc4edefe096ff2e58d00523bd7e8a6, total transactions 1, work 2, stake version 0
...
2019-03-30 14:39:04.752 [DBG] MINR: Received block template building on 5bec7567af40504e0994db3b573c186fffcc4edefe096ff2e58d00523bd7e8a6 (height 0) (1 transactions, 0 stake transactions, 0 votes, -1 in fees, 6 signature operations, 371 bytes, target difficulty 7fffff0000000000000000000000000000000000000000000000000000000000, stake difficulty 0.0002)
New templates are generated based on the genesis block every minute when there is no other activity as expected:
2019-03-30 14:40:06.252 [DBG] MINR: Received block template building on 5bec7567af40504e0994db3b573c186fffcc4edefe096ff2e58d00523bd7e8a6 (height 0) (1 transactions, 0 stake transactions, 0 votes, -1 in fees, 6 signature operations, 371 bytes, target difficulty 7fffff0000000000000000000000000000000000000000000000000000000000, stake difficulty 0.0002)
...
2019-03-30 14:41:07.252 [DBG] MINR: Received block template building on 5bec7567af40504e0994db3b573c186fffcc4edefe096ff2e58d00523bd7e8a6 (height 0) (1 transactions, 0 stake transactions, 0 votes, -1 in fees, 6 signature operations, 371 bytes, target difficulty 7fffff0000000000000000000000000000000000000000000000000000000000, stake difficulty 0.0002)
Receiving a mined block 1 generates a template immediately as expected:
2019-03-30 14:41:33.608 [INF] MINR: Block submitted via CPU miner accepted (hash 4240d8416f3d2d829bcc0c930f8e65648e50891faf97c5e190c2f1c913fc3ce1, height 1, amount 300000 DCR)
...
2019-03-30 14:41:33.609 [DBG] MINR: Received block template building on 4240d8416f3d2d829bcc0c930f8e65648e50891faf97c5e190c2f1c913fc3ce1 (height 1) (1 transactions, 0 stake transactions, 0 votes, 0 in fees, 2 signature operations, 358 bytes, target difficulty 7fffff0000000000000000000000000000000000000000000000000000000000, stake difficulty 0.0002)
Pruning of the template pool prior to SVH-1 works as expected:
2019-03-30 14:43:28.843 [DBG] MINR: Pruning template cf3570b4326635b627546b14d857eff2df9636b4353d2e27c46629285f6657aa (height 10)
2019-03-30 14:43:28.843 [INF] MINR: Block submitted via CPU miner accepted (hash 213894d52014ae7c4ac7ee51cf36950460b2c35552b5b902fa6fd22fe3b81e09, height 12, amount 350 DCR)
...
2019-03-30 14:43:30.981 [DBG] MINR: Pruning template 6b070ca5bfb15e8152f0b6fe3559b9ed5c7e4aee064bd69c157abbe441197bbb (height 11)
New templates are generated every 30 seconds when there is transaction activity and it returns to generating them every minute when there is no activity as expected:
2019-03-30 14:46:19.257 [DBG] MINR: Received block template building on 11444e9ad31a3d4287eeffb52d40a6e0bd1cd7e1cd2e851347b6ab47f29088b3 (height 32) (2 transactions, 20 stake transactions, 0 votes, 68970 in fees, 63 signature operations, 7226 bytes, target difficulty 7fffff0000000000000000000000000000000000000000000000000000000000, stake difficulty 0.0002)
...
2019-03-30 14:46:50.262 [DBG] MINR: Received block template building on 11444e9ad31a3d4287eeffb52d40a6e0bd1cd7e1cd2e851347b6ab47f29088b3 (height 32) (3 transactions, 20 stake transactions, 0 votes, 179160 in fees, 364 signature operations, 18243 bytes, target difficulty 7fffff0000000000000000000000000000000000000000000000000000000000, stake difficulty 0.0002)
...
2019-03-30 14:47:21.301 [DBG] MINR: Received block template building on 11444e9ad31a3d4287eeffb52d40a6e0bd1cd7e1cd2e851347b6ab47f29088b3 (height 32) (3 transactions, 20 stake transactions, 0 votes, 179160 in fees, 364 signature operations, 18243 bytes, target difficulty 7fffff0000000000000000000000000000000000000000000000000000000000, stake difficulty 0.0002)
...
2019-03-30 14:48:22.266 [DBG] MINR: Received block template building on 11444e9ad31a3d4287eeffb52d40a6e0bd1cd7e1cd2e851347b6ab47f29088b3 (height 32) (3 transactions, 20 stake transactions, 0 votes, 179160 in fees, 364 signature operations, 18243 bytes, target difficulty 7fffff0000000000000000000000000000000000000000000000000000000000, stake difficulty 0.0002)
Initial startup with tip at SVH-2 (just before the first template which will require votes) generates a template immediately as expected as does receiving the block:
2019-03-30 14:53:51.096 [INF] MINR: Block submitted via CPU miner accepted (hash 70488ee35cb39cedd78a29b5628e474a89bd1684979c88900cd9a55f41b9dbcb, height 142, amount 346.53465346 DCR)
...
2019-03-30 14:53:51.102 [DBG] MINR: Received block template building on 70488ee35cb39cedd78a29b5628e474a89bd1684979c88900cd9a55f41b9dbcb (height 142) (1 transactions, 20 stake transactions, 0 votes, 59600 in fees, 42 signature operations, 6278 bytes, target difficulty 7fffff0000000000000000000000000000000000000000000000000000000000, stake difficulty 1.04072117)
...
2019-03-30 14:56:44.427 [INF] CHAN: Chain state: height 142, hash 70488ee35cb39cedd78a29b5628e474a89bd1684979c88900cd9a55f41b9dbcb, total transactions 1427, work 286, stake version 0
...
2019-03-30 14:56:44.440 [DBG] MINR: Received block template building on 70488ee35cb39cedd78a29b5628e474a89bd1684979c88900cd9a55f41b9dbcb (height 142) (1 transactions, 0 stake transactions, 0 votes, 0 in fees, 2 signature operations, 358 bytes, target difficulty 7fffff0000000000000000000000000000000000000000000000000000000000, stake difficulty 1.04072117)
The template pool is properly storing variants prior to SVH-1:
2019-03-30 15:07:21.015 [DBG] MINR: Adding template for block height 143 with key 88a5cd89d222f0ddc823796dd3e2017a15ed49a74e1a136f34d49bd6e4b60ee874af7687d2f752b137a641ac30bbec70e0b37385a5c29defe4acfa310137dc82007a21fb56d59015 (size: 6)
...
2019-03-30 15:08:22.021 [DBG] MINR: Adding template for block height 143 with key f7abedfb777acb5d7fbdf4cc942e1f3a5fc288e927db547124573dc2f53b31a4f6f0d96b177b24a606af9ad024cd8be2fd2a5262ca3147f856f7431eb3ce80da009c032f65d59015 (size: 7)
...
2019-03-30 15:09:23.018 [DBG] MINR: Adding template for block height 143 with key 890ff2bc9aa8682a478ca07e455e71ccb529941200b1edbafeeba194544fc9e992264b58f2e19a750d2abfde4798a7f8fb34b7b23e388b21590a02f6be0ba93400bee56273d59015 (size: 8)
New templates are generated at SVH-1 (the first template that requires votes) every minute when there is no other activity as expected:
2019-03-30 15:11:28.014 [DBG] MINR: Received block template building on 6c144d5ac0702ab004ce6dd2b518bf34018bd4b57be47f5f6f107241b2e4604f (height 143) (2 transactions, 5 stake transactions, 5 votes, 0 in fees, 28 signature operations, 3026 bytes, target difficulty 7fffff0000000000000000000000000000000000000000000000000000000000, stake difficulty 3.40653966)
...
2019-03-30 15:12:29.013 [DBG] MINR: Received block template building on 6c144d5ac0702ab004ce6dd2b518bf34018bd4b57be47f5f6f107241b2e4604f (height 143) (2 transactions, 5 stake transactions, 5 votes, 0 in fees, 28 signature operations, 3026 bytes, target difficulty 7fffff0000000000000000000000000000000000000000000000000000000000, stake difficulty 3.40653966)
Initial startup at SVH-1 (with wallet disconnected to ensure votes are obtained via getminingstate) generates a template upon receiving the 3rd and subsequent votes as expected:
2019-03-30 15:16:45.612 [INF] CHAN: Chain state: height 143, hash 6c144d5ac0702ab004ce6dd2b518bf34018bd4b57be47f5f6f107241b2e4604f, total transactions 1448, work 288, stake version 0
...
2019-03-30 15:16:56.212 [INF] BMGR: New valid peer 127.0.0.1:54972 (inbound) (/dcrwire:0.3.0/dcrd:1.5.0(pre)/)
2019-03-30 15:16:59.214 [DBG] TXMP: Accepted vote 64d0f7f89cbf0ae9948d9930c21f497c25ce4f04b95dc39a34e9e6cb86522a58 for block hash 6c144d5ac0702ab004ce6dd2b518bf34018bd4b57be47f5f6f107241b2e4604f (height 143), voting true on the transaction tree
2019-03-30 15:16:59.215 [DBG] TXMP: Accepted vote 6ab201ea5a74f295f9a0c905d90b99c9e22bc11e3f673e74b227670baf33f41d for block hash 6c144d5ac0702ab004ce6dd2b518bf34018bd4b57be47f5f6f107241b2e4604f (height 143), voting true on the transaction tree
2019-03-30 15:16:59.216 [DBG] TXMP: Accepted vote c1d8ea4a050733737f170eaa2cf4eb060d3b1573d9322dc212ad87a0a1bd449d for block hash 6c144d5ac0702ab004ce6dd2b518bf34018bd4b57be47f5f6f107241b2e4604f (height 143), voting true on the transaction tree
2019-03-30 15:16:59.217 [DBG] TXMP: Accepted vote d1cc3ec92cd2cc42023c5d3cc66c63e873aa3047114a5c55a78a577be188e338 for block hash 6c144d5ac0702ab004ce6dd2b518bf34018bd4b57be47f5f6f107241b2e4604f (height 143), voting true on the transaction tree
2019-03-30 15:16:59.218 [DBG] TXMP: Accepted vote 89d59c97a20f3919721b11697cde3347b2bda3269db2a92151003f6bf6b31369 for block hash 6c144d5ac0702ab004ce6dd2b518bf34018bd4b57be47f5f6f107241b2e4604f (height 143), voting true on the transaction tree
2019-03-30 15:17:00.219 [DBG] MINR: Received block template building on 6c144d5ac0702ab004ce6dd2b518bf34018bd4b57be47f5f6f107241b2e4604f (height 143) (1 transactions, 5 stake transactions, 5 votes, 0 in fees, 7 signature operations, 2091 bytes, target difficulty 7fffff0000000000000000000000000000000000000000000000000000000000, stake difficulty 3.40653966)
New templates are generated at SVH-1 (the first template that requires votes) every 30 seconds when there is transaction activity as expected:
2019-03-30 15:20:03.123 [DBG] MINR: Received block template building on 6c144d5ac0702ab004ce6dd2b518bf34018bd4b57be47f5f6f107241b2e4604f (height 143) (1 transactions, 5 stake transactions, 5 votes, 0 in fees, 7 signature operations, 2091 bytes, target difficulty 7fffff0000000000000000000000000000000000000000000000000000000000, stake difficulty 3.40653966)
...
2019-03-30 15:20:16.331 [DBG] TXMP: Accepted transaction 8e0bfdb04c5d0eff69cb258ec8e8b4b76e67ee8e2be8fdda5acca9369d19afd1 (pool size: 6)
...
2019-03-30 15:20:34.127 [DBG] MINR: Received block template building on 6c144d5ac0702ab004ce6dd2b518bf34018bd4b57be47f5f6f107241b2e4604f (height 143) (2 transactions, 5 stake transactions, 5 votes, 0 in fees, 28 signature operations, 3026 bytes, target difficulty 7fffff0000000000000000000000000000000000000000000000000000000000, stake difficulty 3.40653966)
The template pool is properly storing variants at and after SVH-1:
2019-03-30 15:20:34.127 [DBG] MINR: Adding template for block height 144 with key 959738bcf522d3a8ba784553e2b66c37f9aa18e0bf72f6613a92c2b76bcccfd693c6a30164ac3eb938790e8a342ef98f77cf6bca7555d603d37860e0b8324ae100349d9d0fd69015 (size: 6)
...
2019-03-30 15:22:36.126 [DBG] MINR: Adding template for block height 144 with key 5fcda3a6792d982cd244171de1274736adfdf68355ab2c0be0be272c15bde7a493c6a30164ac3eb938790e8a342ef98f77cf6bca7555d603d37860e0b8324ae1007861052cd69015 (size: 8)
Pruning of the template pool after SVH-1 works as expected:
2019-03-30 15:27:39.961 [DBG] MINR: Pruning template ed98962fc807a1b5034f2a171d0a12ba8effab405644b191ad34b4af1962a0b8 (height 144)
2019-03-30 15:27:39.961 [DBG] MINR: Pruning template 38457ac13384aa778a8d1f9f8cc75fd7f22353ecdfed844a1e0b98d1a8f34735 (height 144)
2019-03-30 15:27:39.961 [DBG] MINR: Pruning template c305af57c1d6c2759964b56e7f271092323f4a598f78e27b8ea19176d3a67276 (height 144)
2019-03-30 15:27:39.961 [DBG] MINR: Pruning template 420cf2595724a5bee915fbab10442d320fe6b811ef7230de70d513370aa6a770 (height 144)
2019-03-30 15:27:39.961 [DBG] MINR: Pruning template fd584a8c2a8fb20e8370e4fd66b8a657c9e22ec0d2ecf210afe904716fab04c2 (height 144)
2019-03-30 15:27:39.961 [DBG] MINR: Pruning template b77d9fe307c629bf3126988ce435a8236b756a02a31d0fb7c6f419041bced287 (height 144)
2019-03-30 15:27:39.961 [DBG] MINR: Pruning template a5adbca4e19d1663d83b59d3c4f189571906b922fc2090c4b422a35e6b161109 (height 144)
2019-03-30 15:27:39.961 [DBG] MINR: Pruning template 1b8838d8b94511e1a4d34aa35a39356154a5535651c0239158373913fb443de9 (height 144)
2019-03-30 15:27:39.961 [DBG] MINR: Pruning template f5f85cc5b6d94259c36f21f4829f9f7436a83e3217d7cb4b8cc8d4aed6eb98a7 (height 144)
2019-03-30 15:27:39.961 [DBG] MINR: Pruning template 55863401cd15d245cf385d91f7e6ba56043596558367c40a0b833cedeea516da (height 144)
2019-03-30 15:27:39.961 [DBG] MINR: Pruning template 33004ce9ab1f5ac3488f887db71b466bdd872081e2bf8dabf70ab39e1cb972f4 (height 144)
2019-03-30 15:27:39.961 [INF] MINR: Block submitted via CPU miner accepted (hash 611522ddc689c90bf0fa72d821285ea4404c5978e7ea3ecb743c342bd303098d, height 146, amount 346.53534316 DCR)
New templates are generated when a reorganization is encountered:
2019-03-30 15:39:53.736 [INF] MINR: Block submitted via CPU miner accepted (hash 416de4225962f30dd2447e03009604c00ab219a0312a3d05cc687068adae2732, height 155, amount 346.53465346 DCR)
2019-03-30 15:39:55.941 [INF] CHAN: FORK: Block 34c230242f66d96a9fa686ff40aadf587e026319f0f1a2c24f51bf5c916c2f7e (height 155) forks the chain at height 154/block 285116378ed43edf04d8c812788f032df23f908daeaa1ca0e7f842d3140ad84d, but does not cause a reorganize
2019-03-30 15:39:55.941 [INF] BMGR: Processed 2 blocks in the last 1m27.96s (4 transactions, 40 tickets, 10 votes, 0 revocations, height 155, 2019-03-30 15:39:49 -0500 CDT)
2019-03-30 15:39:55.942 [INF] CHAN: REORGANIZE: Block 0930ebabe85e98ad1b6280a88841d799feaae9055dac083ed2bab19cf60238af is causing a reorganize.
2019-03-30 15:39:55.958 [INF] CHAN: REORGANIZE: Chain forks at 285116378ed43edf04d8c812788f032df23f908daeaa1ca0e7f842d3140ad84d (height 154)
2019-03-30 15:39:55.958 [INF] CHAN: REORGANIZE: Old best chain tip was 416de4225962f30dd2447e03009604c00ab219a0312a3d05cc687068adae2732 (height 155)
2019-03-30 15:39:55.958 [INF] CHAN: REORGANIZE: New best chain tip is 0930ebabe85e98ad1b6280a88841d799feaae9055dac083ed2bab19cf60238af (height 156)
...
2019-03-30 15:39:55.960 [DBG] MINR: Adding template for block height 156 with key 14e4478b7365ddb119b14bc5dadabfe2a2030d4184344039ae11268110737e1c469622c87707f8b4ac0fade1d6b6a751435db4301e5e6a4b2ed606b9287f8d7e004e9bee1dd79015 (size: 1)
2019-03-30 15:39:55.960 [DBG] MINR: Received block template building on 34c230242f66d96a9fa686ff40aadf587e026319f0f1a2c24f51bf5c916c2f7e (height 155) (1 transactions, 25 stake transactions, 5 votes, 0 in fees, 0 signature operations, 8020 bytes, target difficulty 7fffff0000000000000000000000000000000000000000000000000000000000, stake difficulty 11.97546649)
...
2019-03-30 15:39:56.978 [DBG] MINR: Adding template for block height 156 with key 8c76b4e4c4ca1ea62282c87229cde7dbf00f866bc4924f1bef133e5ec8d1c315469622c87707f8b4ac0fade1d6b6a751435db4301e5e6a4b2ed606b9287f8d7e0018362a1ed79015 (size: 2)
2019-03-30 15:39:56.978 [DBG] MINR: Received block template building on 34c230242f66d96a9fa686ff40aadf587e026319f0f1a2c24f51bf5c916c2f7e (height 155) (1 transactions, 25 stake transactions, 5 votes, 0 in fees, 0 signature operations, 8020 bytes, target difficulty 7fffff0000000000000000000000000000000000000000000000000000000000, stake difficulty 11.97546649)
...
2019-03-30 15:39:58.948 [DBG] MINR: Adding template for block height 157 with key 2bd3c6fc3be80e44c1c013bdeba46d5add8ee4b5d7cfaf1c90a8ad7a86e1f310b37cfea2cfb8f4059fb0a05998ad35b710b2cc58430397dee62b623c31803b2f00ac6ba11ed79015 (size: 3)
2019-03-30 15:39:58.948 [DBG] MINR: Received block template building on 0930ebabe85e98ad1b6280a88841d799feaae9055dac083ed2bab19cf60238af (height 156) (1 transactions, 5 stake transactions, 5 votes, 0 in fees, 7 signature operations, 2090 bytes, target difficulty 7fffff0000000000000000000000000000000000000000000000000000000000, stake difficulty 11.97546649)
...
2019-03-30 15:39:59.948 [DBG] MINR: Adding template for block height 157 with key bff68740581d48e721873f329b096150a4bf840184fc56105ac17c77d86fb002b37cfea2cfb8f4059fb0a05998ad35b710b2cc58430397dee62b623c31803b2f007606dd1ed79015 (size: 4)
2019-03-30 15:39:59.948 [DBG] MINR: Received block template building on 0930ebabe85e98ad1b6280a88841d799feaae9055dac083ed2bab19cf60238af (height 156) (1 transactions, 5 stake transactions, 5 votes, 0 in fees, 7 signature operations, 2090 bytes, target difficulty 7fffff0000000000000000000000000000000000000000000000000000000000, stake difficulty 11.97546649)
As can be seen from the test results, there are still a couple of non-critical behaviors that should be addressed via separate PRs:
- Two templates are being generated back to back (1 second apart) despite the first one already being a valid template with 5 votes
- The intermediate nodes in a reorganization are generating block templates instead of only the final block
- This will require a more thorough examination as it might not be every intermediate node, rather only the one prior to the final block and is likely related to lack of enough votes on the final tip
JFixby
pushed a commit
to JFixby/dcrd
that referenced
this pull request
Sep 20, 2019
…timization build: update to latest version of btcutil
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR requires #1417, #1418, #1419, #1420, #1421, #1422 and #1423
BgBlkTmplGeneratorrepresents the background process that generates block templates and notifies all subscribed clients on template regeneration. It generates new templates based on mempool activity for vote and non-vote transactions and the time elapsed since last template regeneration.