[gov] Migrate to new politeia tlog API

[thi4go]

This diff implements the changes necessary to migrate to the new politeia tlog APIs. All data is synced through politeia APIs and stored on stormdb for now. Talked to @chappjc and we might want to drop the stormdb dependency in the near future, so this data might go to a pg table instead of being stored on storm. Major changes on this PR were:

• Drops go-piparser that dcrdata needed as part of the logic of fetching the git repos and parsing votes data.
• The pg tables proposals and proposal_votes are no longer used anywhere in the code.
• Chart data is now pre-parsed from the api and stored on stormdb with the proposal object
• Code cleanups and refactoring of the gov module for improved readability and reasoning.

I think there isn't a need for the gov package to be a standalone module anymore. I left it be, but if you agree on this, I can make it part of this PR.

About supporting the legacy proposals, politeiagui is using legacyproposals.json and legacyvotestatuses.json to display part of the data. These would be enough for the dcrdata UI if we import them, except for the tickets vote data for the charts. So I think we have two options:

• Show charts on legacy proposals, and keep proposals and proposal_votes tables.
• Decide to not show charts on legacy proposals, and nuke the pg tables completely.

With this migration we lose the ability to reconstruct the pg tables if we ever need it, since we dropped the go-piparser dep and all the legacy code behind it, so I'm not sure how I feel about the first option. The data will always be availabe on git if someone ever wants the parse it themselves and verify it, so I tend to think it's okay not showing charts for the legacy props.

Adding support for legacy proposals will be done next in a fresh PR in order to facilitate review.

closes #1816 and #1823

[thi4go]

This is ready for a review @chappjc and @buck54321 if also interested mate. Need to delete proposals.db from dcrdata data folder before testing it out.

[chappjc]

Thanks, @thi4go . I got it.

[chappjc]

Going to review this today after I push a couple fixes. Don't worry about the conflicts for now @thi4go

[thi4go]

Sounds good mate @chappjc 👍

[chappjc]

I'm started reviewing, but the merge commit from master makes squashing it a little trickier than usual. Please prefer rebase to merge in the future.

But even with numerous commits, it's often easier to squash first without changing the base commit and then rebase rather than doing it in one step with git rebase -i.

What I'm reviewing now is this PR squashed, then rebased on master. I chose to do it this time like the following (after pulling latest master):

git checkout tlog
git branch tlog-pre-squash
git reset --hard master             # start tlog branch over at master
git merge --squash tlog-pre-squash  # replay tlog branch commits in one commit
# then resolve conflicts

but could also have done:

git checkout tlog
git reset --soft ec6b0e6  # the initial branch point, with tlog's working tree and index kept
git commit                # single commit for entire branch
git rebase master         # replay the commit on master
# then resolve conflicts and git rebase --continue

This is the single rebased commit I came up with: chappjc@6e5d940
Please see if that looks right and see if you can squash and rebase this PR down similarly, thanks.

[chappjc]

I had about 30 other line comments that github just discarded. Here's one and I'll write my review again later.

[thi4go]

Hey mate @chappjc thanks for the review. This is ready for another look, comments addressed.

[chappjc]

Great. Apologies for failing to alert you to the axios elimination.

[thi4go]

Hey @chappjc thx again for the review. Second round of comments addressed 👍

[chappjc]

I think there isn't a need for the gov package to be a standalone module anymore. I left it be, but if you agree on this, I can make it part of this PR.

This was to isolate the storm dependency too.

[chappjc]

On the Daylight Robery prop that's voting right now, I'm getting the following at http://127.0.0.1:7777/proposal/ae609f183a031c34:

Seeing the following on the Dubai prop that's not voting yet (http://127.0.0.1:7777/proposal/51c412822e5e4b59), but also on the Storyteller prop:

Seeing this on an accepted prop like Automatic Ticket Revocations (http://127.0.0.1:7777/proposal/e2d7b7d6b837431f), same as the ones that are currently voting:

I've rebuilt everything a couple times.

[thi4go]

The charts broke after I ditched axios, should have changed the way we get the response, my bad mate @chappjc . Pushed a fix for it. The Error: Missing target element "proposal.token" we are also getting on master, still investigating what it might be.

[chappjc]

Ahhh, yes, having removed a ton of .datas when axios was dropped, I should have noticed that. Thanks for fixing.

[chappjc]

I'm just hacking wildly around Stimulus now, but I got around the proposal.token error with the following:

diff --git a/cmd/dcrdata/public/js/controllers/proposal_controller.js b/cmd/dcrdata/public/js/controllers/proposal_controller.js
index b27fb022..a24cc85f 100644
--- a/cmd/dcrdata/public/js/controllers/proposal_controller.js
+++ b/cmd/dcrdata/public/js/controllers/proposal_controller.js
@@ -118,20 +118,20 @@ export default class extends Controller {
   }
 
   async connect () {
-    if (this.hasApprovalMeterTarget) {
-      const d = this.approvalMeterTarget.dataset
-      const opts = {
-        darkMode: darkEnabled(),
-        segments: [
-          { end: d.threshold, color: '#ed6d47' },
-          { end: 1, color: '#2dd8a3' }
-        ]
-      }
-      this.approvalMeter = new MiniMeter(this.approvalMeterTarget, opts)
+    if (!this.hasApprovalMeterTarget) return // there will be no meter or charts
+
+    const d = this.approvalMeterTarget.dataset
+    const opts = {
+      darkMode: darkEnabled(),
+      segments: [
+        { end: d.threshold, color: '#ed6d47' },
+        { end: 1, color: '#2dd8a3' }
+      ]
     }
+    this.approvalMeter = new MiniMeter(this.approvalMeterTarget, opts)
 
     chartData = await requestJSON('/api/proposal/' + this.tokenTarget.dataset.hash)
-
+    if (!chartData) return
     Dygraph = await getDefault(
       import(/* webpackChunkName: "dygraphs" */ '../vendor/dygraphs.min.js')
     )

[chappjc]

Looks good now, and working well.

Some follow-up for me:

• the stimulus fix described in my last comment for when there are no votes and thus no approval meter or charts
• DB upgrade to drop the orphaned proposals table
• Somehow snapshotting the old props (vote data extracted from git repo), and importing all that into the new ProposalsDB @thi4go is working on loading the legacy json files from the politeiagui repo into the actual tlog so we will eventually get the legacy prop info for free

[xaur]

closes #1816 and #1823

Not sure if this PR is deployed yet, but just in case I can still reproduce #1823 at explorer.dcrdata.org (that is, Discussion links still use wrong paths).

[xaur]

Nevermind, just learned about tip.dcrdata.org and Discussion links are correct there so I'll just wait for the deployment.