-
Notifications
You must be signed in to change notification settings - Fork 75
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
multi: Add cache layer #660
Conversation
Is this change going to be backward compatible for non-politeiawww clients of politeiad? |
@sndurkin the politeiad The way its written now, a cache must be used with politeiad. It could be updated to have some type of Edit: the cache now defaults to being disabled on politeiad. The |
e4b4cb0
to
0006904
Compare
This commit adds a cache layer to politeiad and politeiawww. The politeiad git repositories serve as the source of truth for proposal data and a CockroachDB database acts as a cache for proposal data. politeiad has read and write access to the cache. politeiawww has only read access to the cache. The flow of data is as follows: 1. politeiawww receives a command from a user 2. politeiawww creates a politeiad request for the command and sends it 3. politeiad writes the command data to the git repository and updates the cache 4. politeiad returns a response to politeiawww 5. politeiawww reads the cache if needed 6. politeiawww returns a response to the user The cache is automatically rebuilt on cache version change. You can also force the cache to rebuilt by using the flag --buildcache on politeiad startup. The politeiad record and all record metadata has been removed from the politeiawww inventory. The only thing left in the inventory is decred plugin data. The decred plugin data will be added to the cache and removed from the inventory in a future PR. Since this commit touched so much of the code for the `/proposals` endpoints, I began cleaning up `backend.go` and moving the updated `/proposals` enpoint code to `proposals.go`. I had to delete all of the tests that relied on the proposal record in the inventory. I re-wrote most of these tests in `proposals_test.go`.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I love this! Just a couple of nits.
decredplugin/decredplugin.go
Outdated
// InventoryReply returns the decred plugin inventory. | ||
type InventoryReply struct { | ||
Comments []Comment `json:"comments"` // Comments | ||
CommentLikes []LikeComment `json:"commentlikes"` // Comment likes |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The asymmetry is bothering my OCD!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed.
politeiad/backend/gitbe/decred.go
Outdated
decredPluginJournals = "journals" | ||
decredPluginIdentity = "fullidentity" | ||
decredPluginJournals = "journals" | ||
decredPluginInventoryCmd = "inventorycmd" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why is this postfixed cmd?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It doesn't need to be. It has been shortened to "inventory".
politeiad/cache/cache.go
Outdated
Settings []PluginSetting // Settings | ||
} | ||
|
||
// PluginDriver is the interface that all cache plugins must implement. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Explain what this does and why it exists.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added doco.
} | ||
defer rows.Close() | ||
|
||
records := make([]Record, 0, 1024) // PNOOMA |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Glad you picked up on the PNOOMA development methodology!
* fix prop display * clear warning
Closes #546.
This PR adds a cache layer to politeiad and politeiawww. The politeiad git repositories serve as the source of truth for proposal data and a CockroachDB database acts as a cache for proposal data.
The cache is not required if you're running just politeiad. politeiad has the cache disable by default. If you're running the full politeia stack, politeiad and politeiawww, running the cache is required.
politeiad has read and write access to the cache. politeiawww has only read access to the cache. The flow of data is as follows:
The cache will build itself automatically on startup if it hasn't been built yet or it the version changes. You can also force a cache rebuild by using the
--buildcache
flag when starting politeiad.This PR does not attempt to optimize the cache performance at all. Bottlenecks will be identified and performance will be optimized in a future PR.
Other major changes this PR makes:
SetUnvettedStatus
,SetVettedStatus
, andUpdateRecord
. These responses no longer return the full record. They only return the status of the update.votedetails
,getcomment
,commentlikes
, andinventory
.