kvs: store data pointed by valref raw and unencoded in content store #1214
Conversation
Change internal json_t *o field to void *data, so the internally cached data is not type specific.
Convert cache_entry_create() to not accept any parameters, it now only creates an empty cache entry. Create new function cache_entry_create_json(), that takes an optional json object to hold the cache entry data. If the user passes in NULL, it behaves identically to calling cache_entry_create(). Adjust callers of cache_entry_create() appropriately throughout. Add unit tests and coverage appropriately.
Support type field in cache entries, so different types of data can be stored in the cache entry. Add new functions cache_entry_type() and cache_entry_is_type_json() to retrieve/test type. Add unit tests appropriately.
Add support for cache entries that support storing raw data. Included in this patch is new function cache_entry_create_raw(), cache_entry_is_type_raw(), cache_entry_get_raw(), and cache_entry_set_raw(). Add unit tests appropriately.
Update internal lookup API to handle valref references pointing to raw unencoded data in the content store. Most notably, missing references have an additional boolean indicating if data is raw, so that on data load it can be handled appropriately. In addition, when returning data pointed to by a valref, add checks to ensure the data is raw in the KVS cache. Adjust lookup unit tests appropriately.
Update internal commit API to handle valref references pointing to raw unencoded data in the content store. Most notably, create cache entries with raw data when appropriate, so it will ultimately get flushed to the content store as raw unencoded data.
Update lookup, commit, and KVS cache calls to handle potential for data being raw or json. The major change being valref treeobj objects pointing to raw unencoded data in the content store. Fixes flux-framework#1187
Codecov Report
@@ Coverage Diff @@
## master #1214 +/- ##
==========================================
- Coverage 78.23% 78.22% -0.01%
==========================================
Files 158 158
Lines 29142 29283 +141
==========================================
+ Hits 22800 22908 +108
- Misses 6342 6375 +33
|
|
just realized I didn't do one specific test. I should add some tests to try and handle the specific case of putting content into the content store via |
| { | ||
| struct cache_entry *hp; | ||
|
|
||
| if ((data && len <= 0) || (!data && len)) { |
There was a problem hiding this comment.
The content cache allows a size = 0 blob to be stored. Probably we should allow it in the KVS and add test coverage. Not sure if it worked before, but this would prevent it I guess.
I'll review more closely tomorrow - just noticed that on a quick skim.
There was a problem hiding this comment.
the intent was if len == 0, data should == NULL. Would this be ok?
There was a problem hiding this comment.
Oh duh, sorry, I guess I didn't look closely enough there!
There was a problem hiding this comment.
I'll open a separate bug on ability to store zero length data under a key as there do seem to be some issues there, unrelated to this PR.
|
OK, I opened #1215 on zero length values as a test coverage issue, since it seems to work OK actually. I made a pass through the code and didn't spot anything wrong. Since the test coverage is pretty good, I'd be inclined to give the thumbs up, though I think you're proposed test addition is a good one to add before we merge. I do apologize that you had to do so much work to accomplish a seemingly simple change. As we discussed earlier, I think the ad-hoc, cache-centric mechanism used in the KVS to implement the reactive idiom is at fault. We most likely don't benefit from caching raw values (as opposed to decoded JSON metadata), but we're forced into it because that's the way we stall/restart message handlers while we wait for content RPC's. I'll try to get a bug open on that work so we can track it in the plan and decide if it's worth doing sooner rather than later. Had we done it before this, I think you would have had an easier time. |
Add tests to ensure data written to content store in raw form via the kvs can be read outside of the KVS. Also verify that valref tree objects can be put into the KVS to point to data already in the content store.
|
restarted some builds, write error one and a #1169 one. |
|
OK, thanks @chu11 this is progress! |
This series of patches updates the KVS so that treeobj valref objects will read/write data to the content store in raw/unencoded form.
The key work was to ensure the internal KVS cache could handle storing raw data instead of only json. After this work, the remaining work sort of "fell out" of this. On lookup, create cache entries that contain raw data and handle them appropriately for valref objects. On commits, cache entries with raw data will be written out as raw data (instead of everything being assumed json).