[wip] Refactor carmen-cache #116
Conversation
…sce is done now and the util files are started
…p/hpp files; update other files for appropriate includes"
…sed parameters, which I'm squashing with pragmas since they're mandated by outside libraries
…reachable code (mostly extra returns at the ends of functions)
…lly-committed package-lock.json
…comments for the C++-internal parts of coalesce, as well as node_util
There was a problem hiding this comment.
@aaaandrea here are some notes on the docs for MemoryCache, RocksDBCache, and cpp_util. Sorry for the wall of text 😄
src/memorycache.cpp
Outdated
| * @name pack | ||
| * @memberof MemoryCache | ||
| * @param {String}, filename | ||
| * @returns {String}, filename |
There was a problem hiding this comment.
This doesn't return anything:
info.GetReturnValue().Set(Nan::Undefined());
src/memorycache.cpp
Outdated
| } | ||
|
|
||
| /** | ||
| * lists the data in the memory cache object |
There was a problem hiding this comment.
This only lists the keys in the store, not the values; "data" is ambiguous
src/memorycache.cpp
Outdated
| } | ||
|
|
||
| /** | ||
| * replaces the data in the object |
There was a problem hiding this comment.
This function doesn't set all the data, just the data for a given key. It might replace, or it might append to what's already there, depending on the value of the append argument
src/memorycache.cpp
Outdated
| * @name set | ||
| * @memberof MemoryCache | ||
| * @param {String} id | ||
| * @param {Array}, data |
There was a problem hiding this comment.
It takes up to four arguments; the third and fourth are optional (append is a boolean for whether to append or replace and languages is a language ID array). Also it would be good to say what kind of array the data is (an array of numbers), and that each number represents a grid.
src/memorycache.cpp
Outdated
| * @memberof MemoryCache | ||
| * @param {String} id | ||
| * @param {Array}, data | ||
| * @returns {String} |
There was a problem hiding this comment.
Again, doesn't return anything
src/cpp_util.cpp
Outdated
|
|
||
| namespace carmen { | ||
|
|
||
| // derives grid values |
There was a problem hiding this comment.
Covers are a representation of (relev, score, x, y, feature_id) packed into a single 64-bit integer. This function converts from the packed integer into the full structure where you can access each property.
src/cpp_util.cpp
Outdated
| return cover; | ||
| } | ||
|
|
||
| // Compute distance between coordinates |
There was a problem hiding this comment.
This isn't a distance calculation. It converts from the ZXY coordinates of the tile that contains a proximity point to a ZXY at the appropriate zoom level necessary for a given coalesce operation (which may examine grids at multiple zoom levels that might not be the same as the zoom level used to specify the proximity location).
src/cpp_util.cpp
Outdated
| } | ||
|
|
||
|
|
||
| // Compute distance between coords + zoom multiplier |
There was a problem hiding this comment.
Again, doesn't calculate a distance, but rather, calculates a ZXY for appropriate for a given coalesce operation out of the supplied bbox ZXY coordinates.
src/cpp_util.cpp
Outdated
| return score > scoredist ? score : scoredist; | ||
| } | ||
|
|
||
| // database lookup |
There was a problem hiding this comment.
This doesn't look anything up, it just opens the file so it's ready for lookups later. We do this when a carmen instance starts. This one is read-write, so we mostly use it for indexing.
src/cpp_util.cpp
Outdated
| return status; | ||
| } | ||
|
|
||
| // read only database lookup |
There was a problem hiding this comment.
The same, but read-only (used at runtime).
src/normalizationcache.cpp
Outdated
| * const cache = require('@mapbox/carmen-cache'); | ||
| * const nc = new cache.NormalizationCache('file.norm.rocksdb', true); | ||
| * | ||
| * // for a normalization cache with over the dictionary ['main st', 'main street'] |
src/normalizationcache.cpp
Outdated
| * | ||
| * @name writeBatch | ||
| * @memberof NormalizationCache | ||
| * @returns {Array} |
…d ensure that all tape assertions have assertion messages
Fix LTO on linux
…m in all test environments, and document benchmarking process
| constexpr uint64_t POW2_3 = static_cast<uint64_t>(_pow(2.0, 3)); | ||
| constexpr uint64_t POW2_2 = static_cast<uint64_t>(_pow(2.0, 2)); | ||
|
|
||
| struct PhrasematchSubq { |
There was a problem hiding this comment.
We could. I'm inclined not to, though. The typedef describes the way JS objects should be formatted in the phrasematches array on the way into coalesce. They come in as JS objects, and then are processed and their contents stored in C++ structs of this type ^. The two types serve the same purpose, but are not the same: aside from being of different types in a technical sense, they also don't have exactly the same fields, either in terms of what they're called or what type they are. Like, languages on the JS side is an array of integers, but that gets converted into a bitfield on the C++ side and stored in a property called langfield. I sort of think conflating the two is a recipe for confusion. (That said, given that, maybe I should call the JS type something different?)
Also, as a practical matter, at the moment we aren't looking for JSDoc entries in headers, and were also trying to keep the cpp_util.* files generally free of JS-isms. Those are both trivially changeable though, so I think they're less compelling arguments than the above.
There was a problem hiding this comment.
looks like we crossed in the mail, there. sgtm!
There was a problem hiding this comment.
(That said, given that, maybe I should call the JS type something different?)
Would be less confusing if I had API.md open and was browsing C++ code...
There was a problem hiding this comment.
Renamed the JS-accessible version in c6d30e5
| } | ||
| } | ||
|
|
||
| // in general, the key format including language field is <key_text><separator character><8-byte langfield> |
…n tests and docs, to avoid confusion with the C++ PhrasematchSubq struct type, which has slightly different properties
This PR reflects the work-in-progress refactor effort described in #115 .