[processor/logdeup] Add scope aggregator #34607
Conversation
|
|
||
| hasher.Write(attrHash[:]) | ||
| // getScopeKey creates a unique hash for the scope to use as a map key | ||
| func getScopeKey(scope pcommon.InstrumentationScope) uint64 { |
There was a problem hiding this comment.
Can we add this to pdatautil?
There was a problem hiding this comment.
Extended pdatautil to do more of the work here instead in a716f65
| attrHash := pdatautil.MapHash(logRecord.Attributes()) | ||
| // getResourceKey creates a unique hash for the resource to use as a map key | ||
| func getResourceKey(resource pcommon.Resource) uint64 { | ||
| hasher := xxhash.New() |
There was a problem hiding this comment.
What's the rationale for wrapping the pdatautil output? In general I think we should standardize where possible. If there's something deficient about the pdatautil implementation, can we fix or enhance it? Maybe just a pdatautil.ResourceHash which aliases MapHash so that there's an unambiguous standard?
| } | ||
|
|
||
| // getLogKey creates a unique hash for the log record to use as a map key | ||
| func getLogKey(logRecord plog.LogRecord) uint64 { |
There was a problem hiding this comment.
I see why this one isn't standard, since we're only accounting for some fields. Maybe an options pattern accounts for it in the future, but I'm find with leaving it alone for now.
|
@djaglowski Moved hash generation into pdatautil, let me know what you think 😄 |
**Description:** Adds a scope aggregator to separate log records by different scopes. Also includes some refactor work: - Extend pdatautil.hash with configurable options of values to hash, includes ability of returning uint64 - Add consistent hash functions for resource, scope and log records that use new hash functions - Update aggregators to keep a copy of the original resource, scope and log record instead of just it's attributes - Simplifies ConsumeLogs & Export functions - Update newLogCounter to set lastObservedTime - Update existing and add more tests **Link to tracking Issue:** - Closes open-telemetry#34606 34606 **Testing:** Updated existing and added more unit tests to verify behaviour.
Description:
Adds a scope aggregator to separate log records by different scopes.
Also includes some refactor work:
Link to tracking Issue:
Testing:
Updated existing and added more unit tests to verify behaviour.