-
Notifications
You must be signed in to change notification settings - Fork 68
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(streaming): raw event connector #1720
base: main
Are you sure you want to change the base?
Conversation
ae3da9b
to
d15c55f
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why did you start passing meter models.Meter
to some functions and kept meterSlug
in others? Why start doing it at all? Can't the connector query the meter from the repository?
// Engine is the aggregation engine to use | ||
Engine AggregationEngine | ||
ClickHouse ClickHouseAggregationConfiguration |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Long term we should probably move the clickhouse config to top level and better organize engine configs.
This change only increases the chaos as it is with aggregation level, but clickhouse specific configs. :/
) | ||
|
||
// ParseEvent validates and parses an event against a meter. | ||
func ParseEvent(meter Meter, ev event.Event) (*float64, *string, map[string]string, error) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why is validate being replaced by parsing?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The new connector required to know value. It was always parsing, just before didn't return the parsed value. This file looks new but it's mostly renamed. I removed the new connector from the PR but would like to keep parsing this way as it helps testing.
7a2161e
to
98d0cbe
Compare
1b4b83e
to
7a1e5cc
Compare
As of today, OpenMeter uses ClickHouse materialized views to track meters. Events are inserted into a raw events table on which MVs are based. This PR moves the event insert logic out from the sink worker to the streaming connector and splits the connector into two different connectors:
PR review guide: