-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
chore(sinks): Update PartitionBatcher
to use BatchConfig
#18792
Conversation
Signed-off-by: Stephen Wakely <[email protected]>
Signed-off-by: Stephen Wakely <[email protected]>
✅ Deploy Preview for vector-project canceled.
|
✅ Deploy Preview for vrl-playground ready!
To edit notification comments on pull requests, go to your Netlify site configuration. |
Signed-off-by: Stephen Wakely <[email protected]>
Signed-off-by: Stephen Wakely <[email protected]>
Signed-off-by: Stephen Wakely <[email protected]>
/ci-run-regression |
/ci-run-integration |
Signed-off-by: Stephen Wakely <[email protected]>
Datadog ReportBranch report: ✅ |
Regression Detector ResultsRun ID: abf3002b-68aa-47b8-acc5-ce1419b05882 ExplanationA regression test is an integrated performance test for Because a target's optimization goal performance in each experiment will vary somewhat each time it is run, we can only estimate mean differences in optimization goal relative to the baseline target. We express these differences as a percentage change relative to the baseline target, denoted "Δ mean %". These estimates are made to a precision that balances accuracy and cost control. We represent this precision as a 90.00% confidence interval denoted "Δ mean % CI": there is a 90.00% chance that the true value of "Δ mean %" is in that interval. We decide whether a change in performance is a "regression" -- a change worth investigating further -- if both of the following two criteria are true:
The table below, if present, lists those experiments that have experienced a statistically significant change in mean optimization goal performance between baseline and comparison SHAs with 90.00% confidence OR have been detected as newly erratic. Negative values of "Δ mean %" mean that baseline is faster, whereas positive values of "Δ mean %" mean that comparison is faster. Results that do not exhibit more than a ±5.00% change in their mean optimization goal are discarded. An experiment is erratic if its coefficient of variation is greater than 0.1. The abbreviated table will be omitted if no interesting change is observed. No interesting changes in experiment optimization goals with confidence ≥ 90.00% and |Δ mean %| ≥ 5.00%. Fine details of change detection per experiment.
|
Signed-off-by: Stephen Wakely <[email protected]>
Signed-off-by: Stephen Wakely <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This LGTM with the exception of a question around the batch settings closure.
/// A closure that retrieves a new [`BatchConfig`] when needed to batch a | ||
/// new partition. | ||
state: F, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
All users of this batcher appear to pass in a state closure that returns the same thing every time. Am I misreading the callers? Could we instead store that thing with a Clone
bound so as to just make copies of it?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Every time it is called it creates a new batch which contains not just the settings (size limits etc) for the batch but also the batch data. A Clone
on that record would also clone that data which would not be desirable - unless we implemented Clone ourselves and forced it to not clone the data. I don't think would be desirable.
I could create a new trait, something like CloneSettingsButNotData
, but I feel the closure is simpler. Do you have strong feelings otherwise?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I was thinking to clone an empty batch, which wouldn't have any data yet. However, what you describe makes sense. I don't have strong feelings otherwise, no.
let batch = if let Some(batch) = this.batches.get_mut(&item_key) { | ||
batch | ||
} else { | ||
// We have no batch yet for this partition key, so | ||
// create one and create the expiration entries as well. | ||
// This allows the batch to expire before filling up, | ||
// and vice versa. | ||
let batch = Batch::new(item_limit, alloc_limit).with(item); | ||
let batch = (this.state)(); | ||
this.batches.insert(item_key.clone(), batch); | ||
this.batches | ||
.get_mut(&item_key) | ||
.expect("batch has just been inserted so should exist") | ||
}; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Would using let batch = this.batches.get_mut(&item_key).unwrap_or_else(|| …);
simplify this any?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The get_mut
takes a mutable borrow of this.batches
which means we aren't able to get another mutable borrow for this.batches.insert
.
I swear I did see a function get_or_insert_with
at some point, but I can't find it now. It's possible that was in another language..
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You mean this?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah, possibly. Wrong container... :-(
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
From an Integrations perspective, the changes look fine to me
Signed-off-by: Stephen Wakely <[email protected]>
Signed-off-by: Stephen Wakely <[email protected]>
Datadog ReportBranch report: ✅ |
# [3.0.0](answerbook/vector@v2.1.2...v3.0.0) (2024-01-17) ### Bug Fixes * **amqp sink**: remove duplicate events (vectordotdev#18932) [a916605](answerbook/vector@a916605) - GitHub * **amqp sink**: remove unnecessary unwrap & emit event dropped errors (vectordotdev#18923) [26f430c](answerbook/vector@26f430c) - GitHub * **amqp sink**: remove unused feature flag (vectordotdev#18948) [cb53588](answerbook/vector@cb53588) - GitHub * **buffers**: apply stricter file permissions to buffer data files when possible (vectordotdev#18895) [cf7298f](answerbook/vector@cf7298f) - GitHub * **clickhouse sink**: fix healthcheck uri (vectordotdev#19067) [7a7b53b](answerbook/vector@7a7b53b) - GitHub * **codecs**: fix 'ProtobufSerializerConfig' input type (vectordotdev#19264) [d2fea65](answerbook/vector@d2fea65) - Jesse Szwedko * **codecs**: native JSON serialization/deserialization for special f64 values (vectordotdev#18650) [39b9298](answerbook/vector@39b9298) - GitHub * **config**: Only try default paths if the path is not specified (vectordotdev#18681) [37fc9db](answerbook/vector@37fc9db) - GitHub * **datadog_agent source, datadog_metrics sink**: handle interval for non-rate series metrics (vectordotdev#18889) [dc9d966](answerbook/vector@dc9d966) - GitHub * **datadog_agent source**: return 200 on empty object payload (vectordotdev#19093) [ab7983a](answerbook/vector@ab7983a) - Jesse Szwedko * **datadog_metrics sink**: evaluate series v1 env var at runtime (vectordotdev#19148) [7b292ce](answerbook/vector@7b292ce) - Jesse Szwedko * **datadog_metrics sink**: improve aggregation performance (vectordotdev#18759) [6a40169](answerbook/vector@6a40169) - GitHub * **datadog_metrics sink**: Revert to using v1 endpoint by default (vectordotdev#19138) [3158f46](answerbook/vector@3158f46) - Jesse Szwedko * **datadog_metrics sink**: the integration tests weren't actually validating anything (vectordotdev#18754) [afc166f](answerbook/vector@afc166f) - GitHub * **datadog_traces sink**: improve request size limiting (vectordotdev#18903) [a477d72](answerbook/vector@a477d72) - GitHub * **debian platform**: Re-add `conf-files` directive for `cargo-deb` (vectordotdev#18726) [e445721](answerbook/vector@e445721) - GitHub * **dev**: Gate config conversion tests (vectordotdev#18698) [c35ae64](answerbook/vector@c35ae64) - GitHub * **dev**: update environment for website development (vectordotdev#18657) [4327776](answerbook/vector@4327776) - GitHub * **dnstap source**: support DNSSEC RRSIG record data (vectordotdev#18878) [ed97f0d](answerbook/vector@ed97f0d) - GitHub * **docker source**: do not emit component error for out of order logs (vectordotdev#18649) [93d7af4](answerbook/vector@93d7af4) - GitHub * **http_server source**: panic when http server receives metric events (vectordotdev#18781) [b107ff7](answerbook/vector@b107ff7) - GitHub * **kafka sink**: Make KafkaService return `Poll::Pending` when producer queue is full (vectordotdev#18770) [a1863e6](answerbook/vector@a1863e6) - GitHub * **kafka source**: fix acknowledgement handling during shutdown and rebalance events (vectordotdev#17497) [f2efb1a](answerbook/vector@f2efb1a) - GitHub * **loki sink**: update to use the global list of compression algorithms (vectordotdev#19099) [218963a](answerbook/vector@218963a) - Jesse Szwedko * **releasing**: Update cargo-deb (vectordotdev#19009) [88194e7](answerbook/vector@88194e7) - GitHub * **releasing**: Update example YAML config data_dir (vectordotdev#18896) [0b27019](answerbook/vector@0b27019) - GitHub * remove gh token call (vectordotdev#19047) [1c864aa](answerbook/vector@1c864aa) - GitHub * **sources**: emit `ComponentEventsDropped` when source send is cancelled (vectordotdev#18859) [a0e2769](answerbook/vector@a0e2769) - GitHub * **tls**: for incoming connection alpn negotiation should be done using set_alpn_select_callback (vectordotdev#18843) [8a5b67e](answerbook/vector@8a5b67e) - GitHub ### Chores * Add SHA256 checksums file to GH releases (vectordotdev#18701) [18f07a0](answerbook/vector@18f07a0) - GitHub * **ci**: Add a summary if the regression workflow is skipped (vectordotdev#18724) [17bd2b1](answerbook/vector@17bd2b1) - GitHub * **ci**: Add a test to assert conf files aren't overwritten (vectordotdev#18728) [3ade682](answerbook/vector@3ade682) - GitHub * **ci**: add dependabot group for futures (vectordotdev#18954) [c23efce](answerbook/vector@c23efce) - GitHub * **ci**: Bump aws-actions/amazon-ecr-login from 1 to 2 (vectordotdev#18752) [8e2032c](answerbook/vector@8e2032c) - GitHub * **ci**: Bump aws-actions/configure-aws-credentials from 4.0.0 to 4.0.1 (vectordotdev#18771) [c9804f0](answerbook/vector@c9804f0) - GitHub * **ci**: Bump bufbuild/buf-setup-action from 1.26.1 to 1.27.0 (vectordotdev#18783) [a784018](answerbook/vector@a784018) - GitHub * **ci**: Bump bufbuild/buf-setup-action from 1.27.0 to 1.27.1 (vectordotdev#18866) [811b7f7](answerbook/vector@811b7f7) - GitHub * **ci**: Bump bufbuild/buf-setup-action from 1.27.1 to 1.27.2 (vectordotdev#18981) [72560b1](answerbook/vector@72560b1) - GitHub * **ci**: Bump check-spelling/check-spelling from 0.0.21 to 0.0.22 (vectordotdev#18723) [f98cd5d](answerbook/vector@f98cd5d) - GitHub * **ci**: Bump MacOS unit test runners to 13 (vectordotdev#18823) [5bf18df](answerbook/vector@5bf18df) - GitHub * **ci**: Bump tspascoal/get-user-teams-membership from 2 to 3 (vectordotdev#18808) [4701bb9](answerbook/vector@4701bb9) - GitHub * **ci**: filter team members from gardener issue comment workflow (vectordotdev#18915) [bf56ac5](answerbook/vector@bf56ac5) - GitHub * **ci**: Fix cookie banner style issues (vectordotdev#18745) [7a55e54](answerbook/vector@7a55e54) - GitHub * **ci**: Remove unusued Dockerfile (vectordotdev#18824) [9d1a676](answerbook/vector@9d1a676) - GitHub * **ci**: Revet bump check-spelling/check-spelling from 0.0.21 to 0.0.22 (vectordotdev#18742) [7dce292](answerbook/vector@7dce292) - GitHub * **ci**: Run deny check nightly instead of on every PR (vectordotdev#18799) [ae117dc](answerbook/vector@ae117dc) - GitHub * **ci**: temporarily peg greptimedb to `v0.4.0` to unblock CI (vectordotdev#18838) [0776cc0](answerbook/vector@0776cc0) - GitHub * convert test config to yaml (vectordotdev#18856) [8b00214](answerbook/vector@8b00214) - GitHub * **core**: Add a CLI flag to allow for empty configs (vectordotdev#19021) [df4921b](answerbook/vector@df4921b) - GitHub * **core**: add more event metadata to proto (vectordotdev#18816) [2deeba1](answerbook/vector@2deeba1) - GitHub * **core**: Refactor `vector-core::stream` into its own package (vectordotdev#18900) [96f4d73](answerbook/vector@96f4d73) - GitHub * **core**: Set up internal topology API (vectordotdev#18919) [c9c184e](answerbook/vector@c9c184e) - GitHub * **datadog_metrics sink**: Set partial Origin Metrics in edge cases (vectordotdev#18677) [3dab239](answerbook/vector@3dab239) - GitHub * **datadog_metrics sink**: support and migrate to the `v2` series API endpoint (vectordotdev#18761) [3485f2c](answerbook/vector@3485f2c) - GitHub * **datadog**: remove deprecated config options (vectordotdev#18940) [f42751d](answerbook/vector@f42751d) - GitHub * **deps**: Add more `dependabot` groups (vectordotdev#18719) [d0e605e](answerbook/vector@d0e605e) - GitHub * **deps**: Bump @babel/traverse from 7.17.0 to 7.23.2 in /website (vectordotdev#18852) [85d2f17](answerbook/vector@85d2f17) - GitHub * **deps**: Bump apache-avro from 0.15.0 to 0.16.0 (vectordotdev#18685) [b37ce3c](answerbook/vector@b37ce3c) - GitHub * **deps**: Bump async-compression from 0.4.3 to 0.4.4 (vectordotdev#18848) [1a8a8cc](answerbook/vector@1a8a8cc) - GitHub * **deps**: Bump async-graphql from 5.0.10 to 6.0.9 (vectordotdev#18988) [4a4eb61](answerbook/vector@4a4eb61) - GitHub * **deps**: Bump async-nats from 0.32.0 to 0.32.1 (vectordotdev#18735) [047c772](answerbook/vector@047c772) - GitHub * **deps**: Bump async-trait from 0.1.73 to 0.1.74 (vectordotdev#18849) [8d82257](answerbook/vector@8d82257) - GitHub * **deps**: Bump base64 from 0.21.4 to 0.21.5 (vectordotdev#18907) [e754dee](answerbook/vector@e754dee) - GitHub * **deps**: Bump bitmask-enum from 2.2.2 to 2.2.3 (vectordotdev#19057) [7a16ee2](answerbook/vector@7a16ee2) - GitHub * **deps**: Bump bstr from 1.6.2 to 1.7.0 (vectordotdev#18810) [91221c6](answerbook/vector@91221c6) - GitHub * **deps**: Bump cached from 0.45.1 to 0.46.0 (vectordotdev#18660) [331c5a0](answerbook/vector@331c5a0) - GitHub * **deps**: Bump cached from 0.46.0 to 0.46.1 (vectordotdev#19058) [51c6b57](answerbook/vector@51c6b57) - GitHub * **deps**: Bump cargo_toml from 0.16.3 to 0.17.0 (vectordotdev#18978) [b55e436](answerbook/vector@b55e436) - GitHub * **deps**: Bump chrono from 0.4.30 to 0.4.31 (vectordotdev#18583) [052ed98](answerbook/vector@052ed98) - GitHub * **deps**: Bump chrono-tz from 0.8.3 to 0.8.4 (vectordotdev#18979) [7f44b4c](answerbook/vector@7f44b4c) - GitHub * **deps**: Bump clap from 4.4.5 to 4.4.6 (vectordotdev#18715) [4d98fdf](answerbook/vector@4d98fdf) - GitHub * **deps**: Bump clap_complete from 4.4.2 to 4.4.3 (vectordotdev#18716) [27b2c93](answerbook/vector@27b2c93) - GitHub * **deps**: Bump console-subscriber from 0.1.10 to 0.2.0 (vectordotdev#18732) [eda0378](answerbook/vector@eda0378) - GitHub * **deps**: Bump csv from 1.2.2 to 1.3.0 (vectordotdev#18768) [7cb8b52](answerbook/vector@7cb8b52) - GitHub * **deps**: Bump dd-rust-license-tool to 1.0.2 (vectordotdev#18711) [570bd52](answerbook/vector@570bd52) - GitHub * **deps**: Bump dyn-clone from 1.0.14 to 1.0.16 (vectordotdev#19040) [8ba28e0](answerbook/vector@8ba28e0) - GitHub * **deps**: Bump fakedata_generator from 0.2.4 to 0.4.0 (vectordotdev#18910) [7debc60](answerbook/vector@7debc60) - GitHub * **deps**: Bump flate2 from 1.0.27 to 1.0.28 (vectordotdev#18850) [b3889bc](answerbook/vector@b3889bc) - GitHub * **deps**: Bump futures-util from 0.3.28 to 0.3.29 (vectordotdev#18951) [741aec3](answerbook/vector@741aec3) - GitHub * **deps**: Bump goauth from 0.13.1 to 0.14.0 (vectordotdev#18872) [1913ee5](answerbook/vector@1913ee5) - GitHub * **deps**: Bump hashbrown from 0.14.0 to 0.14.1 (vectordotdev#18731) [31d92c2](answerbook/vector@31d92c2) - GitHub * **deps**: Bump hashbrown from 0.14.1 to 0.14.2 (vectordotdev#18893) [aebe8db](answerbook/vector@aebe8db) - GitHub * **deps**: Bump indexmap from 2.0.0 to 2.0.1 (vectordotdev#18705) [65643da](answerbook/vector@65643da) - GitHub * **deps**: Bump indexmap from 2.0.1 to 2.0.2 (vectordotdev#18737) [f47df40](answerbook/vector@f47df40) - GitHub * **deps**: Bump inventory from 0.3.12 to 0.3.13 (vectordotdev#19024) [72eacf5](answerbook/vector@72eacf5) - GitHub * **deps**: Bump lading to 0.19.1 (vectordotdev#18869) [2d7c1bb](answerbook/vector@2d7c1bb) - GitHub * **deps**: Bump libc from 0.2.148 to 0.2.149 (vectordotdev#18800) [4002ef0](answerbook/vector@4002ef0) - GitHub * **deps**: Bump libc from 0.2.149 to 0.2.150 (vectordotdev#19059) [611a652](answerbook/vector@611a652) - GitHub * **deps**: Bump lru from 0.11.1 to 0.12.0 (vectordotdev#18767) [c531a3b](answerbook/vector@c531a3b) - GitHub * **deps**: Bump memchr from 2.6.3 to 2.6.4 (vectordotdev#18736) [c7482d0](answerbook/vector@c7482d0) - GitHub * **deps**: Bump memmap2 from 0.7.1 to 0.8.0 (vectordotdev#18659) [ca9e5b4](answerbook/vector@ca9e5b4) - GitHub * **deps**: Bump memmap2 from 0.8.0 to 0.9.0 (vectordotdev#18765) [dcbbb9b](answerbook/vector@dcbbb9b) - GitHub * **deps**: Bump mongodb from 2.6.1 to 2.7.0 (vectordotdev#18703) [539a40f](answerbook/vector@539a40f) - GitHub * **deps**: Bump mongodb from 2.7.0 to 2.7.1 (vectordotdev#19023) [602f630](answerbook/vector@602f630) - GitHub * **deps**: Bump num_enum from 0.7.0 to 0.7.1 (vectordotdev#18975) [c9b6d45](answerbook/vector@c9b6d45) - GitHub * **deps**: Bump num-traits from 0.2.16 to 0.2.17 (vectordotdev#18802) [c4fbc25](answerbook/vector@c4fbc25) - GitHub * **deps**: Bump OpenDAL to v0.41 (vectordotdev#19039) [5655f76](answerbook/vector@5655f76) - GitHub * **deps**: Bump openssl from 0.10.57 to 0.10.58 (vectordotdev#19025) [c4f2d0e](answerbook/vector@c4f2d0e) - GitHub * **deps**: Bump openssl from 0.10.58 to 0.10.59 (vectordotdev#19054) [ee232f8](answerbook/vector@ee232f8) - GitHub * **deps**: Bump openssl-src from 300.1.5+3.1.3 to 300.1.6+3.1.4 (vectordotdev#18936) [a75a043](answerbook/vector@a75a043) - GitHub * **deps**: Bump ordered-float from 4.1.0 to 4.1.1 (vectordotdev#18818) [99643ca](answerbook/vector@99643ca) - GitHub * **deps**: Bump postcss from 8.4.6 to 8.4.31 in /website (vectordotdev#18750) [92d2be9](answerbook/vector@92d2be9) - GitHub * **deps**: Bump proc-macro2 from 1.0.67 to 1.0.69 (vectordotdev#18803) [bc3b3a2](answerbook/vector@bc3b3a2) - GitHub * **deps**: Bump proptest from 1.2.0 to 1.3.1 (vectordotdev#18738) [2b15c63](answerbook/vector@2b15c63) - GitHub * **deps**: Bump pulsar from 6.0.1 to 6.1.0 (vectordotdev#19004) [43f5913](answerbook/vector@43f5913) - GitHub * **deps**: Bump quanta from 0.11.1 to 0.12.0 (vectordotdev#18774) [e9d2dae](answerbook/vector@e9d2dae) - GitHub * **deps**: Bump quanta from 0.12.0 to 0.12.1 (vectordotdev#19005) [051de5a](answerbook/vector@051de5a) - GitHub * **deps**: Bump ratatui from 0.23.0 to 0.24.0 (vectordotdev#18908) [8b56a93](answerbook/vector@8b56a93) - GitHub * **deps**: Bump regex from 1.10.0 to 1.10.2 (vectordotdev#18858) [434849d](answerbook/vector@434849d) - GitHub * **deps**: Bump regex from 1.9.5 to 1.9.6 (vectordotdev#18739) [8e98321](answerbook/vector@8e98321) - GitHub * **deps**: Bump regex from 1.9.6 to 1.10.0 (vectordotdev#18812) [4d02abf](answerbook/vector@4d02abf) - GitHub * **deps**: Bump reqwest from 0.11.20 to 0.11.22 (vectordotdev#18760) [bac60ad](answerbook/vector@bac60ad) - GitHub * **deps**: Bump rustix from 0.37.19 to 0.37.25 (vectordotdev#18879) [3ca32b8](answerbook/vector@3ca32b8) - GitHub * **deps**: Bump semver from 1.0.18 to 1.0.19 (vectordotdev#18662) [aca7753](answerbook/vector@aca7753) - GitHub * **deps**: Bump semver from 1.0.19 to 1.0.20 (vectordotdev#18811) [d9aca80](answerbook/vector@d9aca80) - GitHub * **deps**: Bump serde from 1.0.188 to 1.0.189 (vectordotdev#18834) [0d09898](answerbook/vector@0d09898) - GitHub * **deps**: Bump serde from 1.0.189 to 1.0.190 (vectordotdev#18945) [73afddb](answerbook/vector@73afddb) - GitHub * **deps**: Bump serde_with from 3.3.0 to 3.4.0 (vectordotdev#18874) [33243ac](answerbook/vector@33243ac) - GitHub * **deps**: Bump serde_yaml from 0.9.25 to 0.9.27 (vectordotdev#18956) [2ee96b1](answerbook/vector@2ee96b1) - GitHub * **deps**: Bump serde-toml-merge from 0.3.2 to 0.3.3 (vectordotdev#18804) [abb9101](answerbook/vector@abb9101) - GitHub * **deps**: Bump serde-wasm-bindgen from 0.6.0 to 0.6.1 (vectordotdev#18935) [ffed6f7](answerbook/vector@ffed6f7) - GitHub * **deps**: Bump sha2 from 0.10.7 to 0.10.8 (vectordotdev#18684) [87af0bd](answerbook/vector@87af0bd) - GitHub * **deps**: Bump socket2 from 0.5.4 to 0.5.5 (vectordotdev#18902) [691fdca](answerbook/vector@691fdca) - GitHub * **deps**: Bump syn from 2.0.37 to 2.0.38 (vectordotdev#18789) [ec5238e](answerbook/vector@ec5238e) - GitHub * **deps**: Bump syn from 2.0.38 to 2.0.39 (vectordotdev#19056) [0f0a0b4](answerbook/vector@0f0a0b4) - GitHub * **deps**: Bump tempfile from 3.6.0 to 3.8.0 (vectordotdev#18686) [c0d24b9](answerbook/vector@c0d24b9) - GitHub * **deps**: Bump tempfile from 3.8.0 to 3.8.1 (vectordotdev#18957) [40961ed](answerbook/vector@40961ed) - GitHub * **deps**: Bump the azure group with 4 updates (vectordotdev#18773) [0e61acc](answerbook/vector@0e61acc) - GitHub * **deps**: Bump the azure group with 4 updates (vectordotdev#19052) [53ffbc3](answerbook/vector@53ffbc3) - GitHub * **deps**: Bump the clap group with 1 update (vectordotdev#18906) [cddb835](answerbook/vector@cddb835) - GitHub * **deps**: Bump the clap group with 2 updates (vectordotdev#18925) [78934c2](answerbook/vector@78934c2) - GitHub * **deps**: Bump the futures group with 1 update (vectordotdev#18961) [0a5e3db](answerbook/vector@0a5e3db) - GitHub * **deps**: Bump the tonic group with 2 updates (vectordotdev#18714) [c95df7c](answerbook/vector@c95df7c) - GitHub * **deps**: Bump the zstd group with 1 update (vectordotdev#18826) [96ef9ee](answerbook/vector@96ef9ee) - GitHub * **deps**: Bump thiserror from 1.0.48 to 1.0.49 (vectordotdev#18683) [6b92a83](answerbook/vector@6b92a83) - GitHub * **deps**: Bump thiserror from 1.0.49 to 1.0.50 (vectordotdev#18892) [16df7ea](answerbook/vector@16df7ea) - GitHub * **deps**: Bump tokio from 1.32.0 to 1.33.0 (vectordotdev#18809) [76971bd](answerbook/vector@76971bd) - GitHub * **deps**: Bump tokio-tungstenite from 0.20.0 to 0.20.1 (vectordotdev#18661) [8abed12](answerbook/vector@8abed12) - GitHub * **deps**: Bump toml from 0.8.0 to 0.8.1 (vectordotdev#18687) [efbe673](answerbook/vector@efbe673) - GitHub * **deps**: Bump toml from 0.8.1 to 0.8.2 (vectordotdev#18747) [f9e51e1](answerbook/vector@f9e51e1) - GitHub * **deps**: Bump toml from 0.8.2 to 0.8.3 (vectordotdev#18909) [22402ca](answerbook/vector@22402ca) - GitHub * **deps**: Bump toml from 0.8.3 to 0.8.4 (vectordotdev#18913) [249330a](answerbook/vector@249330a) - GitHub * **deps**: Bump toml from 0.8.4 to 0.8.5 (vectordotdev#18950) [4a525a8](answerbook/vector@4a525a8) - GitHub * **deps**: Bump toml from 0.8.5 to 0.8.6 (vectordotdev#18962) [5e7ae83](answerbook/vector@5e7ae83) - GitHub * **deps**: Bump tracing-log from 0.1.3 to 0.1.4 (vectordotdev#18914) [e4fd78c](answerbook/vector@e4fd78c) - GitHub [log-0](https://logdna.atlassian.net/browse/log-0) [log-0](https://logdna.atlassian.net/browse/log-0) * **deps**: Bump tracing-log from 0.1.4 to 0.2.0 (vectordotdev#18941) [30a1e26](answerbook/vector@30a1e26) - GitHub [log-0](https://logdna.atlassian.net/browse/log-0) [log-0](https://logdna.atlassian.net/browse/log-0) * **deps**: Bump trust-dns-proto from 0.23.0 to 0.23.1 (vectordotdev#18846) [0568d7a](answerbook/vector@0568d7a) - GitHub * **deps**: Bump trust-dns-proto from 0.23.1 to 0.23.2 (vectordotdev#18911) [1eaf8b1](answerbook/vector@1eaf8b1) - GitHub * **deps**: Bump uuid from 1.4.1 to 1.5.0 (vectordotdev#18880) [a025caa](answerbook/vector@a025caa) - GitHub * **deps**: Bump warp from 0.3.5 to 0.3.6 (vectordotdev#18704) [decaaeb](answerbook/vector@decaaeb) - GitHub * **deps**: Bump wasm-bindgen from 0.2.87 to 0.2.88 (vectordotdev#19026) [63bb9e4](answerbook/vector@63bb9e4) - GitHub * **deps**: Bump webpki from 0.22.1 to 0.22.2 (vectordotdev#18744) [4295985](answerbook/vector@4295985) - GitHub * **deps**: Bump wiremock from 0.5.19 to 0.5.21 (vectordotdev#19055) [4622ef6](answerbook/vector@4622ef6) - GitHub * **deps**: Bump zerocopy from 0.7.21 to 0.7.31 (vectordotdev#19394) [74d6cb1](answerbook/vector@74d6cb1) - Jesse Szwedko * **deps**: clean up VRL crate features (vectordotdev#18740) [1452d54](answerbook/vector@1452d54) - GitHub * **deps**: Group csv crate updates (vectordotdev#18797) [98aa157](answerbook/vector@98aa157) - GitHub * **deps**: Remove usages of atty (vectordotdev#18985) [371580c](answerbook/vector@371580c) - GitHub * **deps**: Update dependencies (vectordotdev#18971) [3ead10f](answerbook/vector@3ead10f) - GitHub * **deps**: Update lading to 0.19.0 (vectordotdev#18861) [dc729f5](answerbook/vector@dc729f5) - GitHub * **deps**: Update license-tool.toml webpki version (vectordotdev#18986) [0051ec0](answerbook/vector@0051ec0) - GitHub * **deps**: Update VRL to 0.8.1 (vectordotdev#19011) [f5ea285](answerbook/vector@f5ea285) - GitHub * **dev**: Add `vector-lib` wrapper for three more libs (vectordotdev#18992) [1eb418b](answerbook/vector@1eb418b) - GitHub * **dev**: Add wrapper for `codecs` to `vector-lib` (vectordotdev#18959) [5f30f74](answerbook/vector@5f30f74) - GitHub * **dev**: Add wrapper for `enrichment` to `vector-lib` (vectordotdev#18977) [e61f308](answerbook/vector@e61f308) - GitHub * **dev**: Add wrapper for `file-source` in `vector-lib` (vectordotdev#18984) [f44da16](answerbook/vector@f44da16) - GitHub * **dev**: Add wrapper for `lookup` in `vector-lib` (vectordotdev#18995) [164f1e9](answerbook/vector@164f1e9) - GitHub * **dev**: Add wrapper for `vector-buffers` to `vector-lib` (vectordotdev#18964) [2cef62c](answerbook/vector@2cef62c) - GitHub * **dev**: Detail the format of DEPRECATIONS.md file (vectordotdev#19016) [223dd7b](answerbook/vector@223dd7b) - GitHub * **dev**: Move some macros into `vector-core` (vectordotdev#19002) [4f613ce](answerbook/vector@4f613ce) - GitHub * **dev**: Remove deprecation action item for armv7 RPMs (vectordotdev#19018) [ff7b95f](answerbook/vector@ff7b95f) - GitHub * **dev**: Set up `vector-lib` wrapper crate with `vector-common` (vectordotdev#18927) [00c40d7](answerbook/vector@00c40d7) - GitHub * **dev**: Wrap `vector-config` in `vector-lib` as `configurable` (vectordotdev#18944) [42beb3f](answerbook/vector@42beb3f) - GitHub * **dev**: Wrap `vector-core` in `vector-lib` (vectordotdev#18934) [270fdfd](answerbook/vector@270fdfd) - GitHub * **dev**: Wrap `vector-stream` in `vector-lib` (vectordotdev#18953) [8a02b16](answerbook/vector@8a02b16) - GitHub * **docs**: Add alpha to traces and beta to metrics in descriptions (vectordotdev#19139) [1b9fb9b](answerbook/vector@1b9fb9b) - Jesse Szwedko * **docs**: add highlight post for secrets in disk buffers (vectordotdev#18994) [0cc9389](answerbook/vector@0cc9389) - GitHub * **docs**: Add spec for `listen` option (vectordotdev#18080) [29e5e22](answerbook/vector@29e5e22) - GitHub * **docs**: Replace setup.vector.dev references (vectordotdev#19080) [def235e](answerbook/vector@def235e) - Jesse Szwedko * **docs**: update a few more examples to YAML (vectordotdev#19103) [a59329a](answerbook/vector@a59329a) - Jesse Szwedko * **docs**: update quickstart.md to use YAML (vectordotdev#18796) [0e76fe0](answerbook/vector@0e76fe0) - GitHub * **external docs**: First batch of editorial edits for the Functions doc (vectordotdev#18780) [1da9005](answerbook/vector@1da9005) - GitHub * **external docs**: Remove or replace mentions of vector in functions doc (vectordotdev#18679) [7ad4112](answerbook/vector@7ad4112) - GitHub * Follow redirects for `sh.vector.dev` (vectordotdev#19000) [9893b86](answerbook/vector@9893b86) - GitHub * **gcp_stackdriver_metrics sink**: rewrite to stream based sink (vectordotdev#18749) [92268e4](answerbook/vector@92268e4) - GitHub * **kubernetes**: Regenerate manifests from 0.27.0 chart (vectordotdev#19001) [5e9dd1d](answerbook/vector@5e9dd1d) - GitHub * **metrics**: improve creation of Origin metadata structures (vectordotdev#18788) [f0adce7](answerbook/vector@f0adce7) - GitHub * Note the version to remove the v1 metrics support from the Datadog Metrics sink (vectordotdev#19017) [be9f229](answerbook/vector@be9f229) - GitHub * **observability, blackhole sink**: Don't report by default (vectordotdev#18963) [3b85b48](answerbook/vector@3b85b48) - GitHub * **observability**: deprecate obsolete http metrics (vectordotdev#18972) [f33dce2](answerbook/vector@f33dce2) - GitHub * **observability**: fix tokio unstable (vectordotdev#18776) [67c4beb](answerbook/vector@67c4beb) - GitHub * **observability**: remove `peer_addr` internal metric tag (vectordotdev#18982) [b9447f6](answerbook/vector@b9447f6) - GitHub * **observability**: remove deprecated `component_name` metric tag (vectordotdev#18942) [c6f5d2b](answerbook/vector@c6f5d2b) - GitHub * **observability**: remove metrics replaced by component_errors_total (vectordotdev#18965) [17f4ed2](answerbook/vector@17f4ed2) - GitHub * **prometheus_remote_write sink**: remote write sink rewrite (vectordotdev#18676) [5c1707f](answerbook/vector@5c1707f) - GitHub * **regression**: Unmark regression tests as erratic now (vectordotdev#19020) [2cdf654](answerbook/vector@2cdf654) - GitHub * **releasing, kubernetes**: Update manifests to v0.26.0 of the chart (vectordotdev#18694) [5f4c3ba](answerbook/vector@5f4c3ba) - GitHub * **releasing**: Add deprecation note about respositories.timber.io deprecation (vectordotdev#19078) [09df599](answerbook/vector@09df599) - Jesse Szwedko * **releasing**: Add known issue for 0.33.0 debian packaging regression (vectordotdev#18727) [ff745ab](answerbook/vector@ff745ab) - GitHub * **releasing**: Add known issue for Datadog Metrics sink in v0.34.0 (vectordotdev#19122) [cee9d07](answerbook/vector@cee9d07) - Jesse Szwedko * **releasing**: Add known issue for protobuf encoder in v0.34.0 (vectordotdev#19244) [f7c3824](answerbook/vector@f7c3824) - Jesse Szwedko * **releasing**: Add upgrade note about TOML breaking change to v0.34.0 (vectordotdev#19120) [dba0ba1](answerbook/vector@dba0ba1) - Jesse Szwedko * **releasing**: Bump Vector version to v0.34.0 (vectordotdev#18693) [1c2f970](answerbook/vector@1c2f970) - GitHub * **releasing**: Fix changelog not for kafka fix in 0.33.1 (vectordotdev#19032) [2501049](answerbook/vector@2501049) - GitHub * **releasing**: Fix formatting for v0.34.0 release note (vectordotdev#19085) [3569271](answerbook/vector@3569271) - Jesse Szwedko * **releasing**: Prepare v0.33.0 release [682f0e0](answerbook/vector@682f0e0) - Jesse Szwedko * **releasing**: Prepare v0.33.1 release [409b69d](answerbook/vector@409b69d) - Jesse Szwedko * **releasing**: Prepare v0.34.0 release [c909b66](answerbook/vector@c909b66) - Jesse Szwedko * **releasing**: Prepare v0.34.1 release [86f1c22](answerbook/vector@86f1c22) - Jesse Szwedko * **releasing**: Prepare v0.34.2 release [d685a16](answerbook/vector@d685a16) - Jesse Szwedko * **releasing**: Typo in v0.33.1 release docs (vectordotdev#18987) [36974a0](answerbook/vector@36974a0) - GitHub * Remove @spencergilbert from CODEOWNERS (vectordotdev#18778) [f300c85](answerbook/vector@f300c85) - GitHub * Remove armv7 RPM package (vectordotdev#18837) [eca8c76](answerbook/vector@eca8c76) - GitHub * remove config/vector.toml (vectordotdev#18833) [efb0d1a](answerbook/vector@efb0d1a) - GitHub * **security**: Ignore RUSTSEC-2023-0071 for now (vectordotdev#19263) [e27b7bd](answerbook/vector@e27b7bd) - Jesse Szwedko * **security**: Remove legacy OpenSSL provider flags (vectordotdev#19015) [2bba40a](answerbook/vector@2bba40a) - GitHub * **sinks**: Update `PartitionBatcher` to use `BatchConfig` (vectordotdev#18792) [4a7d0c3](answerbook/vector@4a7d0c3) - GitHub * Update *-release.md issue templates for vector.dev package release (vectordotdev#18814) [ab8f8d2](answerbook/vector@ab8f8d2) - GitHub * **vrl**: Revive old remap tests (vectordotdev#18678) [53cad38](answerbook/vector@53cad38) - GitHub * **website**: Set download page dropdown to latest version (vectordotdev#18758) [23745f2](answerbook/vector@23745f2) - GitHub * **websites**: Setup preview site workflows (vectordotdev#18924) [9d006c7](answerbook/vector@9d006c7) - GitHub * **websites**: Workflow updates (vectordotdev#19036) [b4ca866](answerbook/vector@b4ca866) - GitHub * **website**: WEB-4247 | Update references from s3 to setup.vector.dev (vectordotdev#19149) [9e1ad37](answerbook/vector@9e1ad37) - Jesse Szwedko * **website**: WEB-4275 | Update Navigation (vectordotdev#19186) [73a668c](answerbook/vector@73a668c) - Doug Smith * **website**: Workflow fixes (vectordotdev#19046) [fb63f8e](answerbook/vector@fb63f8e) - GitHub ### Continuous Integration * mark otlp_http_to_blackhole experiment erratic (vectordotdev#18786) [54b54a5](answerbook/vector@54b54a5) - GitHub ### Features * add PR comment trigger for the workload checks workflow (vectordotdev#18839) [11bc5d9](answerbook/vector@11bc5d9) - GitHub * **amqp**: added integration test for TLS (vectordotdev#18813) [3c4ae86](answerbook/vector@3c4ae86) - GitHub [LOG-16435](https://logdna.atlassian.net/browse/LOG-16435) * **ci**: Add Vector workload checks (vectordotdev#18569) [e2b7de0](answerbook/vector@e2b7de0) - GitHub * **codecs**: add support for protobuf encoding (vectordotdev#18598) [737f5c3](answerbook/vector@737f5c3) - GitHub * **docs**: add fallibility examples (vectordotdev#18931) [08b45a5](answerbook/vector@08b45a5) - GitHub * **journald source**: Add emit_cursor option (vectordotdev#18882) [74051dc](answerbook/vector@74051dc) - GitHub * **regression**: convert all regression cases configs to YAML (vectordotdev#18825) [1fb0f0d](answerbook/vector@1fb0f0d) - GitHub * **timestamp encoding**: add unixtime formats (vectordotdev#18817) [53039e7](answerbook/vector@53039e7) - GitHub ### Miscellaneous * Merge pull request vectordotdev#396 from answerbook/feature/LOG-18978 [2b046bf](answerbook/vector@2b046bf) - GitHub [LOG-18978](https://logdna.atlassian.net/browse/LOG-18978) * Merge tag 'v0.34.2' into feature/LOG-18978 [a88d5ed](answerbook/vector@a88d5ed) - Darin Spivey [LOG-18978](https://logdna.atlassian.net/browse/LOG-18978) [LOG-18978](https://logdna.atlassian.net/browse/LOG-18978) * Update RUM domain (vectordotdev#19367) [4acb9de](answerbook/vector@4acb9de) - Jesse Szwedko * chore(docs):Add Obs Pipelines to docs (vectordotdev#19201) [8f3f160](answerbook/vector@8f3f160) - Jesse Szwedko * updating the doc, 2 urls were 404 (vectordotdev#18949) [bf58b06](answerbook/vector@bf58b06) - GitHub * chore!(config, docs): delete deprecated vector.toml code (vectordotdev#18795) [c8557d0](answerbook/vector@c8557d0) - GitHub * [WEB-3464] Adds TrustArc cookie consent banner (vectordotdev#18741) [43428d3](answerbook/vector@43428d3) - GitHub * Add announcement for new repository URLs (vectordotdev#18798) [239cf94](answerbook/vector@239cf94) - GitHub * **examples**: Convert config/examples from TOML to YAML (vectordotdev#18832) [6ffb072](answerbook/vector@6ffb072) - GitHub * **external docs**: Fix metrics test example (vectordotdev#18725) [96def01](answerbook/vector@96def01) - GitHub * fix truncate arguments (vectordotdev#19068) [9a5cb51](answerbook/vector@9a5cb51) - Jesse Szwedko * **gcp_pubsub source**: Add required fields to documentation examples (vectordotdev#18998) [21f741d](answerbook/vector@21f741d) - GitHub * **nats source**: add subscriber_capacity option (vectordotdev#18899) [e7b563d](answerbook/vector@e7b563d) - GitHub * **sources, sinks**: add telemetry to http and grpc servers (vectordotdev#18887) [e779019](answerbook/vector@e779019) - GitHub * **tls**: add new dedicated page for TLS configuration (vectordotdev#18844) [625e4bd](answerbook/vector@625e4bd) - GitHub * **vrl**: add an example of parsing upstreaminfo with parse_nginx_log (vectordotdev#18815) [774094e](answerbook/vector@774094e) - GitHub ### **BREAKING CHANGES** * **security:** Remove legacy OpenSSL provider flags (vectordotdev#19015) * **observability:** remove metrics replaced by component_errors_total (vectordotdev#18965) * **observability:** remove `peer_addr` internal metric tag (vectordotdev#18982) * **observability, blackhole sink:** Don't report by default (vectordotdev#18963) * **observability:** remove deprecated `component_name` metric tag (vectordotdev#18942) * **datadog:** remove deprecated config options (vectordotdev#18940)
This updates
PartitionBatcher
to useBatchConfig
.I need this because the Prometheus RemoteWrite sink will need to use a reducer to aggregate the metrics. That sink uses
PartitionBatcher
.