Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[receiver/solacereceiver] Added configurable flow control interval, baggage unmarshalling #16570

Merged
merged 13 commits into from
Dec 5, 2022

Conversation

mcardy
Copy link
Contributor

@mcardy mcardy commented Dec 1, 2022

Description:

  1. Updated payload unmarshalling to include baggage supporting Solace PubSub+ Event Broker version 10.2.1
  2. Added configuration knob for flow control retry interval and stats to help tune interval

Link to tracking Issue: N/A

Testing: Updated tests accordingly to add tests for new unmarshalling behaviour, new configuration options and new stats.

Documentation: Added new section for flow control configuration in README, bumped status to beta since configuration should now be stable.

@djaglowski FYI

mcardy and others added 7 commits October 26, 2022 14:37
* Added new configuration for backpressure

SOL-79626, SOL-79630

* Added handling for delayed retry in receiver

SOL-79627, SOL-79631

* Added new flow control metrics to aid in flow control tuning

SOL-79628, SOL-79629, SOL-79632

* Added flowcontrol info to readme

Also fixed some linting errors

SOL-80997
… for baggage (#1615)

SOL-79113, SOL-79114, SOL-79115
…rted topic

Only applies to messages starting with _telemetry

SOL-81980
@mcardy mcardy requested a review from a team December 1, 2022 16:09
@mcardy mcardy requested a review from djaglowski as a code owner December 1, 2022 16:09
@runforesight
Copy link

runforesight bot commented Dec 1, 2022

Foresight Summary

    
Major Impacts

build-and-test-windows duration(4 seconds) has decreased 32 minutes 48 seconds compared to main branch avg(32 minutes 52 seconds).
View More Details

⭕  build-and-test-windows workflow has finished in 4 seconds (32 minutes 48 seconds less than main branch avg.) and finished at 5th Dec, 2022.


Job Failed Steps Tests
windows-unittest -     🔗  N/A See Details

✅  tracegen workflow has finished in 20 minutes 13 seconds (⚠️ 17 minutes 52 seconds more than main branch avg.) and finished at 5th Dec, 2022.


Job Failed Steps Tests
publish-latest -     🔗  N/A See Details
publish-stable -     🔗  N/A See Details
build-dev -     🔗  N/A See Details

✅  check-links workflow has finished in 20 minutes 39 seconds (⚠️ 18 minutes 30 seconds more than main branch avg.) and finished at 5th Dec, 2022.


Job Failed Steps Tests
changed files -     🔗  N/A See Details
check-links -     🔗  N/A See Details

✅  changelog workflow has finished in 20 minutes 56 seconds (⚠️ 15 minutes 46 seconds more than main branch avg.) and finished at 5th Dec, 2022.


Job Failed Steps Tests
changelog -     🔗  N/A See Details

✅  prometheus-compliance-tests workflow has finished in 29 minutes 16 seconds (⚠️ 21 minutes 12 seconds more than main branch avg.) and finished at 5th Dec, 2022.


Job Failed Steps Tests
prometheus-compliance-tests -     🔗  ✅ 21  ❌ 0  ⏭ 0    🔗 See Details

✅  load-tests workflow has finished in 1 hour 1 minute 6 seconds (⚠️ 46 minutes 50 seconds more than main branch avg.) and finished at 5th Dec, 2022.


Job Failed Steps Tests
loadtest (TestIdleMode) -     🔗  ✅ 1  ❌ 0  ⏭ 0    🔗 See Details
loadtest (TestMetric10kDPS|TestMetricsFromFile) -     🔗  ✅ 6  ❌ 0  ⏭ 0    🔗 See Details
loadtest (TestTraceAttributesProcessor) -     🔗  ✅ 3  ❌ 0  ⏭ 0    🔗 See Details
loadtest (TestMetricResourceProcessor|TestTrace10kSPS) -     🔗  ✅ 12  ❌ 0  ⏭ 0    🔗 See Details
loadtest (TestBallastMemory|TestLog10kDPS) -     🔗  ✅ 19  ❌ 0  ⏭ 0    🔗 See Details
loadtest (TestTraceBallast1kSPSWithAttrs|TestTraceBallast1kSPSAddAttrs) -     🔗  ✅ 10  ❌ 0  ⏭ 0    🔗 See Details
loadtest (TestTraceNoBackend10kSPS|TestTrace1kSPSWithAttrs) -     🔗  ✅ 8  ❌ 0  ⏭ 0    🔗 See Details
setup-environment -     🔗  N/A See Details

 build-and-test workflow has finished in 1 hour 3 minutes 5 seconds (⚠️ 8 minutes 29 seconds more than main branch avg.) and finished at 5th Dec, 2022.


Job Failed Steps Tests
correctness-metrics N/A  ✅ 0  ❌ 0  ⏭ 0    🔗 See Details
correctness-traces N/A  ✅ 0  ❌ 0  ⏭ 0    🔗 See Details
unittest-matrix (1.18, internal) N/A  ✅ 592  ❌ 0  ⏭ 0    🔗 See Details
unittest-matrix (1.18, extension) N/A  ✅ 528  ❌ 0  ⏭ 0    🔗 See Details
unittest-matrix (1.18, processor) N/A  ✅ 1464  ❌ 0  ⏭ 0    🔗 See Details
unittest-matrix (1.19, internal) N/A  ✅ 592  ❌ 0  ⏭ 0    🔗 See Details
unittest-matrix (1.19, extension) N/A  ✅ 528  ❌ 0  ⏭ 0    🔗 See Details
unittest-matrix (1.18, receiver-0) N/A  ✅ 2528  ❌ 0  ⏭ 0    🔗 See Details
unittest-matrix (1.19, processor) N/A  ✅ 1464  ❌ 0  ⏭ 0    🔗 See Details
unittest-matrix (1.19, receiver-0) N/A  ✅ 2527  ❌ 0  ⏭ 0    🔗 See Details
unittest-matrix (1.19, exporter) N/A  ✅ 2412  ❌ 0  ⏭ 0    🔗 See Details
unittest-matrix (1.18, exporter) N/A  ✅ 2412  ❌ 0  ⏭ 0    🔗 See Details
integration-tests N/A  ✅ 59  ❌ 0  ⏭ 0    🔗 See Details
unittest-matrix (1.19, other) N/A  ✅ 4350  ❌ 0  ⏭ 0    🔗 See Details
unittest-matrix (1.18, other) N/A  ✅ 4345  ❌ 0  ⏭ 0    🔗 See Details
unittest-matrix (1.18, receiver-1) N/A  ✅ 1845  ❌ 0  ⏭ 0    🔗 See Details
unittest-matrix (1.19, receiver-1) N/A  ✅ 1845  ❌ 0  ⏭ 0    🔗 See Details

🔎 See details on Foresight

*You can configure Foresight comments in your organization settings page.

@github-actions github-actions bot requested a review from djaglowski December 1, 2022 17:09
@mcardy
Copy link
Contributor Author

mcardy commented Dec 2, 2022

@djaglowski seems both test runs failed with

make[1]: Leaving directory '/home/runner/work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/receiver/riakreceiver'
make[1]: *** [../../Makefile.Common:77: do-integration-tests-with-cover] Error 1
make: *** [Makefile:169: receiver/riakreceiver] Error 2
Error: Process completed with exit code 2.

I don't know how my changes would have affected this. My tests pass locally. I'm going to try and pull in the latest changes from main and see if that helps.

Copy link
Member

@TylerHelmuth TylerHelmuth left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is updating the payload unmarshalling and the new flow control config related? Could they be broken into separate PRs?

@mcardy
Copy link
Contributor Author

mcardy commented Dec 2, 2022

@TylerHelmuth we wanted to deliver both changes at the same time to align with the next release of our event broker and having a single artifact to validate is more straight forward for us. It would be somewhat of a pain to separate these out into two PRs at this point, but going forward we can strive to have work items in separate PRs if that is better for the review process.

@djaglowski djaglowski merged commit 27d5190 into open-telemetry:main Dec 5, 2022
shalper2 pushed a commit to shalper2/opentelemetry-collector-contrib that referenced this pull request Dec 6, 2022
…aggage unmarshalling (open-telemetry#16570)

* Added Flow Control (open-telemetry#1614)

* Added new configuration for backpressure

SOL-79626, SOL-79630

* Added handling for delayed retry in receiver

SOL-79627, SOL-79631

* Added new flow control metrics to aid in flow control tuning

SOL-79628, SOL-79629, SOL-79632

* Added flowcontrol info to readme

Also fixed some linting errors

SOL-80997

* Updated protobuf spec to contain baggage string, added handling+tests for baggage (open-telemetry#1615)

SOL-79113, SOL-79114, SOL-79115

* Require an upgrade when receiving a telemetry message with an unsupported topic

Only applies to messages starting with _telemetry

SOL-81980

* Change stability from Alpha to Beta as configuration is now stable

SOL-81470

* Added changelog entries

* Fixed linting failure in tests for unchecked error

* Updated protobuf spec with latest comments
@plantfansam plantfansam mentioned this pull request Jul 21, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants