[receiver/fluentforward] Fix performance degradation

[TylerHelmuth]

Description:
Fixes a performance degradation caused by #18860. This change ensures the long-lived context is not mutated for each event. See issues for details.

Link to tracking Issue:
Fixes #19798

Testing:
tested using load balancing tests

[runforesight]

Foresight Summary

    
Major Impacts

build-and-test-windows duration(6 seconds) has decreased 32 minutes 6 seconds compared to main branch avg(32 minutes 12 seconds).

View More Details

⭕  build-and-test-windows workflow has finished in 6 seconds (32 minutes 1 second less than main branch avg.) and finished at 5th Apr, 2023.

---

Job	Failed Steps	Tests
windows-unittest-matrix	-     🔗	N/A	See Details
windows-unittest	-     🔗	N/A	See Details

---

✅  check-links workflow has finished in 51 seconds and finished at 5th Apr, 2023.

---

Job	Failed Steps	Tests
changed files	-     🔗	N/A	See Details
check-links	-     🔗	N/A	See Details

---

✅  telemetrygen workflow has finished in 1 minute 8 seconds and finished at 5th Apr, 2023.

---

Job	Failed Steps	Tests
build-dev	-     🔗	N/A	See Details
publish-latest	-     🔗	N/A	See Details
publish-stable	-     🔗	N/A	See Details

---

✅  changelog workflow has finished in 2 minutes 8 seconds and finished at 5th Apr, 2023.

---

Job	Failed Steps	Tests
changelog	-     🔗	N/A	See Details

---

✅  prometheus-compliance-tests workflow has finished in 3 minutes 44 seconds (2 minutes 36 seconds less than main branch avg.) and finished at 5th Apr, 2023.

---

Job	Failed Steps	Tests
prometheus-compliance-tests	-     🔗	N/A	See Details

---

✅  load-tests workflow has finished in 6 minutes 34 seconds (3 minutes 47 seconds less than main branch avg.) and finished at 5th Apr, 2023.

---

Job	Failed Steps	Tests
setup-environment	-     🔗	N/A	See Details
loadtest (TestIdleMode)	-     🔗	N/A	See Details
loadtest (TestBallastMemory|TestLog10kDPS)	-     🔗	N/A	See Details
loadtest (TestMetric10kDPS|TestMetricsFromFile)	-     🔗	N/A	See Details
loadtest (TestMetricResourceProcessor|TestTrace10kSPS)	-     🔗	N/A	See Details
loadtest (TestTraceNoBackend10kSPS|TestTrace1kSPSWithAttrs)	-     🔗	N/A	See Details
loadtest (TestTraceBallast1kSPSWithAttrs|TestTraceBallast1kSPSAddAttrs)	-     🔗	N/A	See Details
loadtest (TestTraceAttributesProcessor)	-     🔗	N/A	See Details

---

✅  e2e-tests workflow has finished in 12 minutes 6 seconds (1 minute 59 seconds less than main branch avg.) and finished at 5th Apr, 2023.

---

Job	Failed Steps	Tests
kubernetes-test (v1.26.0)	-     🔗	N/A	See Details
kubernetes-test (v1.25.3)	-     🔗	N/A	See Details
kubernetes-test (v1.24.7)	-     🔗	N/A	See Details
kubernetes-test (v1.23.13)	-     🔗	N/A	See Details

---

✅  build-and-test workflow has finished in 33 minutes 11 seconds (13 minutes 15 seconds less than main branch avg.) and finished at 5th Apr, 2023.

---

Job	Failed Steps	Tests
setup-environment	-     🔗	N/A	See Details
check-collector-module-version	-     🔗	N/A	See Details
checks	-     🔗	N/A	See Details
build-examples	-     🔗	N/A	See Details
check-codeowners	-     🔗	N/A	See Details
correctness-metrics	-     🔗	N/A	See Details
correctness-traces	-     🔗	N/A	See Details
integration-tests	-     🔗	N/A	See Details
lint-matrix (receiver-0)	-     🔗	N/A	See Details
lint-matrix (receiver-1)	-     🔗	N/A	See Details
lint-matrix (processor)	-     🔗	N/A	See Details
lint-matrix (exporter)	-     🔗	N/A	See Details
lint-matrix (extension)	-     🔗	N/A	See Details
lint-matrix (connector)	-     🔗	N/A	See Details
lint-matrix (internal)	-     🔗	N/A	See Details
lint-matrix (other)	-     🔗	N/A	See Details
unittest-matrix (1.20, receiver-0)	-     🔗	N/A	See Details
unittest-matrix (1.20, receiver-1)	-     🔗	N/A	See Details
unittest-matrix (1.20, processor)	-     🔗	N/A	See Details
unittest-matrix (1.20, exporter)	-     🔗	N/A	See Details
unittest-matrix (1.20, extension)	-     🔗	N/A	See Details
unittest-matrix (1.20, connector)	-     🔗	N/A	See Details
unittest-matrix (1.20, internal)	-     🔗	N/A	See Details
unittest-matrix (1.20, other)	-     🔗	N/A	See Details
unittest-matrix (1.19, receiver-0)	-     🔗	N/A	See Details
unittest-matrix (1.19, receiver-1)	-     🔗	N/A	See Details
unittest-matrix (1.19, processor)	-     🔗	N/A	See Details
unittest-matrix (1.19, exporter)	-     🔗	N/A	See Details
unittest-matrix (1.19, extension)	-     🔗	N/A	See Details
unittest-matrix (1.19, connector)	-     🔗	N/A	See Details
unittest-matrix (1.19, internal)	-     🔗	N/A	See Details
unittest-matrix (1.19, other)	-     🔗	N/A	See Details
lint	-     🔗	N/A	See Details
unittest (1.20)	-     🔗	N/A	See Details
unittest (1.19)	-     🔗	N/A	See Details
cross-compile (darwin, amd64)	-     🔗	N/A	See Details
cross-compile (darwin, arm64)	-     🔗	N/A	See Details
cross-compile (linux, 386)	-     🔗	N/A	See Details
cross-compile (linux, amd64)	-     🔗	N/A	See Details
cross-compile (linux, arm)	-     🔗	N/A	See Details
cross-compile (linux, arm64)	-     🔗	N/A	See Details
cross-compile (linux, ppc64le)	-     🔗	N/A	See Details
cross-compile (windows, 386)	-     🔗	N/A	See Details
cross-compile (windows, amd64)	-     🔗	N/A	See Details
build-package (deb)	-     🔗	N/A	See Details
build-package (rpm)	-     🔗	N/A	See Details
windows-msi	-     🔗	N/A	See Details
publish-check	-     🔗	N/A	See Details
publish-dev	-     🔗	N/A	See Details
publish-stable	-     🔗	N/A	See Details
rotate-milestone	-     🔗	N/A	See Details

---

🔎 See details on Foresight

^{*You can configure Foresight comments in your organization settings page.}

[codeboten]

What's the benchmark before/after this change?

[dmitryax]

Thanks, @TylerHelmuth

[dmitryax]

The load test is back to normal:

Test                                    |Result|Duration|CPU Avg%|CPU Max%|RAM Avg MiB|RAM Max MiB|Sent Items|Received Items|
----------------------------------------|------|-------:|-------:|-------:|----------:|----------:|---------:|----
Log10kDPS/FluentForward-SplunkHEC       |PASS  |     15s|    19.1|    19.3|         55|         81|    147600|        147600|

[TylerHelmuth]

What's the benchmark before/after this change?

This is current main branch:

Test                                    |Result|Duration|CPU Avg%|CPU Max%|RAM Avg MiB|RAM Max MiB|Sent Items|Received Items|
----------------------------------------|------|-------:|-------:|-------:|----------:|----------:|---------:|-------------:|
Log10kDPS/FluentForward-SplunkHEC       |PASS  |     15s|    28.6|    31.7|         60|         88|    147500|        147500|