Collector docs on single-writer principle #4433
Conversation
michael2893
changed the title
michael2893
requested review from
a team and
atoulme
and removed request for
a team
|
@open-telemetry/collector-approvers ptal |
|
|
||
| There is a gateway deployment configured to handle all traffic for three other collectors in the same system. | ||
| If the collectors are not uniquely identified and the SDK fails to distinguish between them, they may | ||
| send identical data to the gateway collector from different points in time. In this scenario, |
There was a problem hiding this comment.
Can you give a more concrete example here? Having multiple instances of a collector behind a load-balancer is certainly common practice, and there's no inherent problem in having a SDK sending data via this load-balancer, causing different data points for the same workload to land at different collector instances.
There are a few situations that need to be accounted for when scaling, like using target-allocator for pull-based scraping (nothing to do with OTLP though), or tail-sampling (due to the statefulness characteristic of this component).
There was a problem hiding this comment.
Yeah - what I have here isn't really specific enough. I can certainly provide an example here
| There are patterns in the data that may provide some insight into whether this is happening or not. | ||
| For example, upon visual inspection, a series with unexplained gaps or jumps in the same series may be a clue that | ||
| multiple collectors are sending the same samples. Unexplained behavior in a time series could potentially | ||
| point to the backend scraping data from multiple sources. |
There was a problem hiding this comment.
Another common way to find this out is when the backend complains about "out of order samples" -- if a data point for the state of a counter at T2 was received, and later a data point for the state of the same counter at T1 was received, a backend might say that the late data point is discarded.
michael2893
changed the title
Single-writer principle
…lier in the doc that these considerations should be made
chalin
force-pushed
the
michael2893-update-collector-documentation
branch
from
18efeda to
f1a7d4b
Compare
chalin
changed the title
Single-writer principle
|
/fix:all |
|
You triggered fix:all action run at https://github.com/open-telemetry/opentelemetry.io/actions/runs/9427840190 |
Co-authored-by: opentelemetrybot <[email protected]>
Co-authored-by: Phillip Carter <[email protected]>
Co-authored-by: opentelemetrybot <[email protected]> Co-authored-by: Phillip Carter <[email protected]>
Co-authored-by: opentelemetrybot <[email protected]> Co-authored-by: Phillip Carter <[email protected]>
Co-authored-by: Emídio Neto <[email protected]> Co-authored-by: Luiz Aoqui <[email protected]>
…pen-telemetry#5134) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…etry#5136) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…7f) (open-telemetry#5140) Co-authored-by: Fabrizio Ferri-Benedetti <[email protected]>
…ation (open-telemetry#5130) Co-authored-by: opentelemetrybot <[email protected]> Co-authored-by: Fabrizio Ferri-Benedetti <[email protected]>
Co-authored-by: Juraci Paixão Kröhling <[email protected]> Co-authored-by: Fabrizio Ferri-Benedetti <[email protected]>
Co-authored-by: Fabrizio Ferri-Benedetti <[email protected]> Co-authored-by: opentelemetrybot <[email protected]>
Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
open-telemetry#5142) Co-authored-by: Tiffany Hrabusa <[email protected]>
) Signed-off-by: svrnm <[email protected]> Co-authored-by: Fabrizio Ferri-Benedetti <[email protected]> Co-authored-by: Arve Knudsen <[email protected]> Co-authored-by: Cijo Thomas <[email protected]> Co-authored-by: Goutham Veeramachaneni <[email protected]> Co-authored-by: Jacob Aronoff <[email protected]> Co-authored-by: Adriana Villela <[email protected]> Co-authored-by: Severin Neumann <[email protected]> Co-authored-by: Patrice Chalin <[email protected]> Co-authored-by: opentelemetrybot <[email protected]>
Co-authored-by: Tiffany Hrabusa <[email protected]> Co-authored-by: Alex Boten <[email protected]> Co-authored-by: opentelemetrybot <[email protected]> Co-authored-by: Phillip Carter <[email protected]>
…-telemetry#5156) Co-authored-by: Tiffany Hrabusa <[email protected]> Co-authored-by: opentelemetrybot <[email protected]>
…github.com/michael2893/opentelemetry.io into michael2893-update-collector-documentation
…github.com/michael2893/opentelemetry.io into michael2893-update-collector-documentation
…github.com/michael2893/opentelemetry.io into michael2893-update-collector-documentation
|
there was an issue with squashing the commits from the invalid email here, so I just moved the change to here: |
Summary
This change addresses the request for documentation on the Single-Writer principle. #4368
Description
deployment/gatewayOpen questions