Move Collector architecture documentation and diagrams from Github to opentelemetry.io #4029

tiffany76 · 2024-02-20T19:35:21Z

This PR creates a new docs page (Architecture) in the Collector documentation on opentelemetry.io. The content is being moved from Github (docs/design.md) to the OTel website to supplement existing documentation.

The file has been copyedited and modified to fit website heading conventions.

The diagram image files have been replaced in favor of Mermaid diagrams.

(Note: Work to remove outdated content and merge what remains with existing opentelemetry.io pages will be handled in a future issue/PR.)

linux-foundation-easycla · 2024-02-20T19:35:26Z

The committers listed above are authorized under a signed CLA.

✅ login: tiffany76 / name: Tiffany Hrabusa (1b7117e, 87e96b2, 282fbf9, 4b8eff0, 0a016d4, 0a3184f, e09a8bf, 3dcfa04, a9b4f7c, 677af81, 1cc36cd, 64865d0, 2236830, f3abc95, 2740396, a84d8b6, 3a46360, ff7579f, 392906b, ef248c2, d9858b7, 8e12f26)
✅ login: theletterf / name: Fabrizio Ferri-Benedetti (6072452)
✅ login: cartermp / name: Phillip Carter (d0aeb5a)

content/en/docs/collector/architecture.md

svrnm · 2024-02-21T10:59:02Z

content/en/docs/collector/architecture.md

+For details of config file format see
+[this document](https://docs.google.com/document/d/1NeheFG7DmcUYo_h2vLtNRlia9x5wOJMlV4QKEK05FhQ/edit#).


why is this in a google doc and not in a markdown file in the collector repo? I guess this has historical reasons, maybe we can migrate it as well?

I'm not sure about the document's origin, but I'm happy to convert it to Markdown and move it to the Collector repo if that's the consensus.

Good question where this should live, this is up to the @open-telemetry/collector-approvers

I'd remove it altogether. It's so old that a few things are already wrong.

Removed in 2740396. Thanks!

content/en/docs/collector/architecture.md

svrnm

This is a really great start, we might need to do some passes to make this content up-to-date.

svrnm · 2024-02-21T11:03:08Z

@tiffany76 no need to keep this as a draft from my point of view! This will also trigger some review requests with @open-telemetry/collector-approvers who should definitely provide an early review.

Co-authored-by: Severin Neumann <[email protected]>

tiffany76 · 2024-02-21T22:25:40Z

content/en/docs/collector/architecture.md

+    traces: # a pipeline of “traces” type
+      receivers: [otlp]
+      processors: [memory_limiter, batch]
+      exporters: [jaeger]


I count 11 more mentions of jaeger in the document, including several in diagrams (two of which I'm still working on).

We do not need to remove all of them, only the instances where the jaeger exporter is used since it has been deprecated a while ago already, the receivers are still fine, but we still could consider using OTLP there. Jaeger can remain if named as a potential backend (which now supports OTLP)

I changed mentions of jaeger exporter to otlp in 8e12f26.

theletterf

Nice work, thank you! Just some suggestions and nits.

content/en/docs/collector/architecture.md

Co-authored-by: Fabrizio Ferri-Benedetti <[email protected]>

content/en/docs/collector/architecture.md

tiffany76 · 2024-02-28T20:03:08Z

/fix:format

github-actions · 2024-02-28T20:03:39Z

You triggered fix:format action run at https://github.com/open-telemetry/opentelemetry.io/actions/runs/8086355257

mx-psi · 2024-02-29T15:31:08Z

Is there a preview on how this will look on the opentelemetry.io website?

tiffany76 · 2024-02-29T15:45:10Z

@mx-psi Try this one.

jpkrohling

I didn't finish reviewing yet, but here are a few early comments.

jpkrohling · 2024-03-01T16:11:55Z

content/en/docs/collector/architecture.md

+For details of config file format see
+[this document](https://docs.google.com/document/d/1NeheFG7DmcUYo_h2vLtNRlia9x5wOJMlV4QKEK05FhQ/edit#).


I'd remove it altogether. It's so old that a few things are already wrong.

content/en/docs/collector/architecture.md

jpkrohling · 2024-03-01T16:14:01Z

content/en/docs/collector/architecture.md

+
+> The configuration uses composite key names in the form of `type[/name]` as
+> defined in
+> [this document](https://docs.google.com/document/d/1NeheFG7DmcUYo_h2vLtNRlia9x5wOJMlV4QKEK05FhQ/edit#)).


Again, I'd remove mentions to this file. While it has parts that clarify this particular trait, other parts of the doc are outdated and might confuse users.

I've removed mention of the file (2740396) but left the config explanation.

jpkrohling · 2024-03-01T16:14:54Z

content/en/docs/collector/architecture.md

+flowchart LR
+    R1("`#quot;opentelemetry-collector#quot; Receiver`") --> FO((fan-out))
+    FO -->|Pipeline 'traces'| P1["`#quot;memory_limiter#quot; Processor`"]
+    FO -->|Pipeline 'traces/2'| P2["`#quot;tags#quot; Processor`"]


What's this tags processor?

The 'tags' processor is mentioned in two diagrams from the original document: the receiver diagram and the exporter diagram.

I've recreated them as Mermaid diagrams, so we can make changes easily if the 'tags' processor is no longer valid or relevant.

I don't remember about that processor, and I would expect this diagram to be related to the config that was shared in the previous example.

Maybe we can say 'transform' instead of tags here?

The collector config coming right before this would need to be updated as well. I believe this diagram here is a representation of that config.

Updated configs and diagrams to replace tags with transform: ff7579f.

Co-authored-by: Juraci Paixão Kröhling <[email protected]>

jpkrohling · 2024-03-05T09:58:46Z

I'm missing "Connectors" here, and I think @djaglowski's talk at KubeCon Amsterdam was great in that area, where he not only mentioned how connectors work, but how they differ from the collector on seemingly identical features (same receiver on two pipelines vs. forward connector, I believe). It might be good to incorporate that, or at least link the presentation there.

https://www.youtube.com/watch?v=uPpZ23iu6kI

svrnm · 2024-03-05T12:05:32Z

I'm missing "Connectors" here, and I think @djaglowski's talk at KubeCon Amsterdam was great in that area, where he not only mentioned how connectors work, but how they differ from the collector on seemingly identical features (same receiver on two pipelines vs. forward connector, I believe). It might be good to incorporate that, or at least link the presentation there.

https://www.youtube.com/watch?v=uPpZ23iu6kI

I wonder if we should do this in a follow up PR?, so step 1 is to bring over that content from the collector repo to docs, step2 is removing that file from the collector repo, step3 is some additional scrubbing of this file (incl. adding some words about connectors)

mx-psi

Agree with Severin that we can move forward with this PR and leave connectors to a follow up PR.

mx-psi · 2024-03-05T13:00:53Z

content/en/docs/collector/architecture.md

+flowchart LR
+    R1("`#quot;opentelemetry-collector#quot; Receiver`") --> FO((fan-out))
+    FO -->|Pipeline 'traces'| P1["`#quot;memory_limiter#quot; Processor`"]
+    FO -->|Pipeline 'traces/2'| P2["`#quot;tags#quot; Processor`"]


Maybe we can say 'transform' instead of tags here?

svrnm

I am happy with this as is, we will need to do some follow up work on it, but I prefer merging this in and then incrementally improve.

tiffany76 added 3 commits February 16, 2024 15:04

Create new file with design.md content

a9b4f7c

Copyedit text copied from design.md

2236830

Add TODOs

4b8eff0

Add first of six Mermaid diagrams

87e96b2

tiffany76 commented Feb 20, 2024

View reviewed changes

content/en/docs/collector/architecture.md Show resolved Hide resolved

svrnm reviewed Feb 21, 2024

View reviewed changes

content/en/docs/collector/architecture.md Outdated Show resolved Hide resolved

svrnm reviewed Feb 21, 2024

View reviewed changes

content/en/docs/collector/architecture.md Outdated Show resolved Hide resolved

svrnm reviewed Feb 21, 2024

View reviewed changes

content/en/docs/collector/architecture.md Outdated Show resolved Hide resolved

svrnm reviewed Feb 21, 2024

View reviewed changes

content/en/docs/collector/architecture.md Outdated Show resolved Hide resolved

svrnm reviewed Feb 21, 2024

View reviewed changes

content/en/docs/collector/architecture.md Outdated Show resolved Hide resolved

svrnm reviewed Feb 21, 2024

View reviewed changes

content/en/docs/collector/architecture.md Outdated Show resolved Hide resolved

svrnm reviewed Feb 21, 2024

View reviewed changes

content/en/docs/collector/architecture.md Outdated Show resolved Hide resolved

svrnm reviewed Feb 21, 2024

View reviewed changes

Apply suggestions from initial review

e09a8bf

Co-authored-by: Severin Neumann <[email protected]>

tiffany76 marked this pull request as ready for review February 21, 2024 18:49

tiffany76 requested review from a team and bogdandrutu and removed request for a team February 21, 2024 18:49

Update and add Mermaid diagrams

0a3184f

tiffany76 commented Feb 21, 2024

View reviewed changes

theletterf reviewed Feb 23, 2024

View reviewed changes

theletterf and others added 4 commits February 23, 2024 18:06

Merge branch 'main' into collector_architecture

6072452

Apply suggestions from follow-up review

a84d8b6

Co-authored-by: Fabrizio Ferri-Benedetti <[email protected]>

Make small copyedits and formatting changes

1b7117e

Add Receivers mermaid diagram

3dcfa04

theletterf approved these changes Feb 27, 2024

View reviewed changes

tiffany76 commented Feb 27, 2024

View reviewed changes

content/en/docs/collector/architecture.md Show resolved Hide resolved

tiffany76 and others added 6 commits February 27, 2024 15:30

Neutralize formatting of Agent diagram

677af81

Clean up and add style to Agent mermaid diagram

282fbf9

Add Service mermaid diagram

1cc36cd

Change mentions of jaeger exporter to otlp

8e12f26

Make Prettier fixes

d9858b7

Merge branch 'main' into collector_architecture

64865d0

jpkrohling reviewed Mar 1, 2024

View reviewed changes

tiffany76 and others added 3 commits March 4, 2024 09:32

Update content/en/docs/collector/architecture.md

f3abc95

Co-authored-by: Juraci Paixão Kröhling <[email protected]>

Merge branch 'main' into collector_architecture

392906b

Remove mention of google doc

2740396

mx-psi reviewed Mar 5, 2024

View reviewed changes

tiffany76 and others added 3 commits March 6, 2024 15:13

Merge branch 'main' into collector_architecture

3a46360

Replace mentions of tags processor

ff7579f

Fix Exporter diagram

ef248c2

tiffany76 requested review from jpkrohling, svrnm and mx-psi March 7, 2024 01:46

svrnm approved these changes Mar 7, 2024

View reviewed changes

mx-psi approved these changes Mar 7, 2024

View reviewed changes

Merge branch 'main' into collector_architecture

d0aeb5a

cartermp merged commit 7996562 into open-telemetry:main Mar 7, 2024
14 checks passed

tiffany76 deleted the collector_architecture branch March 7, 2024 18:40

tiffany76 mentioned this pull request Mar 7, 2024

Fix: Remove the docs in design.md folder open-telemetry/opentelemetry-collector#9021

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Move Collector architecture documentation and diagrams from Github to opentelemetry.io #4029

Move Collector architecture documentation and diagrams from Github to opentelemetry.io #4029

tiffany76 commented Feb 20, 2024

linux-foundation-easycla bot commented Feb 20, 2024 •

edited

Loading

svrnm Feb 21, 2024

tiffany76 Feb 21, 2024

svrnm Feb 26, 2024

jpkrohling Mar 1, 2024

tiffany76 Mar 4, 2024

svrnm left a comment

svrnm commented Feb 21, 2024

tiffany76 Feb 21, 2024

svrnm Feb 22, 2024

tiffany76 Feb 28, 2024

theletterf left a comment

tiffany76 commented Feb 28, 2024

github-actions bot commented Feb 28, 2024

mx-psi commented Feb 29, 2024

tiffany76 commented Feb 29, 2024 •

edited

Loading

jpkrohling left a comment

jpkrohling Mar 1, 2024

jpkrohling Mar 1, 2024

tiffany76 Mar 4, 2024

jpkrohling Mar 1, 2024

tiffany76 Mar 4, 2024

jpkrohling Mar 5, 2024

mx-psi Mar 5, 2024

jpkrohling Mar 5, 2024

tiffany76 Mar 7, 2024

jpkrohling commented Mar 5, 2024

svrnm commented Mar 5, 2024

mx-psi left a comment •

edited

Loading

mx-psi Mar 5, 2024

svrnm left a comment

		For details of config file format see
		[this document](https://docs.google.com/document/d/1NeheFG7DmcUYo_h2vLtNRlia9x5wOJMlV4QKEK05FhQ/edit#).

Move Collector architecture documentation and diagrams from Github to opentelemetry.io #4029

Move Collector architecture documentation and diagrams from Github to opentelemetry.io #4029

Conversation

tiffany76 commented Feb 20, 2024

linux-foundation-easycla bot commented Feb 20, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

svrnm left a comment

Choose a reason for hiding this comment

svrnm commented Feb 21, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

theletterf left a comment

Choose a reason for hiding this comment

tiffany76 commented Feb 28, 2024

github-actions bot commented Feb 28, 2024

mx-psi commented Feb 29, 2024

tiffany76 commented Feb 29, 2024 • edited Loading

jpkrohling left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jpkrohling commented Mar 5, 2024

svrnm commented Mar 5, 2024

mx-psi left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

svrnm left a comment

Choose a reason for hiding this comment

linux-foundation-easycla bot commented Feb 20, 2024 •

edited

Loading

tiffany76 commented Feb 29, 2024 •

edited

Loading

mx-psi left a comment •

edited

Loading