Project Tracking: HTTP semantic convention stability

[trask]

Description

Get the HTTP semantic conventions across the finish line to stability, and afterwards update existing HTTP instrumentation across all OpenTelemetry repositories to conform with the stable conventions.

Project Board

https://github.com/orgs/open-telemetry/projects/41

Deliverables

• Mark the HTTP semantic conventions as stable.
• Update existing HTTP instrumentation across all OpenTelemetry repositories to conform with the stable conventions.

Staffing / Help Wanted

The goal is to follow @tedsuo's proposed Semantic Convention Process.

• Stage 1: Working Group Preparation
• Stage 2: Specification
• Stage 3: Implementation

Required staffing for Stage 1 and 2

• @trask project lead
• @denisivan0v domain expert
• @lmolkova domain expert
• @mateuszrzeszutek prototyping in Java
• @reyang technical committee sponsor
• @jsuereth technical committee sponsor
• Need: more domain experts
• Need: engineers willing to write prototypes in at least one more language other than Java. Language should be fairly different from Java (for example: Python).
• Need: maintainers or approvers from those languages committed to reviewing the prototypes.

Required staffing for Stage 3

• @trask project lead
• @mateuszrzeszutek updating Java instrumentation
• Need: engineers across all languages willing to update existing HTTP instrumentation across all OpenTelemetry repositories to conform with the stable conventions.
• Need: maintainers or approvers across all languages committed to reviewing the updates.

Meeting Times

Once a project is started, the working group should meet regularly for discussion. These meeting times should be posted on the OpenTelemetry public calendar.

Timeline

Stage 1 is happening now.

Stage 2 will begin as soon as there are two technical committee sponsors and we coordinate a weekly meeting time.

Since there are still several issues and unknowns around marking (any) semantic conventions stable, it's unclear if we will be able to fast track this effort by meeting multiple times per week as recommended in the Semantic Convention Process, and we will likely need to coordinate heavily cross-group with the Semantic Conventions Working Group.

At the same time, a ton of work has already gone into moving the HTTP semantic conventions close to stability already, and so we may still be able to target the suggested 3 months total duration for Stage 2.

Stage 3 will begin as soon as the HTTP semantic conventions are marked stable, and it will target the suggested 3 months for completion.

Labels

The tracking issue should be properly labeled to indicate what parts of the specification it is focused on.

Linked Issues and PRs

All PRs, Issues, and OTEPs related to the project should link back to the tracking issue, so that they can be easily found.

[mateuszrzeszutek]

• Need: engineers willing to write prototypes in at least two languages. Languages should be fairly different from each other (for example: Java and Python).

The Java instrumentation repo already contains an (almost) up-to-date implementation of the current HTTP spec, and I try to keep it up-to-date as much as my "free" time allows it. So, if you need volunteers for that, you can sign me up.

[reyang]

I'm willing to sponsor this from TC.

[MrAlias]

Is this for tracing only, or does it include metrics?

[trask]

Is this for tracing only, or does it include metrics?

the desire is for it to include stability of at least http.server.duration and http.client.duration on the metrics side

[reyang]

Discussed during the Jan. 23, 2023 meeting, created the project board https://github.com/orgs/open-telemetry/projects/41.

[trask]

if you are interested and able to participate in the HTTP semantic convention stability (sub-)group, please fill out the meeting time poll below.

we would like to meet 2-3 times a week for 30 min each for 6 weeks (starting next week) with at least one APAC- and one EU-friendly time each week.

https://doodle.com/meeting/organize/id/dw0ko0ge

[trask]

another doodle, if you can fill this one out also ❤️ https://doodle.com/meeting/organize/id/aA6Qv6zb

[alolita]

@trask Awesome initiative! I'm interested in supporting and participating in this workgroup.

[chameleon82]

From user experience I would like to use convention metrics (and tags) to build graphs like https://grafana.com/grafana/plugins/novatec-sdg-panel/
one metric i would suggest to discuss http.server.rate/http.client.rate with status attribute to cover request/error rates with relation between service.name and peer.service

[trask]

Thanks all for filling out the doodles 🧑‍🎨

Please book your calendars for the next 6 weeks (starting Jan 30):

• Mon 3-3:30
• Wed 3-3:30
• Fri 9:30-10

These meetings are now on the OTel calendar.

I will send an OTel blog post shortly announcing this effort in case we can pick up anyone else who may be interested and able to participate as well.

[Emily-Jiang]

Can we also add jvm metrics to the stability list?

[trask]

Can we also add jvm metrics to the stability list?

hey @Emily-Jiang, I opened #3311 for this

[dyladan]

@trask I wasn't really sure which issue to bring this feedback to so I'll add it here. If this is the wrong place let me know and I can move it.

Last week at the Kubecon OpenTelemetry Community Meeting @tedsuo asked for feedback from the community about breaking semantic conventions in response to the ECS merger. The general feedback we received at the meeting was that users were very apprehensive about any breaking changes to telemetry data. Users at the meeting were encouraged to provide their feedback to the spec, but as mentioned in the maintainers call yesterday they seem quite hesitant to do so. It is worth pointing out that the users who attend the community meeting are likely skewed towards early adopters who already have significant investment in processes, pipelines, and analytics on this data and are likely to be more impacted than the average user.

Given this feedback from the meeting, I think we should consider being much stricter about which breaking changes are accepted and set a bar for the decision significantly higher than just aligning with ECS. A simple example is the net -> network rename. As far as I can tell, the only benefits to this change are alignment with ECS and possibly very slightly more clarity by avoiding abbreviations. Since this change doesn't introduce any new capabilities or remove any limitations, I would propose it as an example for a change we should not make.

/cc @Aneurysm9

[dyladan]

Given the feedback received from @trask in the spec meeting tuesday, I feel it necessary to revise my opinion here. The reasoning given for each change is sound in my opinion and I feel that the semconv team has done a great job given the very hard task of deciding which breaking changes are worth making and which are not. I still think the feedback given in the project meeting is valuable and should be considered, but no longer feel that any of the proposed changes should be blocked.

Slide deck here: https://docs.google.com/presentation/d/1QUeV4v9tlCRB9X3G2wrbjAKmCLaFN5qDKAa9wS9jx0s/edit#slide=id.p

[trask]

Thanks for bringing the feedback above @dyladan!

It made it clear that we hadn't communicated the changes and their reasonings clearly enough to the broader community, which brought about the presentation we gave to the specification group yesterday (slides and recording available).

It also made us think even more deeply about the migration plan for users, which resulted in a third iteration of the proposed transition plan: #3443

[arminru]

The work is done and this project can be closed. The OTel HTTP semantic conventions were declared stable in November 2023.

https://opentelemetry.io/blog/2023/http-conventions-declared-stable/
https://github.com/open-telemetry/semantic-conventions/blob/main/docs/http/migration-guide.md

https://github.com/open-telemetry/semantic-conventions/releases/tag/v1.23.0
https://github.com/open-telemetry/semantic-conventions/releases/tag/v1.23.1

Thanks to everyone who contributed to it! 🙌

[arminru]

Reopening until we move it to or create a project tracker in the community repo, which will also track the implementation and adoption of the updated semantics (stage 3 above).

cc @trask

[trask]

discussing how to track implementation adoption now in open-telemetry/semantic-conventions#534