Publish stable modules with otel release versions and not just release candidates

[braydonk]

Is your feature request related to a problem? Please describe.
When I want to update the go.opentelemetry.io/collector dependencies in my project, I'd like to just go get all the dependencies @latest. With featuregate and pdata being marked as stable, they are only being published as release candidates. This means @latest doesn't work properly; while all the other collector latest dependencies are v0.81.0 as of writing, featuregate for example latest resolves to v0.77.0. This ruins the process of just updating all collector dependencies, because featuregate requires lower versions of the other depedencies meaning they resolve in various incorrect ways.

Describe the solution you'd like
It would be great if latest could resolve to the same thing for these two stable modules as all the others, even if the tags aren't exactly the same thing.

Describe alternatives you've considered
The current strategy will be to exclude these from the automatic upgrade process and manually grab the latest tags for these modules. This is manual work in what in the past has always been a simple automatic upgrade. This also means I'll presumably need to do this with any future modules being marked stable.

Additional context
I am able to provide screenshots or pastebins similar to https://cloud-native.slack.com/archives/C01N6P7KR6W/p1689092712052069 if it will help.

[mx-psi]

It would be great if latest could resolve to the same thing for these two stable modules as all the others, even if the tags aren't exactly the same thing.

I think unfortunately this is how the go command works and there is not much we can do about it. Once we start releasing 1.x versions for pdata and featuregate this should go away but for now I don't see a way to solve this.

[braydonk]

I think unfortunately this is how the go command works and there is not much we can do about it.

Yeah I understand that, but imo the fact that this is how the go command works exposes a flaw in the release process. The fact that the "latest release" in go mod's eyes is not the same for the modules marked stable is pretty unintuitive in my opinion.

Once we start releasing 1.x versions for pdata and featuregate this should go away

Is there a timeline for this? And will this same release candidate interim happen for each module being marked stable? If not, I have a proposed solution that would mitigate the problem.

If these modules that have "release candidate" versions were also simultaneously published under a "release" tag that synced up with all the other modules, this wouldn't be a problem. The restriction of not keeping release versions updated for modules that are marked stable feels like an adverse side effect.

I don't know much about how releases are done, but I can try to learn such that I'd contribute it myself if the maintainers would be okay with this approach.

[mx-psi]

Yeah I understand that, but imo the fact that this is how the go command works exposes a flaw in the release process. The fact that the "latest release" in go mod's eyes is not the same for the modules marked stable is pretty unintuitive in my opinion.

IMO it exposes a flaw in the go command and Go's ecosystem general lack of good support for pre-releases 😄 but I understand that improving go get is much harder than changing our release process (although maybe a new latest-pre string is not entirely out of the question?).

Is there a timeline for this? And will this same release candidate interim happen for each module being marked stable? If not, I have a proposed solution that would mitigate the problem.

@dmitryax may be able to speak on pdata's stabilization, but in general I don't think we are able to provide a timeline for this at the moment (you are welcome to help us figure one out!).

If these modules that have "release candidate" versions were also simultaneously published under a "release" tag that synced up with all the other modules, this wouldn't be a problem. The restriction of not keeping release versions updated for modules that are marked stable feels like an adverse side effect.

Do you have examples of other repositories doing this? I am personally not entirely opposed to the idea but I haven't seen anyone else do this (e.g. opentelemetry-go's process is similar to ours)

[braydonk]

IMO it exposes a flaw in the go command and Go's ecosystem general lack of good support for pre-releases

Also fully agree! A similar latest-pre string would be quite useful, I wonder if there's a proposal for something like this already. 🤔

but in general I don't think we are able to provide a timeline for this at the moment (you are welcome to help us figure one out!).

That is understandable and fine for this case; mostly wanted to get a sense for whether it was worth pursuing a solution to the release weirdness or if this was just something that was incoming and I complained for nothing 😁
If I can help with pdata in any way I'm happy to connect outside of this issue on it.

Do you have examples of other repositories doing this?

I don't know if there are any other large multi-module projects putting themselves in the situation that this repo is currently in, so I've been unable to find any examples. I'm going to spin up my own proof-of-concept multi-module repo with the release shape that I'm proposing to make sure it would work the way I think it does.

[dmitryax]

#6794 is the only issue that needs to be resolved for pdata 1.0, and it's a pretty big one. We ran through a few iterations and tried several solutions. All of them have some cons. The most promising copy-on-write approach introduces a performance hit. We discussed with @bogdandrutu another option requiring generating protobuf files with pdatagen. I'll try this once after some other work in the collector (the new exporter helper). Regardless, we should move from RC this year. I agree it's been too long.

[mx-psi]

prometheus/prometheus/pull/12623 is related to this, either because of go get behavior or because of confusion regarding the release candidates.

[braydonk]

So I spun up https://github.com/braydonk/go-multimod-rc-releases to demonstrate what I mean by publishing modules under release and release candidate at the same time.

I made two modules, mod_a and mod_b. I published a primary release tag, v0.1.0 and made tags to version each module mod_a/v0.1.0 and mod_b/v0.1.0. I also made a "release candidate" tag for mod_a by making the tag mod_a/v1.0.0-rc1 (this is kind of arbitrary; just need to demonstrate this with any pre-release identifier).

To test this, you can make a new directory and go mod init in it.

Get mod_a at latest:

braydon@pop-os:~/Work/temp_go$ go get github.com/braydonk/go-multimod-rc-releases/mod_a@latest
go: downloading github.com/braydonk/go-multimod-rc-releases/mod_a v0.1.0
go: downloading github.com/braydonk/go-multimod-rc-releases v0.1.0
go: added github.com/braydonk/go-multimod-rc-releases/mod_a v0.1.0

Get mod_b at latest:

braydon@pop-os:~/Work/temp_go$ go get github.com/braydonk/go-multimod-rc-releases/mod_b@latest
go: downloading github.com/braydonk/go-multimod-rc-releases/mod_b v0.1.0
go: added github.com/braydonk/go-multimod-rc-releases/mod_b v0.1.0

The release candidate version of mod_a exists in harmony and doesn't affect our ability to pull down latest for mod_a. I can update to the release candidate if I want:

braydon@pop-os:~/Work/temp_go$ go get -u github.com/braydonk/go-multimod-rc-releases/[email protected]
go: downloading github.com/braydonk/go-multimod-rc-releases/mod_a v1.0.0-rc1
go: upgraded github.com/braydonk/go-multimod-rc-releases/mod_a v0.1.0 => v1.0.0-rc1

But mod_a@latest will still be what I want it to be:

braydon@pop-os:~/Work/temp_go$ go get -u github.com/braydonk/go-multimod-rc-releases/mod_a@latest
go: downgraded github.com/braydonk/go-multimod-rc-releases/mod_a v1.0.0-rc1 => v0.1.0

(this output looks a bit different, I had already downloaded v0.1.0 from the previous commands so it was in my package cache)

Hopefully this little proof-of-concept repo demonstrates what I'm proposing: if we were hypothetically publishing the next pdata package, we would have tags for pdata/v1.0.0-rcv00015 as we currently would, but we would also have pdata/v0.84.0 so that if we try to get pdata@latest, latest will point to v0.84.0 for that module just as it would for any other module in this repo.

[mx-psi]

On today's SIG we decided to release pdata 1.0 after we have merged #8494 and have done at least one release to make sure we get feedback

[braydonk]

Releasing pdata 1.0 is a good thing, but it is orthogonal to this issue itself. This is about the structure of releases, which will continue being a problem as any package moves to rcv. The structure of releases would need to change to what I suggested in the above comment to remediate the underlying problem.

[braydonk]

Discussed in the Oct 25 Collector SIG meeting. After pdata and featuregate go stable, we will be reconsidering the strategy around release candidates in general in a way that will hopefully avoid this issue with Go modules in the future.

I can either leave this issue open and idle until we reach that point, or I can open a new one now and close this one.