Use go workspaces to simplify dependency versioning

[jpkrohling]

During the SIG meeting on August 10, we agreed that we'd do a proof-of-concept of Go Workspaces to simplify the management of versions (especially replaces statements) in the Go modules for a repository.

I'll work on the PoC, while @bryan-aguilar was requested to help with the tooling, given Bryan's experience with similar tools.

[jpkrohling]

I'll rebase the go-work PR today and figure out what else is needed (build, release, ...).

@bryan-aguilar, could you help me with this by assessing what needs to change regarding tooling?

@codeboten, any hints you can give me as part of a similar PR you have for the contrib?

[bryan-aguilar]

I created an issue against go-build-tools about adding workspace support to crosslink.

[codeboten]

@jpkrohling here's what i've done so far in my pr open-telemetry/opentelemetry-collector-contrib#10904:

• created a workspace: go work init
• added all the components: go work use -r .
• removed all replace statements
• synced workspace & modules: go work sync

One thing I've noticed is that the go.sum files for many modules now include lines for components within the repository which they did not use to:
https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/fb991d141f409ea16e8292decf5a612988c4e180/exporter/elasticsearchexporter/go.sum#L213-L214

It's unclear to me whether this is a problem or not. I tested making a change locally to a component used by another module, and building the module picked up the local change, which is what we want.

Some of the things I have left to check on my list:

• will dependabot continue to work?
• are there any issues w/ go mod tidy + go workspaces?

[bogdandrutu]

Other question:

1. How do we do a release? We need to depend on an unreleased version, and seems that because there is a need to add the sum entry, that is not possible. Can you check the "prepare" PR with workspaces?

[jpkrohling]

Thanks, @codeboten. That's almost verbatim what I did, including the manual test to validate that changes in one module are visible in another.

I'll check dependabot, and I have not found evidence that go mod tidy wouldn't work or would do unexpected things. When workspaces are active, the module resolution uses the workspace as the main context, so go mod commands will just use it when relevant.

I remember seeing that there's something special about releasing modules that are part of workspaces, but I'll read the docs again and summarize my understanding.

[codeboten]

I remember seeing that there's something special about releasing modules that are part of workspaces, but I'll read the docs again and summarize my understanding.

Based on what I'm reading here, it seems that in order for us to make use of go workspaces, we would have to release any modules that have dependencies before being able to update the dependencies themselves to that version. If so this seems like it will complicate the process quite a bit.

[jpkrohling]

I ran a few experiments, and here's what I have to share at this point:

• the page @codeboten linked is exactly what I had in mind. I decided to create a script to demonstrate what it would take for us to properly release following those instructions: https://github.com/jpkrohling/collector-workspaces/blob/main/scripts/release.sh . We certainly would need some automation for that, as the inter-dependencies can be quite intricate. Everything is doable except for pushing commits directly to the main branch. We would need a way to bypass our current restrictions without opening us up for trouble. Without the intermediary tags and commits, we wouldn't be able to run "go mod tidy" for instance, as we would be pointing to non-existing versions (on remote).
• Then, I realized that we should be having the exact same problem today, and checked which types of commits we have during the release. We actually only touch the go.mod files: https://github.com/open-telemetry/opentelemetry-collector/pull/6079/files . It just so happens that we have replaces in the go.mod files pointing to the local modules, which makes it all work. I tried that with he second script and it "worked", except that it yields changes to the go.sum right after the change. So, we would probably need one step after that: https://github.com/jpkrohling/collector-workspaces/blob/main/scripts/dirty.sh
• We then go back to the replaces statements: go.work does support that as well, except that it requires specific versions to be replaced. Using the same replacements we have currently on core returns a message like:

go: workspace module github.com/jpkrohling/collector-workspaces is replaced at all versions in the go.work file. To fix, remove the replacement from the go.work file or specify the version at which to replace the module.

I'm now at a point where I'm doubting we will be gaining any improvements by using go.work, given that our main goal was to indeed reduce the replace statements across modules.

I'm stopping the experiment here so that we can regroup and discuss what we want to do next.

[codeboten]

I'm now at a point where I'm doubting we will be gaining any improvements by using go.work, given that our main goal was to indeed reduce the replace statements across modules.

I'm at the same point in the contrib repo. Until we decide to automate the release of individual components (if we decide to do so), there seems to be very little to gain from workspaces if we end up having to use these replace statements anyways.

[jpkrohling]

I think this was worth the experiment, but it's clear by now that we aren't gaining much from using go workspaces.