maintenance: postmortem for v0.23.1 release trouble

[hyangah]

Summary

vscode-go/v0.23.1 release includes dependency updates that made it incompatible with gopls earlier than v0.6.6. gopls/v0.6.6 was released less than a month ago and is still the latest version (as of 03/16/2021). Some users who didn’t update gopls observed the language server failed to start and language features stopped working.

Root cause

vscode-go was using the prerelease version of LSP 3.16 to use new features like call hierarchy and semantic tokens since v0.18.0 (release: 10/23/2020). LSP 3.16 was finalized and officially released on 12/14/2020. The final version contains changes incompatible with the prerelease version, more specifically, PrepareSupportDefaultBehavior field’s type was changed from boolean to enum. See microsoft/language-server-protocol/commit/ed36538.

Gopls does not depend on this particular field internally, but it still needs to decode the message if it is received. This decoding logic doesn’t tolerate type mismatches. Unfortunately, VSCode sets this field in the initialization message unconditionally and the extension has no control over this part of message exchange.

The gopls and vscode-go teams already discovered the risk of this incompatible change at the end of Jan 2021. We initially planned to update the LSP library dependency in vscode-go/v0.23.0 (release: 03/04/2021) to fix bugs in the prerelease version of LSP client (e.g. #1153). But due to this incompatibility issue, we decided to hold off until gopls/v0.6.6 with the updated LSP/3.16 spec is released.

gopls/v0.6.6 was released on Feb 23, 2021, and vscode-go/v0.23.1 with the updated LSP 3.16 client library was released on March 11, 2021, 16 days later. Some users were still using older versions of gopls.

We hoped users with old versions of gopls to be prompted to update it when the failure due to this incompatibility is detected, but this logic didn’t work as expected for some users because:

• Functioning module proxies are not accessible.
• vscode-go/v0.23.1 had gopls/v0.6.4 as the hard-coded latest version so the fallback mechanism for version check when module proxies are unreachable didn’t work. \

We didn’t choose to rollback because the path to fix is well-defined: update the gopls using Go extension’s update/install tool command, or rollback the extension to the previous version using VSCode’s “Install Another Version..” functionality. But maybe some users who are new to vscode or vscode-go may benefit from the rollback.

• Simply pulling out v0.23.1 from the marketplace and republishing v0.23.0 isn’t possible - the marketplace stores all the published versions and rejects existing versions. The tool (vsce) supports unpublishing the entire extension, but does not allow to delete only a specific version from the marketplace.
• The rollback should be done as a form of roll-forward - publishing v0.23.0 as v0.23.2. That may be confusing, so it’s better to avoid unless there is no easy way to workaround.

Impact

• ~50 users sent automated issue reports and 2 users filed issues manually. (We do not know the actual number of affected users).
• V0.23.2 was released (not perfect fix, but allows users to know how to fix).
• V0.23.3 was another followup release to enforce checking gopls version before triggering the automated issue report logic.

Lesson Learned

• Gopls tool update is slower than we expected. Auto-update can fail for many reasons.
• Many users do not differentiate between the extension and the gopls. (Often, vscode, go versions)
• Current extension’s tool version check logic is not sufficient to handle compatibility issues.

Things That Went Well

• The risk was already detected by presubmit tests during vscode-go/v0.23.0 dev cycle and the change was held off for a month. As part of vscode-go/v0.23.1 release preparation, we checked and verified major container-based editor services (Github codespaces and Google CloudShell IDE) updated to use gopls/v0.6.6.
• Automated issue report logic could successfully capture the relevant information (i.e. logs indicating LSP message decoding failure, gopls/extension versions), and many users opted to report the issues.
• Automated issue report logic prompts users to update the gopls if the version is older than the latest gopls version. 14 automated issues were filed after the emergency vscode-go/v0.23.2 release that updated the hard-coded latest gopls version. 31 issues were filed with vscode-go/v0.23.1.

Things That Went Poorly

• Even though we held off vscode-go’s LSP dependency update a bit, that was still too soon - gopls/v0.6.6 was released less than a month ago. On the other hand, we saw many users who reported the issue were still using gopls/v0.6.4. Given that gopls/v0.6.5 was released on Feb 3 (more than a month ago), we suspect the gopls update check wasn’t effective for some users.
• There is no minimum required tool version enforcement.
  • Before starting gopls, vscode-go checks if the local gopls is recent, but this still suggests updates by prompting users and may continue to start gopls even before the prompt shows up. This version check and suggestion logic runs only once. It is designed for gopls-by-default change so it’s likely the code path for this check was never exercised this time.
  • Regular gopls update check starts 10 minutes after the vscode-go is activated, and does not prevent users from using old versions of gopls.
  • Update is still voluntary - the extension will use the available gopls regardless of users’ response to the update suggestion prompt.
• Many individuals still use old versions of Gopls
  • Some users turned off the gopls update prompt feature.
  • v0.23.2 and earlier versions of vscode-go depend on module proxy’s /@v/list endpoint to discover the latest version of gopls. Some users do not have access to proxy.golang.org or a good module proxy.
  • Some users use private dev-containers where gopls is baked in, and disabled gopls version check.
  • There is a hard-coded latest version just in case users don’t have access to a good module proxy. But v0.23.1 was released without updating the hard-coded latest version.

Where We Got Lucky

• Some of us were available late night for emergency release and could respond to open issues. Thanks .

Timeline (EST)

02/23 gopls/v0.6.6 with the updated LSP 3.16 spec was released.
03/11 15:00 v0.23.1 with the updated LSP 3.16 client library was released.

03/11 18:26 First automated issue report #1296 received.
03/11 20:27 Issue #1300 was filed to provide users with instruction and communication.

03/11 21:23 Prepared the change to update the hard-coded gopls latest version to 0.6.6, to help users to self-diagnose and fix the issue before reporting the issue.
03/12 00:26 v0.23.2 released.

Action Items

• Get the latest gopls version info by using go list instead of depending on proxy’s /@v/list endpoint. That will help users who cannot access module proxies but have to depend on ‘direct’ mode. CL/295418. : release v0.24.0
• Enforce minimum required gopls version. Issue #1334 : partially addressed in v0.23.3
  • Document how to use old go versions that are not compatible with gopls that meets the minimum version requirements. Issue #388
• Investigate why some users with vscode-go/v0.23.2 still ended up filing the automated issue reports. For example, this user gopls: automated issue report (initialization) #1329 (comment) reported there was no prompt. : addressed in v0.23.3
  • One possible explanation is the users may be disabled the version check completely. CL/302832
• Investigate if gopls can tolerate this type of changes and be forward compatible as much as possible.
  • Another option: parsing of fields irrelevant to gopls’s functionality yet may be skipped. golang/go#45316
• Update the hard-coded latest version number automatically when gopls is released. (release: automate latest version string update for gopls, dlv-dap as part of release process #1403)
• Investigate how to monitor gopls version distribution. (Internally in discussion)
• Investigate if we can turn on auto-update by default. (tools: enable auto-update of gopls by default #1405)
• Enable auto-update option in CloudShell IDE and CodeSpace. (they release containers frequently, so it will be ok)
• Releases in AM to increase the chance of detecting/responding to issues before the end of normal work hours. (noted)
• Evaluate/document our rollback strategy