Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow minor releases before the block height #320

Merged
merged 12 commits into from
Oct 4, 2023
86 changes: 60 additions & 26 deletions x/upgrade/abci.go
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@
func BeginBlocker(k keeper.Keeper, ctx sdk.Context, _ abci.RequestBeginBlock) {
defer telemetry.ModuleMeasureSince(types.ModuleName, time.Now(), telemetry.MetricKeyBeginBlocker)

plan, found := k.GetUpgradePlan(ctx)
plan, planFound := k.GetUpgradePlan(ctx)

if !k.DowngradeVerified() {
k.SetDowngradeVerified(true)
Expand All @@ -33,14 +33,14 @@
// 1. If there is no scheduled upgrade.
// 2. If the plan is not ready.
// 3. If the plan is ready and skip upgrade height is set for current height.
if !found || !plan.ShouldExecute(ctx) || (plan.ShouldExecute(ctx) && k.IsSkipHeight(ctx.BlockHeight())) {
if !planFound || !plan.ShouldExecute(ctx) || (plan.ShouldExecute(ctx) && k.IsSkipHeight(ctx.BlockHeight())) {
if lastAppliedPlan != "" && !k.HasHandler(lastAppliedPlan) {
panic(fmt.Sprintf("Wrong app version %d, upgrade handler is missing for %s upgrade plan", ctx.ConsensusParams().Version.AppVersion, lastAppliedPlan))
}
}
}

if !found {
if !planFound {
return
}

Expand All @@ -53,49 +53,83 @@
},
)

// To make sure clear upgrade is executed at the same block
// If the plan's block height has passed, then it must be the executed version
// All major and minor releases are REQUIRED to execute on the scheduled block height
if plan.ShouldExecute(ctx) {
// If skip upgrade has been set for current height, we clear the upgrade plan
if k.IsSkipHeight(ctx.BlockHeight()) {
skipUpgradeMsg := fmt.Sprintf("UPGRADE \"%s\" SKIPPED at %d: %s", plan.Name, plan.Height, plan.Info)
ctx.Logger().Info(skipUpgradeMsg)

// Clear the upgrade plan at current height
k.ClearUpgradePlan(ctx)
skipUpgrade(k, ctx, plan)
return
Comment on lines +61 to 62
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: combine into
return skipUpgrade(k, ctx, plan)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It'd be cool, but this isn't yet legal golang :)

}
// If we don't have an upgrade handler for this upgrade name, then we need to shutdown
if !k.HasHandler(plan.Name) {
panicUpgradeNeeded(k, ctx, plan)
}
applyUpgrade(k, ctx, plan)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

based on the other discussion we were doing, if the upgrade plan gets cleared after applying upgrade, wouldnt this cause issue for generating apphash since it does affect the stored state?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My initial thought was that the upgrade action was stateless since registered plans stay registered, but if we specifically clear stuff out upon running upgrades, this may cause issue.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it - I updated with the new logic (see next commit)

return
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If a node is behind, and it sees the current scheduled plan being a minor version (e.g. 1.3.0->1.3.1) but there is a "future" (future only to that node; it is already the current scheduled plan for up-to-date nodes) major version (e.g. 1.3.1->2.0.0), and it uses the 2.0.0 binary, then it would only panic once it passes the 1.3.1 upgrade height, which might result in bad data being written because of changes in 2.0.0

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We discussed in thread, but for other's purposes, we determined that this scenario and logic exists today and are electing to focus on the minimum change to allow the minor releases at the moment.

}

// If running a pending minor release, apply the upgrade if handler is present
// Minor releases are allowed to run before the scheduled upgrade height, but not required to.
if plan.UpgradeDetails().IsMinorRelease() {
// if the pending upgrade is skipped, don't apply it
if k.IsSkipHeight(plan.Height) {
skipUpgrade(k, ctx, plan)
stevenlanders marked this conversation as resolved.
Show resolved Hide resolved
return
}
// if not yet present, then emit a scheduled log (every 100 blocks, to reduce logs)
if !k.HasHandler(plan.Name) {
// Write the upgrade info to disk. The UpgradeStoreLoader uses this info to perform or skip
// store migrations.
err := k.DumpUpgradeInfoWithInfoToDisk(ctx.BlockHeight(), plan.Name, plan.Info)
if err != nil {
panic(fmt.Errorf("unable to write upgrade info to filesystem: %s", err.Error()))
if ctx.BlockHeight()%100 == 0 {
ctx.Logger().Info(BuildUpgradeScheduledMsg(plan))
}

upgradeMsg := BuildUpgradeNeededMsg(plan)
// We don't have an upgrade handler for this upgrade name, meaning this software is out of date so shutdown
ctx.Logger().Error(upgradeMsg)

panic(upgradeMsg)
return
}
// We have an upgrade handler for this upgrade name, so apply the upgrade
ctx.Logger().Info(fmt.Sprintf("applying upgrade \"%s\" at %s", plan.Name, plan.DueAt()))
ctx = ctx.WithBlockGasMeter(sdk.NewInfiniteGasMeter())
k.ApplyUpgrade(ctx, plan)
applyUpgrade(k, ctx, plan)
return
}

// if we have a pending upgrade, but it is not yet time, make sure we did not
// set the handler already
// if we have a handler for a non-minor upgrade, that means it updated too early and must stop
if k.HasHandler(plan.Name) {
downgradeMsg := fmt.Sprintf("BINARY UPDATED BEFORE TRIGGER! UPGRADE \"%s\" - in binary but not executed on chain", plan.Name)
ctx.Logger().Error(downgradeMsg)
panic(downgradeMsg)
}
}

// panicUpgradeNeeded shuts down the node and prints a message that the upgrade needs to be applied.
func panicUpgradeNeeded(k keeper.Keeper, ctx sdk.Context, plan types.Plan) {
// Write the upgrade info to disk. The UpgradeStoreLoader uses this info to perform or skip
// store migrations.
err := k.DumpUpgradeInfoWithInfoToDisk(ctx.BlockHeight(), plan.Name, plan.Info)
if err != nil {
panic(fmt.Errorf("unable to write upgrade info to filesystem: %s", err.Error()))

Check warning on line 105 in x/upgrade/abci.go

View check run for this annotation

Codecov / codecov/patch

x/upgrade/abci.go#L105

Added line #L105 was not covered by tests
}

upgradeMsg := BuildUpgradeNeededMsg(plan)
ctx.Logger().Error(upgradeMsg)

panic(upgradeMsg)
}

func applyUpgrade(k keeper.Keeper, ctx sdk.Context, plan types.Plan) {
ctx.Logger().Info(fmt.Sprintf("applying upgrade \"%s\" at %s", plan.Name, plan.DueAt()))
ctx = ctx.WithBlockGasMeter(sdk.NewInfiniteGasMeter())
k.ApplyUpgrade(ctx, plan)
}

// skipUpgrade logs a message that the upgrade has been skipped and clears the upgrade plan.
func skipUpgrade(k keeper.Keeper, ctx sdk.Context, plan types.Plan) {
skipUpgradeMsg := fmt.Sprintf("UPGRADE \"%s\" SKIPPED at %d: %s", plan.Name, plan.Height, plan.Info)
ctx.Logger().Info(skipUpgradeMsg)
k.ClearUpgradePlan(ctx)
}

// BuildUpgradeNeededMsg prints the message that notifies that an upgrade is needed.
func BuildUpgradeNeededMsg(plan types.Plan) string {
return fmt.Sprintf("UPGRADE \"%s\" NEEDED at %s: %s", plan.Name, plan.DueAt(), plan.Info)
}

// BuildUpgradeScheduledMsg prints upgrade scheduled message
func BuildUpgradeScheduledMsg(plan types.Plan) string {
return fmt.Sprintf("UPGRADE \"%s\" SCHEDULED at %s: %s", plan.Name, plan.DueAt(), plan.Info)
}
106 changes: 106 additions & 0 deletions x/upgrade/abci_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,8 @@ type TestSuite struct {
ctx sdk.Context
}

const minorUpgradeInfo = `{"upgradeType":"minor"}`

var s TestSuite

func setupTest(height int64, skip map[int64]bool) TestSuite {
Expand Down Expand Up @@ -254,6 +256,37 @@ func TestContains(t *testing.T) {
require.False(t, s.keeper.IsSkipHeight(4))
}

func TestSkipUpgradeSkipMinor(t *testing.T) {
var (
planHeight int64 = 15
height int64 = 10
name = "minor"
)
s := setupTest(height, map[int64]bool{planHeight: true})

newCtx := s.ctx

req := abci.RequestBeginBlock{Header: newCtx.BlockHeader()}
err := s.handler(s.ctx, &types.SoftwareUpgradeProposal{Title: "prop", Plan: types.Plan{
Name: name,
Height: planHeight,
Info: minorUpgradeInfo,
}})
require.NoError(t, err)

t.Log("Verify if skip upgrade flag clears minor upgrade plan")
VerifySet(t, map[int64]bool{planHeight: true})

newCtx = newCtx.WithBlockHeight(height)
require.NotPanics(t, func() {
s.module.BeginBlock(newCtx, req)
})

t.Log("Verify minor proposal is cleared")
VerifyCleared(t, s.ctx)
VerifyNotDone(t, s.ctx, name)
}

func TestSkipUpgradeSkippingAll(t *testing.T) {
var (
skipOne int64 = 11
Expand Down Expand Up @@ -462,6 +495,79 @@ func TestBinaryVersion(t *testing.T) {
},
true,
},
{
"test not panic: minor version upgrade in future and not applied",
func() (sdk.Context, abci.RequestBeginBlock) {
err := s.handler(s.ctx, &types.SoftwareUpgradeProposal{
Title: "Upgrade test",
Plan: types.Plan{Name: "test2", Height: 105, Info: minorUpgradeInfo},
})
require.NoError(t, err)

newCtx := s.ctx.WithBlockHeight(100)
req := abci.RequestBeginBlock{Header: newCtx.BlockHeader()}
return newCtx, req
},
false,
},
{
"test panic: minor version upgrade is due",
func() (sdk.Context, abci.RequestBeginBlock) {
err := s.handler(s.ctx, &types.SoftwareUpgradeProposal{
Title: "Upgrade test",
Plan: types.Plan{Name: "test2", Height: 13, Info: minorUpgradeInfo},
})
require.NoError(t, err)

newCtx := s.ctx.WithBlockHeight(13)
req := abci.RequestBeginBlock{Header: newCtx.BlockHeader()}
return newCtx, req
},
true,
},
{
"test not panic: minor upgrade is in future and already applied",
func() (sdk.Context, abci.RequestBeginBlock) {
s.keeper.SetUpgradeHandler("test3", func(_ sdk.Context, _ types.Plan, vm module.VersionMap) (module.VersionMap, error) {
return vm, nil
})

err := s.handler(s.ctx, &types.SoftwareUpgradeProposal{
Title: "Upgrade test",
Plan: types.Plan{Name: "test3", Height: s.ctx.BlockHeight() + 10, Info: minorUpgradeInfo},
})
require.NoError(t, err)

newCtx := s.ctx.WithBlockHeight(12)
s.keeper.ApplyUpgrade(newCtx, types.Plan{
Name: "test3",
Height: 12,
})

req := abci.RequestBeginBlock{Header: newCtx.BlockHeader()}
return newCtx, req
},
false,
},
{
"test not panic: minor upgrade should apply",
func() (sdk.Context, abci.RequestBeginBlock) {
s.keeper.SetUpgradeHandler("test4", func(_ sdk.Context, _ types.Plan, vm module.VersionMap) (module.VersionMap, error) {
return vm, nil
})

err := s.handler(s.ctx, &types.SoftwareUpgradeProposal{
Title: "Upgrade test",
Plan: types.Plan{Name: "test4", Height: s.ctx.BlockHeight() + 10, Info: minorUpgradeInfo},
})
require.NoError(t, err)

newCtx := s.ctx.WithBlockHeight(12)
req := abci.RequestBeginBlock{Header: newCtx.BlockHeader()}
return newCtx, req
},
false,
},
}

for _, tc := range testCases {
Expand Down
4 changes: 2 additions & 2 deletions x/upgrade/keeper/keeper.go
Original file line number Diff line number Diff line change
Expand Up @@ -385,7 +385,7 @@ func (k Keeper) DumpUpgradeInfoToDisk(height int64, name string) error {

// Deprecated: DumpUpgradeInfoWithInfoToDisk writes upgrade information to UpgradeInfoFileName.
// `info` should be provided and contain Plan.Info data in order to support
// auto download functionality by cosmovisor and other tools using upgarde-info.json
// auto download functionality by cosmovisor and other tools using upgrade-info.json
// (GetUpgradeInfoPath()) file.
func (k Keeper) DumpUpgradeInfoWithInfoToDisk(height int64, name string, info string) error {
upgradeInfoFilePath, err := k.GetUpgradeInfoPath()
Expand Down Expand Up @@ -457,7 +457,7 @@ type upgradeInfo struct {
Name string `json:"name,omitempty"`
// Height has types.Plan.Height value
Height int64 `json:"height,omitempty"`
// Height has types.Plan.Height value
// Info has types.Plan.Info value
Info string `json:"info,omitempty"`
}

Expand Down
24 changes: 24 additions & 0 deletions x/upgrade/types/plan.go
Original file line number Diff line number Diff line change
@@ -1,7 +1,9 @@
package types

import (
"encoding/json"
"fmt"
"strings"

sdk "github.com/cosmos/cosmos-sdk/types"
sdkerrors "github.com/cosmos/cosmos-sdk/types/errors"
Expand Down Expand Up @@ -45,3 +47,25 @@ func (p Plan) ShouldExecute(ctx sdk.Context) bool {
func (p Plan) DueAt() string {
return fmt.Sprintf("height: %d", p.Height)
}

// UpgradeDetails is a struct that represents the details of an upgrade
// This is held in the Info object of an upgrade Plan
type UpgradeDetails struct {
stevenlanders marked this conversation as resolved.
Show resolved Hide resolved
UpgradeType string `json:"upgradeType"`
}

// UpgradeDetails parses and returns a details struct from the Info field of a Plan
// The upgrade.pb.go is generated from proto, so this is separated here
func (p Plan) UpgradeDetails() UpgradeDetails {
var details UpgradeDetails
if err := json.Unmarshal([]byte(p.Info), &details); err != nil {
// invalid json, assume no upgrade details
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add an error log here for failed to unmarshal upgrade info so that we know something bad is happening??

return UpgradeDetails{}
}
return details
}

// IsMinorRelease returns true if the upgrade is a minor release
func (ud UpgradeDetails) IsMinorRelease() bool {
return strings.EqualFold(ud.UpgradeType, "minor")
}
81 changes: 81 additions & 0 deletions x/upgrade/types/plan_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -154,3 +154,84 @@ func TestShouldExecute(t *testing.T) {
})
}
}

func TestUpgradeDetails(t *testing.T) {
tests := []struct {
name string
plan types.Plan
want types.UpgradeDetails
wantErr bool
}{
{
name: "valid upgrade details",
plan: types.Plan{
Info: `{"upgradeType":"minor"}`,
},
want: types.UpgradeDetails{
UpgradeType: "minor",
},
},
{
name: "invalid json in Info",
plan: types.Plan{
Info: `{upgradeType:"minor"}`,
},
want: types.UpgradeDetails{},
},
}

for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
got := test.plan.UpgradeDetails()
if got != test.want {
t.Errorf("UpgradeDetails() = %v, want %v", got, test.want)
}
})
}
}

func TestIsMinorRelease(t *testing.T) {
tests := []struct {
name string
plan types.Plan
want bool
}{
{
name: "minor release",
plan: types.Plan{
Info: `{"upgradeType":"minor"}`,
},
want: true,
},
{
name: "minor release with extra fields",
plan: types.Plan{
Info: `{"upgradeType":"minor","extra":true}`,
},
want: true,
},
{
name: "not a minor release",
plan: types.Plan{
Info: `{"upgradeType":"major"}`,
},
want: false,
},
{
name: "default to major release",
plan: types.Plan{
Info: "",
},
want: false,
},
}

for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
ud := test.plan.UpgradeDetails()
if got := ud.IsMinorRelease(); got != test.want {
t.Errorf("IsMinorRelease() = %v, want %v", got, test.want)
}
})
}
}
Loading