Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: auto-refresh on new revisions during sync retries (Alpha) (#11494) #15603

Open
wants to merge 5 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions assets/swagger.json
Original file line number Diff line number Diff line change
Expand Up @@ -8583,6 +8583,10 @@
"description": "Limit is the maximum number of attempts for retrying a failed sync. If set to 0, no retries will be performed.",
"type": "integer",
"format": "int64"
},
"refresh": {
"type": "boolean",
"title": "Refresh indicates if a new revision should trigger a new sync (default: false)"
}
}
},
Expand Down
11 changes: 11 additions & 0 deletions cmd/util/app.go
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,7 @@ type AppOptions struct {
retryBackoffDuration time.Duration
retryBackoffMaxDuration time.Duration
retryBackoffFactor int64
retryRefresh bool
ref string
}

Expand Down Expand Up @@ -136,6 +137,7 @@ func AddAppFlags(command *cobra.Command, opts *AppOptions) {
command.Flags().DurationVar(&opts.retryBackoffDuration, "sync-retry-backoff-duration", argoappv1.DefaultSyncRetryDuration, "Sync retry backoff base duration. Input needs to be a duration (e.g. 2m, 1h)")
command.Flags().DurationVar(&opts.retryBackoffMaxDuration, "sync-retry-backoff-max-duration", argoappv1.DefaultSyncRetryMaxDuration, "Max sync retry backoff duration. Input needs to be a duration (e.g. 2m, 1h)")
command.Flags().Int64Var(&opts.retryBackoffFactor, "sync-retry-backoff-factor", argoappv1.DefaultSyncRetryFactor, "Factor multiplies the base duration after each failed sync retry")
command.Flags().BoolVar(&opts.retryRefresh, "sync-retry-refresh", false, "Set if a new revision should trigger a new sync")
command.Flags().StringVar(&opts.ref, "ref", "", "Ref is reference to another source within sources field")
}

Expand Down Expand Up @@ -220,6 +222,7 @@ func SetAppSpecOptions(flags *pflag.FlagSet, spec *argoappv1.ApplicationSpec, ap
MaxDuration: appOpts.retryBackoffMaxDuration.String(),
Factor: ptr.To(appOpts.retryBackoffFactor),
},
Refresh: appOpts.retryRefresh,
}
} else if appOpts.retryLimit == 0 {
if spec.SyncPolicy.IsZero() {
Expand All @@ -230,6 +233,14 @@ func SetAppSpecOptions(flags *pflag.FlagSet, spec *argoappv1.ApplicationSpec, ap
} else {
log.Fatalf("Invalid sync-retry-limit [%d]", appOpts.retryLimit)
}
case "sync-retry-refresh":
if spec.SyncPolicy == nil {
spec.SyncPolicy = &argoappv1.SyncPolicy{}
}
if spec.SyncPolicy.Retry == nil {
spec.SyncPolicy.Retry = &argoappv1.RetryStrategy{}
}
spec.SyncPolicy.Retry.Refresh = appOpts.retryRefresh
}
})
if flags.Changed("auto-prune") {
Expand Down
7 changes: 7 additions & 0 deletions cmd/util/app_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -227,6 +227,13 @@ func Test_setAppSpecOptions(t *testing.T) {
assert.NoError(t, f.SetFlag("sync-retry-limit", "0"))
assert.Nil(t, f.spec.SyncPolicy.Retry)
})
t.Run("RetryRefresh", func(t *testing.T) {
assert.NoError(t, f.SetFlag("sync-retry-refresh", "true"))
assert.True(t, f.spec.SyncPolicy.Retry.Refresh)

assert.NoError(t, f.SetFlag("sync-retry-refresh", "false"))
assert.False(t, f.spec.SyncPolicy.Retry.Refresh)
})
t.Run("Kustomize", func(t *testing.T) {
assert.NoError(t, f.SetFlag("kustomize-replica", "my-deployment=2"))
assert.NoError(t, f.SetFlag("kustomize-replica", "my-statefulset=4"))
Expand Down
9 changes: 9 additions & 0 deletions controller/appcontroller.go
Original file line number Diff line number Diff line change
Expand Up @@ -1294,6 +1294,15 @@ func (ctrl *ApplicationController) processRequestedAppOperation(app *appv1.Appli
ctrl.requestAppRefresh(app.QualifiedName(), CompareWithLatest.Pointer(), &retryAfter)
return
} else {
// abort ongoing operation and retry if auto-refresh is enabled and a new revision is available
if state.Phase == synccommon.OperationRunning && state.Operation.Retry.Refresh && app.Status.Sync.Revision != state.Operation.Sync.Revision {
logCtx.Infof("A new revision is available, refreshing and terminating app, was phase: %s, message: %s", state.Phase, state.Message)
ctrl.requestAppRefresh(app.QualifiedName(), CompareWithLatest.Pointer(), nil)
aslafy-z marked this conversation as resolved.
Show resolved Hide resolved
state.Phase = synccommon.OperationTerminating
state.Message = "Operation forced to terminate (new revision available)"
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I could not find this message on the UI, is there a way to make it appear?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we have to move setOperationState above the requestAppRefresh, since otherwise there can be a race condition by the refresh processor kicking in and message not being ready yet.

ctrl.setOperationState(app, state)
return
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's a setOperationState below as well, let's combine the two.

// retrying operation. remove previous failure time in app since it is used as a trigger
// that previous failed and operation should be retried
state.FinishedAt = nil
Expand Down
18 changes: 18 additions & 0 deletions docs/user-guide/auto_sync.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,24 @@ spec:
selfHeal: true
```

## Automatic Retry Refresh on new revisions

This feature allows users to configure their applications to refresh on new revisions when the current sync is retrying. To enable automatic refresh during sync retries, run:


```bash
argocd app set <APPNAME> --sync-retry-refresh
```

Or by setting the `retry.refresh` option to `true` in the sync policy:

```yaml
spec:
syncPolicy:
retry:
refresh: true
```

## Automated Sync Semantics

* An automated sync will only be performed if the application is OutOfSync. Applications in a
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,7 @@ argocd admin app generate-spec APPNAME [flags]
--sync-retry-backoff-factor int Factor multiplies the base duration after each failed sync retry (default 2)
--sync-retry-backoff-max-duration duration Max sync retry backoff duration. Input needs to be a duration (e.g. 2m, 1h) (default 3m0s)
--sync-retry-limit int Max number of allowed sync retries
--sync-retry-refresh Set if a new revision should trigger a new sync
--validate Validation of repo and cluster (default true)
--values stringArray Helm values file(s) to use
--values-literal-file string Filename or URL to import as a literal Helm values block
Expand Down
1 change: 1 addition & 0 deletions docs/user-guide/commands/argocd_app_add-source.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,7 @@ argocd app add-source APPNAME [flags]
--sync-retry-backoff-factor int Factor multiplies the base duration after each failed sync retry (default 2)
--sync-retry-backoff-max-duration duration Max sync retry backoff duration. Input needs to be a duration (e.g. 2m, 1h) (default 3m0s)
--sync-retry-limit int Max number of allowed sync retries
--sync-retry-refresh Set if a new revision should trigger a new sync
--validate Validation of repo and cluster (default true)
--values stringArray Helm values file(s) to use
--values-literal-file string Filename or URL to import as a literal Helm values block
Expand Down
1 change: 1 addition & 0 deletions docs/user-guide/commands/argocd_app_create.md
Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,7 @@ argocd app create APPNAME [flags]
--sync-retry-backoff-factor int Factor multiplies the base duration after each failed sync retry (default 2)
--sync-retry-backoff-max-duration duration Max sync retry backoff duration. Input needs to be a duration (e.g. 2m, 1h) (default 3m0s)
--sync-retry-limit int Max number of allowed sync retries
--sync-retry-refresh Set if a new revision should trigger a new sync
--upsert Allows to override application with the same name even if supplied application spec is different from existing spec
--validate Validation of repo and cluster (default true)
--values stringArray Helm values file(s) to use
Expand Down
1 change: 1 addition & 0 deletions docs/user-guide/commands/argocd_app_set.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,7 @@ argocd app set APPNAME [flags]
--sync-retry-backoff-factor int Factor multiplies the base duration after each failed sync retry (default 2)
--sync-retry-backoff-max-duration duration Max sync retry backoff duration. Input needs to be a duration (e.g. 2m, 1h) (default 3m0s)
--sync-retry-limit int Max number of allowed sync retries
--sync-retry-refresh Set if a new revision should trigger a new sync
--validate Validation of repo and cluster (default true)
--values stringArray Helm values file(s) to use
--values-literal-file string Filename or URL to import as a literal Helm values block
Expand Down
Loading
Loading