Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Show failed command after atmos workflow failure #767

Merged
merged 11 commits into from
Dec 5, 2024
13 changes: 13 additions & 0 deletions internal/exec/help.go
Original file line number Diff line number Diff line change
Expand Up @@ -103,6 +103,19 @@ func processHelp(
"If the workspace does not exist, the command creates it by executing the 'terraform workspace new' command.\n\n" +
"Usage: atmos terraform workspace <component> -s <stack>\n\n" +
"For more details refer to https://atmos.tools/cli/commands/terraform/workspace\n")
} else if componentType == "workflow" && command == "" {
u.PrintMessage("\nAtmos workflow commands support failure handling and resume functionality:\n\n" +
"When a workflow step fails:\n" +
" - The failed step name and command will be displayed\n" +
" - A resume command will be provided to restart from the failed step\n\n" +
"Example:\n" +
"Step 'deploy-vpc' failed!\n" +
"Error: Error applying plan:\n" +
"1 error occurred: AWS API call failed\n\n" +
"Failed command: terraform apply vpc -auto-approve\n\n" +
osterman marked this conversation as resolved.
Show resolved Hide resolved
"To resume the workflow from this step, run:\n" +
"atmos workflow deploy-infra -f workflow1 --from-step deploy-vpc\n\n" +
"For more details refer to https://atmos.tools/cli/commands/workflow/resume\n")
osterman marked this conversation as resolved.
Show resolved Hide resolved
} else {
u.PrintMessage(fmt.Sprintf("\nAtmos supports native '%s' commands with all the options, arguments and flags.\n", componentType))
u.PrintMessage("In addition, 'component' and 'stack' are required in order to generate variables for the component in the stack.\n")
Expand Down
27 changes: 21 additions & 6 deletions internal/exec/workflow_utils.go
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ import (
"fmt"
"os"
"path"
"path/filepath"
"sort"
"strings"

Expand Down Expand Up @@ -70,11 +71,10 @@ func ExecuteWorkflow(
commandType = "atmos"
}

var err error
if commandType == "shell" {
commandName := fmt.Sprintf("%s-step-%d", workflow, stepIdx)
if err := ExecuteShell(cliConfig, command, commandName, ".", []string{}, dryRun); err != nil {
return err
}
err = ExecuteShell(cliConfig, command, commandName, ".", []string{}, dryRun)
} else if commandType == "atmos" {
args := strings.Fields(command)

Expand All @@ -101,12 +101,27 @@ func ExecuteWorkflow(
logFunc(cliConfig, fmt.Sprintf("Stack: %s", finalStack))
}

if err := ExecuteShellCommand(cliConfig, "atmos", args, ".", []string{}, dryRun, ""); err != nil {
return err
}
err = ExecuteShellCommand(cliConfig, "atmos", args, ".", []string{}, dryRun, "")
} else {
return fmt.Errorf("invalid workflow step type '%s'. Supported types are 'atmos' and 'shell'", commandType)
}

if err != nil {
workflowFileName := filepath.Base(workflowPath)
workflowFileName = strings.TrimSuffix(workflowFileName, filepath.Ext(workflowFileName))

failedMsg := color.New(color.FgRed).Sprintf("\nStep '%s' failed!", step.Name)
cmdMsg := color.New(color.FgYellow).Sprintf("Failed command: %s", command)
Cerebrovinny marked this conversation as resolved.
Show resolved Hide resolved
resumeMsg := color.New(color.FgGreen).Sprintf(
"\nTo resume the workflow from this step, run:\ncd %s && atmos workflow %s -f %s --from-step %s",
Cerebrovinny marked this conversation as resolved.
Show resolved Hide resolved
filepath.Dir(workflowPath),
workflow,
workflowFileName,
step.Name,
)

return fmt.Errorf("%s\n%s\n%s\nError: %v", failedMsg, cmdMsg, resumeMsg, err)
}
}

return nil
Expand Down
32 changes: 32 additions & 0 deletions website/docs/core-concepts/workflows/workflows.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -371,6 +371,38 @@ The Atmos stack used by the workflow commands of type `atmos` can be specified i
atmos workflow my-workflow -f workflow1 -s tenant1-ue2-dev
```

## Workflow Failure Handling and Resume

When a workflow step fails, Atmos will:
1. Display which step failed
2. Show the exact command that failed
3. Provide a ready-to-use command to resume the workflow from the failed step

Given this workflow:

```yaml
Cerebrovinny marked this conversation as resolved.
Show resolved Hide resolved
workflows:
test-1:
Cerebrovinny marked this conversation as resolved.
Show resolved Hide resolved
description: "Deploy test components"
Cerebrovinny marked this conversation as resolved.
Show resolved Hide resolved
steps:
- command: terraform plan test/test-component-override-1
name: step-1
- command: terraform plan test/test-component-override-2
name: step-2
- command: terraform plan test/test-component-override-3
name: step-3
```

If step-2 fails, you'll see:

```console
Step 'step-2' failed!
Failed command: terraform plan test/test-component-override-2
Cerebrovinny marked this conversation as resolved.
Show resolved Hide resolved

To resume the workflow from this step, run:
atmos workflow test-1 -f workflow1 --from-step step-2
Cerebrovinny marked this conversation as resolved.
Show resolved Hide resolved
Cerebrovinny marked this conversation as resolved.
Show resolved Hide resolved
```

### Stack Precedence

The stack defined inline in the command itself has the lowest priority, it can and will be overridden by any other stack definition.
Expand Down