Locked Workspace ignores previously generated plan file

[pauloconnor]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request. Searching for pre-existing feature requests helps us consolidate datapoints for identical requirements into a single place, thank you!
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.

---

Overview of the Issue

If a PR has a plan currently being run, and another change is force pushed, Atlantis will reply with The default workspace is currently locked by another command that is running for this pull request. Wait until the previous command is complete and try again.. When the original plan operation is completed, and atlantis apply is run, it results in Ran Apply for 0 projects:, despite there being changes in the original plan output.

Reproduction Steps

• Open a PR with changes
• While the plan is running, force push another change to the PR
• Observe The default workspace is currently locked by another command that is running for this pull request. Wait until the previous command is complete and try again.
• Wait until the original plan has commented it's changes
• Run atlantis apply
• Atlantis will comment Ran Apply for 0 projects:

Logs

Logs

{
    "level": "debug",
    "ts": "2022-04-08T20:40:41.511Z",
    "caller": "events/working_dir.go:84",
    "msg": "clone directory \"/home/atlantis/.atlantis/repos/org/repo/12484/default\" already exists, checking if it's at the right commit",
    "json": {
        "repo": "org/repo",
        "pull": "12484"
    }
}
{
    "level": "debug",
    "ts": "2022-04-08T20:40:41.530Z",
    "caller": "events/working_dir.go:111",
    "msg": "repo was already cloned but is not at correct commit, wanted \"aa1084d52f0d2931f6bac77898c6b2db361567ed\" got \"ba228982bcc1bc493a8bc1ec9dbdccebf445dd6a\"",
    "json": {
        "repo": "org/repo",
        "pull": "12484"
    }
}
{
    "level": "info",
    "ts": "2022-04-08T20:41:19.505Z",
    "caller": "events/working_dir.go:202",
    "msg": "creating dir \"/home/atlantis/.atlantis/repos/org/repo/12484/default\"",
    "json": {
        "repo": "org/repo",
        "pull": "12484"
    }
}
{
    "level": "debug",
    "ts": "2022-04-08T20:42:43.569Z",
    "caller": "events/working_dir.go:268",
    "msg": "ran: git clone --branch nl/irsa-for-scala-thrift-service --depth=1 --single-branch https://orgdeploy:<redacted>@github.com/org/repo.git /home/atlantis/.atlantis/repos/org/repo/12484/default. Output: Cloning into '/home/atlantis/.atlantis/repos/org/repo/12484/default'...\nUpdating files:   1% (90/6268)\rUpdating files:   2% (126/6268)\rUpdating files:   2% (176/6268)\rUpdating files:   3% (189/6268)\rUpdating files:   4% (251/6268)\rUpdating files:   4% (260/6268)\rUpdating files:   5% (314/6268)\rUpdating files:   5% (351/6268)\rUpdating files:   6% (377/6268)\rUpdating files:   6% (430/6268)\rUpdating files:   7% (439/6268)\rUpdating files:   7% (501/6268)\rUpdating files:   8% (502/6268)\rUpdating files:   9% (565/6268)\rUpdating files:   9% (575/6268)\rUpdating files:  10% (627/6268)\rUpdating files:  10% (650/6268)\rUpdating files:  11% (690/6268)\rUpdating files:  11% (725/6268)\rUpdating files:  12% (753/6268)\rUpdating files:  12% (797/6268)\rUpdating files:  13% (815/6268)\rUpdating files:  13% (862/6268)\rUpdating files:  14% (878/6268)\rUpdating files:  14% (930/6268)\rUpdating files:  15% (941/6268)\rUpdating files:  15% (999/6268)\rUpdating files:  16% (1003/6268)\rUpdating files:  16% (1064/6268)\rUpdating files:  17% (1066/6268)\rUpdating files:  18% (1129/6268)\rUpdating files:  18% (1135/6268)\rUpdating files:  19% (1191/6268)\rUpdating files:  19% (1205/6268)\rUpdating files:  20% (1254/6268)\rUpdating files:  20% (1286/6268)\rUpdating files:  21% (1317/6268)\rUpdating files:  21% (1363/6268)\rUpdating files:  22% (1379/6268)\rUpdating files:  22% (1431/6268)\rUpdating files:  23% (1442/6268)\rUpdating files:  23% (1500/6268)\rUpdating files:  24% (1505/6268)\rUpdating files:  25% (1567/6268)\rUpdating files:  25% (1577/6268)\rUpdating files:  26% (1630/6268)\rUpdating files:  26% (1649/6268)\rUpdating files:  27% (1693/6268)\rUpdating files:  27% (1723/6268)\rUpdating files:  28% (1756/6268)\rUpdating files:  28% (1790/6268)\rUpdating files:  29% (1818/6268)\rUpdating files:  29% (1862/6268)\rUpdating files:  30% (1881/6268)\rUpdating files:  30% (1927/6268)\rUpdating files:  31% (1944/6268)\rUpdating files:  31% (2001/6268)\rUpdating files:  32% (2006/6268)\rUpdating files:  33% (2069/6268)\rUpdating files:  33% (2078/6268)\rUpdating files:  34% (2132/6268)\rUpdating files:  34% (2148/6268)\rUpdating files:  35% (2194/6268)\rUpdating files:  35% (2223/6268)\rUpdating files:  36% (2257/6268)\rUpdating files:  36% (2317/6268)\rUpdating files:  37% (2320/6268)\rUpdating files:  38% (2382/6268)\rUpdating files:  38% (2400/6268)\rUpdating files:  39% (2445/6268)\rUpdating files:  39% (2473/6268)\rUpdating files:  40% (2508/6268)\rUpdating files:  40% (2546/6268)\rUpdating files:  41% (2570/6268)\rUpdating files:  41% (2618/6268)\rUpdating files:  42% (2633/6268)\rUpdating files:  43% (2696/6268)\rUpdating files:  43% (2704/6268)\rUpdating files:  44% (2758/6268)\rUpdating files:  44% (2771/6268)\rUpdating files:  45% (2821/6268)\rUpdating files:  45% (2842/6268)\rUpdating files:  46% (2884/6268)\rUpdating files:  46% (2918/6268)\rUpdating files:  47% (2946/6268)\rUpdating files:  47% (2993/6268)\rUpdating files:  48% (3009/6268)\rUpdating files:  48% (3067/6268)\rUpdating files:  49% (3072/6268)\rUpdating files:  50% (3134/6268)\rUpdating files:  50% (3141/6268)\rUpdating files:  51% (3197/6268)\rUpdating files:  51% (3228/6268)\rUpdating files:  52% (3260/6268)\rUpdating files:  52% (3299/6268)\rUpdating files:  53% (3323/6268)\rUpdating files:  53% (3369/6268)\rUpdating files:  54% (3385/6268)\rUpdating files:  54% (3412/6268)\rUpdating files:  55% (3448/6268)\rUpdating files:  55% (3478/6268)\rUpdating files:  56% (3511/6268)\rUpdating files:  56% (3549/6268)\rUpdating files:  57% (3573/6268)\rUpdating files:  57% (3617/6268)\rUpdating files:  58% (3636/6268)\rUpdating files:  58% (3697/6268)\rUpdating files:  59% (3699/6268)\rUpdating files:  60% (3761/6268)\rUpdating files:  60% (3764/6268)\rUpdating files:  61% (3824/6268)\rUpdating files:  61% (3839/6268)\rUpdating files:  62% (3887/6268)\rUpdating files:  62% (3911/6268)\rUpdating files:  63% (3949/6268)\rUpdating files:  63% (3983/6268)\rUpdating files:  64% (4012/6268)\rUpdating files:  64% (4072/6268)\rUpdating files:  65% (4075/6268)\rUpdating files:  66% (4137/6268)\rUpdating files:  66% (4140/6268)\rUpdating files:  67% (4200/6268)\rUpdating files:  67% (4222/6268)\rUpdating files:  68% (4263/6268)\rUpdating files:  68% (4306/6268)\rUpdating files:  69% (4325/6268)\rUpdating files:  70% (4388/6268)\rUpdating files:  70% (4393/6268)\rUpdating files:  71% (4451/6268)\rUpdating files:  71% (4482/6268)\rUpdating files:  72% (4513/6268)\rUpdating files:  73% (4576/6268)\rUpdating files:  74% (4639/6268)\rUpdating files:  74% (4669/6268)\rUpdating files:  75% (4701/6268)\rUpdating files:  75% (4762/6268)\rUpdating files:  76% (4764/6268)\rUpdating files:  77% (4827/6268)\rUpdating files:  77% (4857/6268)\rUpdating files:  78% (4890/6268)\rUpdating files:  78% (4950/6268)\rUpdating files:  79% (4952/6268)\rUpdating files:  80% (5015/6268)\rUpdating files:  80% (5046/6268)\rUpdating files:  81% (5078/6268)\rUpdating files:  82% (5140/6268)\rUpdating files:  83% (5203/6268)\rUpdating files:  83% (5234/6268)\rUpdating files:  84% (5266/6268)\rUpdating files:  84% (5327/6268)\rUpdating files:  85% (5328/6268)\rUpdating files:  86% (5391/6268)\rUpdating files:  86% (5423/6268)\rUpdating files:  87% (5454/6268)\rUpdating files:  88% (5516/6268)\rUpdating files:  88% (5518/6268)\rUpdating files:  89% (5579/6268)\rUpdating files:  89% (5606/6268)\rUpdating files:  90% (5642/6268)\rUpdating files:  90% (5700/6268)\rUpdating files:  91% (5704/6268)\rUpdating files:  92% (5767/6268)\rUpdating files:  92% (5793/6268)\rUpdating files:  93% (5830/6268)\rUpdating files:  93% (5888/6268)\rUpdating files:  94% (5892/6268)\rUpdating files:  95% (5955/6268)\rUpdating files:  95% (5982/6268)\rUpdating files:  96% (6018/6268)\rUpdating files:  96% (6071/6268)\rUpdating files:  97% (6080/6268)\rUpdating files:  98% (6143/6268)\rUpdating files:  98% (6152/6268)\rUpdating files:  99% (6206/6268)\rUpdating files:  99% (6240/6268)\rUpdating files: 100% (6268/6268)\rUpdating files: 100% (6268/6268), done.",
    "json": {
        "repo": "org/repo",
        "pull": "12484"
    }
}
{
    "level": "info",
    "ts": "2022-04-08T20:42:50.808Z",
    "caller": "runtime/pre_workflow_hook_runner.go:50",
    "msg": "successfully ran \"/usr/local/bin/generate_atlantis_server_yaml.sh\" in \"/home/atlantis/.atlantis/repos/org/repo/12484/default\"",
    "json": {
        "repo": "org/repo",
        "pull": "12484"
    }
}
{
    "level": "debug",
    "ts": "2022-04-08T20:42:51.104Z",
    "caller": "vcs/github_client.go:244",
    "msg": "GET /repos/org/repo/pulls/12484/reviews",
    "json": {}
}
{
    "level": "debug",
    "ts": "2022-04-08T20:42:56.480Z",
    "caller": "vcs/github_client.go:176",
    "msg": "GET /repos/org/repo/issues/12484/comments",
    "json": {}
}
{
    "level": "debug",
    "ts": "2022-04-08T20:42:57.383Z",
    "caller": "vcs/github_client.go:163",
    "msg": "POST /repos/org/repo/issues/12484/comments",
    "json": {}
}
{
    "level": "debug",
    "ts": "2022-04-08T20:42:58.006Z",
    "caller": "events/db_updater.go:24",
    "msg": "updating DB with pull results",
    "json": {
        "repo": "org/repo",
        "pull": "12484"
    }
}
{
    "level": "debug",
    "ts": "2022-04-08T20:42:58.276Z",
    "caller": "vcs/github_client.go:163",
    "msg": "POST /repos/org/repo/issues/12484/comments",
    "json": {}
}
{
    "level": "info",
    "ts": "2022-04-08T20:42:58.861Z",
    "caller": "events/automerger.go:31",
    "msg": "automerging pull request",
    "json": {
        "repo": "org/repo",
        "pull": "12484"
    }
}
{
    "level": "debug",
    "ts": "2022-04-08T20:42:58.984Z",
    "caller": "vcs/github_client.go:365",
    "msg": "PUT /repos/github.User{Login:\"org\", ID:175478, NodeID:\"MDEyOk9yZ2FuaXphdGlvbjE3NTQ3OA==\", AvatarURL:\"https://avatars.githubusercontent.com/u/175152?v=4\", HTMLURL:\"https://github.com/org\", GravatarID:\"\", Type:\"Organization\", SiteAdmin:false, URL:\"https://api.github.com/users/org\", EventsURL:\"https://api.github.com/users/org/events{/privacy}\", FollowingURL:\"https://api.github.com/users/org/following{/other_user}\", FollowersURL:\"https://api.github.com/users/org/followers\", GistsURL:\"https://api.github.com/users/org/gists{/gist_id}\", OrganizationsURL:\"https://api.github.com/users/org/orgs\", ReceivedEventsURL:\"https://api.github.com/users/org/received_events\", ReposURL:\"https://api.github.com/users/org/repos\", StarredURL:\"https://api.github.com/users/org/starred{/owner}{/repo}\", SubscriptionsURL:\"https://api.github.com/users/org/subscriptions\"}/0xc000843be0/pulls/12484/merge",
    "json": {}
}
{
    "level": "debug",
    "ts": "2022-04-08T20:43:38.092Z",
    "caller": "vcs/github_client.go:163",
    "msg": "POST /repos/org/repo/issues/12484/comments",
    "json": {}
}
{
    "level": "info",
    "ts": "2022-04-08T20:43:38.732Z",
    "caller": "events/events_controller.go:360",
    "msg": "deleted locks and workspace for repo org/repo, pull 12484",
    "json": {}
}

Environment details

Running in ECS with an EFS filesystem for consistency between container replacements

• Atlantis version: 0.18.1
• If not running the latest Atlantis version have you tried to reproduce this issue on the latest version: Not yet, will do
• Atlantis flags:

Environment

"environment": [
  {
    "name": "ATLANTIS_ALLOW_DRAFT_PRS",
    "value": "true"
  },
  {
    "name": "ATLANTIS_ALLOW_REPO_CONFIG",
    "value": "false"
  },
  {
    "name": "ATLANTIS_ATLANTIS_URL",
    "value": "https://atlantis.org.com"
  },
  {
    "name": "ATLANTIS_AUTOMERGE",
    "value": "true"
  },
  {
    "name": "ATLANTIS_BITBUCKET_BASE_URL",
    "value": ""
  },
  {
    "name": "ATLANTIS_BITBUCKET_USER",
    "value": ""
  },
  {
    "name": "ATLANTIS_ENABLE_DIFF_MARKDOWN_FORMAT",
    "value": "true"
  },
  {
    "name": "ATLANTIS_GH_APP_SLUG",
    "value": "orgdeploy"
  },
  {
    "name": "ATLANTIS_GH_USER",
    "value": "orgdeploy"
  },
  {
    "name": "ATLANTIS_GITLAB_HOSTNAME",
    "value": "gitlab.com"
  },
  {
    "name": "ATLANTIS_GITLAB_USER",
    "value": ""
  },
  {
    "name": "ATLANTIS_HIDE_PREV_PLAN_COMMENTS",
    "value": "true"
  },
  {
    "name": "ATLANTIS_LOG_LEVEL",
    "value": "debug"
  },
  {
    "name": "ATLANTIS_PORT",
    "value": "4141"
  },
  {
    "name": "ATLANTIS_REPO_ALLOWLIST",
    "value": "github.com/org/repo"
  },
  {
    "name": "ATLANTIS_REPO_CONFIG_JSON",
    "value": "{\"automerge\":true,\"repos\":[{\"allow_custom_workflows\":false,\"allowed_overrides\":[\"apply_requirements\",\"workflow\",\"delete_source_branch_on_merge\"],\"allowed_workflows\":[\"terragrunt\"],\"apply_requirements\":[\"approved\",\"mergeable\"],\"branch\":\"/.*/\",\"delete_source_branch_on_merge\":true,\"id\":\"github.com/org/repo\",\"pre_workflow_hooks\":[{\"run\":\"/usr/local/bin/generate_atlantis_server_yaml.sh\"}],\"workflow\":\"terragrunt\"}],\"version\":3,\"workflows\":{\"standard\":{\"apply\":{\"steps\":[{\"run\":\"echo Applying\"},\"apply\"]},\"plan\":{\"steps\":[{\"run\":\"rm -rf .terraform\"},\"init\",{\"run\":\"echo Running tflint...\"},{\"run\":\"tflint --module\"},\"plan\"]}},\"terragrunt\":{\"apply\":{\"steps\":[{\"env\":{\"command\":\"echo \\\"terraform${ATLANTIS_TERRAFORM_VERSION}\\\"\",\"name\":\"TERRAGRUNT_TFPATH\"}},{\"run\":\"echo Applying\"},{\"run\":\"terragrunt apply -auto-approve -compact-warnings -no-color $PLANFILE\"}]},\"plan\":{\"steps\":[{\"env\":{\"command\":\"echo \\\"terraform${ATLANTIS_TERRAFORM_VERSION}\\\"\",\"name\":\"TERRAGRUNT_TFPATH\"}},{\"run\":\"/usr/local/bin/clean_terragrunt.sh $PLANFILE\"}]}}}}"
  }
],

Atlantis.yaml

---
version: 3
projects:
  - dir: terraform/acct-infra/env-prod/us-east-1/eks
    workflow: terragrunt
    terraform_version: v1.1.2
    autoplan:
      enabled: true
      when_modified: ["*.tf*", "*.hcl"]
    apply_requirements: [approved, mergeable]

Additional Context

[evanstachowiak]

I am also running into this on v0.19.2. I commented also on issue: #773

I'm going to repeat that comment here, because I think it's relevant:
I am using an atlantis.yaml that was previously working. I think around v0.19.* this started breaking. It is about 50 projects, each with its own project name so that the -p wildcard flag can be used. The pattern for the naming is ${service_name}-${environment}.

I discovered that if i run atlantis apply -p *-environment, then the command will run, but it will run for ALL projects, regardless of what files have changed.

I have autoplan on, but if I run atlantis plan manually, it doesn't seem to make a difference.

Also of note, I am using custom workflows, not sure if that makes a difference.

[jamengual]

@evanstachowiak Please test with the pre-release image, we did some bug fixes there and I wonder if that could be the issue:

docker pull ghcr.io/runatlantis/atlantis:v0.19.3-pre.20220408

[jamengual]

is this still an issue with v0.19.8?