aws_volume_attachment Error waiting for Volume #4770

jrobison-sb · 2018-06-06T19:58:24Z

Terraform Version

Terraform v0.11.7

provider.aws v1.20.0

Affected Resource(s)

aws_volume_attachment

Actual Behavior

When running a terraform destroy, in which you want to destroy an aws_instance and it's aws_volume_attachment, terraform (or the AWS API) will attempt to destroy the volume attachment first, then then instance second. This often fails because Error waiting for Volume (vol-010b3d979b1027867) to detach from Instance.

Expected Behavior

If an aws_volume_attachment was going to be destroyed, and the associated aws_instance was also going to be destroyed, it would make sense to destroy the instance first and then the aws_volume_attachment second, which would never run any risk of timing out.

Steps to Reproduce

Create an aws_ebs_volume, an aws_instance, and a aws_volume_attachment.
terraform destroy

The text was updated successfully, but these errors were encountered:

YektaLeblebici · 2018-06-12T12:16:38Z

I can confirm this for our infra too. On latest Terraform, destroying aws_volume_attachment and aws_instance together fails with this timeout error.

alexaandru · 2018-07-27T20:31:11Z

Running into this as well.

Hashfyre · 2018-08-13T12:35:25Z

➜ tfa
data.aws_availability_zones.az: Refreshing state...
data.aws_route53_zone.int-orgaws: Refreshing state...
data.aws_route53_zone.orgaws: Refreshing state...
data.aws_vpc.stage-org: Refreshing state...
data.aws_ami.stage-mongodb: Refreshing state...
data.aws_subnet.stage-mongodb: Refreshing state...
data.aws_security_group.stage-mongodb: Refreshing state...
aws_instance.stage-mongodb: Refreshing state... (ID: i-xxxxxxxxxxxxxxxxxxx)
aws_route53_record.stage-mongodb-int: Refreshing state... (ID: ACCID-stage-mongodb-int_A)
aws_ebs_volume.stage-mongodb-volume-a: Refreshing state... (ID: vol-xxxxxxxxxxx)
aws_volume_attachment.stage-mongodb-volume-attachment-a: Refreshing state... (ID: vai-xxxxxxxxx)

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  - destroy

Terraform will perform the following actions:

  - aws_ebs_volume.stage-mongodb-volume-a

  - aws_instance.stage-mongodb

  - aws_route53_record.stage-mongodb-int

  - aws_volume_attachment.stage-mongodb-volume-attachment-a


Plan: 0 to add, 0 to change, 4 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

aws_volume_attachment.stage-mongodb-volume-attachment-a: Destroying... (ID: vai-xxxxxxxxxxx)
aws_route53_record.stage-mongodb-int: Destroying... (ID: ACCID-stage-mongodb-int_A)
aws_volume_attachment.stage-mongodb-volume-attachment-a: Still destroying... (ID: vai-385500658, 10s elapsed)
aws_route53_record.stage-mongodb-int: Still destroying... (ID: ACCID-stage-mongodb-int_A, 10s elapsed)
aws_route53_record.stage-mongodb-int: Still destroying... (ID: ACCID-stage-mongodb-int_A, 20s elapsed)
aws_route53_record.stage-mongodb-int: Still destroying... (ID: ACCID-stage-mongodb-int_A, 30s elapsed)
aws_route53_record.stage-mongodb-int: Still destroying... (ID: ACCID-stage-mongodb-int_A, 40s elapsed)
aws_route53_record.stage-mongodb-int: Still destroying... (ID: ACCID-stage-mongodb-int_A, 50s elapsed)
aws_route53_record.stage-mongodb-int: Still destroying... (ID: ACCID-stage-mongodb-int_A, 1m0s elapsed)
aws_route53_record.stage-mongodb-int: Still destroying... (ID: ACCID-stage-mongodb-int_A, 1m10s elapsed)
aws_route53_record.stage-mongodb-int: Still destroying... (ID: ACCID-stage-mongodb-int_A, 1m20s elapsed)
aws_route53_record.stage-mongodb-int: Still destroying... (ID: ACCID-stage-mongodb-int_A, 1m30s elapsed)
aws_route53_record.stage-mongodb-int: Still destroying... (ID: ACCID-stage-mongodb-int_A, 1m40s elapsed)
aws_route53_record.stage-mongodb-int: Destruction complete after 1m40s

Error: Error applying plan:

1 error(s) occurred:

* aws_volume_attachment.stage-mongodb-volume-attachment-a (destroy): 1 error(s) occurred:

* aws_volume_attachment.stage-mongodb-volume-attachment-a: Error waiting for Volume (vol-xxxxxxxxxxxxx) to detach from Instance: i-xxxxxxxxxxxxx

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

queglay · 2019-02-04T13:24:43Z

this is also a problem if ebs volume ids are replaced in the template, you'll get a similar error. if a volume can't be detached, it would be good to have a variable that did both these things-
shutdown instances if volumes can't be detached
startup instances after volume detachments and attachments are complete.

this would also help when doing terraform destroy.

I've attached an example below from changing the aws_volume_attachement name that causes the same problem.

Error: Error applying plan:

5 error(s) occurred:

module.softnas.aws_volume_attachment.ebs_att[0] (destroy): 1 error(s) occurred:
aws_volume_attachment.ebs_att.0: Error waiting for Volume (vol-0dfdb289f3a00f63c) to detach from Instance: i-055c1f1bd7d1b417c
module.softnas.aws_cloudformation_stack.SoftNAS1Stack: 1 error(s) occurred:
aws_cloudformation_stack.SoftNAS1Stack: Creating CloudFormation stack failed: AlreadyExistsException: Stack [FCB-SoftNAS1Stack] already exists
status code: 400, request id: 303555c9-287f-11e9-b20c-4d54279ee980
module.softnas.aws_volume_attachment.ebs_att[1] (destroy): 1 error(s) occurred:
aws_volume_attachment.ebs_att.1: Error waiting for Volume (vol-071219d59b8153d6a) to detach from Instance: i-055c1f1bd7d1b417c
module.softnas.aws_volume_attachment.ebs_att[3] (destroy): 1 error(s) occurred:
aws_volume_attachment.ebs_att.3: Error waiting for Volume (vol-0c0703a9c333cef86) to detach from Instance: i-055c1f1bd7d1b417c
module.softnas.aws_volume_attachment.ebs_att[2] (destroy): 1 error(s) occurred:
aws_volume_attachment.ebs_att.2: Error waiting for Volume (vol-0a21cf466c807a753) to detach from Instance: i-055c1f1bd7d1b417c

mikaelforeseeti · 2019-04-16T08:04:17Z

I ran into this today, is there a workaround? I tried running terraform destroy --target=module.machines.ep_instance but it still tries to destroy the volume_attachment object first.

paulrowlands · 2019-05-14T12:11:14Z

FYI, I ran into this when implementing Windows servers with MSSQL.

I found that adding a null_resource which has a depends_on for the instance and a remote-exec provisioner which includes a 'when = "destroy"' option and executes a Powershell command stop the MSSQLSERVER service on the host.

After this, terraform destroy will destroy the attachment.

You could replace the MSSQL service with relevant shell commands to stop services, unmount disks, etc.

yum-dev · 2019-06-12T15:19:41Z

Even I'm facing this issue.

Terraform v0.12.1

provider.aws v2.14.0

DrHashi · 2019-07-11T21:00:38Z

Terraform v0.12.3

provider.aws v2.18.0

Still failing...

kingindanord · 2019-07-31T19:34:39Z

I have the same issue

Terraform v0.12.5

provider.aws v2.20.0

twarnock · 2019-08-07T20:26:58Z

We are facing this issue as well.

Terraform v0.12.5
provider.aws v2.22

AndreiF8 · 2019-08-28T17:02:44Z

Facing same issue

Terraform v0.11.14

provider.aws v2.25.0

pitthecat · 2019-09-10T08:30:21Z

We have the same problem

Terraform 0.12.7 and provider AWS 2.25.0

RiceAndMeet · 2019-11-25T21:09:01Z

Facing the same issue here, we worked around it by including skip_destroy = true in resource aws_volume_attachment.

According to the documentation, setting skip_destroy = true will remove resource aws_volume_attachment from the state file, and this will unblock terraform destroy operation.

kreempuff · 2019-12-25T14:26:20Z

Hey all, Happy Holidays.
What fixed it for me was the first line here:

You can detach an Amazon EBS volume from an instance explicitly or by terminating the instance. However, if the instance is running, you must first unmount the volume from the instance.

If the volume is still mounted on the instance, it will never detach (it even shows gray in the AWS console, so terraform may be able to help us here). Debugging terraform with export TF_LOG=TRACE showed that the api returned a 200 OK for the DetachVolume api call, but subsequent calls to DescribeVolume showed the volume as busy.

After I unmounted the volume, terraform destroyed the volume attachment successfully.

twarnock · 2020-01-10T17:00:20Z

If we're listing workarounds, shutting down the instance before the destroy will work as well. The detach will work since the volume would not be "busy" at that point.

edsale · 2020-01-28T13:09:55Z

Same problem here. Only been a bug 19 months. Could we get Terraform to STOP instances with attached storage, DETACH/DELETE all volumes and then TERMINATE the instance? Seems like a fundamental function fails often in this circumstance.

yogeshsajanikar · 2020-03-28T11:50:23Z

Probably there should be a provision to run a script just before volume is detached.

janosmiko · 2020-06-11T09:11:11Z

+1 here
Terraform version: 0.12.25
AWS Provider version: 2.65.0

ghost · 2020-07-04T16:02:31Z

another day, another issue with super basic stuff expected from TF. No response from TF also.

For the record:

Terraform v0.12.26

Issue persists.

onezeroecho · 2020-07-12T06:12:36Z

Got this issue too.

Terraform v0.12.28
+ provider.aws v2.63.0

shrilesh1986 · 2020-08-06T19:43:23Z

Encountered Same issue, have to manually unmount to volume and then proceed with terraform destroy.

stijnopheide · 2020-08-26T17:22:10Z

persists in

terraform v0.13.0
provider.aws v2.77.0

Jouo · 2020-10-13T23:50:53Z

Problem still here
Terraform v0.13.4
AWS v3.10.0

juloi · 2020-11-24T21:57:23Z

Same problem!!!

terraform v0.13.2
provider.aws  ~> 3.0

dogewithit · 2020-12-30T10:34:10Z

Same issue for me

maciejjakub · 2021-02-04T14:27:29Z

Same issue. We're using Terraform v0.13.4.

dstamen · 2021-07-20T14:13:37Z

still hitting this with terraform 1.0.2 and AWS provider 3.5. stopping the instance first allowed me to destroy.

Terraform v1.0.2
on darwin_amd64

provider registry.terraform.io/hashicorp/aws v3.50.0

Illutax · 2021-08-18T08:25:08Z

Couldn't detatch volume, not even with EC2 Management Console. The volume attachment was in state detaching. I manually stopped the instance, then the attachment was successfully detached by itself.
After that i started the instance again and the terraform apply ran successfully.

Was using Terraform v0.12.31
aws-cli v2.2.18

RawthiL · 2021-09-06T20:03:07Z

Same here, using:

Terraform v1.0.5
hashicorp/aws 3.24.1
aws-cli/2.2.16 Python/3.8.8 Linux/5.11.0-27-generic exe/x86_64.ubuntu.20 prompt/off (I'm not sure if this is relevant)

If the volume is mounted the destruction fails, however if I unmount the volume then the attachment destruction, and later the instance destruction, both succeed. While this is logical, I would expect terraform to first destroy the instance (unmounting the volume in the process) and then destroy the volume attachment.

This logic was used to ensure that the EC2 instance for an AWS EBS volume attachment was in a terminated state so that the volume attachment could be destroyed successfully. This has not been an issue in more recent COOL configurations so we are removing it here. In the event that this is still an problem a relevant issue can be found at: hashicorp/terraform-provider-aws#4770

ughstudios · 2021-09-26T02:28:10Z

I'm also having this issue in terraform 1.0.7

…lume By stopping the instance, the volume is unmounted in the instance and the detaching of the volume doesn't run into a timeout fixes #6673 fixes #2084 fixes #2957 fixes #4770 fixes #288 fixes #1017

YakDriver · 2021-10-05T00:06:21Z

Thank you for reporting this and your continued interest. This issue should be resolved with the feature added in #21144. If that does not resolve the issue, please open a new issue and reference this one.

github-actions · 2021-10-08T00:59:55Z

This functionality has been released in v3.62.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

github-actions · 2022-06-03T02:29:44Z

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

radeksimko added bug Addresses a defect in current functionality. service/ec2 Issues and PRs that pertain to the ec2 service. labels Jun 13, 2018

f3lang mentioned this issue May 10, 2019

[WIP] Add option to stop an instance before trying to remove an attached volume #8602

Closed

bflad mentioned this issue Nov 13, 2019

Error when destroying an EC2 instance with a volume attached #6673

Closed

YakDriver self-assigned this Oct 4, 2021

YakDriver mentioned this issue Oct 4, 2021

Add option to stop an instance before trying to remove an attached volume #21144

Merged

YakDriver closed this as completed in #21144 Oct 4, 2021

github-actions bot added this to the v3.62.0 milestone Oct 5, 2021

github-actions bot locked as resolved and limited conversation to collaborators Jun 3, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

aws_volume_attachment Error waiting for Volume #4770

aws_volume_attachment Error waiting for Volume #4770

jrobison-sb commented Jun 6, 2018

YektaLeblebici commented Jun 12, 2018

alexaandru commented Jul 27, 2018

Hashfyre commented Aug 13, 2018

queglay commented Feb 4, 2019

mikaelforeseeti commented Apr 16, 2019

paulrowlands commented May 14, 2019

yum-dev commented Jun 12, 2019

DrHashi commented Jul 11, 2019 •

edited

Loading

kingindanord commented Jul 31, 2019

twarnock commented Aug 7, 2019

AndreiF8 commented Aug 28, 2019

pitthecat commented Sep 10, 2019

RiceAndMeet commented Nov 25, 2019

kreempuff commented Dec 25, 2019 •

edited

Loading

twarnock commented Jan 10, 2020 •

edited

Loading

edsale commented Jan 28, 2020

yogeshsajanikar commented Mar 28, 2020

janosmiko commented Jun 11, 2020

ghost commented Jul 4, 2020

onezeroecho commented Jul 12, 2020

shrilesh1986 commented Aug 6, 2020

stijnopheide commented Aug 26, 2020 •

edited

Loading

Jouo commented Oct 13, 2020 •

edited

Loading

juloi commented Nov 24, 2020

dogewithit commented Dec 30, 2020

maciejjakub commented Feb 4, 2021

dstamen commented Jul 20, 2021

Illutax commented Aug 18, 2021 •

edited

Loading

RawthiL commented Sep 6, 2021

ughstudios commented Sep 26, 2021

YakDriver commented Oct 5, 2021

github-actions bot commented Oct 8, 2021

github-actions bot commented Jun 3, 2022

aws_volume_attachment Error waiting for Volume #4770

aws_volume_attachment Error waiting for Volume #4770

Comments

jrobison-sb commented Jun 6, 2018

Terraform Version

Affected Resource(s)

Actual Behavior

Expected Behavior

Steps to Reproduce

YektaLeblebici commented Jun 12, 2018

alexaandru commented Jul 27, 2018

Hashfyre commented Aug 13, 2018

queglay commented Feb 4, 2019

mikaelforeseeti commented Apr 16, 2019

paulrowlands commented May 14, 2019

yum-dev commented Jun 12, 2019

DrHashi commented Jul 11, 2019 • edited Loading

kingindanord commented Jul 31, 2019

twarnock commented Aug 7, 2019

AndreiF8 commented Aug 28, 2019

pitthecat commented Sep 10, 2019

RiceAndMeet commented Nov 25, 2019

kreempuff commented Dec 25, 2019 • edited Loading

twarnock commented Jan 10, 2020 • edited Loading

edsale commented Jan 28, 2020

yogeshsajanikar commented Mar 28, 2020

janosmiko commented Jun 11, 2020

ghost commented Jul 4, 2020

onezeroecho commented Jul 12, 2020

shrilesh1986 commented Aug 6, 2020

stijnopheide commented Aug 26, 2020 • edited Loading

Jouo commented Oct 13, 2020 • edited Loading

juloi commented Nov 24, 2020

dogewithit commented Dec 30, 2020

maciejjakub commented Feb 4, 2021

dstamen commented Jul 20, 2021

Illutax commented Aug 18, 2021 • edited Loading

RawthiL commented Sep 6, 2021

ughstudios commented Sep 26, 2021

YakDriver commented Oct 5, 2021

github-actions bot commented Oct 8, 2021

github-actions bot commented Jun 3, 2022

DrHashi commented Jul 11, 2019 •

edited

Loading

kreempuff commented Dec 25, 2019 •

edited

Loading

twarnock commented Jan 10, 2020 •

edited

Loading

stijnopheide commented Aug 26, 2020 •

edited

Loading

Jouo commented Oct 13, 2020 •

edited

Loading

Illutax commented Aug 18, 2021 •

edited

Loading