NixOS EBS AMI has a default root filesystem size that's uncomfortably small

[copumpkin]

In general, people can customize their filesystem sizes when spinning up an AMI, but if someone isn't paying close attention and blindly spins one up with the defaults, they'll end up with a root filesystem that's only 3GB, and will fill up immediately. Might put off newcomers, and is a trivial fix.

I think the fix would boil down to changing https://github.com/NixOS/nixpkgs/blob/master/nixos/maintainers/scripts/ec2/create-amis.sh#L185 to a fixed size. It needs to be at least the snapshot size, but can be significantly larger. I'd probably start at 10GB or 20 just to give the Nix store some room to breathe.

cc @edolstra @rbvermaa

[edolstra]

This would undo #15148 which I don't think is a good idea.

You can increase the root size by setting deployment.ec2.ebsInitialRootDiskSize in NixOps. But you can't decrease it, so it makes sense to have the default AMI size small. (We could set a default value for deployment.ec2.ebsInitialRootDiskSize though.)

[copumpkin]

I'm not saying the snapshot should get bigger, just that the default disk mappings in the AMI are larger. There are two quantities here:

1. The size of the EBS volume we create the snapshot from, which is a lower bound on the final EBS volume size
2. The size of the EBS volume mapping (metadata) we add to the AMI specification

I'm saying to change #2, which doesn't prevent you from shrinking it if you want.

[copumpkin]

Also, not everyone uses NixOps 😄

[edolstra]

Well, that was the issue: apparently you cannot decrease the size in the default mapping, only increase it (#15148 (comment)). I may have been wrong though.

[copumpkin]

Looks like you're right 😡. I've submitted a bug and I'll let you know what AWS says about whether it's intentional or not. Very annoying!

[edolstra]

@copumpkin Cool, thanks! Would be great if they fix this.

[copumpkin]

No response yet but will keep you posted :)

…

On Mon, Jan 30, 2017 at 12:00 Eelco Dolstra ***@***.***> wrote: @copumpkin <https://github.com/copumpkin> Cool, thanks! Would be great if they fix this. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#22211 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAKP3u252SaqSDP6w-aTC9FmT0Dnm61ks5rXcLbgaJpZM4LwIQj> .

[knedlsepp]

I'm currently experimenting with NixOS on AWS and just ran into this issue. I'm not using nixops and would like to increase the root partition. This seems to be possible by modifying the EBS Volume via the AWS console. However I have not yet figured out what I need in my configuration.nix so the root partition can make use of the additionally allocated space.
Edit: I've found that autoResize should be enabled, but nixos-rebuild switch doesn't seem to care.
Edit2: autoResize works fine for me since #26912

[copumpkin]

I think this is now fixed on the AWS side, so I'm reopening it

[chris-martin]

This issue bit me - now how do I resize my existing EBS volume?

I've found that autoResize should be enabled

How?

We appear to be talking about this NixOS option but I'm not sure how to set it, since NixOps is the one generating the fileSystems config.

[chris-martin]

I'm all for sensible defaults, but I don't think there is such a thing for disk size. IMO NixOps ought to refuse to create a volume if you don't give it a size.

[copumpkin]

@chris-martin AWS actually supports live resizing of EBS volumes but I don't know if NixOps exposes that. If you just go into the console and highlight your EBS volume, you'll see an option to change its size though. Then you'll want resize2fs on the machine and run that.

Agree on more general point though. It was a bit of a hack and the justification for it has gone away, so I think we should: 1) provide a sensible default size 2) do what you're saying about asking folks to be explicit.

[chris-martin]

Hmmm, after the EBS resize finished, I tried this and it doesn't seem to have any effect.

# resize2fs /dev/disk/by-label/nixos
resize2fs 1.43.4 (31-Jan-2017)
The filesystem is already 786171 (4k) blocks long.  Nothing to do!

[bjornfor]

I think you must resize the partition first, then use resize2fs after that. "parted" has a "resizepart" command that can be used. Perhaps parted /dev/sda resizepart 1 100% (untested!).

[chris-martin]

There's a stack overflow answer that suggests using growpart which seems to be available in the cloud-utils package, but it seems broken.

# growpart /dev/xvda 1
/nix/store/hw5ydc3i92q686sb3yr275qihdhx2m5j-cloud-utils-0.29/bin/.growpart-wrapped: line 257: gnused: command not found
FAILED: gnused failed on dump output

[copumpkin]

Sounds like our growpart wrapper is b0rked (probably needs an issue), but parted is probably still fine.

[chris-martin]

NixOS appears to have inserted an ad hoc workaround in a module that uses cloud-utils, without actually bothering to fix the package :(

nixpkgs/nixos/modules/virtualisation/grow-partition.nix

Lines 28 to 31 in b902a88

	substitute "${pkgs.cloud-utils}/bin/.growpart-wrapped" "$out/bin/growpart" \
	--replace "${pkgs.bash}/bin/sh" "/bin/sh" \
	--replace "awk" "gawk" \
	--replace "sed" "gnused"

[copumpkin]

ugh 😦

[copumpkin]

@Mic92 what was your reasoning for that? It looks like you changed both the module and the package and set dontBuild so I'm assuming there's something not obvious going on here.

[chris-martin]

Related issue: #15736

[chris-martin]

With gnused on the PATH, this seems to work:

# mkdir /tmp/abc
# cd /tmp/abc
# ln -s /run/current-system/sw/bin/sed gnused
# env PATH="$PATH:/tmp/abc" growpart /dev/xvda 1
CHANGED: partition=1 start=2048 old: size=6289375 end=6291423 new: size=62912479,end=62914527
# resize2fs /dev/xvda1
resize2fs 1.43.4 (31-Jan-2017)
Filesystem at /dev/xvda1 is mounted on /; on-line resizing required
old_desc_blocks = 1, new_desc_blocks = 4
The filesystem on /dev/xvda1 is now 7864059 (4k) blocks long.

[orivej]

@copumpkin @Mic92 fixups are only necessary in the initrd where busybox versions of awk and sed are not good enough for growpart. dontBuild is there just because the old buildPhase was deleted and there is no Makefile.

@chris-martin Thanks to #30018, you should be able to install cloud-utils and use its grawpart without installing gnused beforehand.

[chris-martin]

I think the problem is that #30018 never made it into 17.09.

[orivej]

@fpletz Let's release 2354e0f

[knedlsepp]

Not sure if it applies to your case, but I solved resizing the root partition using a nixos-rebuild and reboot with the following configuration.nix: knedlsepp/knedlsepp.at@7348c06

[ORESoftware]

Here is what df says on an EC2 instance:

[root@ip-172-xx-xx-139:~]# df -H
Filesystem                Size  Used Avail Use% Mounted on
devtmpfs                  1.7G     0  1.7G   0% /dev
tmpfs                      17G     0   17G   0% /dev/shm
tmpfs                     8.3G  4.4M  8.3G   1% /run
tmpfs                      17G  369k   17G   1% /run/wrappers
/dev/disk/by-label/nixos  3.2G  3.1G     0 100% /
tmpfs                      17G     0   17G   0% /sys/fs/cgroup
tmpfs                     3.3G     0  3.3G   0% /run/user/0

does anyone know how to resize it so I can use more than just 3.2 GB?

[Azulinho]

@ORESoftware https://tech.mybuilder.com/how-to-expand-a-linux-aws-ebs-root-partition-volume-without-rebooting/

[stale]

Thank you for your contributions.
This has been automatically marked as stale because it has had no activity for 180 days.
If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.
Here are suggestions that might help resolve this more quickly:

1. Search for maintainers and people that previously touched the
   related code and @ mention them in a comment.
2. Ask on the NixOS Discourse. 3. Ask on the #nixos channel on
   irc.freenode.net.

[steve-chavez]

One issue I've faced is that after adding some packages on a NixOS instance, I run short of space, so I resize the EBS volume and try to install cloud-utils(growpart) but Nix fails with "no storage left on device".

I've just destroyed the instance in that case, but maybe the NixOS AMI could include cloud-utils by default?

[Mic92]

that would make sense.

[stale]

I marked this as stale due to inactivity. → More info

[karantan]

Not sure if this is still relevant. I successfully increased disk size on NixOS 21.05 with the following commands:

$ nix-shell -p cloud-utils
...

[nix-shell:~]# lsblk
NAME    MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda    202:0    0  50G  0 disk 
├─xvda1 202:1    0   1M  0 part 
└─xvda2 202:2    0  35G  0 part /

[nix-shell:~]# growpart /dev/xvda 2
CHANGED: partition=2 start=4096 old: size=73396191 end=73400287 new: size=104853471 end=104857567

[nix-shell:~]# resize2fs /dev/xvda2
resize2fs 1.46.2 (28-Feb-2021)
Filesystem at /dev/xvda2 is mounted on /; on-line resizing required
old_desc_blocks = 5, new_desc_blocks = 7
The filesystem on /dev/xvda2 is now 13106683 (4k) blocks long.

[jshcmpbll]

For my teams configuration we are seeing the boot disk size automatically grow from the snapshot size of 3gb to our configured EBS size of 40gb

Here are some dmesg logs:

[    0.534760] GPT:Alternate GPT header not at the end of the disk.
[    1.810611] stage-1-init: [Fri Apr 28 20:53:29 UTC 2023] checking /dev/disk/by-label/nixos...
[    1.813888] stage-1-init: [Fri Apr 28 20:53:29 UTC 2023] [fsck.ext4 (1) -- /mnt-root/] fsck.ext4 -a /dev/disk/by-label/nixos
[    2.149314] stage-1-init: [Fri Apr 28 20:53:30 UTC 2023] resizing /dev/disk/by-label/nixos...
[   31.169463] stage-1-init: [Fri Apr 28 20:53:59 UTC 2023] Resizing the filesystem on /dev/disk/by-label/nixos to 10423291 (4k) blocks.
[   31.170949] stage-1-init: [Fri Apr 28 20:53:59 UTC 2023] The filesystem on /dev/disk/by-label/nixos is now 10423291 (4k) blocks long.
[   31.171956] stage-1-init: [Fri Apr 28 20:53:59 UTC 2023] mounting /dev/disk/by-label/nixos on /...

Looks like autoResize is configured by default in the amazon image module:

nixpkgs/nixos/modules/virtualisation/amazon-image.nix

Line 35 in 358a179

autoResize = true;

I think this issue is good to be closed unless anyone has conflicting results

Thanks @sarcasticadmin for helping with this 🥳

[Mic92]

I'd say that this issue has been resolved.