[Bug] v21.09.x Ubuntu Server 20.04 IP changes after reboot causing failures

[akatukiyou]

Code of Conduct

• I have read and agree to the project's Code of Conduct.

VMware vSphere

7.0.2

HashiCorp Packer

1.7.4

HashiCorp Packer Plugin for VMware vSphere

1.0.1

Guest Operating System

Ubuntu 20.04

Environment Details

Building ubunto 20.04 image
vSphere 7.0.2
Packer 1.7.4
Terraform 1.0.7
Remote PC: Win10
Everything is in the same subnet: 172.16.2.x including: DNS, DHCP, vCetner, ESXI & Win10PC

Description

The previous release worked fine till today. I built images 20 days ago, but failed today, nothing changed....It failed at below

"sudo apt-get upgrade y"
"APT-GET upgrade could not get lock /var/lib/dpkg/lock-frontend. It is held by process 1935 (apt-get)"

So I downloaded the new release to try it out. the issue is DHCP IP address changes after server reboot.
==> vsphere-iso.linux-ubuntu-server: HTTP server is working at http://172.16.2.200:8078/ ==> vsphere-iso.linux-ubuntu-server: Typing boot command... ==> vsphere-iso.linux-ubuntu-server: Waiting for IP... ==> vsphere-iso.linux-ubuntu-server: IP address: 172.16.2.249 ==> vsphere-iso.linux-ubuntu-server: Using SSH communicator to connect: 172.16.2.249 ==> vsphere-iso.linux-ubuntu-server: Waiting for SSH to become available...

First I got an IP 172.16.2.249, then waiting for SSH to connect, there was no response after a long time waiting, so I logged in and checked the IP, turned out the server got a different IP address. 172.16.2.250

I tired 3 times, all the same issue.

Expected Behavior

Expected behavior:

Stay with the same IP

Actual Behavior

Server got a different IP after reboot

Steps to Reproduce

I only changed the vcenter variables in the code, everything else remained the same.

Log Fragments and Files

==> vsphere-iso.linux-ubuntu-server: Creating VM...
==> vsphere-iso.linux-ubuntu-server: Customizing hardware...
==> vsphere-iso.linux-ubuntu-server: Adding SATA controller...
==> vsphere-iso.linux-ubuntu-server: Mounting ISO images...
==> vsphere-iso.linux-ubuntu-server: Adding configuration parameters...
==> vsphere-iso.linux-ubuntu-server: Starting HTTP server on port 8078
==> vsphere-iso.linux-ubuntu-server: Set boot order...
==> vsphere-iso.linux-ubuntu-server: Power on VM...
==> vsphere-iso.linux-ubuntu-server: Waiting 5s for boot...
==> vsphere-iso.linux-ubuntu-server: HTTP server is working at http://172.16.2.200:8078/
==> vsphere-iso.linux-ubuntu-server: Typing boot command...
==> vsphere-iso.linux-ubuntu-server: Waiting for IP...
==> vsphere-iso.linux-ubuntu-server: IP address: 172.16.2.249
==> vsphere-iso.linux-ubuntu-server: Using SSH communicator to connect: 172.16.2.249
==> vsphere-iso.linux-ubuntu-server: Waiting for SSH to become available...
==> vsphere-iso.linux-ubuntu-server: Timeout waiting for SSH.
==> vsphere-iso.linux-ubuntu-server: Power off VM...
==> vsphere-iso.linux-ubuntu-server: Destroying VM...
Build 'vsphere-iso.linux-ubuntu-server' errored after 31 minutes 12 seconds: Timeout waiting for SSH.

==> Wait completed after 31 minutes 12 seconds

==> Some builds didn't complete successfully and had errors:
--> vsphere-iso.linux-ubuntu-server: Timeout waiting for SSH.

==> Builds finished but no artifacts were created.
Done.

Screenshots

No response

Additional Context

No response

[tenthirtyam]

v21.09 release or a clone of today's main?

Ryan

[tenthirtyam]

If you're not a clone, can you test with main? Pushed a recent change to address the locks.

Ryan

[akatukiyou]

Hi there,

The lock issue I had was with version 21.08. It was weird that it worked fine for while and suddenly breaks.

The IP change issue I had was with the latest version 21.09.1 main

[tenthirtyam]

Thanks for the update. It seems that these would be two separate issues and should be tracked separately.

With regards to "v21.09.1 main" did you download the v21.09.1 release package or git clone a copy of main? As the latter would be "top of tree" and be included in the next release.

Apologies, I'm trying to determine where on the commit timeline to investigate when time permits.

Ryan

[tenthirtyam]

Also, there's a recent commit (#53) in main that aims to address the locking issue. Essentially the updates were running twice and locking / colliding at times.

Ryan

[akatukiyou]

I did git-clone a copy of v21.09.1 main .

I couldn't verify the locking issue still exists in ver21.09.1 because IP changes after reboot

[tenthirtyam]

If you perform a git clone it will be pre-release as there are commits to main post-release.

Could you please download v21.09.01 from Releases and let me know if you experience the same issue.

Ryan

[tenthirtyam]

I can confirm that I'm seeing a similar issue in main and retesting with v21.08 the issue does not seem to occur.

In v21.09.x it seems to have an issue after installing the desired packages.

Ryan

[akatukiyou]

FYI, one thing I was thinking while troubleshooting:

At the beginning , I thought it was my DHCP issue, because I setup DHCP IP expires in 30 mins, for testing purpose I only have a range of 20 IPs for DHCP.

So I changed the expire setting to 3 days and the issue still the same.

[tenthirtyam]

Addressed in #61 for main.

You can patch v21.09.x with the following in user-data.pkrtpl.hcl for Ubuntu 20.04 LTS.

The locking issue should also be resolved with this change.

#cloud-config
autoinstall:
  version: 1
  early-commands:
    # Ensures that Packer does not connect too soon.
    - sudo systemctl stop ssh
  locale: ${vm_guest_os_language}
  keyboard:
    layout: ${vm_guest_os_keyboard}
  storage:
    layout:
      name: lvm
  identity:
    hostname: ubuntu-server
    username: ${build_username}
    password: ${build_password_encrypted}
  network:
    network:
      version: 2
      ethernets:
        ens33:
          dhcp4: true
  ssh:
    install-server: true
    allow-pw: true
  packages:
    - openssh-server
    - open-vm-tools
  user-data:
    disable_root: false
  late-commands:
    - 'sed -i "s/dhcp4: true/&\n      dhcp-identifier: mac/" /target/etc/netplan/00-installer-config.yaml'
    - echo '${build_username} ALL=(ALL) NOPASSWD:ALL' > /target/etc/sudoers.d/${build_username}
    - curtin in-target --target=/target -- chmod 440 /etc/sudoers.d/${build_username}
  timezone: ${vm_guest_os_timezone}

Ryan

[akatukiyou]

Thanks, I see the only change is ens192 to ens33

[tenthirtyam]

Yes, but there are additional changes in #93 and #96 in 'main' that will be in the next release. Addresses theses two issues and others.

Ryan

[akatukiyou]

Thanks, waiting for the next release. :)

[akatukiyou]

Just off the topic, can we have an option to choose where the templates stay? currently it goes to "content lib" , can it just stay in the "template" folder? Thx

[tenthirtyam]

While the intent for this project was adjacent to my work as an architect on the VMware Validated Designs and VMware Validated Solutions and use with vRealize Automation's integration with the content library, this enhancement was added and approved in #44.

You'd modify config/common.pkvars.hcl in the next release.

// Template and Content Library Settings
common_template_conversion     = false
common_content_library_name    = "sfo-w01-lib01"
common_content_library_ovf     = false
common_content_library_destroy = false

You can always discuss in the GitHub Discussions.

Ryan

[akatukiyou]

OK, confirmed, all working fine. Nice work!!

`==> Wait completed after 8 minutes 7 seconds

==> Builds finished. The artifacts of successful builds are:
--> vsphere-iso.linux-ubuntu-server: linux-ubuntu-server-20-04-lts
--> vsphere-iso.linux-ubuntu-server: linux-ubuntu-server-20-04-lts
Done.
Press Enter to continue.
`

[tenthirtyam]

Excellent! Thanks for testing!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.