v3.0.1-rc3: IP not found and agent_timeout not being used #1106

arty-hlr · 2024-09-17T12:46:37Z

Contrary to the PR #1016 and to what the documentation https://registry.terraform.io/providers/telmate/proxmox/3.0.1-rc3/docs/resources/vm_qemu says, agent_timeout is not being used, which results in VMs being created and then the following error:

I do use agent_timeout in the module:

but it's not being shown in the plan output:

The only difference with previous templates I tried to clone is that there is now SSH installed. I am not sure why it cannot find the IP address as it shows it in the GUI. I would expect agent_timeout to help, as the error suggests, but it does not do anything unfortunately. It also sometimes work on one VM or the other, pretty randomly as it seems, so doing multiple terraform apply helps, kinda.

Here is the full output with TF_VAR_LOG=DEBUG:

To reproduce:

create a windows template with SSH server installed and clone it (linked clone) with terraform
create a VM from a template with terraform and use agent_timeout

The text was updated successfully, but these errors were encountered:

GMZwinge · 2024-10-02T15:16:23Z

Test without Cloud-init (os_type = "centos"):

3.0.1-rc3 provides an IP address:

2024-10-02T10:56:03.512-0400 [INFO]  provider.terraform-provider-proxmox_v3.0.1-rc3.exe: 2024/10/02 10:56:03 [DEBUG] VM is running, checking the IP: timestamp=2024-10-02T10:56:03.512-0400
2024-10-02T10:56:03.512-0400 [INFO]  provider.terraform-provider-proxmox_v3.0.1-rc3.exe: 2024/10/02 10:56:03 [INFO][initConnInfo] trying to get vm ip address for provisioner: timestamp=2024-10-02T10:56:03.512-0400
2024-10-02T10:56:03.512-0400 [INFO]  provider.terraform-provider-proxmox_v3.0.1-rc3.exe: 2024/10/02 10:56:03 [DEBUG][initConnInfo] retrying for at most  20m0s minutes before giving up: timestamp=2024-10-02T10:56:03.512-0400
2024-10-02T10:56:03.512-0400 [INFO]  provider.terraform-provider-proxmox_v3.0.1-rc3.exe: 2024/10/02 10:56:03 [DEBUG][initConnInfo] retries will end at 2024-10-02 11:16:03.5122731 -0400 EDT m=+1224.259660501: timestamp=2024-10-02T10:56:03.512-0400
2024-10-02T10:56:06.525-0400 [INFO]  provider.terraform-provider-proxmox_v3.0.1-rc3.exe: 2024/10/02 10:56:06 [INFO][getPrimaryIP] check ip result error 500 QEMU guest agent is not running: timestamp=2024-10-02T10:56:06.524-0400
2024-10-02T10:56:14.566-0400 [INFO]  provider.terraform-provider-proxmox_v3.0.1-rc3.exe: 2024/10/02 10:56:14 [INFO][getPrimaryIP] check ip result error 500 QEMU guest agent is not running: timestamp=2024-10-02T10:56:14.565-0400
2024-10-02T10:56:19.712-0400 [INFO]  provider.terraform-provider-proxmox_v3.0.1-rc3.exe: 2024/10/02 10:56:19 [INFO][getPrimaryIP] QEMU Agent interfaces found: [{00:00:00:00:00:00 [127.0.0.1 ::1] lo <nil>} {<MacAddress> [<Ipv6Addresses>] ens18 <nil>}]: timestamp=2024-10-02T10:56:19.712-0400
2024-10-02T10:56:24.922-0400 [INFO]  provider.terraform-provider-proxmox_v3.0.1-rc3.exe: 2024/10/02 10:56:24 [INFO][getPrimaryIP] QEMU Agent interfaces found: [{00:00:00:00:00:00 [127.0.0.1 ::1] lo <nil>} {<MacAddress> [<Ipv4Address> <Ipv6Addresses>] ens18 <nil>}]: timestamp=2024-10-02T10:56:24.922-0400
2024-10-02T10:56:24.922-0400 [INFO]  provider.terraform-provider-proxmox_v3.0.1-rc3.exe: 2024/10/02 10:56:24 [DEBUG][initConnInfo] this is the vm configuration: <Ipv4Address> 22: timestamp=2024-10-02T10:56:24.922-0400

3.0.1-rc4 doesn't provide an IP address:

2024-10-02T10:53:14.311-0400 [INFO]  provider.terraform-provider-proxmox_v3.0.1-rc4.exe: 2024/10/02 10:53:14 [DEBUG] VM is running, checking the IP: timestamp=2024-10-02T10:53:14.311-0400
2024-10-02T10:53:14.311-0400 [INFO]  provider.terraform-provider-proxmox_v3.0.1-rc4.exe: 2024/10/02 10:53:14 [INFO][initConnInfo] trying to get vm ip address for provisioner: timestamp=2024-10-02T10:53:14.311-0400
2024-10-02T10:53:14.311-0400 [INFO]  provider.terraform-provider-proxmox_v3.0.1-rc4.exe: 2024/10/02 10:53:14 [DEBUG][initConnInfo] retrying for at most  20m0s minutes before giving up: timestamp=2024-10-02T10:53:14.311-0400
2024-10-02T10:53:14.311-0400 [INFO]  provider.terraform-provider-proxmox_v3.0.1-rc4.exe: 2024/10/02 10:53:14 [DEBUG][initConnInfo] retries will end at 2024-10-02 11:13:14.3111935 -0400 EDT m=+1221.822484201: timestamp=2024-10-02T10:53:14.311-0400
2024-10-02T10:53:14.311-0400 [INFO]  provider.terraform-provider-proxmox_v3.0.1-rc4.exe: 2024/10/02 10:53:14 [INFO][getPrimaryIP] vm has a cloud-init configuration: timestamp=2024-10-02T10:53:14.311-0400
2024-10-02T10:53:14.311-0400 [INFO]  provider.terraform-provider-proxmox_v3.0.1-rc4.exe: 2024/10/02 10:53:14 [DEBUG][initConnInfo] this is the vm configuration:  22: timestamp=2024-10-02T10:53:14.311-0400

Also, 3.0.1-rc4 displays those warning: Warning: Cloud-init is enabled but no IP config is set and Cloud-init is enabled in your configuration but no static IP address is set, nor is the DHCP option enabled. Was able to get rid of those warning with define_connection_info = false, but it still doesn't provide an IP address:

2024-10-02T11:09:14.978-0400 [INFO]  provider.terraform-provider-proxmox_v3.0.1-rc4.exe: 2024/10/02 11:09:14 [DEBUG] VM is running, checking the IP: timestamp=2024-10-02T11:09:14.978-0400
2024-10-02T11:09:14.978-0400 [INFO]  provider.terraform-provider-proxmox_v3.0.1-rc4.exe: 2024/10/02 11:09:14 [INFO][initConnInfo] define_connection_info is false, no further action: timestamp=2024-10-02T11:09:14.978-0400

Not sure if that's the correct way to NOT use Cloud-init though.

Tinyblargon · 2024-10-03T17:05:35Z

Found the commit that most likely broke it.
Ironically it separated the logic so it would be easier to test.

18095e5#diff-104365919f693375882979581d9f36f5266991eef361948e56c220931fe886ddL1916

Tinyblargon · 2024-10-05T11:03:57Z

@GMZwinge could you check if #1120 fixes your issue?

Tinyblargon · 2024-10-05T11:30:30Z

@arty-hlr

agent_timeout is being used, it could be that endTime on L1970 is shorter, but by default this is 20 minutes.

terraform-provider-proxmox/proxmox/resource_vm_qemu.go

Lines 1970 to 1994 in a9a0bad

    
           for time.Now().Before(endTime) { 
        
           	var interfaces []pxapi.AgentNetworkInterface 
        
           	interfaces, err = vmr.GetAgentInformation(client, false) 
        
           	if err != nil { 
        
           		if !strings.Contains(err.Error(), ErrorGuestAgentNotRunning) { 
        
           			return primaryIPs{}, diag.FromErr(err) 
        
           		} 
        
           		log.Printf("[INFO][getPrimaryIP] check ip result error %s", err.Error()) 
        
           		logger.Debug().Int("vmid", vmr.VmId()).Msgf("check ip result error %s", err.Error()) 
        
           	} else { // vm is running and reachable 
        
           		if len(interfaces) > 0 { // agent returned some information 
        
           			log.Printf("[INFO][getPrimaryIP] QEMU Agent interfaces found: %v", interfaces) 
        
           			logger.Debug().Int("vmid", vmr.VmId()).Msgf("QEMU Agent interfaces found: %v", interfaces) 
        
           			conn = conn.parsePrimaryIPs(interfaces, primaryMacAddress) 
        
           			if conn.hasRequiredIP() { 
        
           				return conn.IPs, diag.Diagnostics{} 
        
           			} 
        
           		} 
        
           		if waitedTime > agentTimeout { 
        
           			break 
        
           		} 
        
           		waitedTime += additionalWait 
        
           	} 
        
           	time.Sleep(time.Duration(additionalWait) * time.Second) 
        
           }

However, it the log your showed does show the information in it's raw form.
I think we are parsing the MAC address wrong and therefore we can't match it with an interface returned by the guest-agent.

terraform-provider-proxmox/proxmox/resource_vm_qemu.go

Lines 1965 to 1968 in a9a0bad

    
           if _, ok := vmConfig["net"+strconv.Itoa(i)]; ok { 
        
           	primaryMacAddress = macAddressRegex.FindString(vmConfig["net"+strconv.Itoa(i)].(string)) 
        
           	break 
        
           }

Upstream I'm working on re-implementingthe network interfaces to get rid of this regex parsing of the MAC address on L1966.

Tinyblargon · 2024-10-05T15:41:47Z

@arty-hlr can you check if #1120 fixes your issue?

arty-hlr · 2024-10-10T11:56:00Z

@Tinyblargon How do I tell terraform/opentofu to use your branch? I only see source and version in the terraform provider config. Should I compile it, and copy the executable to the plugin directory like here?

arty-hlr · 2024-10-10T11:57:19Z

about agent_timeout: I did try to change its value, but it didn't make any difference, so I assumed it wasn't used at all as it wasn't in the logs.

Tinyblargon · 2024-10-14T15:39:31Z

@arty-hlr when you compile the branch, the compiled binary has to be renamed to linux_amd64, then you can put at .terraform/providers/terraform.local/local/proxmox/1.0.0/linux_amd64 inside your terraform project developer.md has more information about this.

arty-hlr · 2024-10-23T13:00:17Z

Hi @Tinyblargon, that didn't work, I had to change the .terraformrc to add a filesystem mirror, unfortunately that's not mentioned anywhere in the docs.

I just ran a test with your branch and unfortunately it still doesn't work, here's the relevant part of the output:

It seems that looping over terraform apply (which worked before) doesn't anymore either.

Tinyblargon · 2024-10-23T14:17:09Z

@arty-hlr

Checking if i have the correct situation.

os_type == "cloud-init"
a cloud-init disk is configured
skip_ipv4 == true
skip_ipv6 == false

The error is telling me that ipconfig0 is not configured.

arty-hlr · 2024-10-23T14:38:25Z

@Tinyblargon The same happens with os_type as cloud-init or cloud-init not set at all, the above screenshots from the run were with cloud-init not set at all. There is no cloud-init disk configured, the VMs are just cloned from the template and get a dynamic IP address from the DHCP server, so I don't use ipconfig0 as there's no actual cloud-init.

I'm actually not sure why it says Cloud-init is enabled because it is not in the terraform config and the templates don't have a cloud-init drive.

arty-hlr · 2024-10-23T14:42:17Z

The code here https://github.com/Telmate/terraform-provider-proxmox/blob/master/proxmox/resource_vm_qemu.go#L1935 seems to indicate that this shouldn't happen when cloud-init is not set

Tinyblargon · 2024-10-31T21:43:59Z

@arty-hlr made some changes and tested locally.
It now works as expected and reliably gets an IP address.
Did 2 test, one VM with cloud-init, and one VM without cloud-init with pre-configured DHCP.

Tinyblargon · 2024-11-27T23:00:12Z

@GMZwinge @arty-hlr could you test if the latest release solves your issue?

arty-hlr · 2024-12-10T13:47:57Z

Sorry for the late answer, will do today and let you know

arty-hlr · 2024-12-11T12:36:46Z

@Tinyblargon Unfortunately same problem with the latest release (rc6) 😢 the provider couldn't get the IP of one of the VMs even though it was up and showed an IP on proxmox.

Here the next test run, I was checking the VMs in proxmox as they were created and all 5 got an IP and displayed it, somehow the provider couldn't find the IP for 3 of them:

Tinyblargon · 2024-12-11T12:58:11Z

@arty-hlr could you set the agent_timeout really high and see if it has different results?
Does the vm have a net0 network interface? I believe it only checks for that one. The GUI does display the ip of the lowest net adapter.

arty-hlr · 2024-12-11T13:49:59Z

They have a net0 network interface, I had agent_timeout at 120, just set it to 3600, it made no difference unfortunately. The EX-1 VM was not the last to be created, had an IP in proxmox, but the provider couldn't get it.

This is the full terraform file for a windows VM we use, if that helps:

terraform {
  required_providers {
    proxmox = {
      source  = "telmate/proxmox"
      version = "3.0.1-rc6"
    }
  }
}

resource "proxmox_vm_qemu" "windows-vm" {
  name          = var.name
  target_node   = var.proxmox_host
  clone         = var.template
  full_clone    = "false"
  agent         = 1
  agent_timeout = 3600
  os_type       = "cloud-init"
  cores         = 4
  sockets       = 1
  cpu_type      = "host"
  bios          = "ovmf"
  memory        = 4096
  scsihw        = "virtio-scsi-pci"
  bootdisk      = "scsi0"
  pool          = var.pool
  skip_ipv6     = "true"

  disks {
    sata {
      sata0 {
        disk {
          size       = "128G"
          storage    = "loc_pve-xxx_r1"
          emulatessd = true
          discard    = true
        }
      }
    }
  }

  network {
    id     = 0
    model  = "virtio"
    bridge = var.nic_name
    tag    = var.vlan_num
  }
  lifecycle {
    ignore_changes = [
      network,
    ]
  }
}

Tinyblargon · 2024-12-15T12:39:42Z

@arty-hlr Could you check if the debug logs show the following errors? It will show the raw data of the qemu-guest-agent.

https://github.com/Telmate/terraform-provider-proxmox/blame/7e667094714d35f66c8d45ef08400895e1ad6804/proxmox/resource_vm_qemu.go#L1679-L1680

arty-hlr · 2024-12-16T14:23:39Z

@Tinyblargon So, I think we're getting closer. This is how it looks like when the IP is found:

and this is when the IP is not found:

Here's the whole log file: https://gist.github.com/arty-hlr/6c25344a04e9fcfdebcbb02c5a1830f5

For me it looks like the qemu guest agent returns a temporary IP address (169.254.XXX.XXX) which is then not parsed correctly somehow. I'd suggest maybe waiting until an actual private IP is returned by the qemu guest agent?

Tinyblargon · 2024-12-16T14:32:21Z

@arty-hlr 192.254.xxx.xxx is used by windows for local addresses when it didn't get dhcp yet?

arty-hlr · 2024-12-16T15:44:19Z

@Tinyblargon 169.254.xxx.xxx, yes. When I look at the VM in proxmox in real time, I usually see a second or so with that IP before the actual IP is given by the DHCP server. It could be a "slow" race condition between when the provider checks and when the IP is given? Then maybe checking if the IP is in 10.0.0.0/8, 172.16.0.0/12, or 192.168.0.0/16 could be worth doing? But that implies it's supposed to be a private IP though. Then maybe just ignoring 169.254.xxx.xxx would be better? Just throwing ideas

Tinyblargon · 2024-12-16T17:29:26Z

@arty-hlr After doing some more research these are Link-Local addresses and should be ignored.
https://en.wikipedia.org/wiki/Link-local_address

arty-hlr · 2024-12-17T13:07:22Z

@Tinyblargon Yes exactly. Do you mean they should already be ignored, or it should be fixed so that they are? From what I can see it is reported by getPrimaryIP but not printed in initConnInfo.

Tinyblargon · 2024-12-17T13:36:43Z

@arty-hlr it should be fixed so they are ignored.

Tinyblargon added the issue/investigate label Sep 17, 2024

Tinyblargon mentioned this issue Oct 28, 2024

Terraform Destroy not working #1116

Open

Tinyblargon added a commit to Tinyblargon/terraform-provider-proxmox that referenced this issue Oct 31, 2024

Merge branch 'Telmate:master' into Telmate#1106

9db77c8

Tinyblargon added a commit to Tinyblargon/terraform-provider-proxmox that referenced this issue Nov 19, 2024

Merge branch 'master' into Telmate#1106

0c58b70

Tinyblargon added type/bug resource/qemu Issue or PR related to Qemu resource issue/confirmed Issue has been reviewed and confirmed to be present or accepted to be implemented and removed issue/investigate labels Nov 27, 2024

Tinyblargon self-assigned this Dec 16, 2024

Tinyblargon mentioned this issue Dec 16, 2024

[3.0.1-rc7] Status #1180

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

v3.0.1-rc3: IP not found and agent_timeout not being used #1106

v3.0.1-rc3: IP not found and agent_timeout not being used #1106

arty-hlr commented Sep 17, 2024

GMZwinge commented Oct 2, 2024 •

edited

Loading

Tinyblargon commented Oct 3, 2024

Tinyblargon commented Oct 5, 2024

Tinyblargon commented Oct 5, 2024

Tinyblargon commented Oct 5, 2024

arty-hlr commented Oct 10, 2024

arty-hlr commented Oct 10, 2024

Tinyblargon commented Oct 14, 2024

arty-hlr commented Oct 23, 2024

Tinyblargon commented Oct 23, 2024

arty-hlr commented Oct 23, 2024

arty-hlr commented Oct 23, 2024

Tinyblargon commented Oct 31, 2024

Tinyblargon commented Nov 27, 2024

arty-hlr commented Dec 10, 2024

arty-hlr commented Dec 11, 2024

Tinyblargon commented Dec 11, 2024

arty-hlr commented Dec 11, 2024

Tinyblargon commented Dec 15, 2024

arty-hlr commented Dec 16, 2024

Tinyblargon commented Dec 16, 2024

arty-hlr commented Dec 16, 2024

Tinyblargon commented Dec 16, 2024

arty-hlr commented Dec 17, 2024

Tinyblargon commented Dec 17, 2024

v3.0.1-rc3: IP not found and agent_timeout not being used #1106

v3.0.1-rc3: IP not found and agent_timeout not being used #1106

Comments

arty-hlr commented Sep 17, 2024

To reproduce:

GMZwinge commented Oct 2, 2024 • edited Loading

Tinyblargon commented Oct 3, 2024

Tinyblargon commented Oct 5, 2024

Tinyblargon commented Oct 5, 2024

Tinyblargon commented Oct 5, 2024

arty-hlr commented Oct 10, 2024

arty-hlr commented Oct 10, 2024

Tinyblargon commented Oct 14, 2024

arty-hlr commented Oct 23, 2024

Tinyblargon commented Oct 23, 2024

arty-hlr commented Oct 23, 2024

arty-hlr commented Oct 23, 2024

Tinyblargon commented Oct 31, 2024

Tinyblargon commented Nov 27, 2024

arty-hlr commented Dec 10, 2024

arty-hlr commented Dec 11, 2024

Tinyblargon commented Dec 11, 2024

arty-hlr commented Dec 11, 2024

Tinyblargon commented Dec 15, 2024

arty-hlr commented Dec 16, 2024

Tinyblargon commented Dec 16, 2024

arty-hlr commented Dec 16, 2024

Tinyblargon commented Dec 16, 2024

arty-hlr commented Dec 17, 2024

Tinyblargon commented Dec 17, 2024

GMZwinge commented Oct 2, 2024 •

edited

Loading