Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

InvalidLoadBalancerProfile in AKS #6525

Closed
MikeMichel opened this issue Apr 17, 2020 · 28 comments · Fixed by #6534
Closed

InvalidLoadBalancerProfile in AKS #6525

MikeMichel opened this issue Apr 17, 2020 · 28 comments · Fixed by #6534

Comments

@MikeMichel
Copy link

After deplyoing an app with a load balancer into a fresh AKS cluster with no load balancer profile configured (only sku to Standard) and then changing for example number of nodes gives me:

Error: Code="InvalidLoadBalancerProfile" Message="Load balancer profile must
specify one of ManagedOutboundIPs, OutboundIPPrefixes and OutboundIPs." 
Target="networkProfile.loadBalancerProfile"

Output of terrafom apply which is causing it I guess:

            load_balancer_profile {
                effective_outbound_ips    = [
                    "/subscriptions/..../publicIPAddresses/.....",
                ]
                managed_outbound_ip_count = 1
                outbound_ip_address_ids   = []
                outbound_ip_prefix_ids    = []
            }
        }

Community Note

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
  • Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform (and AzureRM Provider) Version

Terraform v0.12.23

  • provider.azurerm v2.6.0
  • #0000
@ablyler
Copy link

ablyler commented Apr 17, 2020

I am having the same issue after upgrading to azure provider v2.6.0. I have tried to fix the issue by adding the load_balancer_profile block as so:

  network_profile {
    network_plugin    = "azure"
    load_balancer_sku = "standard"
    load_balancer_profile {
      managed_outbound_ip_count = 1
    }
  }

However, I get the same error in the description of this issue.

@asubmani
Copy link
Contributor

I am getting the same error. I am using terraform v0.12.24 and using azurerm provider v2.6.0

Provisioned the AKS cluster successfully.
Terraform Plan about "Changing" SPN for AKS RBAC. # I did not make any changes to SPN details as my cluster is using Managed Identities.

I get the error only during terraform apply

module.AKSClus.azurerm_kubernetes_cluster.TerraAKSwithRBAC: Still modifying
... [id=/subscriptions/xxxx-1193-4xxx-xxxxx7-...rService/managedClusters/
jb-aks-eus-qa, 1m0s elapsed]

Error: updating Managed Kubernetes Cluster "jb-aks-eus-qa" (Resource Group
"xx-rg-aks-qa"): containerservice.ManagedClustersClient#CreateOrUpdate: Fai
lure sending request: StatusCode=400 -- Original Error: Code="InvalidLoadBa
lancerProfile" Message="Load balancer profile must specify one of ManagedOu
tboundIPs, OutboundIPPrefixes and OutboundIPs." Target="networkProfile.load
BalancerProfile"

@aristosvo
Copy link
Collaborator

Ah, shoot! I just pushed a fix PR.

I've spent some thoughts on this scenario during implementation of the feature causing this issue, but didn't implement the right test for it and forgot!

@gitflo1
Copy link

gitflo1 commented Apr 20, 2020

Getting the same error as described above for the following scenario, using terraform version v.0.12.24 while upgrading from AKS version 0.15.7 to 0.16.7 with the following config:

  network_profile {
    network_plugin     = "azure"
    network_policy     = "azure"
    dns_service_ip     = var.aks_dns_service_ip
    docker_bridge_cidr = var.aks_docker_bridge_cidr
    service_cidr       = var.aks_service_cidr
    outbound_type = "loadBalancer"

    // For egress configuration
    load_balancer_sku  = "Standard"
    load_balancer_profile {
      outbound_ip_address_ids = [ var.aks_egress_ip_id ]
    }
  }

Terraform tries an update in place and fails with the error message displayed in the original post.

@bytebounder
Copy link

As an interim measure, I rolled back my azurerm provider version to 2.5.0.

@aristosvo
Copy link
Collaborator

@tombuildsstuff Can we tag this as a /bug? The fix is already available, including a test reproducing the failure.

@robologic et all: as a workaround you can also change the number of outbound IP's for every change, the problem shows when no change in the load_balancer_profile is applied. In that case the load_balancer_profile configuration is not included in the request towards Azure 😢

@tezzigator
Copy link

tezzigator commented Apr 22, 2020

As an interim measure, I rolled back my azurerm provider version to 2.5.0.

This doesnt work for me as I need 2.6.0 for the kubelet exports.

I believe I have the same issue here on 2.6.0; I can create the AKS in Devops no problems, but if later then I try to change on that AKS with terraform apply via Devops for any change on that AKS, it doesnt have to be LB or IP related, for example, changing the IP blocks allowed into the K8S API, then Devops fails with

2020-04-22T03:33:15.8987559Z �[1m�[31mError: �[0m�[0m�[1mupdating Managed Kubernetes Cluster "REDACTED" (Resource Group "REDACTED"): containerservice.ManagedClustersClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="InvalidLoadBalancerProfile" Message="Load balancer profile must specify one of ManagedOutboundIPs, OutboundIPPrefixes and OutboundIPs." Target="networkProfile.loadBalancerProfile"�[0m
2020-04-22T03:33:15.8989767Z 
2020-04-22T03:33:15.8990917Z �[0m  on modules/aks/main.tf line 7, in resource "azurerm_kubernetes_cluster" "aks":
2020-04-22T03:33:15.8992918Z    7: resource "azurerm_kubernetes_cluster" "aks" �[4m{�[0m
2020-04-22T03:33:15.8993692Z �[0m
2020-04-22T03:33:15.8994180Z �[0m�[0m
2020-04-22T03:33:16.0389462Z ##[error]Error: The process '/opt/hostedtoolcache/terraform/0.12.24/x64/terraform' failed with exit code 1

@chrishatfield86
Copy link

chrishatfield86 commented Apr 23, 2020

I'm seeing the same behaviour on 2.6.0 and don't wish to roll back at this stage. I can initially provision a cluster fine with a load_balancer_profile block but any subsequent terraform apply will fail. These subsequent applies having nothing in the plan indicating any change required for the network_profile and load_balancer_profile

##[error]Terraform command 'apply' failed with exit code '1'.:  updating Managed Kubernetes Cluster "my_cluster" (Resource Group "my_rg"): containerservice.ManagedClustersClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="InvalidLoadBalancerProfile" Message="Load balancer profile must specify one of ManagedOutboundIPs, OutboundIPPrefixes and OutboundIPs." Target="networkProfile.loadBalancerProfile"

@titilambert
Copy link
Contributor

Hello,
I just tested the master branch (v2.6 ++ ) and it worked !
Thanks @aristosvo

@asubmani
Copy link
Contributor

I am still getting this error with provider 2.7.0. I have to keep changing managed_outbound_ip_count = 1 to a different number each time I run my plan & apply.
The plan does not catch this (unless I change the value); but the terraform apply command throws this error.

@aristosvo
Copy link
Collaborator

@asubmani I'm sorry, it is not merged yet. If any of you can thoroughly review #6534 it might speed up things a bit, but other than that I can do nothing.

👍 on the issue is the best way for the maintainers to indicate the need for the fix.

@jmcshane
Copy link
Contributor

jmcshane commented Apr 27, 2020

I'm still having an issue now on 2.7.0, my load balancer profile just sets managed_outbound_ip_count but it is failing with the same message.

@asubmani
Copy link
Contributor

asubmani commented Apr 29, 2020

@asubmani I'm sorry, it is not merged yet. If any of you can thoroughly review #6534 it might speed up things a bit, but other than that I can do nothing.

👍 on the issue is the best way for the maintainers to indicate the need for the fix.

Thank you... so I fork your repo from #6534 and build it using make for golang? I am not a developer, but if you could point me to some basic doc I can do my bit.

@angelbarrera92
Copy link

This was working in 2.5.0

@aristosvo
Copy link
Collaborator

It is still working in 2.5.0!

To make it possible to change your loadbalancer outbound IP settings without recreating the cluster I filled a PR, which is merged into 2.6.0. Due to this PR the loadbalancer outbound settings can be changed. But if these settings are not changed a bug in the code causes your AKS cluster to think you have no outbound loadbalancer settings, which blocks any other kind of change you want to make to your cluster.

The good news is 2.5.0 is still working and just limited functionality has made it past that. If you need 2.6+ functionality, this sad issue is blocking cluster changes after cluster creation.

@jmcshane
Copy link
Contributor

So does this mean that clusters created with the provider before 2.5 are not able to be modified if the provider version is bumped above 2.6? Is there an ignore value that we can set?

@markslater
Copy link

markslater commented May 11, 2020

This issue (in my experience), means that all versions of the provider > 2.5 are broken for AKS, regardless of the version of the provider used to create the AKS cluster.

It's actually a rather insidious break, because you can create AKS clusters using newer versions of the provider, but the first time you have a modification, terraform apply will crap out every time.

To me, this appears to be a grade A drop-everything-and-fix-it issue because to all intents and purposes the past four provider releases are fundamentally broken for all AKS users... please excuse me if I've got the wrong end of the stick and there's a workaround I missed!

@aristosvo
Copy link
Collaborator

I discussed it with the maintainers, if it doesn’t make it into 2.10.0 (May 14th) expect it for 2.11.0.

@atrauzzi
Copy link

Just encountered this. This is a pretty nasty issue and I have to agree with @markslater, this needs to be fixed and released sooner than a regular release cycle.

Can we ping the maintainers in here for discussion? Do they ascertain the severity of this or can they provide sound reasoning?

@djsly
Copy link
Contributor

djsly commented May 12, 2020

is this PR the official one ? #6534. we need a custom build as this is breaking all cluster update :(

@viresh-contino
Copy link

viresh-contino commented May 12, 2020

I have just observed this with 2.9.0.
this is the error message:
containerservice.ManagedClustersClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="InvalidLoadBalancerProfile" Message="Load balancer profile must specify one of ManagedOutboundIPs

I basically did a terraform plan, terraform apply and then another terraform plan and terraform apply.

@ghost
Copy link

ghost commented May 15, 2020

This has been released in version 2.10.0 of the provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading. As an example:

provider "azurerm" {
    version = "~> 2.10.0"
}
# ... other configuration ...

@atrauzzi
Copy link

Just linked in the issue above. If you're looking to upgrade and are using the Basic SKU of postgres, you'll encounter another show-stopper bug.

@aristosvo
Copy link
Collaborator

It’s quit easy to separate the versions between two resources, although I feel your pain.

Bugs like these on advanced resources are not always easy to catch though..

@atrauzzi
Copy link

To be honest, I don't feel like I've been doing anything advanced. I've hit so many bugs and gotchas just trying to provision a resource group with a database, gateway, AKS and key store.

All low tier and lots of defaults.

@nivhty
Copy link

nivhty commented May 19, 2020

I'm having the same issue as well on:

  • Terraform v0.12.24
  • Azurerm provider 2.8.0
    When I do a re-apply after adding a tags block in the code, I was getting the same error message:
Error: Code="InvalidLoadBalancerProfile" Message="Load balancer profile must
specify one of ManagedOutboundIPs, OutboundIPPrefixes and OutboundIPs." 
Target="networkProfile.loadBalancerProfile"

My network_profile block is below:

  network_profile {
    network_plugin     = var.cluster_network_plugin
    dns_service_ip     = var.cluster_dns_service_ip
    docker_bridge_cidr = var.cluster_docker_bridge_cidr
    service_cidr       = var.cluster_service_cidr

    # Standard LoadBalancer is required for multiple nodepools.
    load_balancer_sku = "standard"

    # Use a single IP for outbound traffic
    load_balancer_profile {
      outbound_ip_address_ids = [azurerm_public_ip.outbound.id]
    }
  }

@aristosvo
Copy link
Collaborator

I'm having the same issue as well on:

  • Terraform v0.12.24

  • Azurerm provider 2.8.0

@nivhty upgrade to 2.10.0 for the fix.

@ghost
Copy link

ghost commented Jun 12, 2020

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!

@ghost ghost locked and limited conversation to collaborators Jun 12, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet