[Question] how to update the airflow version in Self-managed Apache Airflow deployment for EKS.

[taeyoung94]

• ✋ I have searched the open/closed issues and my issue is not listed.

i am wondering how to update the airflow version in airflow on eks.

While using the given version which is 2.3.0, I was getting the error "Prevent KubernetesJobWatcher getting stuck on resource too old". apache/airflow#23521
this error was fixed in the 2.3.1 airflow version, so I want to update the version as well.

here is what I tried.
i thought I should change this, but it didn't work.
my error was like this.

"│ Error: Failed to query available provider packages
│
│ Could not retrieve the list of available versions for provider hashicorp/local: the previously-selected version 2.3.1 is no longer available"

not only 2.3.1 version as well as versions higher than 2.3 failed.

provider "registry.terraform.io/hashicorp/local" {
version = "2.3.0"
constraints = ">= 2.1.0"
hashes = [
"h1:+l9ZTDGmGdwnuYI5ftUjwP8UgoLw4f4V9xoCzal4LW0=",
"zh:1f1920b3f78c31c6b69cdfe1e016a959667c0e2d01934e1a084b94d5a02cd9d2",
"zh:550a3cdae0ddb350942624e7b2e8b31d28bc15c20511553432413b1f38f4b214",
"zh:68d1d9ccbfce2ce56b28a23b22833a5369d4c719d6d75d50e101a8a8dbe33b9b",
"zh:6ae3ad6d865a906920c313ec2f413d080efe32c230aca711fd106b4cb9022ced",
"zh:78d5eefdd9e494defcb3c68d282b8f96630502cac21d1ea161f53cfe9bb483b3",
"zh:a0f413d50f54124057ae3dcd9353a797b84e91dc34bcf85c34a06f8aef1f9b12",
"zh:a2ac6d4088ceddcd73d88505e18b8226a6e008bff967b9e2d04254ef71b4ac6b",
"zh:a851010672e5218bdd4c4ea1822706c9025ef813a03da716d647dd6f8e2cffb0",
"zh:aa797561755041ef2fad99ee9ffc12b5e724e246bb019b21d7409afc2ece3232",
"zh:c6afa960a20d776f54bb1fc260cd13ead17280ebd87f05b9abcaa841ed29d289",
"zh:df0975e86b30bb89717b8c8d6d4690b21db66de06e79e6d6cfda769f3304afe6",
"zh:f0d3cc3da72135efdbe8f4cfbfb0f2f7174827887990a5545e6db1981f0d3a7c",
]
}

is there anyway to update version of airflow?

appreciate it.

[vara-bonthu]

Airflow Blueprint is using Airflow 2.3.3 currently. You can update the Helm Values here

Airflow Helm Version can be updated here

[taeyoung94]

yes it says 2.3.3, but when I deployed it, it wasn't.

here is the image of the deployment of airflow-scheduler

so I tried to extract yaml file from the deployment of airflow-scheduler, and change the airflow version.
but it didn't work as well. the pod could not initialize

is there anyway to update version besides modifying Helm the values?
and Am I the only one experiencing this?

thank you for you help.

[taeyoung94]

I found the updated document of Airflow Helm Chart 1.6.0. It says that the default version of airflow image is 2.3.0, which might be the cause of the problem.

Could you please explain how to update the airflow helm chart to 1.7? I would appreciate your help.

Or do you know when the next version will be released? I can wait for it if it’s coming soon.

[vara-bonthu]

Airflow Blueprint is using Airflow 2.3.3 currently. You can update the Helm Values here

Airflow Helm Version can be updated here

See this reply and the links. You just have to bump the latest version of Helm Chart for Airflow and the values file.

NOTE: We haven't tested the upgrade yet. Please feel free to test latest version and let us know how it went.

[mfjackson]

@taeyoung94 Just to expand on what @vara-bonthu said, the value of airflowVersion in your values.yaml file must match the version of Airflow being deployed in the Docker image being referenced by the Helm chart. So it would be 2.3.3 based on the current image used by this project or if you were to use a custom Docker image built on, say, apache/airflow:2.5.0-python3.10, the value of airflowVersion in the values.yaml chart would have to be explicitly set to 2.5.0.

[vara-bonthu]

FYI...I am working on updating the Airflow blueprint with latest versions. I will keep you posted once its ready.

[vara-bonthu]

@jagpk Please validate this issue. If its already addressed with you recent upgrade then it can be closed.

[jagpk]

@taeyoung94 we have upgraded the Airflow blueprint to Helm chart 1.9.0 and Airflow application version 2.5.3 in #230. There are major changes in the Airflow values.yaml in the version 2.5.3, also the updated blueprint uses the new centralized terraform-aws-eks-data-addons modules to deploy Apache Airflow Helm Chart. Please ensure you use the latest Airflow blueprint to upgrade Airflow instead of just changing Airflow version in previous version of blueprint. Once you have updated the blueprint as per #230 then going forward to upgrade Airflow application version you need to do ensure that below steps are done correctly:

1. Update Airflow Application version by changingairflow_version value in schedulers/terraform/self-managed-airflow/locals.tf
2. Update Airflow Helm chart version by changing the value in modules/terraform-aws-eks-data-addons/main.tf.
3. Update the airflow-values.yaml in the blueprint with matching Airflow values.yaml.

[jagpk]

@vara-bonthu the question has been addressed by the latest upgrade #230. We can close this issue now.

[github-actions]

Issue closed due to inactivity.