New blog post about Che deployment on AKS (#29)

Co-authored-by: Ilya Buziuk <[email protected]>
Co-authored-by: Mario Loriedo <[email protected]>

_posts/2022-07-25-installing-eclipse-che-on-aks.adoc

@@ -0,0 +1,195 @@
+---
+title: Installing Eclipse Che on the Azure Kubernetes Service (AKS)
+layout: post
+author: Serguei Gorokhov and Piotr Karatkevich
+description: >-
+  In this blog post, we'll tell the story how we installed Eclipse Che on Microsoft Azure, what challenges we met and how solved them.
+categories: []
+keywords: ['AKS', 'Azure Kubernetes Service', 'AAD', 'Azure Active Directory']
+slug: /@karatkep/installing-eclipse-che-on-aks
+---
+
+We installed {prod} on AKS (Azure Kubernetes Service) cluster integrated with AAD (Azure Active Directory) according to link:https://www.eclipse.org/che/docs/che-7/installation-guide/installing-che-on-microsoft-azure/[these old (7.41) instructions] (there is no recent version of AKS installation documentation). And to our surprise it did not work. Here at Epam Systems we love complex challenges, so we decided not to stop and deep dived into {prod} world to improve interaction with Microsoft Azure.
+
+NOTE: Taking into account that {prod} calls {kubernetes} API server using user's identity tokens generated by Identity Provider, AKS needs to be integrated with AAD, otherwise AKS will reject the token. Details how to integrate AKS with AAD can be found link:https://docs.microsoft.com/en-us/azure/aks/managed-aad[here].
+
+Architecture diagram below can be used to get a context about our case. Please note that some of the components are omitted for the sake of simplicity. But it should be enough to get up to speed and dive into {prod} world together with us.
+
+.Eclipse Che components deployed on AKS
+image::/assets/img/installing-eclipse-che-on-aks/che-in-aks.png[Eclipse Che components deployed on AKS]
+
+== Challenge 1: AKS does not accept `id_token`
+link:https://oauth2-proxy.github.io/oauth2-proxy/[OAuth2 Proxy] is configured to send an `id_token` to Traefik for installations made for {kubernetes}. link:https://traefik.io/traefik/[Traefik], in turn, sends the `id_token` to all upstream services, including components such as Che server and User Dashboard. Che server and User Dashboard call AKS API Server using the provided `id_token`. But when AKS is configured to use AAD, its API Server does not accept `id_token` (it only accepts `access_token`).
+
+=== How did we solve the first challenge
+We need a way to ask OAuth2 Proxy to send an `access_token` instead of an `id_token`. Token replacement is a OAuth2 Proxy alpha feature, hence it cannot be used in production. The alternative is to configure it to pass `access_token` via X-Forwarded-Access-Token header, but the `id_token` in the Authorization header won't be replaced. So, first step is to add `access_token` via X-Forwarded-Access-Token header:
+
+[source]
+----
+pass_access_token = true
+----
+
+Then we need somehow to replace Authorization header on X-Forwarded-Access-Token header. Fortunately, we have Traefik in place. So, let's ask Traefik to make a necessary replacement:
+[source,yaml]
+----
+http:
+  middlewares:
+    che-header-rewrite:
+      plugin:
+        header-rewrite:
+          from: X-Forwarded-Access-Token
+          prefix: 'Bearer '
+          to: Authorization
+----
+
+And voila, we made it work, like magic. But how can we make it usable for other users who wants to install {prod} on AKS? We already mentioned, that at Epam Systems we love complex challenges. And of course we love coding. So, we opened the link:https://github.com/eclipse/che/issues/21450[che-operator enhancements] and proposed the link:https://github.com/eclipse-che/che-operator/pull/1400[improvements].
+
+As a consequence of the proposed improvements, a new configuration field `identityToken` has been introduced into {prod} starting from version 7.50. A {prod} administrator can easily configure what token will be passed to upstream servicees in CheCluster CR (Custom Resource):
+[source,yaml]
+----
+spec:
+  networking:
+    auth:
+      identityToken: access_token
+----
+There are two types of token supported: `id_token` and `access_token`. Default value is `id_token`.
+
+NOTE: The field `identityToken` is specific to {prod} installations made for Kubernetes only and ignored for OpenShift.
+
+== Challenge 2: No way to obtain a proper access token accepted by AKS
+This is a follow up of Challenge 1: after configuring {prod} to pass an `access_token` instead of an `id_token` we need to make sure that AKS considers it valid.
+
+When AKS integration with AAD is enabled, AKS expects OpenID Connect (OIDC) tokens to be issued from an application, pre-registered in AAD: 'Azure Kubernetes Service AAD Server'.
+
+.Azure Kubernetes Service AAD Server application
+image::/assets/img/installing-eclipse-che-on-aks/aks-aad-server-app.png[Azure Kubernetes Service AAD Server]
+
+{prod} Gateway OAuth2 Proxy with the default settings uses a standard set of OIDC scopes and AAD will return an `access_token` that can be used against Microsoft Graph but not against AKS.
+
+In order to return a token that will be accepted by AKS, OAuth2 Proxy should be configured with an additional OIDC scope, representing Azure Kubernetes Service AAD Server application:
+[source,yaml]
+----
+6dae42f8-4368-4678-94ff-3960e28e3630/user.read
+----
+where the ID is the Application ID of the AKS instance registered in AAD.
+
+=== How did we solve the second challenge
+A new configuration field named `oAuthScope` has been introduced into {prod} starting from version 7.50. That has been specified by the already opened link:https://github.com/eclipse/che/issues/21450[che-operator enhancements] and implemented through the provided link:https://github.com/eclipse-che/che-operator/pull/1400[improvement]. An {prod} administrator can easily configure authorization scopes in CheCluster CR:
+[source,yaml]
+----
+spec:
+  networking:
+    auth:
+      oAuthScope: openid email profile 6dae42f8-4368-4678-94ff-3960e28e3630/user.read
+----
+NOTE: The field `oAuthScope` is specific to {prod} installations made for {kubernetes} only and ignored for OpenShift.
+
+== Challenge 3: Che server requires mandatory 'email' claim which is absent in AKS access_token
+Besides passing OIDC token to AKS for authorization purposes, Che server also uses it for creating a 'user' record in PostgreSQL database. There are two pieces of information (claims) about current user that Che server expects to extract from the token: name and email. Che allows configuring of user name claim via CheCluster CR in AKS (`CHE_OIDC_USERNAME__CLAIM` according to link:https://www.eclipse.org/che/docs/next/administration-guide/advanced-configuration-options-for-the-che-server-component/#_che_oidc_username_claim[documentation]), but email claim name is hardcoded to `email` and cannot be changed. This hardcode becomes a problem for AKS setup since `access_token` returned by AAD for AKS does not contain `email` claim (see current structure below), and customization is not possible here since AKS application registration in AAD is maintained by Microsoft.
+[source,jwt]
+----
+{
+  "typ": "JWT",
+  "alg": "RS256",
+  "x5t": "2ZQpJ3UpbjAYXYGaXEJl8lV0TOI",
+  "kid": "2ZQpJ3UpbjAYXYGaXEJl8lV0TOI"
+}.{
+  "aud": "6dae42f8-4368-4678-94ff-3960e28e3630",
+  "iss": "https://sts.windows.net/f974903a-f693-4ca2-aa63-45e5bd2d1318/",
+  "iat": 1656412511,
+  "nbf": 1656412511,
+  "exp": 1656417802,
+  "acr": "1",
+  "aio": "AUQAu/8TAABB1o0F/h6L8SJWLibrQqPKrVF7qpdIFiQ0qeIj/wNUfcLkM87v+wgvQ+I8uj0WC2Y6WEoHlBcsJdt0m+kJ4dFeNw==",
+  "amr": [
+    "pwd"
+  ],
+  "appid": "1c0b88a0-8ec2-4cca-a2b0-4f468110ef86",
+  "appidacr": "1",
+  "family_name": "Kluklu",
+  "given_name": "Tratata",
+  "groups": [
+    "73fdccdf-e10d-45f9-b7f6-31848842999f",
+    "5988d043-3af9-4e81-b041-90b3456f9f4e",
+    "bddcb049-0337-4dec-bbaf-7600b8c12623"
+  ],
+  "ipaddr": "10.123.51.3",
+  "name": "Tratata Kluklu",
+  "oid": "2cf0521c-c76d-4e7c-b41f-863674057db3",
+  "onprem_sid": "S-2-4-31-6364504-298352422-13854118-387761",
+  "puid": "3213CDFA3CAF2IA5",
+  "rh": "0.AQkA1NIbtJ39JkuKaqbJ82fCHscCrm1oG3hTlP47YOQHDjAJAHg.",
+  "scp": "user.read",
+  "sub": "qweBLvHX49QA5WlXpJzq_erXQ2NldnSqpgY93oALLDY",
+  "tid": "a385e78a-aedc-4033-82ba-e6ef88120591",
+  "unique_name": "[email protected]",
+  "upn": "[email protected]",
+  "uti": "lfZmPsgcWmS3dG78GpMjRA",
+  "ver": "1.0",
+  "wids": [
+    "c79abafb-610b-4a34-82e2-ef7a293db6ca"
+  ]
+}.[Signature]
+----
+
+
+=== How did we solve the third challenge
+As for the previous challenges, we need some enhancements on {prod} side here too. We want to allow administrators to configure what token claim need to be used to extract user email. As we did it before, we opened the link:https://github.com/eclipse/che/issues/21515[che-server enhancement] and proposed the link:https://github.com/eclipse-che/che-server/pull/324[improvement].
+
+Now {prod} adminstrators can configure the email claim to be used when parsing the JWT token:
+[source,yaml]
+----
+spec:
+  components:
+    cheServer:
+      extraProperties:
+        CHE_OIDC_EMAIL__CLAIM: unique_name
+----
+If not defined, the fallback value is `email`.
+
+== Conclusion
+In this post, we walked through the challenges we faced to install {prod} on AKS and how we contributed back to the project to address the issues.
+
+Now, user has all needed things configurable to be able to run successfully {prod} on AKS. For example, in our particular case we prepared yaml file that overrides the default values in CheCluster CR.
+[source,yaml]
+----
+spec:
+  networking:
+    auth:
+      identityProviderURL: https://sts.windows.net/{TENANT_ID}/v2.0/
+      identityToken: access_token
+      oAuthClientName: {CLIENT_ID}
+      oAuthSecret: {CLIENT_SECRET}
+      oAuthScope: openid email profile 6dae42f8-4368-4678-94ff-3960e28e3630/user.read
+  components:
+    cheServer:
+      extraProperties:
+        CHE_OIDC_AUTH__SERVER__URL: https://sts.windows.net/{TENANT_ID}/v2.0/
+        CHE_OIDC_EMAIL__CLAIM: unique_name 
+----
+
+* `TENANT_ID` - Directory (tenant) ID, see Figure 3.
+* `CLIENT_ID` - Application (client) ID, see Figure 3.
+* `CLIENT_SECRET` - Client secret, you can manage it in 'Certificates & secret' section
+
+.Registered Che application
+image::/assets/img/installing-eclipse-che-on-aks/azure-che-demo-app.png[Registered Che application]
+
+WARNING: Don't forget to configure API permissions to authorize your application to call AKS Server API.
+
+.AKS API permissions
+image::/assets/img/installing-eclipse-che-on-aks/aks-api-permissions.png[AKS API permissions]
+
+After the {prod} App configuration in Azure is completed, the command `chectl server:deploy` can be used to install {prod} on AKS using the `YAML` file above:
+[source,shell]
+----
+chectl server:deploy \
+       --platform=k8s \
+       --installer=operator \
+       --che-operator-cr-patch-yaml=che.yaml \
+       --skip-oidc-provider-check \
+       --skip-cert-manager \
+       --domain=eclipse-che-demo.mydomain.com
+----
+
+NOTE: In our case we already configured `cert-manager` and created `domain` according to the link:https://www.eclipse.org/che/docs/che-7/installation-guide/installing-che-on-microsoft-azure/[old (7.41) instructions].