Not able to install eclipse che V7 in azure cloud environment

[echvsg]

Summary

Not able install the eclipse che v7 in azure environment.

Relevant information

I have followed the exact instructions from the eclipse che website to set up in the azure environment and I am stuck at this below error.

Followed the detailed steps as mentioned in this link (https://www.eclipse.org/che/docs/che-7/installing-eclipse-che-on-microsoft-azure).

The domain name that I have created for this test run is "azr.eclispecheide-demo.cloud".

Error:

CertficateYaml.txt
cluster-issuer yaml.txt
dns.go:119] cert-manager/controller/challenges/Check "level"=0 "msg"="checking DNS propagation" "dnsName"="azr.eclipsecheide-demo.cloud" "domain"="azr.eclipsecheide-demo.cloud" "resource_kind"="Challenge" "resource_name"="che-tls-2114626333-0" "resource_namespace"="che" "type"="dns-01" "nameservers"=["10.0.0.10:53"] E1022 15:16:06.259899 1 sync.go:183] cert-manager/controller/challenges "msg"="propagation check failed" "error"="DNS record for "azr.eclipsecheide-demo.cloud" not yet propagated" "dnsName"="azr.eclipsecheide-demo.cloud" "resource_kind"="Challenge" "resource_name"="che-tls-2114626333-0" "resource_namespace"="che" "type"="dns-01" I1022 15:16:06.259963 1 controller.go:135] cert-manager/controller/challenges "level"=0 "msg"="finished processing work item" "key"="che/che-tls-2114626333-0"

I have upgraded the cert-manager from v0.8.1 to v.0.10.1 but no luck. It is stuck in the same place. Let me know if anyone has any suggestions to try and get me out of this error.

[l0rd]

@sagargrsi can you be more precise: on which step of the guide are you getting this error?

[echvsg]

Hi @l0rd ,
I am able to go thru all the steps but encountered error in this section - installing cert-manager (https://www.eclipse.org/che/docs/che-7/installing-eclipse-che-on-microsoft-azure/#installing-cert-manager-on-azure_installing-che-on-microsoft-azure). I am getting stuck at step 8 (Certificate issuance in progress. Temporary certificate issued.), it is not moving forward with the step 9 where it should say "Certificate is up to date and has not expired". When I check the cert-manager pod log files, I can find that the dns domain propagation failure. I do not see a reason for this failure. I have recreated everything with v.0.10.1 certmanager(step 2). The error remains the same.

[sleshchenko]

@sagargrsi Unfortunately, there is no clear hint of what can be wrong in your case.
Could you check that

      clientSecretSecretRef:
            name: azuredns-config
            key: CLIENT_SECRET

if available for the certificate manager? It had to be created during the following step https://www.eclipse.org/che/docs/che-7/installing-eclipse-che-on-microsoft-azure/#creating-a-service-account-secret-on-azure_installing-che-on-microsoft-azure

[echvsg]

@sleshchenko Yes, i have verified and created the client_secret key before applying cluster-issuer and certificate. Do you think any other additional step/instruction that is required is missing in the documentation?. I am creating this in a brand new environment and no other dependencies exists.

[sleshchenko]

@benoitf and me tested it on vanilla Azure K8s installation without dependencies as well. Maybe something is lost during docs migration, maybe something changed.
As an alternative, I can propose you to generate let's encrypt wildcard certificate by hand and provide it for che with secret. And then you would need to regenerate them manually after expiring.

The issue you faced is not really related to Che but to AWS + DNS Challenge + CertManager, so oyu can start with manually generated cert, play with Che and solve certManager issue separately.

Let me know if you need info - how to generate let's encrypt cert manually.

[echvsg]

@sleshchenko, can you please send me the instructions to generate the certs DNS validation manually for azure cloud specific. We have to get this working in azure only.
I was trying to see if any one encountered the same issue but not finding enough posts/links to debug further. The nslookup against the ACME Challenge domain name is also working but the cert is not moving from temporary to final stage. Thanks.

[sleshchenko]

@sagargrsi
I created certificate manually with certbot

certbot certonly --preferred-challenges=dns --manual [email protected] --server https://acme-v02.api.letsencrypt.org/directory -d *.che.azure.codenvy-dev.com --agree-tos

Create TLS secret for che

kubectl create secret tls che-tls --key=privkey.pem --cert=fullchain.pem  -n che

and everything works just fine.
From #13869 (comment)
Certbot is https://certbot.eff.org/

[echvsg]

@sleshchenko
I just have to add these additional steps to go thru the DNS wild card challenge along with the existing documentation?

[sleshchenko]

It's an alternative for steps Enabling the TLS and DNS challenge on Azure and Creating a Service Account Secret on Azure.
Since you already tried to automate certificate generation process, I suppose it should work after you remove cert and tls secret

kubectl delete certificate che-tls -n che
kubectl delete secret che-tls -n che

After that you can jump to
https://www.eclipse.org/che/docs/che-7/installing-eclipse-che-on-microsoft-azure/#installing-che-on-azure-using-the-chectl-command_installing-che-on-microsoft-azure

[echvsg]

@sleshchenko
Can you give permission to view the google document that was created for azure?
I have requested permission from my google account.
https://docs.google.com/document/d/1WSB5VTS0sBask5lE0pyhH5Gp-8qC4xXr8NgckF0b0Z8/edit

[sleshchenko]

@sagargrsi I'm not the owner of that document. Could you share your email here or in Eclipse Mattermost and I believe I'm able to give you read-only access.

[sleshchenko]

cc @benoitf AFAIS you're the document owner

[echvsg]

@sleshchenko, @benoitf : Here is my email address([email protected]).

[benoitf]

@sagargrsi Permissions granted

[echvsg]

@sleshchenko: Do you have any certbot installation instructions that work on azure cloud shell? There is no sudo permissions from the azure cloud shell.

[echvsg]

@sleshchenko : I have used the following command to install in azure cloud shell.
pip install certbot-azure --user
Even after installing certbot-azure, I am getting "certbot: command not found" error. Is there any way to work this without installing the certboot?

[sleshchenko]

@sagargrsi

@sleshchenko: Do you have any certbot installation instructions that work on azure cloud shell? There is no sudo permissions from the azure cloud shell.

You don't need certbot installed on azure cloud. You should be able to install cerbot locally, generate certificate and then upload it to azure k8s as secret. Sounds like it should work for you?

[echvsg]

@sleshchenko: I was trying to install that as well. Do we have certbot installation for windows? I could not find those.

[sleshchenko]

@sagargrsi seems you're right and there is no certbot for windows. Maybe you can use https://certifytheweb.com/ or https://github.com/ebekker/ACMESharp instead but I'm not sure.

[echvsg]

@sleshchenko: I have verified the eclipseche installation documentation from the shared google
docs as well. It is same like from the eclipseche website. Perhaps something was missed or any dependency prior to starting those instructions? Meanwhile i will try the other two options you just mentioned.

[monaka]

@benoitf @sagargrsi It looks there is no CAA record settings in the document.
refs: https://community.letsencrypt.org/t/acme-caa-validationmethods-support/63125

[monaka]

For example. CAA record on pizzafactory.jp that is hosting PizzaFactory/Camino Che based PaaS.
The backend is by AzureDNS + AKS.

$ dig caa pizzafactory.jp

; <<>> DiG 9.10.3-P4-Debian <<>> caa pizzafactory.jp
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 34641
;; flags: qr rd ra; QUERY: 1, ANSWER: 5, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1452
;; QUESTION SECTION:
;pizzafactory.jp.               IN      CAA

;; ANSWER SECTION:
pizzafactory.jp.        3600    IN      CAA     0 issue "digicert.com"
pizzafactory.jp.        3600    IN      CAA     0 issue "letsencrypt.org"
pizzafactory.jp.        3600    IN      CAA     0 issuewild "digicert.com"
pizzafactory.jp.        3600    IN      CAA     0 issuewild "letsencrypt.org"
pizzafactory.jp.        3600    IN      CAA     0 iodef "mailto:[email protected]"

[che-bot]

Issues go stale after 180 days of inactivity. lifecycle/stale issues rot after an additional 7 days of inactivity and eventually close.

Mark the issue as fresh with /remove-lifecycle stale in a new comment.

If this issue is safe to close now please do so.

Moderators: Add lifecycle/frozen label to avoid stale mode.