Avoid requesting new certs inside letsencrypt:enable if valid certs exist

[ikalkov]

Description of problem

The dokku letsencrypt:enable command seems to request new certificates from the server each time it is executed, regardless of whether or not valid certs already exist for that app. I am not sure if this is the desired behavior or if it makes sense to check whether a valid certificate already exists and not to request a new one if the existing one is fine?

We use letsencrypt:enable in our CI scripts during automatic deployment of review branches (which are being updated several times a day). It works fine for the first 5 commits, but then we run into rate limits (see results below). I am aware that we could extend our scripts to somehow check whether the branch was already deployed in the past and use letsencrypt:auto-renew instead, but it sounds way more complicated than implementing one additional check inside letsencrypt:enable.

How reproducible

1. Create a new app or clone an existing app
2. Run letsencrypt:enable for that app 1st time (succeeds)
3. Run letsencrypt:enable for that app 2nd time (succeeds)
4. Run letsencrypt:enable for that app 3rd time (succeeds)
5. Run letsencrypt:enable for that app 4th time (succeeds)
6. Run letsencrypt:enable for that app 5th time (succeeds)
7. Run letsencrypt:enable for that app 6th time (fails):

$ dokku letsencrypt:enable web-test
=====> Enabling letsencrypt for web-test
-----> Enabling ACME proxy for web-test...
       Reloading nginx configuration (via systemctl): nginx.service.
-----> Getting letsencrypt certificate for web-test...
        - Domain 'web-test.stage.com'
       2022/05/05 08:03:56 [INFO] [web-test.stage.com] acme: Obtaining bundled SAN certificate
       2022/05/05 08:03:56 Could not obtain certificates:
        acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rateLimited :: Error creating new order :: too many certificates (5) already issued for this exact set of domains in the last 168 hours: web-test.stage.com: see https://letsencrypt.org/docs/rate-limits/
-----> Certificate retrieval failed!
-----> Disabling ACME proxy for web-test...
       Reloading nginx configuration (via systemctl): nginx.service.
 !     Failed to setup letsencrypt
 !     Check log output for further information on failure

Actual Results

First deployment is fine:

$ ssh "ssh://dokku@$(parse-ssh-host):$(parse-ssh-port)" -- letsencrypt:enable $REVIEW_APP_NAME
=====> Enabling letsencrypt for web-review-setup-secu-9glcxl
-----> Enabling ACME proxy for web-review-setup-secu-9glcxl...
       Reloading nginx configuration (via systemctl): nginx.service.
-----> Getting letsencrypt certificate for web-review-setup-secu-9glcxl...
        - Domain 'web-review-setup-secu-9glcxl.stage.com'
       2022/04/28 10:44:55 No key found for account [email protected]. Generating a P256 key.
       2022/04/28 10:44:55 Saved key to /certs/accounts/acme-v02.api.letsencrypt.org/[email protected]/keys/[email protected]
       2022/04/28 10:44:55 [INFO] acme: Registering account for [email protected]
       !!!! HEADS UP !!!!
       
       Your account credentials have been saved in your Let's Encrypt
       configuration directory at "/certs/accounts".
       
       You should make a secure backup of this folder now. This
       configuration directory will also contain certificates and
       private keys obtained from Let's Encrypt so making regular
       backups of this folder is ideal.
       2022/04/28 10:44:55 [INFO] [web-review-setup-secu-9glcxl.stage.com] acme: Obtaining bundled SAN certificate
       2022/04/28 10:44:56 [INFO] [web-review-setup-secu-9glcxl.stage.com] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/102952361997
       2022/04/28 10:44:56 [INFO] [web-review-setup-secu-9glcxl.stage.com] acme: Could not find solver for: tls-alpn-01
       2022/04/28 10:44:56 [INFO] [web-review-setup-secu-9glcxl.stage.com] acme: use http-01 solver
       2022/04/28 10:44:56 [INFO] [web-review-setup-secu-9glcxl.stage.com] acme: Trying to solve HTTP-01
       2022/04/28 10:44:56 [INFO] [web-review-setup-secu-9glcxl.stage.com] Served key authentication
       2022/04/28 10:44:57 [INFO] [web-review-setup-secu-9glcxl.stage.com] Served key authentication
       2022/04/28 10:44:57 [INFO] [web-review-setup-secu-9glcxl.stage.com] Served key authentication
       2022/04/28 10:44:57 [INFO] [web-review-setup-secu-9glcxl.stage.com] Served key authentication
       2022/04/28 10:45:02 [INFO] [web-review-setup-secu-9glcxl.stage.com] The server validated our request
       2022/04/28 10:45:02 [INFO] [web-review-setup-secu-9glcxl.stage.com] acme: Validations succeeded; requesting certificates
       2022/04/28 10:45:03 [INFO] [web-review-setup-secu-9glcxl.stage.com] Server responded with a certificate.
-----> Certificate retrieved successfully.

Later deployments (6+) fail:

$ ssh "ssh://dokku@$(parse-ssh-host):$(parse-ssh-port)" -- letsencrypt:enable $REVIEW_APP_NAME
=====> Enabling letsencrypt for web-review-setup-secu-9glcxl
-----> Enabling ACME proxy for web-review-setup-secu-9glcxl...
       Reloading nginx configuration (via systemctl): nginx.service.
-----> Getting letsencrypt certificate for web-review-setup-secu-9glcxl...
        - Domain 'web-review-setup-secu-9glcxl.stage.com'
       2022/05/02 15:02:54 [INFO] [web-review-setup-secu-9glcxl.stage.com] acme: Obtaining bundled SAN certificate
       2022/05/02 15:02:54 Could not obtain certificates:
       	acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rateLimited :: Error creating new order :: too many certificates (5) already issued for this exact set of domains in the last 168 hours: web-review-setup-secu-9glcxl.stage.com: see https://letsencrypt.org/docs/rate-limits/
-----> Certificate retrieval failed!
-----> Disabling ACME proxy for web-review-setup-secu-9glcxl...
       Reloading nginx configuration (via systemctl): nginx.service.
 !     Failed to setup letsencrypt
 !     Check log output for further information on failure

Expected Results

Reuse existing certificates in letsencrypt:enable instead of requesting new ones.

Environment Information

Limited to relevant information. I will provide more, if required:

$ dokku report web-review-setup-secu-9glcxl
-----> uname: Linux stage 5.4.0-96-generic #109-Ubuntu SMP Wed Jan 12 16:49:16 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
-----> memory: 
                     total        used        free      shared  buff/cache   available
       Mem:          31974        2837        2602          72       26535       28621
       Swap:         16366         509       15857
-----> docker version: 
       Client: Docker Engine - Community
        Version:           20.10.12
        API version:       1.41
        Go version:        go1.16.12
        Git commit:        e91ed57
        Built:             Mon Dec 13 11:45:33 2021
        OS/Arch:           linux/amd64
        Context:           default
        Experimental:      true
       
       Server: Docker Engine - Community
        Engine:
         Version:          20.10.12
         API version:      1.41 (minimum version 1.12)
         Go version:       go1.16.12
         Git commit:       459d0df
         Built:            Mon Dec 13 11:43:42 2021
         OS/Arch:          linux/amd64
         Experimental:     false
        containerd:
         Version:          1.4.12
         GitCommit:        7b11cfaabd73bb80907dd23182b9347b4245eb5d
        runc:
         Version:          1.0.2
         GitCommit:        v1.0.2-0-g52b36a2
        docker-init:
         Version:          0.19.0
         GitCommit:        de40ad0
-----> docker daemon info: 
       Client:
        Context:    default
        Debug Mode: true
        Plugins:
         app: Docker App (Docker Inc., v0.9.1-beta3)
         buildx: Docker Buildx (Docker Inc., v0.7.1-docker)
       
       Server:
        Containers: 79
         Running: 25
         Paused: 0
         Stopped: 54
        Images: 2368
        Server Version: 20.10.12
        Storage Driver: overlay2
         Backing Filesystem: extfs
         Supports d_type: true
         Native Overlay Diff: true
         userxattr: false
        Logging Driver: json-file
        Cgroup Driver: cgroupfs
        Cgroup Version: 1
        Plugins:
         Volume: local
         Network: bridge host ipvlan macvlan null overlay
         Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
        Swarm: inactive
        Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
        Default Runtime: runc
        Init Binary: docker-init
        containerd version: 7b11cfaabd73bb80907dd23182b9347b4245eb5d
        runc version: v1.0.2-0-g52b36a2
        init version: de40ad0
        Security Options:
         apparmor
         seccomp
          Profile: default
        Kernel Version: 5.4.0-96-generic
        Operating System: Ubuntu 20.04.4 LTS
        OSType: linux
        Architecture: x86_64
        CPUs: 8
WARNING: No swap limit support
        Total Memory: 31.23GiB
        Name: stage
        ID: KVOP:J6QX:CGV2:2JNV:QP6B:AU6L:GA5L:YEQF:72F5:NDHM:WUFE:GF2V
        Docker Root Dir: /var/lib/docker
        Debug Mode: false
        Registry: https://index.docker.io/v1/
        Labels:
        Experimental: false
        Insecure Registries:
         127.0.0.0/8
        Live Restore Enabled: false
       
-----> git version: git version 2.25.1
-----> sigil version: 0.7.1build+1ec3006
-----> herokuish version: 
       herokuish: 0.5.34
       buildpacks:
         heroku-buildpack-multi     v1.0.0
         heroku-buildpack-ruby      v236
         heroku-buildpack-nodejs    v189
         heroku-buildpack-clojure   v87
         heroku-buildpack-python    v205
         heroku-buildpack-java      v69
         heroku-buildpack-gradle    v36
         heroku-buildpack-scala     v91
         heroku-buildpack-play      v26
         heroku-buildpack-php       v205
         heroku-buildpack-go        v160
         heroku-buildpack-nginx     v16
         buildpack-null             v3
-----> dokku version: dokku version 0.26.7
-----> plugn version: plugn: 0.7.1build+9bf216b
-----> dokku plugins: 
         00_dokku-standard    0.26.7 enabled    dokku core standard plugin
         20_events            0.26.7 enabled    dokku core events logging plugin
         app-json             0.26.7 enabled    dokku core app-json plugin
         apps                 0.26.7 enabled    dokku core apps plugin
         builder              0.26.7 enabled    dokku core builder plugin
         builder-dockerfile   0.26.7 enabled    dokku core builder-dockerfile plugin
         builder-herokuish    0.26.7 enabled    dokku core builder-herokuish plugin
         builder-null         0.26.7 enabled    dokku core builder-null plugin
         builder-pack         0.26.7 enabled    dokku core builder-pack plugin
         buildpacks           0.26.7 enabled    dokku core buildpacks plugin
         certs                0.26.7 enabled    dokku core certificate management plugin
         checks               0.26.7 enabled    dokku core checks plugin
         common               0.26.7 enabled    dokku core common plugin
         config               0.26.7 enabled    dokku core config plugin
         cron                 0.26.7 enabled    dokku core cron plugin
         docker-options       0.26.7 enabled    dokku core docker-options plugin
         domains              0.26.7 enabled    dokku core domains plugin
         enter                0.26.7 enabled    dokku core enter plugin
         git                  0.26.7 enabled    dokku core git plugin
         letsencrypt          0.13.0 enabled    Automated installation of let's encrypt TLS certificates
         logs                 0.26.7 enabled    dokku core logs plugin
         mysql                1.12.1 enabled    dokku mysql service plugin
         network              0.26.7 enabled    dokku core network plugin
         nginx-vhosts         0.26.7 enabled    dokku core nginx-vhosts plugin
         plugin               0.26.7 enabled    dokku core plugin plugin
         postgres             1.11.6 enabled    dokku postgres service plugin
         proxy                0.26.7 enabled    dokku core proxy plugin
         ps                   0.26.7 enabled    dokku core ps plugin
         registry             0.26.7 enabled    dokku core registry plugin
         repo                 0.26.7 enabled    dokku core repo plugin
         resource             0.26.7 enabled    dokku core resource plugin
         run                  0.26.7 enabled    dokku core run plugin
         scheduler            0.25.6 enabled    dokku core scheduler plugin
         scheduler-docker-local 0.26.7 enabled    dokku core scheduler-docker-local plugin
         scheduler-null       0.26.7 enabled    dokku core scheduler-null plugin
         shell                0.26.7 enabled    dokku core shell plugin
         ssh-keys             0.26.7 enabled    dokku core ssh-keys plugin
         storage              0.26.7 enabled    dokku core storage plugin
         tags                 0.24.10 enabled    dokku core tags plugin
         tar                  0.24.10 enabled    dokku core tar plugin
         trace                0.26.7 enabled    dokku core trace plugin
=====> web-review-setup-secu-9glcxl app-json information
       App json computed selected:    app.json
       App json global selected:      app.json
       App json selected:             
=====> web-review-setup-secu-9glcxl app information
       App created at:                1649754286
       App deploy source:             web-review-setup-secu-9glcxl
       App deploy source metadata:    web-review-setup-secu-9glcxl
       App dir:                       /home/dokku/web-review-setup-secu-9glcxl
       App locked:                    false
=====> web-review-setup-secu-9glcxl builder information
       Builder build dir:             
       Builder computed build dir:    
       Builder computed selected:     
       Builder global build dir:      
       Builder global selected:       
       Builder selected:              
=====> web-review-setup-secu-9glcxl builder-dockerfile information
       Builder dockerfile computed dockerfile path: Dockerfile               
       Builder dockerfile global dockerfile path: Dockerfile               
       Builder dockerfile dockerfile path:                          
=====> web-review-setup-secu-9glcxl builder-pack information
       Builder pack computed projecttoml path: project.toml             
       Builder pack global projecttoml path: project.toml             
       Builder pack projecttoml path:                          
=====> web-review-setup-secu-9glcxl buildpacks information
       Buildpacks computed stack:     gliderlabs/herokuish:latest-20
       Buildpacks global stack:       
       Buildpacks list:               
       Buildpacks stack:              
=====> web-review-setup-secu-9glcxl ssl information
       Ssl dir:                       /home/dokku/web-review-setup-secu-9glcxl/tls
       Ssl enabled:                   true                     
       Ssl hostnames:                 web-review-setup-secu-9glcxl.stage.com
       Ssl expires at:                Jul 11 04:40:40 2022 GMT 
       Ssl issuer:                    C = US, O = Let's Encrypt, CN = R3
       Ssl starts at:                 Apr 12 04:40:41 2022 GMT 
       Ssl subject:                   subject=CN = web-review-setup-secu-9glcxl.stage.com
       Ssl verified:                  self signed              
=====> web-review-setup-secu-9glcxl checks information
       Checks disabled list:          none                     
       Checks skipped list:           none                     
=====> web-review-setup-secu-9glcxl cron information
       Cron task count:               0
=====> web-review-setup-secu-9glcxl docker options information
       ...
=====> web-review-setup-secu-9glcxl domains information
       Domains app enabled:           true                     
       Domains app vhosts:            web-review-setup-secu-9glcxl.stage.com
       Domains global enabled:        true                     
       Domains global vhosts:         stage.com      
=====> web-review-setup-secu-9glcxl git information
       ...
=====> web-review-setup-secu-9glcxl logs information
       Logs computed max size:        10m
       Logs global max size:          10m
       Logs global vector sink:       
       Logs max size:                 
       Logs vector sink:              
=====> web-review-setup-secu-9glcxl network information
       Network attach post create:           
       Network attach post deploy:           
       Network bind all interfaces:          false
       Network computed attach post create:  
       Network computed attach post deploy:  
       Network computed bind all interfaces: false
       Network computed initial network:     
       Network computed tld:                 
       Network global attach post create:    
       Network global attach post deploy:    
       Network global bind all interfaces:   false
       Network global initial network:       
       Network global tld:                   
       Network initial network:              
       Network static web listener:          
       Network tld:                          
       Network web listeners:                172.17.0.19:5000
=====> web-review-setup-secu-9glcxl nginx information
       ...                            
=====> web-review-setup-secu-9glcxl proxy information
       Proxy enabled:                 true
       Proxy port map:                http:80:5000 https:443:5000
       Proxy type:                    nginx
=====> web-review-setup-secu-9glcxl ps information
       Deployed:                      true
       Processes:                     1
       Ps can scale:                  true
       Ps computed procfile path:     Procfile
       Ps global procfile path:       Procfile
       Ps procfile path:              
       Ps restart policy:             on-failure:10
       Restore:                       true
       Running:                       true
       Status web 1:                  running (CID: a4df19ca4af)
=====> web-review-setup-secu-9glcxl registry information
       Registry computed image repo:      dokku/web-review-setup-secu-9glcxl
       Registry computed push on release: false
       Registry computed server:          
       Registry global push on release:   
       Registry global server:            
       Registry image repo:               
       Registry push on release:          
       Registry server:                   
       Registry tag version:              
=====> web-review-setup-secu-9glcxl resource information
=====> web-review-setup-secu-9glcxl scheduler information
       Scheduler computed selected:   docker-local
       Scheduler global selected:     docker-local
       Scheduler selected:            
=====> web-review-setup-secu-9glcxl scheduler-docker-local information
       Scheduler docker local disable chown:                          
       Scheduler docker local parallel schedule count:                          
=====> web-review-setup-secu-9glcxl storage information
       ...

How (deb/make/rpm) and where (AWS, VirtualBox, physical, etc.) was Dokku installed?:

Dedicated physical server.

Additional information

Will be provided if needed.

[ikalkov]

If this is the desired behavior, I'd like to know what is the best way to detect whether letsencrypt is already enabled. If not, we will be happy to submit a PR with a fix.

[josegonzalez]

I would definitely take an MR. Some notes:

• It should take into account the new functionality in post-domains-update. That should probably always request a new cert, or at least check if all domains are covered.
• There should be a way to force-refresh the certificates (either via a flag or a new command).

[odscjames]

There should be a way to force-refresh the certificates (either via a flag or a new command).

+1 - when moving certs between servers (see #233) you might copy certs by hand from old server to new, but then want to run enable to get a new cert on the new machine and make sure auto renewal is set up correctly.