Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Internal Error trying to renew cert #396

Open
AnonJervis opened this issue May 4, 2020 · 66 comments
Open

Internal Error trying to renew cert #396

AnonJervis opened this issue May 4, 2020 · 66 comments
Labels

Comments

@AnonJervis
Copy link

I was trying to renew my cert running version 2.2.1 and the following error popped up:

[5/4/2020] [10:01:54 PM] [SSL      ] › ℹ  info      Renewing Let'sEncrypt certificates for Cert #1: mywebite.com,
[5/4/2020] [10:01:55 PM] [Express  ] › ⚠  warning   Command failed: /usr/bin/certbot renew --non-interactive --config "/etc/letsencrypt.ini" --cert-name "npm-1" --preferred-challenges "dns,http" --disable-hook-validation ,
Saving debug log to /var/log/letsencrypt/letsencrypt.log,
No certificate found with name npm-1 (expected /etc/letsencrypt/renewal/npm-1.conf).

I tried restarting the container to renew again and the log shows:

),
[5/4/2020] [10:00:15 PM] [SSL      ] › ✖  error     Certificate is not valid (Command failed: openssl x509 -in /etc/letsencrypt/live/npm-1/fullchain.pem -subject -noout,
140647724621128:error:2006D080:BIO routines:BIO_new_file:no such file:crypto/bio/bss_file.c:76:,
unable to load certificate,
140647724621128:error:02001002:system library:fopen:No such file or directory:crypto/bio/bss_file.c:69:fopen('/etc/letsencrypt/live/npm-1/fullchain.pem','r'),
Can't open /etc/letsencrypt/live/npm-1/fullchain.pem for reading, No such file or directory,
[5/4/2020] [10:00:15 PM] [SSL      ] › ℹ  info      Renew Complete,
[5/4/2020] [10:00:14 PM] [IP Ranges] › ℹ  info      IP Ranges Renewal Timer initialized,
[5/4/2020] [10:00:14 PM] [SSL      ] › ℹ  info      Let's Encrypt Renewal Timer initialized,
[5/4/2020] [10:00:15 PM] [Nginx    ] › ℹ  info      Reloading Nginx,
[5/4/2020] [10:00:14 PM] [Global   ] › ℹ  info      Backend PID 201 listening on port 3000 ...,
[5/4/2020] [10:00:14 PM] [SSL      ] › ℹ  info      Renewing SSL certs close to expiry...,
[5/4/2020] [10:00:14 PM] [IP Ranges] › ℹ  info      Fetching https://www.cloudflare.com/ips-v6,
[5/4/2020] [10:00:13 PM] [IP Ranges] › ℹ  info      Fetching https://ip-ranges.amazonaws.com/ip-ranges.json,
[5/4/2020] [10:00:14 PM] [IP Ranges] › ℹ  info      Fetching https://www.cloudflare.com/ips-v4,
[5/4/2020] [10:00:13 PM] [IP Ranges] › ℹ  info      Fetching IP Ranges from online services...,
[5/4/2020] [10:00:12 PM] [Migrate  ] › ℹ  info      Current database version: 20200410143839,
❯ Enabling IPV6 in hosts: /data/nginx,
  ❯ /etc/nginx/conf.d/production.conf,
  ❯ /etc/nginx/conf.d/default.conf,
  ❯ /etc/nginx/conf.d/include/resolvers.conf,
  ❯ /etc/nginx/conf.d/include/ip_ranges.conf,
  ❯ /etc/nginx/conf.d/include/proxy.conf,
  ❯ /etc/nginx/conf.d/include/block-exploits.conf,
  ❯ /etc/nginx/conf.d/include/assets.conf,
  ❯ /etc/nginx/conf.d/include/force-ssl.conf,
  ❯ /etc/nginx/conf.d/include/letsencrypt-acme-challenge.conf,
  ❯ /etc/nginx/conf.d/include/ssl-ciphers.conf,
❯ Enabling IPV6 in hosts: /etc/nginx/conf.d,
[services.d] done.,
[services.d] starting services,
[cont-init.d] done.,
[cont-init.d] executing container initialization scripts...,
[fix-attrs.d] done.,
[fix-attrs.d] applying ownership & permissions fixes...,
[s6-init] ensuring user provided files have correct perms...exited 0.,[s6-init] making user provided files available at /var/run/s6/etc...exited 0.

I decided to update to latest and the problem persists, so I completely deleted my npm container and it's data and start a whole new instance. However, I still cannot renew and now my SSL cert expired the moment I tried registering. This is not my only webserver trying to renew SSL and its happening to my other ones as well. I've tried creating with sub.subdomain.duckdns.org and it registered fine. Here is my log after starting everything fresh:

[cont-init.d] done.,
[services.d] starting services,
[services.d] done.,
Generating dummy SSL certificate...,
Generating a RSA private key,
...............................+++++,
............+++++,
writing new private key to '/data/nginx/dummykey.pem',
-----,
Complete,
❯ Enabling IPV6 in hosts: /etc/nginx/conf.d,
  ❯ /etc/nginx/conf.d/include/ssl-ciphers.conf,
  ❯ /etc/nginx/conf.d/include/letsencrypt-acme-challenge.conf,
  ❯ /etc/nginx/conf.d/include/force-ssl.conf,
  ❯ /etc/nginx/conf.d/include/assets.conf,
  ❯ /etc/nginx/conf.d/include/block-exploits.conf,
  ❯ /etc/nginx/conf.d/include/proxy.conf,
  ❯ /etc/nginx/conf.d/include/ip_ranges.conf,
  ❯ /etc/nginx/conf.d/include/resolvers.conf,
  ❯ /etc/nginx/conf.d/default.conf,
  ❯ /etc/nginx/conf.d/production.conf,
❯ Enabling IPV6 in hosts: /data/nginx,
[5/4/2020] [9:50:10 PM] [Global   ] › ✖  error     connect ECONNREFUSED xxx.xxx.xx.x3306,
[5/4/2020] [9:50:11 PM] [Global   ] › ✖  error     connect ECONNREFUSED xxx.xxx.xx.x3306,
[5/4/2020] [9:50:12 PM] [Global   ] › ✖  error     connect ECONNREFUSED xxx.xxx.xx.x3306,
[5/4/2020] [9:50:13 PM] [Migrate  ] › ℹ  info      Current database version: none,
[5/4/2020] [9:50:13 PM] [Migrate  ] › ℹ  info      [initial-schema] Migrating Up...,
[5/4/2020] [9:50:13 PM] [Migrate  ] › ℹ  info      [initial-schema] auth Table created,
[5/4/2020] [9:50:13 PM] [Migrate  ] › ℹ  info      [initial-schema] user Table created,
[5/4/2020] [9:50:13 PM] [Migrate  ] › ℹ  info      [initial-schema] user_permission Table created,
[5/4/2020] [9:50:13 PM] [Migrate  ] › ℹ  info      [initial-schema] proxy_host Table created,
[5/4/2020] [9:50:13 PM] [Migrate  ] › ℹ  info      [initial-schema] redirection_host Table created,
[5/4/2020] [9:50:13 PM] [Migrate  ] › ℹ  info      [initial-schema] dead_host Table created,
[5/4/2020] [9:50:13 PM] [Migrate  ] › ℹ  info      [initial-schema] stream Table created,
[5/4/2020] [9:50:13 PM] [Migrate  ] › ℹ  info      [initial-schema] access_list Table created,
[5/4/2020] [9:50:13 PM] [Migrate  ] › ℹ  info      [initial-schema] certificate Table created,
[5/4/2020] [9:50:13 PM] [Migrate  ] › ℹ  info      [initial-schema] access_list_auth Table created,
[5/4/2020] [9:50:13 PM] [Migrate  ] › ℹ  info      [initial-schema] audit_log Table created,
[5/4/2020] [9:50:13 PM] [Migrate  ] › ℹ  info      [websockets] Migrating Up...,
[5/4/2020] [9:50:13 PM] [Migrate  ] › ℹ  info      [websockets] proxy_host Table altered,
[5/4/2020] [9:50:13 PM] [Migrate  ] › ℹ  info      [forward_host] Migrating Up...,
[5/4/2020] [9:50:13 PM] [Migrate  ] › ℹ  info      [forward_host] proxy_host Table altered,
[5/4/2020] [9:50:13 PM] [Migrate  ] › ℹ  info      [http2_support] Migrating Up...,
[5/4/2020] [9:50:13 PM] [Migrate  ] › ℹ  info      [http2_support] proxy_host Table altered,
[5/4/2020] [9:50:13 PM] [Migrate  ] › ℹ  info      [http2_support] redirection_host Table altered,
[5/4/2020] [9:50:13 PM] [Migrate  ] › ℹ  info      [http2_support] dead_host Table altered,
[5/4/2020] [9:50:13 PM] [Migrate  ] › ℹ  info      [forward_scheme] Migrating Up...,
[5/4/2020] [9:50:13 PM] [Migrate  ] › ℹ  info      [forward_scheme] proxy_host Table altered,
[5/4/2020] [9:50:13 PM] [Migrate  ] › ℹ  info      [disabled] Migrating Up...,
[5/4/2020] [9:50:13 PM] [Migrate  ] › ℹ  info      [disabled] proxy_host Table altered,
[5/4/2020] [9:50:13 PM] [Migrate  ] › ℹ  info      [disabled] redirection_host Table altered,
[5/4/2020] [9:50:13 PM] [Migrate  ] › ℹ  info      [disabled] dead_host Table altered,
[5/4/2020] [9:50:13 PM] [Migrate  ] › ℹ  info      [disabled] stream Table altered,
[5/4/2020] [9:50:13 PM] [Migrate  ] › ℹ  info      [custom_locations] Migrating Up...,
[5/4/2020] [9:50:13 PM] [Migrate  ] › ℹ  info      [custom_locations] proxy_host Table altered,
[5/4/2020] [9:50:13 PM] [Migrate  ] › ℹ  info      [hsts] Migrating Up...,
[5/4/2020] [9:50:13 PM] [Migrate  ] › ℹ  info      [hsts] proxy_host Table altered,
[5/4/2020] [9:50:13 PM] [Migrate  ] › ℹ  info      [hsts] redirection_host Table altered,
[5/4/2020] [9:50:13 PM] [Migrate  ] › ℹ  info      [hsts] dead_host Table altered,
[5/4/2020] [9:50:13 PM] [Migrate  ] › ℹ  info      [settings] Migrating Up...,
[5/4/2020] [9:50:13 PM] [Migrate  ] › ℹ  info      [settings] setting Table created,
[5/4/2020] [9:50:13 PM] [Migrate  ] › ℹ  info      [settings] Default settings added,
[5/4/2020] [9:50:13 PM] [Migrate  ] › ℹ  info      [access_list_client] Migrating Up...,
[5/4/2020] [9:50:13 PM] [Migrate  ] › ℹ  info      [access_list_client] access_list_client Table created,
[5/4/2020] [9:50:13 PM] [Migrate  ] › ℹ  info      [access_list_client] access_list Table altered,
[5/4/2020] [9:50:13 PM] [Setup    ] › ℹ  info      Creating a new JWT key pair...,
[5/4/2020] [9:50:22 PM] [Setup    ] › ℹ  info      Wrote JWT key pair to config file: /app/config/production.json,
[5/4/2020] [9:50:22 PM] [Setup    ] › ⚠  warning   Restarting interface to apply new configuration,
[5/4/2020] [9:50:24 PM] [Migrate  ] › ℹ  info      Current database version: 20200410143839,
[5/4/2020] [9:50:24 PM] [Setup    ] › ℹ  info      Creating a new user: [email protected] with password: changeme,
[5/4/2020] [9:50:26 PM] [Setup    ] › ℹ  info      Initial setup completed,
[5/4/2020] [9:50:26 PM] [IP Ranges] › ℹ  info      Fetching IP Ranges from online services...,
[5/4/2020] [9:50:26 PM] [IP Ranges] › ℹ  info      Fetching https://ip-ranges.amazonaws.com/ip-ranges.json,
[5/4/2020] [9:50:26 PM] [IP Ranges] › ℹ  info      Fetching https://www.cloudflare.com/ips-v4,
[5/4/2020] [9:50:26 PM] [IP Ranges] › ℹ  info      Fetching https://www.cloudflare.com/ips-v6,
[5/4/2020] [9:50:26 PM] [SSL      ] › ℹ  info      Let's Encrypt Renewal Timer initialized,
[5/4/2020] [9:50:26 PM] [SSL      ] › ℹ  info      Renewing SSL certs close to expiry...,
[5/4/2020] [9:50:26 PM] [IP Ranges] › ℹ  info      IP Ranges Renewal Timer initialized,
[5/4/2020] [9:50:26 PM] [Global   ] › ℹ  info      Backend PID 269 listening on port 3000 ...,
[5/4/2020] [9:50:27 PM] [Nginx    ] › ℹ  info      Reloading Nginx,
[5/4/2020] [9:50:27 PM] [SSL      ] › ℹ  info      Renew Complete,
`QueryBuilder#allowEager` method is deprecated. You should use `allowGraph` instead. `allowEager` method will be removed in 3.0,
`QueryBuilder#eager` method is deprecated. You should use the `withGraphFetched` method instead. `eager` method will be removed in 3.0,
QueryBuilder#omit is deprecated. This method will be removed in version 3.0,
[5/4/2020] [9:51:42 PM] [Nginx    ] › ℹ  info      Reloading Nginx,
[5/4/2020] [9:51:42 PM] [SSL      ] › ℹ  info      Requesting Let'sEncrypt certificates for Cert #1: mywebite.com,
[5/4/2020] [9:51:46 PM] [Nginx    ] › ℹ  info      Reloading Nginx,
[5/4/2020] [9:51:46 PM] [Express  ] › ⚠  warning   Command failed: /usr/bin/certbot certonly --non-interactive --config "/etc/letsencrypt.ini" --cert-name "npm-1" --agree-tos --email "email.com" --preferred-challenges "dns,http" --webroot --domains "mywebite.com" ,
Saving debug log to /var/log/letsencrypt/letsencrypt.log,
Plugins selected: Authenticator webroot, Installer None,
Obtaining a new certificate,
Performing the following challenges:,
http-01 challenge for mywebite.com,
Using the webroot path /data/letsencrypt-acme-challenge for all unmatched domains.,
Waiting for verification...,
Challenge failed for domain mywebite.com,
http-01 challenge for mywebite.com,
Cleaning up challenges,
Some challenges have failed.,
,

Could this be possible bug in the latest update? Could it be my domain has already been registered with let's encrypted before, I cannot re-register a new one when I start a new container?

@AnonJervis AnonJervis added the bug label May 4, 2020
@erikalm
Copy link

erikalm commented May 6, 2020

I had a similar issue and fixed it by deleting the certificate that was having issues in "SSL Certificates" tab and requested the certificate again. Before you try this you might want to backup the database and configuration files just in case.

@kizza42
Copy link

kizza42 commented May 8, 2020

I'm having this issue as well, tried restarting the container and clearing out the certificates that keep failing ( I only set this up for the 1st time yesterday). I can't get it to work

When I request I get an error
Error

Then I see the errors in the log like @AnonJervis

If I navigate to SSL Certificates page, I can see an entry still but it wont work
Error2

@CorySanin
Copy link

I'm having the same issue as well. Renewing manually times out and deleting then re-adding the cert gives "internal error".

@Indemnity83
Copy link
Contributor

@kizza42 and @CorySanin I believe the "internal error" notifications may be fixed in PR #407

@CorySanin
Copy link

CorySanin commented May 11, 2020

switched my existing Docker container over to jc21/nginx-proxy-manager:github-pr-407 and I'm still getting "internal error" when manually renewing or adding a new certificate.

Actually, a couple of them were able to renew. But most don't. But the ones that renewed probably would have worked before switching to the tag for the pr.

@Indemnity83
Copy link
Contributor

Are the ones that are failing linked to hosts that have an applied access list?
If so does removing the access list allow it to succeed?
Lastly, can you post the log from the npm container?

@CorySanin
Copy link

I have some with and without access lists, and the ones without are also failing. But looking at the logs I see Another instance of Certbot is already running. I think what I'm experiencing is certbot hangs on one of the certs and then everything else fails. Which doesn't seem like the original issue, I don't think. So if I continue to see this I might open up a new issue.

@kizza42
Copy link

kizza42 commented May 11, 2020

My log after I restart the container:

[s6-finish] sending all processes the TERM signal.
[s6-finish] sending all processes the KILL signal and exiting.
[s6-init] making user provided files available at /var/run/s6/etc...exited 0.
[s6-init] ensuring user provided files have correct perms...exited 0.
[fix-attrs.d] applying ownership & permissions fixes...
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] done.
[services.d] starting services
[services.d] done.
❯ Enabling IPV6 in hosts: /etc/nginx/conf.d
  ❯ /etc/nginx/conf.d/default.conf
  ❯ /etc/nginx/conf.d/production.conf
  ❯ /etc/nginx/conf.d/include/proxy.conf
  ❯ /etc/nginx/conf.d/include/letsencrypt-acme-challenge.conf
  ❯ /etc/nginx/conf.d/include/ssl-ciphers.conf
  ❯ /etc/nginx/conf.d/include/block-exploits.conf
  ❯ /etc/nginx/conf.d/include/force-ssl.conf
  ❯ /etc/nginx/conf.d/include/ip_ranges.conf
  ❯ /etc/nginx/conf.d/include/assets.conf
  ❯ /etc/nginx/conf.d/include/resolvers.conf
❯ Enabling IPV6 in hosts: /data/nginx
  ❯ /data/nginx/default_host/site.conf
  ❯ /data/nginx/proxy_host/2.conf
  ❯ /data/nginx/proxy_host/3.conf
  ❯ /data/nginx/proxy_host/4.conf
  ❯ /data/nginx/proxy_host/5.conf
  ❯ /data/nginx/proxy_host/6.conf
  ❯ /data/nginx/proxy_host/7.conf
[5/11/2020] [11:34:52 PM] [Migrate  ] › ℹ  info      Current database version: 20200410143839
[5/11/2020] [11:34:52 PM] [IP Ranges] › ℹ  info      Fetching IP Ranges from online services...
[5/11/2020] [11:34:52 PM] [IP Ranges] › ℹ  info      Fetching https://ip-ranges.amazonaws.com/ip-ranges.json
[5/11/2020] [11:34:52 PM] [IP Ranges] › ℹ  info      Fetching https://www.cloudflare.com/ips-v4
[5/11/2020] [11:34:52 PM] [IP Ranges] › ℹ  info      Fetching https://www.cloudflare.com/ips-v6
[5/11/2020] [11:34:52 PM] [SSL      ] › ℹ  info      Let's Encrypt Renewal Timer initialized
[5/11/2020] [11:34:52 PM] [SSL      ] › ℹ  info      Renewing SSL certs close to expiry...
[5/11/2020] [11:34:52 PM] [IP Ranges] › ℹ  info      IP Ranges Renewal Timer initialized
[5/11/2020] [11:34:52 PM] [Global   ] › ℹ  info      Backend PID 204 listening on port 3000 ...
[5/11/2020] [11:34:53 PM] [SSL      ] › ✖  error     Error: Command failed: /usr/bin/certbot renew --non-interactive --quiet --config "/etc/letsencrypt.ini" --preferred-challenges "dns,http" --disable-hook-validation  

Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/certbot/_internal/renewal.py", line 64, in _reconstitute
    renewal_candidate = storage.RenewableCert(full_path, config)
  File "/usr/lib/python3.8/site-packages/certbot/_internal/storage.py", line 444, in __init__
    raise errors.CertStorageError(
certbot.errors.CertStorageError: renewal config file {} is missing a required file reference

Renewal configuration file /etc/letsencrypt/renewal/npm-1.conf is broken. Skipping.
Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/certbot/_internal/renewal.py", line 64, in _reconstitute
    renewal_candidate = storage.RenewableCert(full_path, config)
  File "/usr/lib/python3.8/site-packages/certbot/_internal/storage.py", line 444, in __init__
    raise errors.CertStorageError(
certbot.errors.CertStorageError: renewal config file {} is missing a required file reference
Renewal configuration file /etc/letsencrypt/renewal/npm-2.conf is broken. Skipping.
Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/certbot/_internal/renewal.py", line 64, in _reconstitute
    renewal_candidate = storage.RenewableCert(full_path, config)
  File "/usr/lib/python3.8/site-packages/certbot/_internal/storage.py", line 444, in __init__
    raise errors.CertStorageError(
certbot.errors.CertStorageError: renewal config file {} is missing a required file reference
Renewal configuration file /etc/letsencrypt/renewal/npm-3.conf is broken. Skipping.
Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/certbot/_internal/renewal.py", line 64, in _reconstitute
    renewal_candidate = storage.RenewableCert(full_path, config)
  File "/usr/lib/python3.8/site-packages/certbot/_internal/storage.py", line 444, in __init__
    raise errors.CertStorageError(
certbot.errors.CertStorageError: renewal config file {} is missing a required file reference
Renewal configuration file /etc/letsencrypt/renewal/npm-4.conf is broken. Skipping.
Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/certbot/_internal/renewal.py", line 64, in _reconstitute
    renewal_candidate = storage.RenewableCert(full_path, config)
  File "/usr/lib/python3.8/site-packages/certbot/_internal/storage.py", line 444, in __init__
    raise errors.CertStorageError(
certbot.errors.CertStorageError: renewal config file {} is missing a required file reference
Renewal configuration file /etc/letsencrypt/renewal/npm-5.conf is broken. Skipping.
0 renew failure(s), 5 parse failure(s)
    at ChildProcess.exithandler (child_process.js:295:12)
    at ChildProcess.emit (events.js:210:5)
    at maybeClose (internal/child_process.js:1028:16)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:283:5)
`QueryBuilder#allowEager` method is deprecated. You should use `allowGraph` instead. `allowEager` method will be removed in 3.0
`QueryBuilder#eager` method is deprecated. You should use the `withGraphFetched` method instead. `eager` method will be removed in 3.0
QueryBuilder#omit is deprecated. This method will be removed in version 3.0

Those broken files:
/etc/letsencrypt/renewal/npm-3.conf are just empty

So I removed them all and restarted container and tried to create a cert only:

[fix-attrs.d] applying ownership & permissions fixes...,
[fix-attrs.d] done.,
[cont-init.d] executing container initialization scripts...,
[cont-init.d] done.,
[services.d] starting services,
[services.d] done.,
❯ Enabling IPV6 in hosts: /etc/nginx/conf.d,
  ❯ /etc/nginx/conf.d/default.conf,
  ❯ /etc/nginx/conf.d/production.conf,
  ❯ /etc/nginx/conf.d/include/proxy.conf,
  ❯ /etc/nginx/conf.d/include/letsencrypt-acme-challenge.conf,
  ❯ /etc/nginx/conf.d/include/ssl-ciphers.conf,
  ❯ /etc/nginx/conf.d/include/block-exploits.conf,
  ❯ /etc/nginx/conf.d/include/force-ssl.conf,
  ❯ /etc/nginx/conf.d/include/ip_ranges.conf,
  ❯ /etc/nginx/conf.d/include/assets.conf,
  ❯ /etc/nginx/conf.d/include/resolvers.conf,
❯ Enabling IPV6 in hosts: /data/nginx,
  ❯ /data/nginx/default_host/site.conf,
  ❯ /data/nginx/proxy_host/2.conf,
  ❯ /data/nginx/proxy_host/3.conf,
  ❯ /data/nginx/proxy_host/4.conf,
  ❯ /data/nginx/proxy_host/5.conf,
  ❯ /data/nginx/proxy_host/6.conf,
  ❯ /data/nginx/proxy_host/7.conf,
[5/11/2020] [11:49:41 PM] [Migrate  ] › ℹ  info      Current database version: 20200410143839,
[5/11/2020] [11:49:41 PM] [IP Ranges] › ℹ  info      Fetching IP Ranges from online services...,
[5/11/2020] [11:49:41 PM] [IP Ranges] › ℹ  info      Fetching https://ip-ranges.amazonaws.com/ip-ranges.json,
[5/11/2020] [11:49:41 PM] [IP Ranges] › ℹ  info      Fetching https://www.cloudflare.com/ips-v4,
[5/11/2020] [11:49:41 PM] [IP Ranges] › ℹ  info      Fetching https://www.cloudflare.com/ips-v6,
[5/11/2020] [11:49:41 PM] [SSL      ] › ℹ  info      Let's Encrypt Renewal Timer initialized,
[5/11/2020] [11:49:41 PM] [SSL      ] › ℹ  info      Renewing SSL certs close to expiry...,
[5/11/2020] [11:49:41 PM] [IP Ranges] › ℹ  info      IP Ranges Renewal Timer initialized,
[5/11/2020] [11:49:41 PM] [Global   ] › ℹ  info      Backend PID 212 listening on port 3000 ...,
[5/11/2020] [11:49:42 PM] [Nginx    ] › ℹ  info      Reloading Nginx,
[5/11/2020] [11:49:42 PM] [SSL      ] › ℹ  info      Renew Complete,
s6-svscanctl: fatal: unable to control /var/run/s6/services: supervisor not listening,
finish: applet not found,
[cont-finish.d] executing container finish scripts...,
[cont-finish.d] done.,
[s6-finish] waiting for services.,
[s6-finish] sending all processes the TERM signal.,
[s6-finish] sending all processes the KILL signal and exiting.,
[s6-init] making user provided files available at /var/run/s6/etc...exited 0.,
[s6-init] ensuring user provided files have correct perms...exited 0.,
[fix-attrs.d] applying ownership & permissions fixes...,
[fix-attrs.d] done.,
[cont-init.d] executing container initialization scripts...,
[cont-init.d] done.,
[services.d] starting services,
[services.d] done.,
❯ Enabling IPV6 in hosts: /etc/nginx/conf.d,
  ❯ /etc/nginx/conf.d/default.conf,
  ❯ /etc/nginx/conf.d/production.conf,
  ❯ /etc/nginx/conf.d/include/proxy.conf,
  ❯ /etc/nginx/conf.d/include/letsencrypt-acme-challenge.conf,
  ❯ /etc/nginx/conf.d/include/ssl-ciphers.conf,
  ❯ /etc/nginx/conf.d/include/block-exploits.conf,
  ❯ /etc/nginx/conf.d/include/force-ssl.conf,
  ❯ /etc/nginx/conf.d/include/ip_ranges.conf,
  ❯ /etc/nginx/conf.d/include/assets.conf,
  ❯ /etc/nginx/conf.d/include/resolvers.conf,
❯ Enabling IPV6 in hosts: /data/nginx,
  ❯ /data/nginx/default_host/site.conf,
  ❯ /data/nginx/proxy_host/2.conf,
  ❯ /data/nginx/proxy_host/3.conf,
  ❯ /data/nginx/proxy_host/4.conf,
  ❯ /data/nginx/proxy_host/5.conf,
  ❯ /data/nginx/proxy_host/6.conf,
  ❯ /data/nginx/proxy_host/7.conf,
[5/11/2020] [11:50:30 PM] [Migrate  ] › ℹ  info      Current database version: 20200410143839,
[5/11/2020] [11:50:30 PM] [IP Ranges] › ℹ  info      Fetching IP Ranges from online services...,
[5/11/2020] [11:50:30 PM] [IP Ranges] › ℹ  info      Fetching https://ip-ranges.amazonaws.com/ip-ranges.json,
[5/11/2020] [11:50:30 PM] [IP Ranges] › ℹ  info      Fetching https://www.cloudflare.com/ips-v4,
[5/11/2020] [11:50:30 PM] [IP Ranges] › ℹ  info      Fetching https://www.cloudflare.com/ips-v6,
[5/11/2020] [11:50:30 PM] [SSL      ] › ℹ  info      Let's Encrypt Renewal Timer initialized,
[5/11/2020] [11:50:30 PM] [SSL      ] › ℹ  info      Renewing SSL certs close to expiry...,
[5/11/2020] [11:50:30 PM] [IP Ranges] › ℹ  info      IP Ranges Renewal Timer initialized,
[5/11/2020] [11:50:30 PM] [Global   ] › ℹ  info      Backend PID 202 listening on port 3000 ...,
[5/11/2020] [11:50:31 PM] [Nginx    ] › ℹ  info      Reloading Nginx,
[5/11/2020] [11:50:31 PM] [SSL      ] › ℹ  info      Renew Complete,
`QueryBuilder#allowEager` method is deprecated. You should use `allowGraph` instead. `allowEager` method will be removed in 3.0,
`QueryBuilder#eager` method is deprecated. You should use the `withGraphFetched` method instead. `eager` method will be removed in 3.0,
QueryBuilder#omit is deprecated. This method will be removed in version 3.0,
[5/11/2020] [11:51:10 PM] [Nginx    ] › ℹ  info      Reloading Nginx,
[5/11/2020] [11:51:10 PM] [SSL      ] › ℹ  info      Requesting Let'sEncrypt certificates for Cert #6: test.kizza42.com,
[5/11/2020] [11:51:20 PM] [Nginx    ] › ℹ  info      Reloading Nginx,
[5/11/2020] [11:51:20 PM] [Express  ] › ⚠  warning   Command failed: /usr/bin/certbot certonly --non-interactive --config "/etc/letsencrypt.ini" --cert-name "npm-6" --agree-tos --email "[email protected]" --preferred-challenges "dns,http" --webroot --domains "test.kizza42.com" ,
Saving debug log to /var/log/letsencrypt/letsencrypt.log,
Plugins selected: Authenticator webroot, Installer None,
Obtaining a new certificate,
Performing the following challenges:,
http-01 challenge for test.kizza42.com,
Using the webroot path /data/letsencrypt-acme-challenge for all unmatched domains.,
Waiting for verification...,
Cleaning up challenges,
An unexpected error occurred:,
OSError: [Errno 95] Not supported: '../../archive/npm-6/cert1.pem' -> '/etc/letsencrypt/live/npm-6/cert.pem',
Please see the logfiles in /var/log/letsencrypt for more details.,,

And /var/log/letsencrypt/letsencrypt.log

[root@docker-226431607f00:/var/log/letsencrypt]# tail letsencrypt.log
  File "/usr/lib/python3.8/site-packages/certbot/_internal/main.py", line 1237, in certonly
    lineage = _get_and_save_cert(le_client, config, domains, certname, lineage)
  File "/usr/lib/python3.8/site-packages/certbot/_internal/main.py", line 121, in _get_and_save_cert
    lineage = le_client.obtain_and_enroll_certificate(domains, certname)
  File "/usr/lib/python3.8/site-packages/certbot/_internal/client.py", line 430, in obtain_and_enroll_certificate
    return storage.RenewableCert.new_lineage(
  File "/usr/lib/python3.8/site-packages/certbot/_internal/storage.py", line 1022, in new_lineage
    os.symlink(_relpath_from_file(archive_target[kind], target[kind]), target[kind])
OSError: [Errno 95] Not supported: '../../archive/npm-6/cert1.pem' -> '/etc/letsencrypt/live/npm-6/cert.pem'
2020-05-11 23:51:20,159:ERROR:certbot._internal.log:An unexpected error occurred:

@jc21
Copy link
Member

jc21 commented May 20, 2020

Firstly, the error in the issue at the top says that the challenge has failed. This would be due to you asking for a cert for mywebsite.com that doesn't have dns pointing to your setup.

I doubt any of this has anything to do with access lists.

The missing files make me think that maybe the data and/or letsencrypt folders aren't mounted properly. Please paste your docker-compose.yml to be sure.

@kizza42
Copy link

kizza42 commented May 20, 2020

Thankyou for the help @jc21
Here is my compose file:

version: "2"
services:
  nGinx:
    image: jc21/nginx-proxy-manager:2
    restart: always
    networks:
      dockerlan:
        ipv4_address: 192.168.0.193    
    dns:
      - 192.168.0.1
    volumes:
      - /media/k2nas/SSD/Docker/data/Nginx/app/config:/app/config
      - /media/k2nas/SSD/Docker/data/Nginx/data:/data
      - /media/k2nas/SSD/Docker/data/Nginx/letsencrypt:/etc/letsencrypt
  mariadb:
    # Pinned at this version for Innodb Error
    image: jc21/mariadb-aria:10.4.12
    restart: always
    networks:
      dockerlan:
        ipv4_address: 192.168.0.194
    ports:
      - 3306:3306
    environment:
      MYSQL_ROOT_PASSWORD: 'npm'
      MYSQL_DATABASE: 'npm'
      MYSQL_USER: 'npm'
      MYSQL_PASSWORD: 'npm'
    volumes:
      - /media/k2nas/SSD/Docker/data/Nginx/mysql:/var/lib/mysql
networks:
  dockerlan:
       external: true

@pantherale0
Copy link

Also experiencing this issue. From the logs:
[5/25/2020] [6:42:02 PM] [Express ] › ⚠ warning Command failed: /usr/bin/certbot certonly --non-interactive --config "/etc/letsencrypt.ini" --cert-name "npm-1" --agree-tos --email "[EMAIL ADDRESS]" --preferred-challenges "dns,http" --webroot --domains "[DOMAIN]" Saving debug log to /var/log/letsencrypt/letsencrypt.log Plugins selected: Authenticator webroot, Installer None Obtaining a new certificate Performing the following challenges: http-01 challenge for [DOMAIN] Using the webroot path /data/letsencrypt-acme-challenge for all unmatched domains. Waiting for verification... Challenge failed for domain [DOMAIN] http-01 challenge for [DOMAIN] Cleaning up challenges Some challenges have failed.

At first thought this was due to a bad install or something, so have a complete clean install and experiencing this. Checked DNS settings, all records are pointing to the correct IP and also checked firewall rules (not that any of these have changed recently).

Interestingly this worked fine a week ago.

@Xinil
Copy link

Xinil commented May 27, 2020

Also experiencing an Internal Error when I attempt to perform a certificate renewal, or when the certbot attempts for me. letsencrypt.log:

Initialized: <certbot._internal.plugins.webroot.Authenticator object at 0x7f73f85f5df0>
Prep: True
2020-05-27 07:03:04,824:DEBUG:certbot._internal.plugins.selection:Selected authenticator <certbot._internal.plugins.webroot.Authenticator object at 0x7f73f85f5df0> and installer None
2020-05-27 07:03:04,824:INFO:certbot._internal.plugins.selection:Plugins selected: Authenticator webroot, Installer None
2020-05-27 07:03:04,837:WARNING:certbot._internal.renewal:Attempting to renew cert (npm-7) from /etc/letsencrypt/renewal/npm-7.conf produced an unexpected error: [Errno 1] Operation not permitted: '/etc/letsencrypt/accounts/acme-v02.api.letsencrypt.org/directory/2f1011068b45be8e10e11180c968b254/private_key.json'. Skipping.
2020-05-27 07:03:04,837:DEBUG:certbot._internal.renewal:Traceback was:
Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/certbot/_internal/account.py", line 227, in _load_for_server_path
    with open(self._key_path(account_dir_path)) as key_file:
PermissionError: [Errno 1] Operation not permitted: '/etc/letsencrypt/accounts/acme-v02.api.letsencrypt.org/directory/2f1011068b45be8e10e11180c968b254/private_key.json'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/certbot/_internal/renewal.py", line 449, in handle_renewal_request
    main.renew_cert(lineage_config, plugins, renewal_candidate)
  File "/usr/lib/python3.8/site-packages/certbot/_internal/main.py", line 1178, in renew_cert
    le_client = _init_le_client(config, auth, installer)
  File "/usr/lib/python3.8/site-packages/certbot/_internal/main.py", line 607, in _init_le_client
    acc, acme = _determine_account(config)
  File "/usr/lib/python3.8/site-packages/certbot/_internal/main.py", line 511, in _determine_account
    acc = account_storage.load(config.account)
  File "/usr/lib/python3.8/site-packages/certbot/_internal/account.py", line 237, in load
    return self._load_for_server_path(account_id, self.config.server_path)
  File "/usr/lib/python3.8/site-packages/certbot/_internal/account.py", line 232, in _load_for_server_path
    raise errors.AccountStorageError(error)
certbot.errors.AccountStorageError: [Errno 1] Operation not permitted: '/etc/letsencrypt/accounts/acme-v02.api.letsencrypt.org/directory/2f1011068b45be8e10e11180c968b254/private_key.json'

My docker-compose.yml

version: "3"
services:
  app:
    image: jc21/nginx-proxy-manager:latest
    restart: always
    ports:
      - 80:80
      - 81:81
      - 443:443
    volumes:
      - ./config.json:/app/config/production.json
      - ./data:/data
      - ./letsencrypt:/etc/letsencrypt
    depends_on:
      - db
    environment:
    # if you want pretty colors in your docker logs:
    - FORCE_COLOR=1
  db:
    image: mysql:5.7
    restart: always
    environment:
      MYSQL_ROOT_PASSWORD: "npm"
      MYSQL_DATABASE: "npm"
      MYSQL_USER: "npm"
      MYSQL_PASSWORD: "npm"
    volumes:
      - ./data/mysql:/var/lib/mysql

The volumes exist, I have files in the folder '2f1011068b45be8e10e11180c968b254' it describes as a permission issue. Latest v2.2.4.

Any help is appreciated.

@Xinil
Copy link

Xinil commented Jun 10, 2020

Pretty bummed there's no direction on how to resolve this. Anyone thinking about going the manual certificate renewal route? Sounds like a pain, but this is debilitating for my network communication :(

@OuticNZ
Copy link

OuticNZ commented Jun 20, 2020

My certificates are coming up for renewal and it's failing as well. 20 days before they start expiring. Any guidance on how to resolve?

@Kipjr
Copy link

Kipjr commented Jul 5, 2020

Time-consuming workaround for me is:

docker exec -it docker-nginx-proxy_app_1 /bin/bash

certbot certonly --manual --preferred-challenges=dns -m [email protected]  --agree-tos -d YOUR_DOMAIN.TLD

Please deploy a DNS TXT record under the name
_acme-challenge.DOMAIN.TLD with the following value:

AbCdEfGhIjK12345RandomCode

Before continuing, verify the record is deployed.


Press Enter to Continue

Go to your DNS-settings of your Domain and add the following record:
name: __acme-challenge
type: TXT
ttl: 1hr
data: The code generated by certbot

wait 5 minutes and press enter in your CLI to verify your domain

Waiting for verification...
Cleaning up challenges

IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/DOMAIN.TLD-0001/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/DOMAIN.TLD-0001/privkey.pem
   Your cert will expire on 2020-10-03. To obtain a new or tweaked
   version of this certificate in the future, simply run certbot
   again. To non-interactively renew *all* of your certificates, run
   "certbot renew"
 - If you like Certbot, please consider supporting our work by:

   Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
   Donating to EFF:                    https://eff.org/donate-le

Exit the container and go to ./letsencrypt/ and retrieve the cert and cert key.

Go to the Manager and http://192.168.1.100:81/nginx/certificates and add a custom certificate by uploading those files.

@trankillity
Copy link

Would love an update on this very crippling bug, or be pointed in the direction of the last Docker tag that didn't have this.

@Tsunami2056
Copy link

I don't know if it will help, but I ended up giving up. And going to another docker solution that supports dns challenges. Once you configure it is 100% automatic (and also doesn't need you to have your http port open).

Perhaps this could be something to explore for the future of NPM.

@Xinil
Copy link

Xinil commented Jul 10, 2020

I also gave up and moved completely away from Nginx Proxy Manager. Had great success for a year or so, but this SSL issue is crippling. It took me 1hr to install Caddy2 and replicate everything I had.

@dialanothernumb
Copy link

which solution did you go for in the end @Xinil and @Tsunami2056 ?

@Tsunami2056
Copy link

Tsunami2056 commented Jul 10, 2020

which solution did you go for in the end @Xinil and @Tsunami2056 ?

I ended up setting up linuxserver's letsencrypt docker, a little more manual, but supports dns challenges with API plugins for alot of providers. And all is also automated.

If ever at one point there's dns challenging that's automated in npm, I would probably switch back though, it's still alot easier.

@trankillity
Copy link

Funnily enough, the non-official docker image of NPM works totally fine - so I just started using that.

@dfmckay
Copy link

dfmckay commented Jul 31, 2020

It looks like there was a change in the way the certs were written. New certs are written as root vs the uid specified in the docker setup, also the keys that are located in the live directory are now symlinks. I think these issues are the main cause of the errors

My fix was to edit each domain, go into the ssl tab and request a new cert. After that I deleted the old certs in the ssl certificates tab. This didn't get rid of the old conf files, so I opened a shell (I'm using portainer) then ran 'certbot renew' this will give you a list of bad conf files. Then I went to '/etc/letsencrypt/renewal' and deleted the conf files that were giving me errors. To see if it worked I reran 'certbot renew'. The messages came back as skipped which is what I should see.

@janoxakes
Copy link

Does anybody know what is the latest image that does not have this issue? All my certs are already expired and none of the workarounds worked for me (or I didn't understand them). The non-official package does not work on ARM, so no RPi support.

@Kipjr
Copy link

Kipjr commented Oct 3, 2020

Two things:

I'm using v2.5.0 and I might have a workaround:

  • cleared all entries in the MariaDB-database/npm/certificate

  • delete all certificates in /etc/letsencrypt/

  • check firewall to have an open port 80 and port-foward to this docker container. Make sure no other ports are using these.

  1. @jc21 , why is npm-1.conf generated and not domain.tld.conf which I see in the folder /etc/letsencrypt/renewal?

@swingstate
Copy link

swingstate commented Oct 29, 2020

Installed NPM through docker compose a few days ago and have been able to setup the proxy & to get a LetsEncrypt certificate.

Now, a few days later I cannot get any more certificates and I can see exceptions in the logs. When I try to renew a cert I get an internal error.

Removed the whole container and the db, reinstalled with new folders for letsencrypt and config, tried with a new subdomain and still the same issue.
Port 80 and 443 are forwarded to the docker host and access to a jellyfin server (also a container) through HTTP works.
I can also ping acme-v02.api.letsencrypt.org from the host and get an IpV6 reply. Exposing the host to the internet without a firewall in between also didn't help.

What I understand from the log is, that there seem to be connection errors. However, the host can access

@jc21 can you help narrowing this down?

Below the logs from within portainer after I reinstalled the container.


[10/29/2020] [9:29:34 PM] [Migrate  ] › ℹ  info      [http2_support] Migrating Up...,
[10/29/2020] [9:29:34 PM] [Migrate  ] › ℹ  info      [http2_support] proxy_host Table altered,
[10/29/2020] [9:29:34 PM] [Migrate  ] › ℹ  info      [http2_support] redirection_host Table altered,
[10/29/2020] [9:29:34 PM] [Migrate  ] › ℹ  info      [http2_support] dead_host Table altered,
[10/29/2020] [9:29:34 PM] [Migrate  ] › ℹ  info      [forward_scheme] Migrating Up...,
[10/29/2020] [9:29:34 PM] [Migrate  ] › ℹ  info      [forward_scheme] proxy_host Table altered,
[10/29/2020] [9:29:34 PM] [Migrate  ] › ℹ  info      [disabled] Migrating Up...,
[10/29/2020] [9:29:34 PM] [Migrate  ] › ℹ  info      [disabled] proxy_host Table altered,
[10/29/2020] [9:29:34 PM] [Migrate  ] › ℹ  info      [disabled] redirection_host Table altered,
[10/29/2020] [9:29:34 PM] [Migrate  ] › ℹ  info      [disabled] dead_host Table altered,
[10/29/2020] [9:29:34 PM] [Migrate  ] › ℹ  info      [disabled] stream Table altered,
[10/29/2020] [9:29:34 PM] [Migrate  ] › ℹ  info      [custom_locations] Migrating Up...,
[10/29/2020] [9:29:34 PM] [Migrate  ] › ℹ  info      [custom_locations] proxy_host Table altered,
[10/29/2020] [9:29:34 PM] [Migrate  ] › ℹ  info      [hsts] Migrating Up...,
[10/29/2020] [9:29:34 PM] [Migrate  ] › ℹ  info      [hsts] proxy_host Table altered,
[10/29/2020] [9:29:34 PM] [Migrate  ] › ℹ  info      [hsts] redirection_host Table altered,
[10/29/2020] [9:29:34 PM] [Migrate  ] › ℹ  info      [hsts] dead_host Table altered,
[10/29/2020] [9:29:34 PM] [Migrate  ] › ℹ  info      [settings] Migrating Up...,
[10/29/2020] [9:29:34 PM] [Migrate  ] › ℹ  info      [settings] setting Table created,
[10/29/2020] [9:29:34 PM] [Migrate  ] › ℹ  info      [access_list_client] Migrating Up...,
[10/29/2020] [9:29:34 PM] [Migrate  ] › ℹ  info      [access_list_client] access_list_client Table created,
[10/29/2020] [9:29:34 PM] [Migrate  ] › ℹ  info      [access_list_client] access_list Table altered,
[10/29/2020] [9:29:34 PM] [Migrate  ] › ℹ  info      [access_list_client_fix] Migrating Up...,
[10/29/2020] [9:29:34 PM] [Migrate  ] › ℹ  info      [access_list_client_fix] access_list Table altered,
[10/29/2020] [9:29:34 PM] [Migrate  ] › ℹ  info      [pass_auth] Migrating Up...,
[10/29/2020] [9:29:35 PM] [Migrate  ] › ℹ  info      [pass_auth] access_list Table altered,
[10/29/2020] [9:29:35 PM] [Setup    ] › ℹ  info      Creating a new user: [email protected] with password: changeme,
[10/29/2020] [9:29:36 PM] [Setup    ] › ℹ  info      Initial admin setup completed,
[10/29/2020] [9:29:36 PM] [IP Ranges] › ℹ  info      Fetching IP Ranges from online services...,
[10/29/2020] [9:29:36 PM] [IP Ranges] › ℹ  info      Fetching https://ip-ranges.amazonaws.com/ip-ranges.json,
[10/29/2020] [9:29:36 PM] [Setup    ] › ℹ  info      Default settings added,
[10/29/2020] [9:29:41 PM] [IP Ranges] › ✖  error     getaddrinfo EAI_AGAIN ip-ranges.amazonaws.com,
[10/29/2020] [9:29:41 PM] [SSL      ] › ℹ  info      Let's Encrypt Renewal Timer initialized,
[10/29/2020] [9:29:41 PM] [SSL      ] › ℹ  info      Renewing SSL certs close to expiry...,
[10/29/2020] [9:29:41 PM] [IP Ranges] › ℹ  info      IP Ranges Renewal Timer initialized,
[10/29/2020] [9:29:41 PM] [Global   ] › ℹ  info      Backend PID 209 listening on port 3000 ...,
[10/29/2020] [9:29:45 PM] [Nginx    ] › ℹ  info      Reloading Nginx,
[10/29/2020] [9:29:46 PM] [SSL      ] › ℹ  info      Renew Complete,
`QueryBuilder#allowEager` method is deprecated. You should use `allowGraph` instead. `allowEager` method will be removed in 3.0,
`QueryBuilder#eager` method is deprecated. You should use the `withGraphFetched` method instead. `eager` method will be removed in 3.0,
QueryBuilder#omit is deprecated. This method will be removed in version 3.0,
Duplicate relation "access_list" in a relation expression. You should use "a.[b, c]" instead of "[a.b, a.c]". This will cause an error in objection 2.0,
[10/29/2020] [9:35:21 PM] [Nginx    ] › ℹ  info      Reloading Nginx,
[10/29/2020] [9:36:04 PM] [Nginx    ] › ℹ  info      Reloading Nginx,
[10/29/2020] [9:36:04 PM] [SSL      ] › ℹ  info      Requesting Let'sEncrypt certificates for Cert #1: music.removed.de,
[10/29/2020] [9:36:14 PM] [Nginx    ] › ℹ  info      Reloading Nginx,
[10/29/2020] [9:36:14 PM] [Express  ] › ⚠  warning   Command failed: /usr/bin/certbot certonly --non-interactive --config "/etc/letsencrypt.ini" --cert-name "npm-1" --agree-tos --email "[email protected]" --preferred-challenges "dns,http" --domains "music.removed.de" ,
Saving debug log to /var/log/letsencrypt/letsencrypt.log,
Plugins selected: Authenticator webroot, Installer None,
An unexpected error occurred:,
Traceback (most recent call last):,
  File "/usr/lib/python3.8/site-packages/urllib3/connection.py", line 159, in _new_conn,
    conn = connection.create_connection(,
  File "/usr/lib/python3.8/site-packages/urllib3/util/connection.py", line 61, in create_connection,
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):,
  File "/usr/lib/python3.8/socket.py", line 918, in getaddrinfo,
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):,
socket.gaierror: [Errno -3] Try again,
,
During handling of the above exception, another exception occurred:,
,
Traceback (most recent call last):,
  File "/usr/lib/python3.8/site-packages/urllib3/connectionpool.py", line 670, in urlopen,
    httplib_response = self._make_request(,
  File "/usr/lib/python3.8/site-packages/urllib3/connectionpool.py", line 381, in _make_request,
    self._validate_conn(conn),
  File "/usr/lib/python3.8/site-packages/urllib3/connectionpool.py", line 976, in _validate_conn,
    conn.connect(),
  File "/usr/lib/python3.8/site-packages/urllib3/connection.py", line 308, in connect,
    conn = self._new_conn(),
  File "/usr/lib/python3.8/site-packages/urllib3/connection.py", line 171, in _new_conn,
    raise NewConnectionError(,
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0xffff9c5c3370>: Failed to establish a new connection: [Errno -3] Try again,
,
During handling of the above exception, another exception occurred:,
,
Traceback (most recent call last):,
  File "/usr/lib/python3.8/site-packages/requests/adapters.py", line 439, in send,
    resp = conn.urlopen(,
  File "/usr/lib/python3.8/site-packages/urllib3/connectionpool.py", line 724, in urlopen,
    retries = retries.increment(,
  File "/usr/lib/python3.8/site-packages/urllib3/util/retry.py", line 439, in increment,
    raise MaxRetryError(_pool, url, error or ResponseError(cause)),
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='acme-v02.api.letsencrypt.org', port=443): Max retries exceeded with url: /directory (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0xffff9c5c3370>: Failed to establish a new connection: [Errno -3] Try again')),
,
During handling of the above exception, another exception occurred:,
,
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='acme-v02.api.letsencrypt.org', port=443): Max retries exceeded with url: /directory (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0xffff9c5c3370>: Failed to establish a new connection: [Errno -3] Try again')),
Please see the logfiles in /var/log/letsencrypt for more details.,
,
[10/29/2020] [10:12:13 PM] [Nginx    ] › ℹ  info      Reloading Nginx,
[10/29/2020] [10:29:41 PM] [SSL      ] › ℹ  info      Renewing SSL certs close to expiry...,
[10/29/2020] [10:29:45 PM] [Nginx    ] › ℹ  info      Reloading Nginx,
[10/29/2020] [10:29:45 PM] [SSL      ] › ℹ  info      Renew Complete,
[10/29/2020] [10:29:45 PM] [SSL      ] › ✖  error     Certificate is not valid (Command failed: openssl x509 -in /etc/letsencrypt/live/npm-1/fullchain.pem -subject -noout,
Can't open /etc/letsencrypt/live/npm-1/fullchain.pem for reading, No such file or directory,
281472828726840:error:02001002:system library:fopen:No such file or directory:crypto/bio/bss_file.c:69:fopen('/etc/letsencrypt/live/npm-1/fullchain.pem','r'),
281472828726840:error:2006D080:BIO routines:BIO_new_file:no such file:crypto/bio/bss_file.c:76:,
unable to load certificate,
),

my docker-compose:

  GNU nano 3.2                                                           docker-compose.yaml                                                                       

version: "3"
services:
  app:
    image: jc21/nginx-proxy-manager:2
    restart: always
    ports:
      # Public HTTP Port:
      - '80:80'
      # Public HTTPS Port:
      - '443:443'
      # Admin Web Port:
      - '81:81'
    environment:
      # Uncomment this if IPv6 is not enabled on your host
       DISABLE_IPV6: 'true'
    volumes:
      # Make sure this config.json file exists as per instructions above:
      - ./config.json:/app/config/production.json
      - ./data:/data
      - ./letsencrypt:/etc/letsencrypt
    depends_on:
      - db
  db:
    image: webhippie/mariadb:latest
    restart: always
    environment:
      MARIADB_ROOT_PASSWORD: 'npm'
      MARIADB_DATABASE: 'npm'
      MARIADB_USERNAME: 'npm'
      MARIADB_PASSWORD: 'npm'
    volumes:
      - ./data/mysql:/var/lib/mysql

@Vinestou
Copy link

I have this exact same problem. I keep getting an error. I have two instances: one local and one on a vps. vps has no problems. Which logs should i add for info?

@Gieter15
Copy link

Found the solution. Steps I took:

  • Go to cloudflare, set proxy status to "DNS Only"
  • Wait a couple of minutes.
  • Go to Dashboard, Proxy Hosts, disable source, now you can update the ssl certificate.
  • Enable source, and set proxy status back to Proxied on cloudflare.

Hopefully I'm helping someone out there.

@mastan30
Copy link

mastan30 commented Jan 26, 2022

I am still facing this error for creating the Certificate too. This is the error I am seeing, can some one help please?

[1/26/2022] [2:47:11 PM] [Nginx    ] › ℹ  info      Reloading Nginx
[1/26/2022] [2:47:16 PM] [SSL      ] › ℹ  info      Requesting Let'sEncrypt certificates for Cert #1: xxxxxx.duckdns.org
[1/26/2022] [2:47:16 PM] [SSL      ] › ℹ  info      Command: certbot certonly --config "/etc/letsencrypt.ini" --cert-name "npm-1" --agree-tos --authenticator webroot --email "[email protected]" --preferred-challenges "dns,http" --domains "xxxxxx.duckdns.org" 
[1/26/2022] [2:47:29 PM] [Nginx    ] › ℹ  info      Reloading Nginx
[1/26/2022] [2:47:29 PM] [Express  ] › ⚠  warning   Command failed: certbot certonly --config "/etc/letsencrypt.ini" --cert-name "npm-1" --agree-tos --authenticator webroot --email "[email protected]" --preferred-challenges "dns,http" --domains "xxxxxxx.duckdns.org" 
Saving debug log to /var/log/letsencrypt/letsencrypt.log

@LeducH
Copy link

LeducH commented Jan 28, 2022

How to resolve this issue?

@jeb-de
Copy link

jeb-de commented Feb 2, 2022

For me it was a problem with Access Lists.
After changing the "Access" of my proxy host to "Publicly Accessible", I was able to renew the Let's encrypt certificate

@Gh0stRocket
Copy link

Gh0stRocket commented Feb 9, 2022

For those who get an Internal Error while trying to renew certs and have this or a similar error message in their logs:

certbot.errors.CertStorageError: expected /etc/letsencrypt/live/example.com/cert.pem to be a symlink
2021-01-24 11:24:27,110:WARNING:certbot.renewal:Renewal configuration file /etc/letsencrypt/renewal/example.com.conf is broken. Skipping.

Try this #1816 (comment)

@cptskippy
Copy link

@LeducH I was able to resolve the problem through the UI.

  1. Navigate to Proxy Hosts
  2. Edit a Host entry with a bad SSL Cert
  3. Navigate to the SSL Tab
  4. Click on the SSL Certificate field and in the drop down select "Request a new SSL Certificate"
  5. Click Save
  6. Navigate to SSL Certificates
  7. Delete the old Certificate

After doing the above steps for each of my Proxy Hosts, they can be renewed from the GUI. I'm not sure if Auto-renew will work but I guess I'll find out in a couple months.

@oneil1838
Copy link

oneil1838 commented Jun 9, 2022

I found a solution to get a renewal.
) open portainer (if you use it, I use it on every docker maschine)
) navigate to "Containers"-> select the ">_" at the npm container
) navigate with the console to "cd /letsencrypt/live/npm-1"
) run "ln -s cert.pem cert2.pem"
) open NPM -> "SSL Ceritifcates" and try to renew one. This was the solution for me... only the cert in the first "npm-" -folder need a symlink and than the manual renew works. I don't have a cert what will stop working in the next days so I cant test the auto renewal yet.

) mayby a reboot of the nginx service helps to push the new cert

I hope this will help you guys too.

@SydDean
Copy link

SydDean commented Nov 18, 2022

For me this was happening to one specific proxy so I knew it must be down to something special about this host. Checked myself several times, protocol correct, ip correct, port correct. Anyway it was my bad. Although I had DNS setup correctly to point the domain to NGinxPM, I also had a left over NAT rule on my firewall that was forwarding incoming directly to the original machine, not NPM. So could be worth checking your firewall to make sure you've not done something as dumb as I did.

@cptskippy
Copy link

For me this was happening to one specific proxy so I knew it must be down to something special about this host. Checked myself several times, protocol correct, ip correct, port correct. Anyway it was my bad. Although I had DNS setup correctly to point the domain to NGinxPM, I also had a left over NAT rule on my firewall that was forwarding incoming directly to the original machine, not NPM. So could be worth checking your firewall to make sure you've not done something as dumb as I did.

I wish it were that simple. I have a half dozen domains on a single IP going to multiple servers. If I had a NAT rule wrong then only one of those servers would be remotely accessible.

@Elwyr
Copy link

Elwyr commented Jan 16, 2023

I don't know if it's the same error, but I can't renew some certs (not sure why some are affected and some are not) unless I turn off all the SSL settings in the proxy host. Turn off force SSL, turn off HTTP/2, turn off HSTS. Go to certs and renew works, then I have to go back and turn all that back on. Not a great solution.

@D3B453R
Copy link

D3B453R commented Jan 23, 2023

I don't know if it's the same error, but I can't renew some certs (not sure why some are affected and some are not) unless I turn off all the SSL settings in the proxy host. Turn off force SSL, turn off HTTP/2, turn off HSTS. Go to certs and renew works, then I have to go back and turn all that back on. Not a great solution.

I have the same issue. Can't renew specific certs unless I've turned all the SSL settings off. I could provide my logs later that day. Tested it on just one host that was failing, still plenty of hosts to test it. :D

@D3B453R
Copy link

D3B453R commented Jan 23, 2023

I don't know if it's the same error, but I can't renew some certs (not sure why some are affected and some are not) unless I turn off all the SSL settings in the proxy host. Turn off force SSL, turn off HTTP/2, turn off HSTS. Go to certs and renew works, then I have to go back and turn all that back on. Not a great solution.

I have the same issue. Can't renew specific certs unless I've turned all the SSL settings off. I could provide my logs later that day. Tested it on just one host that was failing, still plenty of hosts to test it. :D

Tested it now on a few hosts, I have to disable the "Force SSL" under SSL-Settings to renew my certs. When I don't disable it I'll get the "internal error" with the following logs on version 2.9.19:

letsencrypt.log.txt

`2023-01-23 14:59:03,971:DEBUG:urllib3.connectionpool:https://acme-v02.api.letsencrypt.org:443 "POST /acme/authz-v3/197593605707 HTTP/1.1" 200 805
2023-01-23 14:59:03,972:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Date: Mon, 23 Jan 2023 14:59:03 GMT
Content-Type: application/json
Content-Length: 805
Connection: keep-alive
Boulder-Requester: 286501160
Cache-Control: public, max-age=0, no-cache
Link: https://acme-v02.api.letsencrypt.org/directory;rel="index"
Replay-Nonce: CENSORED
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
"identifier": {
"type": "dns",
"value": "my.domain.tld"
},
"status": "pending",
"expires": "2023-01-30T14:58:59Z",
"challenges": [
{
"type": "http-01",
"status": "pending",
"url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/197593605707/FAxihQ",
"token": "CENSORED"
},
{
"type": "dns-01",
"status": "pending",
"url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/197593605707/EcSxMg",
"token": "CENSORED"
},
{
"type": "tls-alpn-01",
"status": "pending",
"url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/197593605707/NKSXIA",
"token": "CENSORED"
}
]
}
2023-01-23 14:59:03,972:DEBUG:acme.client:Storing nonce: CENSORED
2023-01-23 14:59:06,974:DEBUG:acme.client:JWS payload:
b''
2023-01-23 14:59:06,975:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/authz-v3/197593605707:
{
"protected": "CENSORED",
"signature": "CENSORED",
"payload": ""
}
2023-01-23 14:59:07,128:DEBUG:urllib3.connectionpool:https://acme-v02.api.letsencrypt.org:443 "POST /acme/authz-v3/197593605707 HTTP/1.1" 200 1857
2023-01-23 14:59:07,128:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Date: Mon, 23 Jan 2023 14:59:07 GMT
Content-Type: application/json
Content-Length: 1857
Connection: keep-alive
Boulder-Requester: 286501160
Cache-Control: public, max-age=0, no-cache
Link: https://acme-v02.api.letsencrypt.org/directory;rel="index"
Replay-Nonce: CENSORED
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
"identifier": {
"type": "dns",
"value": "my.domain.tld"
},
"status": "invalid",
"expires": "2023-01-30T14:58:59Z",
"challenges": [
{
"type": "http-01",
"status": "invalid",
"error": {
"type": "urn:ietf:params:acme:error:connection",
"detail": "MY-IP: Fetching https://my.domain.tld/.well-known/acme-challenge/CENSORED: Error getting validation data",
"status": 400
},
"url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/197593605707/FAxihQ",
"token": "CENSORED",
"validationRecord": [
{
"url": "http://my.domain.tld/.well-known/acme-challenge/CENSORED",
"hostname": "my.domain.tld",
"port": "80",
"addressesResolved": [
"MY-IP",
"MY-IP-V6"
],
"addressUsed": "MY-IP-V6"
},
{
"url": "http://my.domain.tld/.well-known/acme-challenge/CENSORED",
"hostname": "my.domain.tld",
"port": "80",
"addressesResolved": [
"MY-IP",
"MY-IP-V6"
],
"addressUsed": "MY-IP"
},
{
"url": "https://my.domain.tld/.well-known/acme-challenge/CENSORED",
"hostname": "my.domain.tld",
"port": "443",
"addressesResolved": [
"MY-IP",
"MY-IP-V6"
],
"addressUsed": "MY-IP-V6"
}
],
"validated": "2023-01-23T14:58:59Z"
}
]
}
2023-01-23 14:59:07,128:DEBUG:acme.client:Storing nonce: CENSORED
2023-01-23 14:59:07,129:INFO:certbot._internal.auth_handler:Challenge failed for domain my.domain.tld
2023-01-23 14:59:07,129:INFO:certbot._internal.auth_handler:http-01 challenge for my.domain.tld
2023-01-23 14:59:07,129:DEBUG:certbot._internal.display.obj:Notifying user:
Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
Domain: my.domain.tld
Type: connection
Detail: MY-IP: Fetching https://my.domain.tld/.well-known/acme-challenge/***CENSORED***: Error getting validation data

Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.

2023-01-23 14:59:07,129:DEBUG:certbot._internal.error_handler:Encountered exception:
Traceback (most recent call last):
File "/opt/certbot/lib/python3.7/site-packages/certbot/_internal/auth_handler.py", line 106, in handle_authorizations
self._poll_authorizations(authzrs, max_retries, best_effort)
File "/opt/certbot/lib/python3.7/site-packages/certbot/_internal/auth_handler.py", line 206, in _poll_authorizations
raise errors.AuthorizationError('Some challenges have failed.')
certbot.errors.AuthorizationError: Some challenges have failed.

2023-01-23 14:59:07,129:DEBUG:certbot._internal.error_handler:Calling registered functions
2023-01-23 14:59:07,129:INFO:certbot._internal.auth_handler:Cleaning up challenges
2023-01-23 14:59:07,129:DEBUG:certbot._internal.plugins.webroot:Removing /data/letsencrypt-acme-challenge/.well-known/acme-challenge/CENSORED
2023-01-23 14:59:07,130:DEBUG:certbot._internal.plugins.webroot:All challenges cleaned up
2023-01-23 14:59:07,130:ERROR:certbot._internal.renewal:Failed to renew certificate npm-59 with error: Some challenges have failed.
2023-01-23 14:59:07,131:DEBUG:certbot._internal.renewal:Traceback was:
Traceback (most recent call last):
File "/opt/certbot/lib/python3.7/site-packages/certbot/_internal/renewal.py", line 484, in handle_renewal_request
main.renew_cert(lineage_config, plugins, renewal_candidate)
File "/opt/certbot/lib/python3.7/site-packages/certbot/_internal/main.py", line 1541, in renew_cert
renewed_lineage = _get_and_save_cert(le_client, config, lineage=lineage)
File "/opt/certbot/lib/python3.7/site-packages/certbot/_internal/main.py", line 129, in _get_and_save_cert
renewal.renew_cert(config, domains, le_client, lineage)
File "/opt/certbot/lib/python3.7/site-packages/certbot/_internal/renewal.py", line 344, in renew_cert
new_cert, new_chain, new_key, _ = le_client.obtain_certificate(domains, new_key)
File "/opt/certbot/lib/python3.7/site-packages/certbot/_internal/client.py", line 442, in obtain_certificate
orderr = self._get_order_and_authorizations(csr.data, self.config.allow_subset_of_names)
File "/opt/certbot/lib/python3.7/site-packages/certbot/_internal/client.py", line 510, in _get_order_and_authorizations
authzr = self.auth_handler.handle_authorizations(orderr, self.config, best_effort)
File "/opt/certbot/lib/python3.7/site-packages/certbot/_internal/auth_handler.py", line 106, in handle_authorizations
self._poll_authorizations(authzrs, max_retries, best_effort)
File "/opt/certbot/lib/python3.7/site-packages/certbot/_internal/auth_handler.py", line 206, in _poll_authorizations
raise errors.AuthorizationError('Some challenges have failed.')
certbot.errors.AuthorizationError: Some challenges have failed.

2023-01-23 14:59:07,131:DEBUG:certbot._internal.display.obj:Notifying user:


2023-01-23 14:59:07,131:ERROR:certbot._internal.renewal:All renewals failed. The following certificates could not be renewed:
2023-01-23 14:59:07,131:ERROR:certbot._internal.renewal: /etc/letsencrypt/live/npm-59/fullchain.pem (failure)
2023-01-23 14:59:07,131:DEBUG:certbot._internal.display.obj:Notifying user: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-01-23 14:59:07,131:DEBUG:certbot._internal.log:Exiting abnormally:
Traceback (most recent call last):
File "/usr/bin/certbot", line 8, in
sys.exit(main())
File "/opt/certbot/lib/python3.7/site-packages/certbot/main.py", line 19, in main
return internal_main.main(cli_args)
File "/opt/certbot/lib/python3.7/site-packages/certbot/_internal/main.py", line 1744, in main
return config.func(config, plugins)
File "/opt/certbot/lib/python3.7/site-packages/certbot/_internal/main.py", line 1630, in renew
renewal.handle_renewal_request(config)
File "/opt/certbot/lib/python3.7/site-packages/certbot/_internal/renewal.py", line 511, in handle_renewal_request
f"{len(renew_failures)} renew failure(s), {len(parse_failures)} parse failure(s)")
certbot.errors.Error: 1 renew failure(s), 0 parse failure(s)
2023-01-23 14:59:07,131:ERROR:certbot._internal.log:1 renew failure(s), 0 parse failure(s)
`

@kmanwar89
Copy link

+1 as someone else experiencing this error - I've been flooded with renewal emails from LetsEncrypt, and it seems the certs don't auto renew...

Fortunately, it was only 2 or 3 certs so I manually deleted them and re-requested them without issues. Is there any workaround identified?

The logs I get are the below, and seem to be continuous:

npm  | 2023-01-27T21:31:34.074189905Z 
npm  | 2023-01-27T21:31:34.074194695Z     at ChildProcess.exithandler (node:child_process:402:12)
npm  | 2023-01-27T21:31:34.074199735Z     at ChildProcess.emit (node:events:513:28)
npm  | 2023-01-27T21:31:34.074204645Z     at maybeClose (node:internal/child_process:1100:16)
npm  | 2023-01-27T21:31:34.074209645Z     at Process.ChildProcess._handle.onexit (node:internal/child_process:304:5)
npm  | 2023-01-27T22:31:32.252441362Z [1/27/2023] [10:31:32 PM] [SSL      ] › ℹ  info      Renewing SSL certs close to expiry...
npm  | 2023-01-27T22:31:32.261066175Z [1/27/2023] [10:31:32 PM] [IP Ranges] › ℹ  info      Fetching IP Ranges from online services...
npm  | 2023-01-27T22:31:32.261095964Z [1/27/2023] [10:31:32 PM] [IP Ranges] › ℹ  info      Fetching https://ip-ranges.amazonaws.com/ip-ranges.json
npm  | 2023-01-27T22:31:32.616221690Z [1/27/2023] [10:31:32 PM] [IP Ranges] › ℹ  info      Fetching https://www.cloudflare.com/ips-v4
npm  | 2023-01-27T22:31:32.786813548Z [1/27/2023] [10:31:32 PM] [IP Ranges] › ℹ  info      Fetching https://www.cloudflare.com/ips-v6
npm  | 2023-01-27T22:31:33.083437363Z [1/27/2023] [10:31:33 PM] [Nginx    ] › ℹ  info      Reloading Nginx
npm  | 2023-01-27T22:31:33.749887794Z [1/27/2023] [10:31:33 PM] [SSL      ] › ✖  error     Error: Command failed: certbot renew --non-interactive --quiet --config "/etc/letsencrypt.ini" --preferred-challenges "dns,http" --disable-hook-validation  
npm  | 2023-01-27T22:31:33.749917303Z Renewal configuration file /etc/letsencrypt/renewal/npm-14.conf is broken.
npm  | 2023-01-27T22:31:33.749922373Z The error was: expected /etc/letsencrypt/live/npm-14/cert.pem to be a symlink
npm  | 2023-01-27T22:31:33.749926401Z Skipping.
npm  | 2023-01-27T22:31:33.749930219Z Renewal configuration file /etc/letsencrypt/renewal/npm-16.conf is broken.
npm  | 2023-01-27T22:31:33.749934127Z The error was: expected /etc/letsencrypt/live/npm-16/cert.pem to be a symlink
npm  | 2023-01-27T22:31:33.749937964Z Skipping.
npm  | 2023-01-27T22:31:33.749941702Z Renewal configuration file /etc/letsencrypt/renewal/npm-19.conf is broken.
npm  | 2023-01-27T22:31:33.749945519Z The error was: expected /etc/letsencrypt/live/npm-19/cert.pem to be a symlink
npm  | 2023-01-27T22:31:33.749949327Z Skipping.
npm  | 2023-01-27T22:31:33.749952954Z Renewal configuration file /etc/letsencrypt/renewal/npm-2.conf is broken.
npm  | 2023-01-27T22:31:33.749956752Z The error was: expected /etc/letsencrypt/live/npm-2/cert.pem to be a symlink
npm  | 2023-01-27T22:31:33.749973365Z Skipping.
npm  | 2023-01-27T22:31:33.749977202Z Renewal configuration file /etc/letsencrypt/renewal/npm-20.conf is broken.
npm  | 2023-01-27T22:31:33.749980960Z The error was: expected /etc/letsencrypt/live/npm-20/cert.pem to be a symlink
npm  | 2023-01-27T22:31:33.749984868Z Skipping.
npm  | 2023-01-27T22:31:33.749988625Z Renewal configuration file /etc/letsencrypt/renewal/npm-21.conf is broken.
npm  | 2023-01-27T22:31:33.749992433Z The error was: expected /etc/letsencrypt/live/npm-21/cert.pem to be a symlink
npm  | 2023-01-27T22:31:33.749996200Z Skipping.
npm  | 2023-01-27T22:31:33.749999838Z Renewal configuration file /etc/letsencrypt/renewal/npm-23.conf is broken.
npm  | 2023-01-27T22:31:33.750003595Z The error was: expected /etc/letsencrypt/live/npm-23/cert.pem to be a symlink
npm  | 2023-01-27T22:31:33.750007373Z Skipping.
npm  | 2023-01-27T22:31:33.750011000Z Renewal configuration file /etc/letsencrypt/renewal/npm-25.conf is broken.
npm  | 2023-01-27T22:31:33.750014857Z The error was: expected /etc/letsencrypt/live/npm-25/cert.pem to be a symlink
npm  | 2023-01-27T22:31:33.750018675Z Skipping.
npm  | 2023-01-27T22:31:33.750022252Z Renewal configuration file /etc/letsencrypt/renewal/npm-28.conf is broken.
npm  | 2023-01-27T22:31:33.750026020Z The error was: expected /etc/letsencrypt/live/npm-28/cert.pem to be a symlink
npm  | 2023-01-27T22:31:33.750029827Z Skipping.
npm  | 2023-01-27T22:31:33.750033404Z Renewal configuration file /etc/letsencrypt/renewal/npm-29.conf is broken.
npm  | 2023-01-27T22:31:33.750037182Z The error was: expected /etc/letsencrypt/live/npm-29/cert.pem to be a symlink
npm  | 2023-01-27T22:31:33.750040959Z Skipping.
npm  | 2023-01-27T22:31:33.750044577Z Renewal configuration file /etc/letsencrypt/renewal/npm-30.conf is broken.
npm  | 2023-01-27T22:31:33.750048374Z The error was: expected /etc/letsencrypt/live/npm-30/cert.pem to be a symlink
npm  | 2023-01-27T22:31:33.750052172Z Skipping.
npm  | 2023-01-27T22:31:33.750056761Z Renewal configuration file /etc/letsencrypt/renewal/npm-31.conf is broken.
npm  | 2023-01-27T22:31:33.750060629Z The error was: expected /etc/letsencrypt/live/npm-31/cert.pem to be a symlink
npm  | 2023-01-27T22:31:33.750064436Z Skipping.
npm  | 2023-01-27T22:31:33.750068003Z Renewal configuration file /etc/letsencrypt/renewal/npm-32.conf is broken.
npm  | 2023-01-27T22:31:33.750071761Z The error was: expected /etc/letsencrypt/live/npm-32/cert.pem to be a symlink
npm  | 2023-01-27T22:31:33.750075599Z Skipping.
npm  | 2023-01-27T22:31:33.750079156Z Renewal configuration file /etc/letsencrypt/renewal/npm-33.conf is broken.
npm  | 2023-01-27T22:31:33.750082933Z The error was: expected /etc/letsencrypt/live/npm-33/cert.pem to be a symlink
npm  | 2023-01-27T22:31:33.750086721Z Skipping.
npm  | 2023-01-27T22:31:33.750090298Z Renewal configuration file /etc/letsencrypt/renewal/npm-34.conf is broken.
npm  | 2023-01-27T22:31:33.750094055Z The error was: expected /etc/letsencrypt/live/npm-34/cert.pem to be a symlink
npm  | 2023-01-27T22:31:33.750103775Z Skipping.
npm  | 2023-01-27T22:31:33.750107612Z Renewal configuration file /etc/letsencrypt/renewal/npm-35.conf is broken.
npm  | 2023-01-27T22:31:33.750111410Z The error was: expected /etc/letsencrypt/live/npm-35/cert.pem to be a symlink
npm  | 2023-01-27T22:31:33.750115749Z Skipping.
npm  | 2023-01-27T22:31:33.750123574Z Renewal configuration file /etc/letsencrypt/renewal/npm-36.conf is broken.
npm  | 2023-01-27T22:31:33.750128795Z The error was: expected /etc/letsencrypt/live/npm-36/cert.pem to be a symlink
npm  | 2023-01-27T22:31:33.750133835Z Skipping.
npm  | 2023-01-27T22:31:33.750138564Z Renewal configuration file /etc/letsencrypt/renewal/npm-39.conf is broken.
npm  | 2023-01-27T22:31:33.750143584Z The error was: expected /etc/letsencrypt/live/npm-39/cert.pem to be a symlink
npm  | 2023-01-27T22:31:33.750148634Z Skipping.
npm  | 2023-01-27T22:31:33.750153424Z Renewal configuration file /etc/letsencrypt/renewal/npm-40.conf is broken.
npm  | 2023-01-27T22:31:33.750159035Z The error was: expected /etc/letsencrypt/live/npm-40/cert.pem to be a symlink
npm  | 2023-01-27T22:31:33.750164726Z Skipping.
npm  | 2023-01-27T22:31:33.750169586Z Renewal configuration file /etc/letsencrypt/renewal/npm-42.conf is broken.
npm  | 2023-01-27T22:31:33.750175417Z The error was: expected /etc/letsencrypt/live/npm-42/cert.pem to be a symlink
npm  | 2023-01-27T22:31:33.750180748Z Skipping.
npm  | 2023-01-27T22:31:33.750185498Z Renewal configuration file /etc/letsencrypt/renewal/npm-43.conf is broken.
npm  | 2023-01-27T22:31:33.750190498Z The error was: expected /etc/letsencrypt/live/npm-43/cert.pem to be a symlink
npm  | 2023-01-27T22:31:33.750195568Z Skipping.
npm  | 2023-01-27T22:31:33.750200317Z Renewal configuration file /etc/letsencrypt/renewal/npm-44.conf is broken.
npm  | 2023-01-27T22:31:33.750205337Z The error was: expected /etc/letsencrypt/live/npm-44/cert.pem to be a symlink
npm  | 2023-01-27T22:31:33.750210387Z Skipping.
npm  | 2023-01-27T22:31:33.750215177Z 0 renew failure(s), 22 parse failure(s)
npm  | 2023-01-27T22:31:33.750220107Z 
npm  | 2023-01-27T22:31:33.750224856Z     at ChildProcess.exithandler (node:child_process:402:12)
npm  | 2023-01-27T22:31:33.750229856Z     at ChildProcess.emit (node:events:513:28)
npm  | 2023-01-27T22:31:33.750234826Z     at maybeClose (node:internal/child_process:1100:16)
npm  | 2023-01-27T22:31:33.750239856Z     at Process.ChildProcess._handle.onexit (node:internal/child_process:304:5)
npm  | 2023-01-27T23:31:32.252935240Z [1/27/2023] [11:31:32 PM] [SSL      ] › ℹ  info      Renewing SSL certs close to expiry...
npm  | 2023-01-27T23:31:33.972358199Z [1/27/2023] [11:31:33 PM] [SSL      ] › ✖  error     Error: Command failed: certbot renew --non-interactive --quiet --config "/etc/letsencrypt.ini" --preferred-challenges "dns,http" --disable-hook-validation  
npm  | 2023-01-27T23:31:33.972396275Z Renewal configuration file /etc/letsencrypt/renewal/npm-14.conf is broken.
npm  | 2023-01-27T23:31:33.972423779Z The error was: expected /etc/letsencrypt/live/npm-14/cert.pem to be a symlink
npm  | 2023-01-27T23:31:33.972429591Z Skipping.
npm  | 2023-01-27T23:31:33.972434581Z Renewal configuration file /etc/letsencrypt/renewal/npm-16.conf is broken.
npm  | 2023-01-27T23:31:33.972439781Z The error was: expected /etc/letsencrypt/live/npm-16/cert.pem to be a symlink
npm  | 2023-01-27T23:31:33.972444951Z Skipping.
npm  | 2023-01-27T23:31:33.972449861Z Renewal configuration file /etc/letsencrypt/renewal/npm-19.conf is broken.
npm  | 2023-01-27T23:31:33.972454931Z The error was: expected /etc/letsencrypt/live/npm-19/cert.pem to be a symlink
npm  | 2023-01-27T23:31:33.972460001Z Skipping.
npm  | 2023-01-27T23:31:33.972464820Z Renewal configuration file /etc/letsencrypt/renewal/npm-2.conf is broken.
npm  | 2023-01-27T23:31:33.972470091Z The error was: expected /etc/letsencrypt/live/npm-2/cert.pem to be a symlink
npm  | 2023-01-27T23:31:33.972475291Z Skipping.
npm  | 2023-01-27T23:31:33.972480181Z Renewal configuration file /etc/letsencrypt/renewal/npm-20.conf is broken.
npm  | 2023-01-27T23:31:33.972485301Z The error was: expected /etc/letsencrypt/live/npm-20/cert.pem to be a symlink
npm  | 2023-01-27T23:31:33.972490381Z Skipping.
npm  | 2023-01-27T23:31:33.972495281Z Renewal configuration file /etc/letsencrypt/renewal/npm-21.conf is broken.
npm  | 2023-01-27T23:31:33.972500361Z The error was: expected /etc/letsencrypt/live/npm-21/cert.pem to be a symlink
npm  | 2023-01-27T23:31:33.972505391Z Skipping.
npm  | 2023-01-27T23:31:33.972513016Z Renewal configuration file /etc/letsencrypt/renewal/npm-23.conf is broken.
npm  | 2023-01-27T23:31:33.972518236Z The error was: expected /etc/letsencrypt/live/npm-23/cert.pem to be a symlink
npm  | 2023-01-27T23:31:33.972523306Z Skipping.
npm  | 2023-01-27T23:31:33.972528186Z Renewal configuration file /etc/letsencrypt/renewal/npm-25.conf is broken.
npm  | 2023-01-27T23:31:33.972533316Z The error was: expected /etc/letsencrypt/live/npm-25/cert.pem to be a symlink
npm  | 2023-01-27T23:31:33.972538396Z Skipping.
npm  | 2023-01-27T23:31:33.972543206Z Renewal configuration file /etc/letsencrypt/renewal/npm-28.conf is broken.
npm  | 2023-01-27T23:31:33.972548326Z The error was: expected /etc/letsencrypt/live/npm-28/cert.pem to be a symlink
npm  | 2023-01-27T23:31:33.972553436Z Skipping.
npm  | 2023-01-27T23:31:33.972558235Z Renewal configuration file /etc/letsencrypt/renewal/npm-29.conf is broken.
npm  | 2023-01-27T23:31:33.972563315Z The error was: expected /etc/letsencrypt/live/npm-29/cert.pem to be a symlink
npm  | 2023-01-27T23:31:33.972568385Z Skipping.
npm  | 2023-01-27T23:31:33.972573205Z Renewal configuration file /etc/letsencrypt/renewal/npm-30.conf is broken.
npm  | 2023-01-27T23:31:33.972578335Z The error was: expected /etc/letsencrypt/live/npm-30/cert.pem to be a symlink
npm  | 2023-01-27T23:31:33.972583425Z Skipping.
npm  | 2023-01-27T23:31:33.972589768Z Renewal configuration file /etc/letsencrypt/renewal/npm-31.conf is broken.
npm  | 2023-01-27T23:31:33.972601641Z The error was: expected /etc/letsencrypt/live/npm-31/cert.pem to be a symlink
npm  | 2023-01-27T23:31:33.972606952Z Skipping.
npm  | 2023-01-27T23:31:33.972611761Z Renewal configuration file /etc/letsencrypt/renewal/npm-32.conf is broken.
npm  | 2023-01-27T23:31:33.972616821Z The error was: expected /etc/letsencrypt/live/npm-32/cert.pem to be a symlink
npm  | 2023-01-27T23:31:33.972621861Z Skipping.
npm  | 2023-01-27T23:31:33.972626681Z Renewal configuration file /etc/letsencrypt/renewal/npm-33.conf is broken.
npm  | 2023-01-27T23:31:33.972631761Z The error was: expected /etc/letsencrypt/live/npm-33/cert.pem to be a symlink
npm  | 2023-01-27T23:31:33.972636821Z Skipping.
npm  | 2023-01-27T23:31:33.972641700Z Renewal configuration file /etc/letsencrypt/renewal/npm-34.conf is broken.
npm  | 2023-01-27T23:31:33.972646851Z The error was: expected /etc/letsencrypt/live/npm-34/cert.pem to be a symlink
npm  | 2023-01-27T23:31:33.972651951Z Skipping.
npm  | 2023-01-27T23:31:33.972657211Z Renewal configuration file /etc/letsencrypt/renewal/npm-35.conf is broken.
npm  | 2023-01-27T23:31:33.972662291Z The error was: expected /etc/letsencrypt/live/npm-35/cert.pem to be a symlink
npm  | 2023-01-27T23:31:33.972667361Z Skipping.
npm  | 2023-01-27T23:31:33.972672211Z Renewal configuration file /etc/letsencrypt/renewal/npm-36.conf is broken.
npm  | 2023-01-27T23:31:33.972677301Z The error was: expected /etc/letsencrypt/live/npm-36/cert.pem to be a symlink
npm  | 2023-01-27T23:31:33.972682621Z Skipping.
npm  | 2023-01-27T23:31:33.972687461Z Renewal configuration file /etc/letsencrypt/renewal/npm-39.conf is broken.
npm  | 2023-01-27T23:31:33.972692621Z The error was: expected /etc/letsencrypt/live/npm-39/cert.pem to be a symlink
npm  | 2023-01-27T23:31:33.972697701Z Skipping.
npm  | 2023-01-27T23:31:33.972702501Z Renewal configuration file /etc/letsencrypt/renewal/npm-40.conf is broken.
npm  | 2023-01-27T23:31:33.972707571Z The error was: expected /etc/letsencrypt/live/npm-40/cert.pem to be a symlink
npm  | 2023-01-27T23:31:33.972712651Z Skipping.
npm  | 2023-01-27T23:31:33.972717480Z Renewal configuration file /etc/letsencrypt/renewal/npm-42.conf is broken.
npm  | 2023-01-27T23:31:33.972722681Z The error was: expected /etc/letsencrypt/live/npm-42/cert.pem to be a symlink
npm  | 2023-01-27T23:31:33.972727751Z Skipping.
npm  | 2023-01-27T23:31:33.972732600Z Renewal configuration file /etc/letsencrypt/renewal/npm-43.conf is broken.
npm  | 2023-01-27T23:31:33.972737740Z The error was: expected /etc/letsencrypt/live/npm-43/cert.pem to be a symlink
npm  | 2023-01-27T23:31:33.972742851Z Skipping.
npm  | 2023-01-27T23:31:33.972747690Z Renewal configuration file /etc/letsencrypt/renewal/npm-44.conf is broken.
npm  | 2023-01-27T23:31:33.972752800Z The error was: expected /etc/letsencrypt/live/npm-44/cert.pem to be a symlink
npm  | 2023-01-27T23:31:33.972757920Z Skipping.
npm  | 2023-01-27T23:31:33.972763161Z 0 renew failure(s), 22 parse failure(s)
npm  | 2023-01-27T23:31:33.972768221Z 
npm  | 2023-01-27T23:31:33.972778411Z     at ChildProcess.exithandler (node:child_process:402:12)
npm  | 2023-01-27T23:31:33.972783591Z     at ChildProcess.emit (node:events:513:28)
npm  | 2023-01-27T23:31:33.972788561Z     at maybeClose (node:internal/child_process:1100:16)
npm  | 2023-01-27T23:31:33.972793621Z     at Process.ChildProcess._handle.onexit (node:internal/child_process:304:5)
npm  | 2023-01-28T00:09:59.625386675Z [1/28/2023] [12:09:59 AM] [Nginx    ] › ℹ  info      Reloading Nginx
npm  | 2023-01-28T00:10:10.815170898Z [1/28/2023] [12:10:10 AM] [Nginx    ] › ℹ  info      Reloading Nginx
npm  | 2023-01-28T00:11:55.099793912Z [1/28/2023] [12:11:55 AM] [Nginx    ] › ℹ  info      Reloading Nginx

@EDIflyer
Copy link

Same issue here - just had 10 renewal notices this morning - it had been working fine and now none have renewed. @jc21 or anyone else, any word on a fix for this? It's getting to be a fairly major issue and is quite frustrating to have to keep recreating all the certs every month!

@romeolazar
Copy link

Yes, indeed. The issue is still present in the latest version. The only option is to delete and request a new certificate. Hope for a fix. THANK YOU.

@momoirodouhu
Copy link

In my case, this issue was solved by deleting the IPv6 address from the DNS record.

@abdros
Copy link

abdros commented Apr 10, 2023

I had the same issue and solved it by adding a DNS CAA record for the HOST.MYDOMAIN.TLD and setting letsencrypt.org as an authorized certificate provider (I use EasyDNS).
What made me think of this was an email that letsencrypt had sent some time ago regarding this soon-to-come requirement from DNS providers.
Nginx Proxy Manager v2.7.1
Hope this helps others.

@EDIflyer
Copy link

Sadly I don't think that's an option for my DNS provider. The thing is it used to work fine and the issue seems to be around https being enforced even for the LetsEncrypt check so I'm hoping it's sortable in the code...

@abdros
Copy link

abdros commented Apr 10, 2023

Sadly I don't think that's an option for my DNS provider. The thing is it used to work fine and the issue seems to be around https being enforced even for the LetsEncrypt check so I'm hoping it's sortable in the code...

I do not know if it helps, but I had the "Force SSL" option selected in npm, and it worked. I do not know either if all DNS providers are enforcing the CAA requirement. I wish you good luck.

@EDIflyer
Copy link

I do not know if it helps, but I had the "Force SSL" option selected in npm, and it worked. I do not know either if all DNS providers are enforcing the CAA requirement. I wish you good luck.

Yep it seems intermittent - when it stopped working I found if I switch that off it seems to work again obtaining certs, but at present both servers have SSL certs far enough away it's not trying to renew them yet

@kaffeepause07
Copy link

"Force SSL" option was the problem on my site.
Fixed it by changing the force ssl config file.
The force ssl file works with this settings:


set $url "${scheme}:${request_uri}";
if ($url ~ "^http:(?!/\.well-known/acme-challenge/(.*))") {
        return 301 https://$host$request_uri;
}

Get this fixe from this site: https://community.alarmiator.de/t/lets-encrypt-zertifikat-wird-von-nginx-proxy-manager-nicht-aktualisiert/380/3

@D3B453R
Copy link

D3B453R commented Aug 27, 2023

"Force SSL" option was the problem on my site.
Fixed it by changing the force ssl config file.
The force ssl file works with this settings:


set $url "${scheme}:${request_uri}";
if ($url ~ "^http:(?!/\.well-known/acme-challenge/(.*))") {
        return 301 https://$host$request_uri;
}

Get this fixe from this site: https://community.alarmiator.de/t/lets-encrypt-zertifikat-wird-von-nginx-proxy-manager-nicht-aktualisiert/380/3

This works for me. I was already thinking about an alternative for npm.

Thank you.

P.S. vielen Dank für die Lösung, die Kaffeepause haben Sie sich definitiv verdient. ;)

@EDIflyer
Copy link

EDIflyer commented Aug 27, 2023

"Force SSL" option was the problem on my site. Fixed it by changing the force ssl config file. The force ssl file works with this settings:


set $url "${scheme}:${request_uri}";
if ($url ~ "^http:(?!/\.well-known/acme-challenge/(.*))") {
        return 301 https://$host$request_uri;
}

Get this fixe from this site: https://community.alarmiator.de/t/lets-encrypt-zertifikat-wird-von-nginx-proxy-manager-nicht-aktualisiert/380/3

Yep - see the PR I did a few weeks ago - #3121 - has been fine for me since then too.

@Orko79
Copy link

Orko79 commented Mar 21, 2024

"Force SSL" option was the problem on my site. Fixed it by changing the force ssl config file. The force ssl file works with this settings:


set $url "${scheme}:${request_uri}";
if ($url ~ "^http:(?!/\.well-known/acme-challenge/(.*))") {
        return 301 https://$host$request_uri;
}

Get this fixe from this site: https://community.alarmiator.de/t/lets-encrypt-zertifikat-wird-von-nginx-proxy-manager-nicht-aktualisiert/380/3

This worked for me too! Just replaced the original file with the one from the link (of course after making a backup ;-) ) Vielen Dank und immer frischen, guten Kaffee!

@EDIflyer
Copy link

Yep - for any not comfortable with making changes themselves, see the PR above I did that made those changes ;)

@abalgo
Copy link

abalgo commented Oct 13, 2024

For me, the problem was different. I've executed the certbot directly in the nginxproxymanager docker container and the problem was more explicit:
ERROR:certbot._internal.renewal:Renewal configuration file /etc/letsencrypt/renewal/npm-3.conf is broken
ERROR:certbot._internal.renewal:The error was: expected /etc/letsencrypt/live/npm-3/cert.pem to be a symlink. Skipping.

So I've simply copied the files in ../../archive/npm-3 (actually it was not necessary because the files were already there with the name cert3.pem, chain3.pem, ...)
I've just removed the files and replaced them by symlinks:

cd /etc/letsencrypt/live/npm-3/
rm *.pem
ln -s ../../archive/npm-3/cert3.pem cert.pem
ln -s ../../archive/npm-3/chain3.pem chain.pem
ln -s ../../archive/npm-3/fullchain3.pem fullchain.pem
ln -s ../../archive/npm-3/privkey3.pem privkey.pem

of course, replace the "3" by the number matching your situation (the biggest one in the archive directory)

and it works. I've renewed the certificates successfully from the user interface.

I hope it will help.

@maurommx
Copy link

I use nginx proxy manager and now it simply does not update the certificates in any way. I would like to know what the problem is, where exactly do I see the error occurring 500 internal error, I want to know what internal error this is, what is the real error message, where can I find it? I cannot renew this certificate manually in some way, I need to renew the certificate and I do not know how to solve it, apparently there are millions of people looking for a solution to the problem and cannot solve it...., in my case both port 80 and 443 are being redirected. Normally I have some hosts running on 80 and others on 443 and others with an expired certificate and I need to renew some solution?, is the problem in letsencrypt or in nginx proxy manager?

@D3B453R
Copy link

D3B453R commented Dec 17, 2024

I use nginx proxy manager and now it simply does not update the certificates in any way. I would like to know what the problem is, where exactly do I see the error occurring 500 internal error, I want to know what internal error this is, what is the real error message, where can I find it? I cannot renew this certificate manually in some way, I need to renew the certificate and I do not know how to solve it, apparently there are millions of people looking for a solution to the problem and cannot solve it...., in my case both port 80 and 443 are being redirected. Normally I have some hosts running on 80 and others on 443 and others with an expired certificate and I need to renew some solution?, is the problem in letsencrypt or in nginx proxy manager?

Hey, you can check the logs of the Docker container with "docker logs %container%" replace %container% with your npm-container-name i.e. npm-app-1.
Now you should see why it's failing to renew your certs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests