Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error "cert created without giving its location header" when using a different CA #265

Closed
AyrA opened this issue Oct 2, 2021 · 22 comments
Closed

Comments

@AyrA
Copy link

AyrA commented Oct 2, 2021

This message is logged for every ACME certificate:

(22)Invalid argument: cert created without giving its location header
(22)Invalid argument: AH10056: processing <scrubbed>: (null)

However, the certificate is still issued and activated. The status page lists the status as "incomplete" and the "Activity" as "finished successfully". Every time the server is restarted it will try to obtain a certificate again.

Configuration for different CA:

MDCertificateAuthority https://api.buypass.com/acme/directory
MDCertificateAgreement https://api.buypass.com/acme/terms/750
  • Server Version: Apache/2.4.49 (Win64) OpenSSL/1.1.1
@icing
Copy link
Owner

icing commented Oct 4, 2021

Hmm, just setup a domain and got a certificate successfully:

[Mon Oct 04 12:29:03.780581 2021] [md:trace2] [pid 2009937:tid 140299590395648] md_crypt.c(1395): read chain with 2 certs
[Mon Oct 04 12:29:03.780591 2021] [md:trace2] [pid 2009937:tid 140299590395648] md_crypt.c(1477): parsed certs from content-type=application/pem-certificate-chain, content-length=4435
[Mon Oct 04 12:29:03.780596 2021] [md:debug] [pid 2009937:tid 140299590395648] md_acme_drive.c(232): 2 certs parsed
[Mon Oct 04 12:29:03.780622 2021] [md:debug] [pid 2009937:tid 140299590395648] md_acme_drive.c(267): poll for cert at https://api.buypass.com/acme-v02/cert/GcLGg8v1zoE
[Mon Oct 04 12:29:03.780628 2021] [md:trace1] [pid 2009937:tid 140299590395648] md_acme_drive.c(444): got chain with 2 certs (0. attempt)
[Mon Oct 04 12:29:03.780632 2021] [md:debug] [pid 2009937:tid 140299590395648] md_acme_drive.c(484): chain retrieved

If you set LogLevel md:trace4could you send me such a failed attempt to "stefan at eissing.org"? thanks.

@AyrA
Copy link
Author

AyrA commented Oct 4, 2021

I can do that but I have to wait a week because the module hit the 20 certs per week limit.

@icing
Copy link
Owner

icing commented Oct 4, 2021

urgs. But they seem to have a test API endoint: https://api.test4.buypass.no/acme/directory

@AyrA
Copy link
Author

AyrA commented Oct 4, 2021

This problem exists on a production system. I can't just switch that over, or everyone is getting SSL errors.

@icing
Copy link
Owner

icing commented Oct 4, 2021

A minimal test setup, I used here:

<MDomainSet buypass.one-of-my-domains.com>
  MDDriveMode always
  MDCertificateAuthority https://api.test4.buypass.no/acme/directory
</MDomainSet>

that will trigger a renewal, even though that domain is not visible in any VirtualHosts. Hope this helps.

@AyrA
Copy link
Author

AyrA commented Oct 4, 2021

I found a domain I could use to create yet another virtual host and sent the log to you. In the mean time, is there a way to edit the json files to convince mod_md that the certs are valid and it should leave the CA alone until renewal is up?

@icing
Copy link
Owner

icing commented Oct 4, 2021

In the logs you provided I see that you have MDMustStaple on configured. This means that the CA should set this extension in the certificates it creates for you. However the certificate Apache gets do not seem to have this set. Therefore, it tries again.

If you configure MDMustStaple off (or comment it out), the renewal attempts should stop.

@AyrA
Copy link
Author

AyrA commented Oct 4, 2021

Seems to work so far.

@icing
Copy link
Owner

icing commented Oct 5, 2021

There it is: https://community.buypass.com/t/60h8l7f/support-for-ocsp-must-staple

This is a bug on their side. If a client submits a certificate request with must-staple set, they can deny it, but they should not pretend all is fine and issue a cert without it.

This makes Apache go into an endless loop until it hits their rate limit.

@AyrA
Copy link
Author

AyrA commented Oct 5, 2021

In that case, adding an appropriate error message like "MDMustStaple is not supported by this CA" would be an acceptable solution.

@icing
Copy link
Owner

icing commented Oct 11, 2021

I agree, if the code could find out that this is the case and not an error on the CA side.

@AyrA
Copy link
Author

AyrA commented Oct 11, 2021

it could also just tell "generated certificate does not match expectations: ".

@icing
Copy link
Owner

icing commented Oct 11, 2021

Yes. What I did already is to tell the reason why a renewal is done. So, in the future, one will see that a new cert is requested because the MustStaple is missing.

But you are right that a check on the retrieved cert with a proper error message would be good as well.

@CRCinAU
Copy link

CRCinAU commented Sep 14, 2023

I just got this error - but I don't have any setting for MustStaple....

I'll quickly try it again with that specifically turned off.

EDIT: It also seems like BuyPass doesn't support secp384r1 either.... So you have to specify secp256r1

@CRCinAU
Copy link

CRCinAU commented Sep 14, 2023

I switched to the Test API URL listed above, and still get:

(22)Invalid argument: cert created without giving its location header
(22)Invalid argument: AH10056: processing <$domain>: (null)

I did eventually get a cert after what looked to be 4 attempts from the testing API.....

I went back to MDCertificateAuthority buypass - and I still get:

(70013)Missing parameter for the specified command line option: AH10056: processing $domain: Too many certificates issued already for requested domains
(70013)Missing parameter for the specified command line option: acme problem urn:ietf:params:acme:error:rateLimited: Too many certificates issued already for requested domains

The config I'm using:

MDBaseServer                    on
MDCertificateProtocol           ACME
MDCAChallenges                  tls-alpn-01 http-01
MDDriveMode                     auto
#MDPrivateKeys                  RSA secp384r1
MDPrivateKeys                   RSA secp256r1
MDRenewWindow                   33%
MDStoreDir                      md
#MDCertificateAuthority         letsencrypt
MDCertificateAuthority          buypass
MDRequireHttps                  permanent
MDCertificateAgreement          accepted
MDMustStaple                    off

@icing
Copy link
Owner

icing commented Sep 14, 2023

rateLimited: Too many certificates issued already for requested domainsmeans that buypass refuses to give new certificates to you as you seem to have reached their rate limit.

They counted the previous attempts as successful. If you have just this one domain, you may try stopping your server, wipe the md directory and start again. This will then create a new account.

@CRCinAU
Copy link

CRCinAU commented Sep 14, 2023

Yeah - I tried this by doing:

$ mv md md.old
$ systemctl restart httpd
wait 5-10 seconds
$ systemctl reload httpd

I think it keeps count of the domain name - cos I'm only testing them with the one system that one a single DNS name...

@CRCinAU
Copy link

CRCinAU commented Sep 23, 2023

I tried this again with buypass today for the same domain.... It created the cert several times and still failed with:

[Sun Sep 24 00:09:35.210334 2023] [md:error] [pid 40123:tid 40126] (22)Invalid argument: cert created without giving its location header
[Sun Sep 24 00:09:35.211559 2023] [md:error] [pid 40123:tid 40126] (22)Invalid argument: AH10056: processing <domain>: (null)
[Sun Sep 24 00:09:52.422743 2023] [md:error] [pid 40464:tid 40469] (22)Invalid argument: cert created without giving its location header
[Sun Sep 24 00:09:52.424005 2023] [md:error] [pid 40464:tid 40469] (22)Invalid argument: AH10056: processing <domain>: (null)
[Sun Sep 24 00:10:10.475356 2023] [md:error] [pid 40769:tid 40772] (22)Invalid argument: cert created without giving its location header
[Sun Sep 24 00:10:10.477005 2023] [md:error] [pid 40769:tid 40772] (22)Invalid argument: AH10056: processing <domain>: (null)
[Sun Sep 24 00:10:34.505717 2023] [md:error] [pid 40769:tid 40772] (22)Invalid argument: cert created without giving its location header
[Sun Sep 24 00:10:34.507411 2023] [md:error] [pid 40769:tid 40772] (22)Invalid argument: AH10056: processing <domain>: (null)
[Sun Sep 24 00:11:26.607397 2023] [md:error] [pid 41162:tid 41165] (22)Invalid argument: cert created without giving its location header
[Sun Sep 24 00:11:26.609041 2023] [md:error] [pid 41162:tid 41165] (22)Invalid argument: AH10056: processing <domain>: (null)
[Sun Sep 24 00:11:40.319072 2023] [md:error] [pid 41162:tid 41165] (70013)Missing parameter for the specified command line option: AH10056: processing <domain>: Too many certificates issued already for requested domains
[Sun Sep 24 00:11:50.131992 2023] [md:error] [pid 41162:tid 41165] (70013)Missing parameter for the specified command line option: AH10056: processing <domain>: Too many certificates issued already for requested domains

This was with the following config:

MDBaseServer                    on
MDCertificateProtocol           ACME
MDCAChallenges                  tls-alpn-01 http-01
MDDriveMode                     auto
#MDPrivateKeys                   RSA secp384r1
MDPrivateKeys                  RSA secp256r1
MDRenewWindow                   33%
MDStoreDir                      md
#MDCertificateAuthority          letsencrypt
MDCertificateAuthority         buypass
MDRequireHttps                  permanent
MDCertificateAgreement          accepted
MDMustStaple                    off

@bitscher
Copy link

bitscher commented Sep 4, 2024

@icing I did encounter this recently and this seems to stem from outdated code in function csr_req

mod_md/src/md_acme_drive.c

Lines 307 to 312 in e5d131b

location = apr_table_get(res->headers, "location");
if (!location) {
md_log_perror(MD_LOG_MARK, MD_LOG_ERR, APR_EINVAL, d->p,
"cert created without giving its location header");
return APR_EINVAL;
}

The Location seems to be the way ACME v1 did provide the download link for the new-cert endpoint but ACMEv2's RFC 8555 does not require it, instead returning an updated order and potentially telling the client to poll the order status until it's valid and a certificate key with the download link is provided in the order json.
It does seem that Boulder did keep the Location header in its ACMEv2 implementation but probably only for historical and compatibility considerations.

https://datatracker.ietf.org/doc/html/rfc8555/#section-7.1

The following table illustrates a typical sequence of requests
   required to establish a new account with the server, prove control of
   an identifier, issue a certificate, and fetch an updated certificate
   some time after issuance.  The "->" is a mnemonic for a Location
   header field pointing to a created resource.

   +-------------------+--------------------------------+--------------+
   | Action            | Request                        | Response     |
   +-------------------+--------------------------------+--------------+
   | Get directory     | GET  directory                 | 200          |
   |                   |                                |              |
   | Get nonce         | HEAD newNonce                  | 200          |
   |                   |                                |              |
   | Create account    | POST newAccount                | 201 ->       |
   |                   |                                | account      |
   |                   |                                |              |
   | Submit order      | POST newOrder                  | 201 -> order |
   |                   |                                |              |
   | Fetch challenges  | POST-as-GET order's            | 200          |
   |                   | authorization urls             |              |
   |                   |                                |              |
   | Respond to        | POST authorization challenge   | 200          |
   | challenges        | urls                           |              |
   |                   |                                |              |
   | Poll for status   | POST-as-GET order              | 200          |
   |                   |                                |              |
   | Finalize order    | POST order's finalize url      | 200          |
   |                   |                                |              |
   | Poll for status   | POST-as-GET order              | 200          |
   |                   |                                |              |
   | Download          | POST-as-GET order's            | 200          |
   | certificate       | certificate url                |              |
   +-------------------+--------------------------------+--------------+

The finalize order step does not require a Location header to be provided.

Now that ACMEv1 support is no more csr_req should be updated to not care about Location and should be updating the order status from the response, if the server is issuing synchronously then the order will go from ready to valid directly and the certificate link will be available in the response.

https://datatracker.ietf.org/doc/html/rfc8555/#section-7.1.6

Note that some of these states may not ever appear in a "status"
field, depending on server behavior. For example, a server that
issues synchronously will never show an order in the "processing"
state. A server that deletes expired authorizations immediately will
never show an authorization in the "expired" state.

This part of the function also seems outdated if RFC8555 is to be trusted a server will not return the certificate as the response for the finalize order.

mod_md/src/md_acme_drive.c

Lines 321 to 335 in e5d131b

/* Check if it already was sent with this response */
ad->chain_up_link = NULL;
if (APR_SUCCESS == (rv = md_cert_read_http(&cert, d->p, res))) {
md_log_perror(MD_LOG_MARK, MD_LOG_DEBUG, rv, d->p, "cert parsed");
apr_array_clear(ad->cred->chain);
APR_ARRAY_PUSH(ad->cred->chain, md_cert_t*) = cert;
get_up_link(d, res->headers);
}
else if (APR_STATUS_IS_ENOENT(rv)) {
rv = APR_SUCCESS;
if (location) {
md_log_perror(MD_LOG_MARK, MD_LOG_DEBUG, rv, d->p,
"cert not in response, need to poll %s", location);
}
}

Also note the issue within md_acmev2_drive_renew

if (MD_ACME_ORDER_ST_READY == ad->order->status) {
rv = md_acme_drive_setup_cred_chain(d, result);
if (APR_SUCCESS != rv) goto leave;
md_log_perror(MD_LOG_MARK, MD_LOG_DEBUG, 0, d->p, "%s: finalized order", d->md->name);
}

The case described above results in the client attempting to finalize the order again and again, even if it was already finalized because function csr_req early returned before the mod_md's order status was updated, so it's still ready on the client side despite being valid on the server side.

In short csr_req needs to be updated to be compliant with RFC 8555 and drop all the assumptions about ACMEv1 that make mod_md incompatible with some servers.

Probably something like this ? (untested+missing error handling, meant as a demonstration of idea more than actual patch)

diff --git a/src/md_acme_drive.c b/src/md_acme_drive.c
index 4bb04f3..fd3c0fe 100644
--- a/src/md_acme_drive.c
+++ b/src/md_acme_drive.c
@@ -295,46 +295,19 @@ static apr_status_t on_init_csr_req(md_acme_req_t *req, void *baton)
     return md_acme_req_body_init(req, jpayload);
 }

-static apr_status_t csr_req(md_acme_t *acme, const md_http_response_t *res, void *baton)
+static apr_status_t csr_req(md_acme_t *acme, apr_pool_t *p, const apr_table_t *hdrs,
+                            md_json_t *body, void *baton)
 {
     md_proto_driver_t *d = baton;
     md_acme_driver_t *ad = d->baton;
-    const char *location;
-    md_cert_t *cert;
-    apr_status_t rv = APR_SUCCESS;
-
+
     (void)acme;
-    location = apr_table_get(res->headers, "location");
-    if (!location) {
-        md_log_perror(MD_LOG_MARK, MD_LOG_ERR, APR_EINVAL, d->p,
-                      "cert created without giving its location header");
-        return APR_EINVAL;
-    }
-    ad->order->certificate = apr_pstrdup(d->p, location);
-    if (APR_SUCCESS != (rv = md_acme_order_save(d->store, d->p, MD_SG_STAGING,
-                                                d->md->name, ad->order, 0))) {
-        md_log_perror(MD_LOG_MARK, MD_LOG_ERR, APR_EINVAL, d->p,
-                      "%s: saving cert url %s", d->md->name, location);
-        return rv;
-    }
-
-    /* Check if it already was sent with this response */
-    ad->chain_up_link = NULL;
-    if (APR_SUCCESS == (rv = md_cert_read_http(&cert, d->p, res))) {
-        md_log_perror(MD_LOG_MARK, MD_LOG_DEBUG, rv, d->p, "cert parsed");
-        apr_array_clear(ad->cred->chain);
-        APR_ARRAY_PUSH(ad->cred->chain, md_cert_t*) = cert;
-        get_up_link(d, res->headers);
-    }
-    else if (APR_STATUS_IS_ENOENT(rv)) {
-        rv = APR_SUCCESS;
-        if (location) {
-            md_log_perror(MD_LOG_MARK, MD_LOG_DEBUG, rv, d->p,
-                          "cert not in response, need to poll %s", location);
-        }
-    }
-
-    return rv;
+    (void)hdrs;
+
+    /* @@@ Not sure about what pool to use */
+    order_update_from_json(ad->order, body, ad->order->p)
+
+    return APR_SUCCESS;
 }

 /**
@@ -382,7 +355,7 @@ apr_status_t md_acme_drive_setup_cred_chain(md_proto_driver_t *d, md_result_t *r

     md_result_activity_printf(result, "Submitting %s CSR to CA", md_pkey_spec_name(spec));
     assert(ad->order->finalize);
-    rv = md_acme_POST(ad->acme, ad->order->finalize, on_init_csr_req, NULL, csr_req, NULL, d);
+    rv = md_acme_POST(ad->acme, ad->order->finalize, on_init_csr_req, csr_req, NULL, NULL, d);

 leave:
     md_acme_report_result(ad->acme, rv, result);
diff --git a/src/md_acmev2_drive.c b/src/md_acmev2_drive.c
index 9dfca96..1c08391 100644
--- a/src/md_acmev2_drive.c
+++ b/src/md_acmev2_drive.c
@@ -164,11 +164,13 @@ retry:
         md_log_perror(MD_LOG_MARK, MD_LOG_DEBUG, 0, d->p, "%s: finalized order", d->md->name);
     }

-    rv = md_acme_order_await_valid(ad->order, ad->acme, d->md,
-                                   ad->authz_monitor_timeout, result, d->p);
-    if (APR_SUCCESS != rv) goto leave;
-
-    if (!ad->order->certificate) {
+    if (MD_ACME_ORDER_ST_PROCESSING == ad->order->status) {
+        rv = md_acme_order_await_valid(ad->order, ad->acme, d->md,
+                                    ad->authz_monitor_timeout, result, d->p);
+        if (APR_SUCCESS != rv) goto leave;
+    }
+
+    if (MD_ACME_ORDER_ST_VALID == ad->order->status && !ad->order->certificate) {
         md_result_set(result, APR_EINVAL, "Order valid, but certificate url is missing.");
         goto leave;
     }

EDIT: the patch is likely missing a save of the updated order json to the store somewhere.

icing added a commit that referenced this issue Sep 6, 2024
header returned by the ACME CA. This was the way it was done in ACME before
it became an IETF standard. Let's Encrypt still supports this, but other
CAs do not. Refs #265.
@icing
Copy link
Owner

icing commented Sep 6, 2024

Thanks @bitscher for pointing out the problem. I just released https://github.com/icing/mod_md/releases/tag/v2.4.27 where the Location header is still used if present, but no longer causes an issue when missing.

Hope this works for you all. Let me know if you can verify it.

@bitscher
Copy link

bitscher commented Sep 7, 2024

Validated for my test case.

@icing
Copy link
Owner

icing commented Sep 13, 2024

Closing as fixed

@icing icing closed this as completed Sep 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants