Emit "days remaining" and count of certificates type performance data metrics

[atc0005]

The question will be whether this applies to any monitored certificate or only the leaf certificate.

Emitting metrics for the specific chain position likely won't be useful, so I'm leaning towards making the assumption that days_remaining (or whatever name is settled on) will be for the certificate in chain position 0.

We could also apply a restriction that the certificate in chain position 0 be a leaf certificate, but that seems fragile.

[atc0005]

Some potential *_days_remaining metrics:

• leaf_cert_days_remaining
• intermediate_certs_days_remaining
• root_cert_days_remaining

Arguments against:

• root_cert_days_remaining
  • including a root certificate in a certificate chain does not appear to be best practice, so emitting a metric for it is questionable
  • emitting the metric conditionally would likely cause issues for tools that always expect a metric to be present
• leaf_cert_days_remaining
  • won't have any use if there are no leaf certs in the chain
    • e.g., if the plugin is used to monitor an intermediates bundle separately
    • not sure if this a real concern?
• intermediate_certs_days_remaining
  • if the certificate is already flagged as problematic, is there value in noting this separately?
    • if a flag is added to ignore expired intermediates, emitting a metric to track the value could be valuable?
  • emitting the metric conditionally would likely cause issues for tools that always expect a metric to be present
    • e.g., if the cert for 20xx contains intermediates, but the replacement cert chain for 20xy contains just a leaf cert

---

All of this said, I'm thinking the best approach is one of:

• name the metric leaf_cert_days_remaining and conditionally emit whenever a leaf certificate is detected
  • this would prevent emitting a metric that communicates "zero days remaining, your cert is about to expire" for (rare?) cases where a cert bundle (e.g., via file flag) is monitored separately
• name the metric days_remaining and document that it is specifically for the certificate in chain position 0
  • this would be regardless of whether the certificate in chain position 0 is a leaf certificate or the outermost intermediate certificate

[atc0005]

Another option is to list metrics for each certificate in the chain based on position.

If we go that route, we still have two options:

• starting at position 0
  • matches openssl output
• starting at position 1
  • matches lscert output
  • matches check_cert plugin output
  • matches ssllabs.com output

Output below for reference.

If we go this route, for consistency we should probably start at position 1 so that there is a strong correlation between the output from the check_cert plugin and lscert CLI tool and the performance data metrics.

---

openssl lists certificates in a chain starting with position 0:

openssl s_client output

$ echo "" | openssl s_client -showcerts -connect www.google.com:443
CONNECTED(0000019C)
---
Certificate chain
 0 s:CN = www.google.com
   i:C = US, O = Google Trust Services LLC, CN = GTS CA 1C3
-----BEGIN CERTIFICATE-----
MIIEhzCCA2+gAwIBAgIQVramuodXnA0KZSP2PDqq9TANBgkqhkiG9w0BAQsFADBG
MQswCQYDVQQGEwJVUzEiMCAGA1UEChMZR29vZ2xlIFRydXN0IFNlcnZpY2VzIExM
QzETMBEGA1UEAxMKR1RTIENBIDFDMzAeFw0yMzAxMDIwODE5MTlaFw0yMzAzMjcw
ODE5MThaMBkxFzAVBgNVBAMTDnd3dy5nb29nbGUuY29tMFkwEwYHKoZIzj0CAQYI
KoZIzj0DAQcDQgAEZ6s7EvAOtBEY/mA7QXOM8eh3zB3CKtQOAPjAgCslU7sXrHM0
fD5nyzdFozwUhSe9t/3FGYvE8NvMHX9dfYBnMKOCAmcwggJjMA4GA1UdDwEB/wQE
AwIHgDATBgNVHSUEDDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMB0GA1UdDgQW
BBTv9gk5apX54rUddZrsu1Obw/jsFzAfBgNVHSMEGDAWgBSKdH+vhc3ulc09nNDi
RhTzcTUdJzBqBggrBgEFBQcBAQReMFwwJwYIKwYBBQUHMAGGG2h0dHA6Ly9vY3Nw
LnBraS5nb29nL2d0czFjMzAxBggrBgEFBQcwAoYlaHR0cDovL3BraS5nb29nL3Jl
cG8vY2VydHMvZ3RzMWMzLmRlcjAZBgNVHREEEjAQgg53d3cuZ29vZ2xlLmNvbTAh
BgNVHSAEGjAYMAgGBmeBDAECATAMBgorBgEEAdZ5AgUDMDwGA1UdHwQ1MDMwMaAv
oC2GK2h0dHA6Ly9jcmxzLnBraS5nb29nL2d0czFjMy9tb1ZEZklTaWEyay5jcmww
ggEEBgorBgEEAdZ5AgQCBIH1BIHyAPAAdQDoPtDaPvUGNTLnVyi8iWvJA9PL0RFr
7Otp4Xd9bQa9bgAAAYVxxzlbAAAEAwBGMEQCIDWx3+yXjTFXE/RB6TC4TdkwC+ZN
mbJZQg3nvd/e/PyDAiBj/jnKBhqFied3XCWeOeRKRA2SUhwhhIARAuvlBVbAIQB3
AHoyjFTYty22IOo44FIe6YQWcDIThU070ivBOlejUutSAAABhXHHOZ4AAAQDAEgw
RgIhAIBXP/MoxR+BO3SUyf6yC2O1jZmOwTo3D0WOal6W6oHGAiEA7U2ZeKuuPeLa
R0z+7b5dXc0Skue3C3dvxY9mGPaUbDAwDQYJKoZIhvcNAQELBQADggEBAEQDzGI/
ofutIT78CFeX23eWbJu8q+4pG/s1XMx7VdfdqySERgX79sYy7xrBVJD2CgGjaEnP
jaFlmfqBKe6crLhvaex4i4s5Rs1Nw6HNWBhYBqbTbvag/85N4HK+y8NxqItbakLT
qsmSi9k0i6Gk5mlUAWGTyYCq893xMwxWtc8qyr6CsVssUGre2KIEZw/wgbMlEZ4L
8aYwd+Bg0XZnvsTCaJfuA8KzJ3R3Wmie3EgQBIudJytATV5wF1zpetffw3TtTjpj
0+IXhUMrVDTDBAs3Gghxyq8+5JjTE0yGEDhup8RpCw2BfWP7c44XBBLHqHiWHD11
AJMyCkuK4vNGWlU=
-----END CERTIFICATE-----
 1 s:C = US, O = Google Trust Services LLC, CN = GTS CA 1C3
   i:C = US, O = Google Trust Services LLC, CN = GTS Root R1
-----BEGIN CERTIFICATE-----
MIIFljCCA36gAwIBAgINAgO8U1lrNMcY9QFQZjANBgkqhkiG9w0BAQsFADBHMQsw
CQYDVQQGEwJVUzEiMCAGA1UEChMZR29vZ2xlIFRydXN0IFNlcnZpY2VzIExMQzEU
MBIGA1UEAxMLR1RTIFJvb3QgUjEwHhcNMjAwODEzMDAwMDQyWhcNMjcwOTMwMDAw
MDQyWjBGMQswCQYDVQQGEwJVUzEiMCAGA1UEChMZR29vZ2xlIFRydXN0IFNlcnZp
Y2VzIExMQzETMBEGA1UEAxMKR1RTIENBIDFDMzCCASIwDQYJKoZIhvcNAQEBBQAD
ggEPADCCAQoCggEBAPWI3+dijB43+DdCkH9sh9D7ZYIl/ejLa6T/belaI+KZ9hzp
kgOZE3wJCor6QtZeViSqejOEH9Hpabu5dOxXTGZok3c3VVP+ORBNtzS7XyV3NzsX
lOo85Z3VvMO0Q+sup0fvsEQRY9i0QYXdQTBIkxu/t/bgRQIh4JZCF8/ZK2VWNAcm
BA2o/X3KLu/qSHw3TT8An4Pf73WELnlXXPxXbhqW//yMmqaZviXZf5YsBvcRKgKA
gOtjGDxQSYflispfGStZloEAoPtR28p3CwvJlk/vcEnHXG0g/Zm0tOLKLnf9LdwL
tmsTDIwZKxeWmLnwi/agJ7u2441Rj72ux5uxiZ0CAwEAAaOCAYAwggF8MA4GA1Ud
DwEB/wQEAwIBhjAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwEgYDVR0T
AQH/BAgwBgEB/wIBADAdBgNVHQ4EFgQUinR/r4XN7pXNPZzQ4kYU83E1HScwHwYD
VR0jBBgwFoAU5K8rJnEaK0gnhS9SZizv8IkTcT4waAYIKwYBBQUHAQEEXDBaMCYG
CCsGAQUFBzABhhpodHRwOi8vb2NzcC5wa2kuZ29vZy9ndHNyMTAwBggrBgEFBQcw
AoYkaHR0cDovL3BraS5nb29nL3JlcG8vY2VydHMvZ3RzcjEuZGVyMDQGA1UdHwQt
MCswKaAnoCWGI2h0dHA6Ly9jcmwucGtpLmdvb2cvZ3RzcjEvZ3RzcjEuY3JsMFcG
A1UdIARQME4wOAYKKwYBBAHWeQIFAzAqMCgGCCsGAQUFBwIBFhxodHRwczovL3Br
aS5nb29nL3JlcG9zaXRvcnkvMAgGBmeBDAECATAIBgZngQwBAgIwDQYJKoZIhvcN
AQELBQADggIBAIl9rCBcDDy+mqhXlRu0rvqrpXJxtDaV/d9AEQNMwkYUuxQkq/BQ
cSLbrcRuf8/xam/IgxvYzolfh2yHuKkMo5uhYpSTld9brmYZCwKWnvy15xBpPnrL
RklfRuFBsdeYTWU0AIAaP0+fbH9JAIFTQaSSIYKCGvGjRFsqUBITTcFTNvNCCK9U
+o53UxtkOCcXCb1YyRt8OS1b887U7ZfbFAO/CVMkH8IMBHmYJvJh8VNS/UKMG2Yr
PxWhu//2m+OBmgEGcYk1KCTd4b3rGS3hSMs9WYNRtHTGnXzGsYZbr8w0xNPM1IER
lQCh9BIiAfq0g3GvjLeMcySsN1PCAJA/Ef5c7TaUEDu9Ka7ixzpiO2xj2YC/WXGs
Yye5TBeg2vZzFb8q3o/zpWwygTMD0IZRcZk0upONXbVRWPeyk+gB9lm+cZv9TSjO
z23HFtz30dZGm6fKa+l3D/2gthsjgx0QGtkJAITgRNOidSOzNIb2ILCkXhAd4FJG
AJ2xDx8hcFH1mt0G/FX0Kw4zd8NLQsLxdxP8c4CU6x+7Nz/OAipmsHMdMqUybDKw
juDEI/9bfU1lcKwrmz3O2+BtjjKAvpafkmO8l7tdufThcV4q5O8DIrGKZTqPwJNl
1IXNDw9bg1kWRxYtnCQ6yICmJhSFm/Y3m6xv+cXDBlHz4n/FsRC6UfTd
-----END CERTIFICATE-----
 2 s:C = US, O = Google Trust Services LLC, CN = GTS Root R1
   i:C = BE, O = GlobalSign nv-sa, OU = Root CA, CN = GlobalSign Root CA
-----BEGIN CERTIFICATE-----
MIIFYjCCBEqgAwIBAgIQd70NbNs2+RrqIQ/E8FjTDTANBgkqhkiG9w0BAQsFADBX
MQswCQYDVQQGEwJCRTEZMBcGA1UEChMQR2xvYmFsU2lnbiBudi1zYTEQMA4GA1UE
CxMHUm9vdCBDQTEbMBkGA1UEAxMSR2xvYmFsU2lnbiBSb290IENBMB4XDTIwMDYx
OTAwMDA0MloXDTI4MDEyODAwMDA0MlowRzELMAkGA1UEBhMCVVMxIjAgBgNVBAoT
GUdvb2dsZSBUcnVzdCBTZXJ2aWNlcyBMTEMxFDASBgNVBAMTC0dUUyBSb290IFIx
MIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAthECix7joXebO9y/lD63
ladAPKH9gvl9MgaCcfb2jH/76Nu8ai6Xl6OMS/kr9rH5zoQdsfnFl97vufKj6bwS
iV6nqlKr+CMny6SxnGPb15l+8Ape62im9MZaRw1NEDPjTrETo8gYbEvs/AmQ351k
KSUjB6G00j0uYODP0gmHu81I8E3CwnqIiru6z1kZ1q+PsAewnjHxgsHA3y6mbWwZ
DrXYfiYaRQM9sHmklCitD38m5agI/pboPGiUU+6DOogrFZYJsuB6jC511pzrp1Zk
j5ZPaK49l8KEj8C8QMALXL32h7M1bKwYUH+E4EzNktMg6TO8UpmvMrUpsyUqtEj5
cuHKZPfmghCN6J3Cioj6OGaK/GP5Afl4/Xtcd/p2h/rs37EOeZVXtL0m79YB0esW
CruOC7XFxYpVq9Os6pFLKcwZpDIlTirxZUTQAs6qzkm06p98g7BAe+dDq6dso499
iYH6TKX/1Y7DzkvgtdizjkXPdsDtQCv9Uw+wp9U7DbGKogPeMa3Md+pvez7W35Ei
Eua++tgy/BBjFFFy3l3WFpO9KWgz7zpm7AeKJt8T11dleCfeXkkUAKIAf5qoIbap
sZWwpbkNFhHax2xIPEDgfg1azVY80ZcFuctL7TlLnMQ/0lUTbiSw1nH69MG6zO0b
9f6BQdgAmD06yK56mDcYBZUCAwEAAaOCATgwggE0MA4GA1UdDwEB/wQEAwIBhjAP
BgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBTkrysmcRorSCeFL1JmLO/wiRNxPjAf
BgNVHSMEGDAWgBRge2YaRQ2XyolQL30EzTSo//z9SzBgBggrBgEFBQcBAQRUMFIw
JQYIKwYBBQUHMAGGGWh0dHA6Ly9vY3NwLnBraS5nb29nL2dzcjEwKQYIKwYBBQUH
MAKGHWh0dHA6Ly9wa2kuZ29vZy9nc3IxL2dzcjEuY3J0MDIGA1UdHwQrMCkwJ6Al
oCOGIWh0dHA6Ly9jcmwucGtpLmdvb2cvZ3NyMS9nc3IxLmNybDA7BgNVHSAENDAy
MAgGBmeBDAECATAIBgZngQwBAgIwDQYLKwYBBAHWeQIFAwIwDQYLKwYBBAHWeQIF
AwMwDQYJKoZIhvcNAQELBQADggEBADSkHrEoo9C0dhemMXoh6dFSPsjbdBZBiLg9
NR3t5P+T4Vxfq7vqfM/b5A3Ri1fyJm9bvhdGaJQ3b2t6yMAYN/olUazsaL+yyEn9
WprKASOshIArAoyZl+tJaox118fessmXn1hIVw41oeQa1v1vg4Fv74zPl6/AhSrw
9U5pCZEt4Wi4wStz6dTZ/CLANx8LZh1J7QJVj2fhMtfTJr9w4z30Z209fOU0iOMy
+qduBmpvvYuR7hZL6Dupszfnw0Skfths18dG9ZKb59UhvmaSGZRVbNQpsg3BZlvi
d0lIKO2d1xozclOzgjXPYovJJIultzkMu34qQb9Sz/yilrbCgj8=
-----END CERTIFICATE-----
---
Server certificate
subject=CN = www.google.com

issuer=C = US, O = Google Trust Services LLC, CN = GTS CA 1C3

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: ECDSA
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 4295 bytes and written 396 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 256 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
depth=2 C = US, O = Google Trust Services LLC, CN = GTS Root R1
verify return:1
depth=1 C = US, O = Google Trust Services LLC, CN = GTS CA 1C3
verify return:1
depth=0 CN = www.google.com
verify return:1
DONE

while tooling from this projects lists certificates starting at 1:

lscert output

$ lscert www.google.com


======================
CERTIFICATES | SUMMARY
======================

- OK: 3 certs retrieved for service running on www.google.com (142.251.15.104) at port 443 using host value "www.google.com"
- OK: Hostname validation using value "www.google.com" successful for leaf certificate
- OK: SANs List validation ignored: 0 SANs entries specified, 1 SANs entries on leaf cert [0 EXPECTED, 0 MISSING, 0 UNEXPECTED]
- OK: Expiration validation successful: leaf cert "www.google.com" expires next with 61d 19h remaining (until 2023-03-27 08:19:18 +0000 UTC) [EXPIRED: 0, EXPIRING: 0, OK: 3]


============================
CERTIFICATES | CHAIN DETAILS
============================

Certificate 1 of 3 (leaf):
        Name: CN=www.google.com
        SANs entries: [www.google.com]
        Issuer: CN=GTS CA 1C3,O=Google Trust Services LLC,C=US
        Serial: 56:B6:A6:BA:87:57:9C:0D:0A:65:23:F6:3C:3A:AA:F5
        Issued On: 2023-01-02 08:19:19 +0000 UTC
        Expiration: 2023-03-27 08:19:18 +0000 UTC
        Status: [OK] 61d 19h remaining

Certificate 2 of 3 (intermediate):
        Name: CN=GTS CA 1C3,O=Google Trust Services LLC,C=US
        SANs entries: []
        Issuer: CN=GTS Root R1,O=Google Trust Services LLC,C=US
        Serial: 02:03:BC:53:59:6B:34:C7:18:F5:01:50:66
        Issued On: 2020-08-13 00:00:42 +0000 UTC
        Expiration: 2027-09-30 00:00:42 +0000 UTC
        Status: [OK] 1709d 11h remaining

Certificate 3 of 3 (intermediate):
        Name: CN=GTS Root R1,O=Google Trust Services LLC,C=US
        SANs entries: []
        Issuer: CN=GlobalSign Root CA,OU=Root CA,O=GlobalSign nv-sa,C=BE
        Serial: 77:BD:0D:6C:DB:36:F9:1A:EA:21:0F:C4:F0:58:D3:0D
        Issued On: 2020-06-19 00:00:42 +0000 UTC
        Expiration: 2028-01-28 00:00:42 +0000 UTC
        Status: [OK] 1829d 11h remaining

[atc0005]

Regardless of which option I go with:

• dipping below 0 into negative days remaining (likely) isn't useful
• perfdata metric(s) should include WARNING and CRITICAL thresholds

[atc0005]

Decided to skip emitting expires_root metric for now. Since I'm using zero as the baseline, emitting 0 when no root certificates are found is likely to be confusing.

Also, as already noted, including the root certificate in the chain is no longer considered a good practice (though it seems it is still common for some CAs to include it in their intermediate bundles).

[atc0005]

Decided to skip emitting expires_root metric for now.

I'll drop support for this from the incoming PR as part of cleaning up the associated branch (e.g., squashing commits). I should look at the ARCHIVE-i445-emit-days-remaining-perfdata branch if I decide to add the support in the future.