Add CRL linting infrastructure #699

aaomidi · 2022-11-30T17:48:51Z

ZLint is an invaluable asset for pre-linting certificate issuances and creating a technical enforcement of the various rules around certificate issuance. The recent incidents 1, 2 involving CRLs have created a need to extend linting to CRLs as well.

This PR aims to bring CRL linting and create the necessary code infrastructure to address #458 to add OCSP linting support to ZLint without having to do any large modifications to the codebase.

The major requirement with this PR is to not unintentionally break the existing lints. The decisions made in this PR are generally a reflection of this goal. The major highlights of these changes are as follows:

Introduce CertificateLintInterface and RevocationListLintInterface. Deprecate the existing LintInterface.
Introduce CertificateLint and RevocationListLint. Deprecate the existing Lint struct.
- As part of this, introduce LintMeta to reduce repeating code between the two new lints.
- Create an adapter to go from the deprecated Lint to CertificateLint and vice versa.
Introduce LinterLookup as a generic interface that acts as the storage backend for the different types of lints.
- The global registry's ByName and BySource methods have now been deprecated, and these methods are now scoped down to the specific stores.
Registering a Lint is now deprecated, and it is recommended to use RegisterCertificateLint and RegisterRevocationListLint for Certificates and CRLs respectively.

Some other notes:

Currently zcrypto does not have support for RevocationLists. A PR will be made to add support for that. In the meantime, ox509 is used to denote "original x509", which has the RevocationList struct.
- Before this PR is submitted, an update to zcrypto should happen so I can replace ox509 with x509.
  - I've started on a potential draft for this update in my fork of zcrypto, although I feel like we should probably merge from upstream and re-apply the zcrypto patches instead of trying to make this work
Maintainers, please comment on how the deprecation of these methods should happen. For now I've added @deprecated to the godoc description of these methods.

Future work:

In a future PR, I will migrate all the existing Lints to be CertificateLint instead. This migration is only necessary due to the introduction of LintMeta.
In a future PR, I will add all the lints that Boulder's CRL Linting currently has.
Adding OCSP linting is not in-scope, and I probably won't get around to doing it.

christopher-henderson

Howdy Amir!

Thank you so much. Not only for your contributions, but also for your clear care and technical due-diligence.

While I indeed have comments to leave behind, this is an important enough change that I believe code collaboration may be the best path forward (in the interest of both time and clarity).

As such, I've opened up my on own PR against your source branch (aaomidi#2). A fork-of-a-fork is somewhat unwieldy, but I believe that we can make it work. At the very least, I believe that it is (one of) the best ways for me to share my inputs. If you have an alternative workflow, then I welcome it!

christopher-henderson · 2022-12-03T20:10:14Z

v3/lint/registration.go

+}
+
+// LinterLookup is an interface describing how registered lints can be looked up.
+type LinterLookup[T Lints] interface {


Ah, so while ZLint does indeed compile against 1.18+, we have yet to do a survey of those who consume ZLint as a library to ensure that everyone else has moved on as well.

As such, we have refrained from using generics so far in order to not cause any breakages.

I've been working on this and it's a bit unwieldy so far, going to try to re-approach it and see where I end up. However I did want to make note that I believe go < 1.18 has hit end of life.

christopher-henderson · 2022-12-03T20:31:50Z

v3/lint/registration.go

+}
+
+type lookupImpl[T Lints] struct {
+	sync.RWMutex


Thank you for considering thread safety in your datastructures!

While the ZLint (as a binary) is single threaded, it is easy to imagine library consumes attempting to run lints in parallel. Indeed, this was a motivator in shifting lints from being singleton structs to storing constructors that can return new structs on a per-lint-run basis.

v3/lint/registration.go

Chris Henderson's Code Suggestions for CRL Linting Infrastructure

christopher-henderson · 2022-12-11T20:09:08Z

@aaomidi the code aspects aside, do you happen to have a notion of a what a good first lint would look like for this infrastructure?

It would serve several purposes:

Give ourselves a concrete example to work with in order to flesh out the rest of the existing infrastructure and tool (e.g. lint generation tools. this codebase has a custom code linter to enforce it's own lint style, integration tests etc.)
Give a clear example that we can use in the README as well as serve as an example to the community on how one would be written.

aaomidi · 2022-12-12T21:08:45Z

Hey @christopher-henderson, my plan is to initially implement the lints Let's Encrypt has implemented: https://github.com/letsencrypt/boulder/blob/main/linter/lints/crl/lints.go

christopher-henderson · 2023-02-05T21:01:37Z

Howdy @aaomidi

I fired up a PR against your branch (aaomidi#4) that pulls in a commit from ZCrypto which itself properly brings in the x509.RevocationList type so that we don't have to muck about with having both Zcrypto and the stdlib operating at the same time.

Speaking of which, I went ahead and imported the RevocationList into Zcrypto (zmap/zcrypto#349). It is not merged yet, athough we can certainly refer to this particular branch in go.mod for the meantime.

I also went ahead and tested the changes against go 1.16 (our minimum required version) and everything appears okay so far (thank you for taking the generics out! I am considering firing up a tracking issue of all of the breaking change that we would like to accomplish a a major update one day).

aaomidi · 2023-02-07T21:36:16Z

Awesome news @christopher-henderson.

Let's maybe brainstorm a bit on on maintaining a patch-list for zcrypto as well, so we can follow upstream a bit more?

I'll review your PR today - cheers!

christopher-henderson · 2023-02-12T19:42:25Z

@aaomidi I believe that I had scraped through all references to the mainline x509 so I don't think that there are any outstanding changes to zcrypto that needs addressing (unless I missed something or there are further requirements/usecases you had in mind).

* Take out generics from the registration struct (#3) * updating to use zcrypto * pointing zcrypto back to master * go tidy up --------- Co-authored-by: Amir Omidi <[email protected]>

aaomidi · 2023-02-15T15:44:34Z

@christopher-henderson Would you like to see this PR bring in some revocation list lints too, or should I make a PR on top of this one?

I was thinking of keeping these separate, but I'm open to opinions.

christopher-henderson · 2023-02-15T16:10:56Z

@aaomidi thank you for asking! I think it would be easier to read if they were branches based off of this one. This change alone touched just enough files that some smaller details can get easily get lost.

aaomidi · 2023-02-28T03:26:26Z

@christopher-henderson I have a new PR open against zmap: zmap/zcrypto#354 this will enable me to write a CRL lint

christopher-henderson · 2023-02-28T15:28:30Z

@aaomidi oh good! I was planning on reaching out today regardless,

Just so that I can make sure that you're never blocked on me, is the plan that we get the changes into zcrypto that we need, write a single lint-or-two, and then that's the it for the merge candidate?

aaomidi · 2023-02-28T16:31:37Z

That's the plan, I started writing one of the lints (checking if CRL hasUpdate is valid, and probably a few more), and realized I need the code to parse a CRL.

I think once that's done we should be able to merge this.

aaomidi · 2023-03-13T06:00:01Z

@christopher-henderson I've added a PR onto the crls branch that adds a single CRL lint: aaomidi#5

I think my plan currently is:

If this looks good, let's get the the single lint merged into this branch. Then maybe let's get this branch merged in? We can then follow up with a few more lints in separate PRs.

My justification is that this is a simple lint and doesn't really need a lot of external review? The other lints might be a bit more complex, and might warrant more review on them. This PR is big enough already.

christopher-henderson · 2023-03-19T18:38:36Z

If this looks good, let's get the the single lint merged into this branch.

Yes, I took a look at the example that you provided and looks perfectly reasonable to me.

Then maybe let's get this branch merged in?

I believe that that would be appropriate as both the lint interface as well as the testing interfaces appear to not be out of line with extant patterns. So I believe the consumers of the library aspect of this repo will be able to succeed.

christopher-henderson · 2023-03-26T18:04:05Z

Thank you @aaomidi! We'll be sure to highlight your change in the next release!

defacto64 · 2023-05-30T12:02:17Z

Dear all,

I am not clear if it is possible to lint a CRL from the command line....?

aaomidi · 2023-05-31T15:09:14Z

@defacto64 I actually am not sure. I don't think I tested this when I was implementing CRL linting. I'll turn that into an issue and write up a PR if the investigation shows that I can't use this through CLI.

defacto64 · 2023-05-31T15:17:53Z

Well, if I feed a CRL in DER encoding to zlint (with option -format der), it shows this kind of error:

time="2023-05-31T17:13:50+02:00" level=fatal msg="unable to parse certificate: asn1: structure error: tags don't match (16 vs {class:0 tag:23 length:13 isCompound:false}) {optional:false explicit:false application:false private:false defaultValue:<nil> tag:<nil> stringType:0 timeType:0 set:false omitEmpty:false} validity @211"

If instead I feed it a CRL in PEM encoding, the error message is like this:
time="2023-05-31T17:15:44+02:00" level=fatal msg="unable to parse PEM"

So I guess at this time Zlint cannot lint a CRL from the command-line....

christopher-henderson · 2023-06-04T18:33:10Z

Indeed this is quite the embarrassing little oversight/bug. Thank you, @defacto64, for your testing!

A patch has been merged at af90382 that allows for PEM encode DERs with the X509 CRL header armor to be accepted and dispatched to the appropriate infrastructure.
A new release candidate has also been pushed to https://github.com/zmap/zlint/releases/tag/v3.5.0-rc2

defacto64 · 2023-06-05T11:23:19Z

Nice!
Now it works, at least with PEM-encoded CRLs, although there is only one CRL lint at this time ("e_crl_has_next_update").
Maybe I will try to develop one for checking valid CRL entry reason codes, if that's not already in progress....

aaomidi added 4 commits November 29, 2022 19:14

Add the skeleton around linting CRLs

0f0f94f

Change the entrypoint of zlint

942739b

Add tests for the new skeleton

074d4f6

Address reviews

ab3ee4a

christopher-henderson self-requested a review December 2, 2022 22:01

starting my own suggestions to work coopertaively on he change

a4c0c02

christopher-henderson requested changes Dec 4, 2022

View reviewed changes

Merge pull request #2 from christopher-henderson/chris/crls

cadcb1c

Chris Henderson's Code Suggestions for CRL Linting Infrastructure

Take out generics from the registration struct (#3)

c26c92d

christopher-henderson self-requested a review February 5, 2023 19:33

christopher-henderson mentioned this pull request Feb 5, 2023

Bring in support for Certificate Revocation Lists zmap/zcrypto#349

Merged

Update to use Zcrypto instead of stdlib crypto for RevocationList (#4)

7042d85

* Take out generics from the registration struct (#3) * updating to use zcrypto * pointing zcrypto back to master * go tidy up --------- Co-authored-by: Amir Omidi <[email protected]>

aaomidi added 4 commits March 13, 2023 00:51

Merge branch 'master' into crls

67d295e

Tidy go mod

d670e8f

Update zcrypto

6af892b

go mod tidy one more time

7776df0

Bypass lint for Registry

fd76904

Add NextUpdate CRL lint (#5)

f84563a

christopher-henderson approved these changes Mar 26, 2023

View reviewed changes

christopher-henderson merged commit 997ad51 into zmap:master Mar 26, 2023

aaomidi mentioned this pull request May 31, 2023

Make CRL Linting possible through CLI #720

Closed

aaomidi mentioned this pull request Nov 10, 2023

Gauging interest in doing a large-scale code change migrating everything from Lints -> CertificateLints #765

Open

This was referenced Nov 13, 2023

CABF SMIME BR 7.1.2.3.m - Adobe Extensions #763

Merged

[LSC] Convert all Lints to CertificateLints #767

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add CRL linting infrastructure #699

Add CRL linting infrastructure #699

aaomidi commented Nov 30, 2022

christopher-henderson left a comment

christopher-henderson Dec 3, 2022

aaomidi Dec 12, 2022

christopher-henderson Dec 3, 2022

christopher-henderson commented Dec 11, 2022

aaomidi commented Dec 12, 2022

christopher-henderson commented Feb 5, 2023

aaomidi commented Feb 7, 2023

christopher-henderson commented Feb 12, 2023

aaomidi commented Feb 15, 2023

christopher-henderson commented Feb 15, 2023

aaomidi commented Feb 28, 2023

christopher-henderson commented Feb 28, 2023

aaomidi commented Feb 28, 2023 •

edited

Loading

aaomidi commented Mar 13, 2023

christopher-henderson commented Mar 19, 2023 •

edited

Loading

christopher-henderson commented Mar 26, 2023

defacto64 commented May 30, 2023

aaomidi commented May 31, 2023

defacto64 commented May 31, 2023 •

edited

Loading

christopher-henderson commented Jun 4, 2023

defacto64 commented Jun 5, 2023

Add CRL linting infrastructure #699

Add CRL linting infrastructure #699

Conversation

aaomidi commented Nov 30, 2022

christopher-henderson left a comment

Choose a reason for hiding this comment

christopher-henderson Dec 3, 2022

Choose a reason for hiding this comment

aaomidi Dec 12, 2022

Choose a reason for hiding this comment

christopher-henderson Dec 3, 2022

Choose a reason for hiding this comment

christopher-henderson commented Dec 11, 2022

aaomidi commented Dec 12, 2022

christopher-henderson commented Feb 5, 2023

aaomidi commented Feb 7, 2023

christopher-henderson commented Feb 12, 2023

aaomidi commented Feb 15, 2023

christopher-henderson commented Feb 15, 2023

aaomidi commented Feb 28, 2023

christopher-henderson commented Feb 28, 2023

aaomidi commented Feb 28, 2023 • edited Loading

aaomidi commented Mar 13, 2023

christopher-henderson commented Mar 19, 2023 • edited Loading

christopher-henderson commented Mar 26, 2023

defacto64 commented May 30, 2023

aaomidi commented May 31, 2023

defacto64 commented May 31, 2023 • edited Loading

christopher-henderson commented Jun 4, 2023

defacto64 commented Jun 5, 2023

aaomidi commented Feb 28, 2023 •

edited

Loading

christopher-henderson commented Mar 19, 2023 •

edited

Loading

defacto64 commented May 31, 2023 •

edited

Loading