initial gRPC API & message definition #771

bobcallaway · 2022-04-12T23:30:07Z

Note: this is targeted at a feature branch, not main. I intend to continue to refine the gRPC work in a branch until we get it ready to merge over into main since it will touch most of Rekor in some way or another.

This starts work on #484; when complete, it also aims to simplify the code generation for rekor-server, custom type definition and easier multi-language client code generation. This PR just shares a draft of the external service & message definitions for feedback, and starts to plumb in the golang client & server side code generation.

TODO (likely incomplete but wanted to start a list):

figure out how to support current v1 API support through legacy gRPC interface (similar to Fulcio)
figure out version implications of gRPC with regards to parsing older log entries
re-envision types interfaces (opportunity for leveraging 1.18 generic support?)
implement an e2e flow for a single custom
implement e2e flows for all custom types
update custom type documentation for devs
add OpenAPI documentation (to be generated from proto files) instead of a hardcoded openapi.yaml file?
remove swagger-based implementation

Signed-off-by: Bob Callaway [email protected]

Signed-off-by: Bob Callaway <[email protected]>

dlorenc · 2022-04-13T00:45:56Z

If you're targeting a feature branch, feel free to just merge away and we can do a larger review after. Feel free to leave them open though if you do want reviews before the merge.

bobcallaway · 2022-04-13T00:59:58Z

I will plan to do that for lots of the functional changes, but I would like some feedback on the external APIs if possible.

priyawadhwa · 2022-04-13T16:13:26Z

rekor_entries.proto

+    /*
+     * The universally unique identifier for the entry within the log shard
+     */
+    string uuid     = 1;


nit: consider renaming uuid to entryID to indicate that we want the universal identifier

if we're assuming that the v2 API will only be run with rekor server >=0.6, then we can probably rename all mentions of uuid to entryID?

yeah, I thought about that as well as normalizing "tree" vs "shard" and trying to make it consistent.

haydentherapper · 2022-04-13T17:15:42Z

pkg/types/alpine/alpine.proto

+    /*
+     * Key / value pairs from .PKGINFO file inside alpine package
+     */
+    map<string, string> pkginfo                      = 3;


Does this let users store arbitrary data?

haydentherapper · 2022-04-13T17:16:55Z

pkg/types/rfc3161/rfc3161.proto

+ *
+ * TimestampRequest uses an identical input and output format;
+ * we create both for potential forward compatibility of separating and changing them;
+ * this also makes the assignment more explicit


I'm a little confused by this - So ProposedTImestampRequest is the request, and TimestampRequest is an internal type?

this is a typo, good catch

haydentherapper · 2022-04-13T17:24:03Z

pkg/types/rpm/rpm.proto

+    /*
+     * Key / value pairs from RPM headers
+     */
+    map<string, string> headers                      = 3;


Same question, does this allow arbitrary data?

same as alpine, the code currently extracts and stores cryptographically-verified metadata from the package headers.

haydentherapper · 2022-04-13T17:25:19Z

rekor_common.proto

+    /*
+     * The public key
+     */
+    bytes content = 1;


This can contain either a public key, certificate or chain, correct? Should we rename this?

well, it could contain a PGP public key, an X509 certificate (or certificate chain), a TUF metadata root, etc. We could use a oneOf with specific types here if you think its better

I'd be in support of switching to a oneOf. I think it makes it more explicit and less prone to error.

FYI this linked issue #809 -- would support switching to a oneof

haydentherapper · 2022-04-13T17:26:32Z

rekor_common.proto

+
+message Signature {
+    /*
+     * The crypto algorithm used to generate the signature


nit: This sounds like the signature algorithm, i.e rsa-pss, ecdsa, whereas it's actually the format to hold the signature. Consider rephrasing

yeah, this is a carry forward from the current implementation... open to alternative suggestions for sure

haydentherapper · 2022-04-13T17:26:56Z

rekor_common.proto

+    MINISIGN                  = 2;
+    X509                      = 3;
+    SSH                       = 4;
+    PKCS7                     = 5;


Does this only work with the JAR type, or any type?

haydentherapper · 2022-04-13T17:27:20Z

rekor_common.proto

+    PublicKey public_key   = 3;
+}
+
+enum PKIAlgorithm {


nit: Comments for each enum

haydentherapper · 2022-04-13T17:28:25Z

rekor_timestamp.proto

+    /*
+     * The set of PEM-encoded certificate chains for the timestamp service;
+     * the chain will start with any intermediate certificates (if present),
+     * and finishwith the root certificate.


finish with

haydentherapper · 2022-04-13T17:28:48Z

rekor_timestamp.proto

+    /*
+     * The raw HTTP body is bound to this field; this is an RFC3161 compliant blob.
+     */
+    google.api.HttpBody http_body = 1 [(google.api.field_behavior) = REQUIRED];


Can it be a byte array, or does it have to be an HTTP body?

well http_body includes a byte array, but in order for rekor to remain an RFC3161 compliant TSA, I think we need to do it this way (could be wrong though, haven't coded it up yet)

This is still WIP, but commiting this iteration. Signed-off-by: Bob Callaway <[email protected]>

asraa · 2022-05-06T16:48:56Z

pkg/types/intoto/v1/intoto.proto

+ */
+message ProposedIntoto {
+    /*
+     * DSSE envelope encompassing the entire Intoto statement


what about reusing the DSSE proto?
https://github.com/secure-systems-lab/dsse/blob/master/envelope.proto

or too much overhead on clients needing to pull that library?

asraa · 2022-05-06T16:53:17Z

pkg/types/intoto/v1/intoto.proto

+ * Intoto represents the structure of the entry persisted in the transparency log for
+ * the Intoto type.
+ */
+message Intoto {


My biggest issue with this is that you can't take an Intoto entry and "verify" it the way that Rekor will verify it.

Can we add the repeated Signatures signature in here too? So that clients can verify the entry signature using the Signatures signing the payload_hash?

asraa · 2022-05-06T16:55:22Z

rekor_common.proto

+    /*
+     * The public key
+     */
+    bytes content = 1;


FYI this linked issue #809 -- would support switching to a oneof

bobcallaway added 2 commits April 12, 2022 17:34

initial GRPC API & message definition

257cc87

Signed-off-by: Bob Callaway <[email protected]>

mark googleapis/proto imports as generated to hide in diffs

2b5b5a1

Signed-off-by: Bob Callaway <[email protected]>

bobcallaway requested review from loosebazooka, asraa and haydentherapper April 12, 2022 23:30

bobcallaway requested review from cpanato and a team as code owners April 12, 2022 23:30

fix import cycle

f1f0b6c

Signed-off-by: Bob Callaway <[email protected]>

priyawadhwa reviewed Apr 13, 2022

View reviewed changes

haydentherapper reviewed Apr 13, 2022

View reviewed changes

Updated form for proto files

adae02a

This is still WIP, but commiting this iteration. Signed-off-by: Bob Callaway <[email protected]>

asraa reviewed May 6, 2022

View reviewed changes

bobcallaway closed this Sep 2, 2022

haydentherapper mentioned this pull request Oct 3, 2023

OpenAPI 3.0 for openapi.yaml? #1729

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

initial gRPC API & message definition #771

initial gRPC API & message definition #771

bobcallaway commented Apr 12, 2022

dlorenc commented Apr 13, 2022

bobcallaway commented Apr 13, 2022

priyawadhwa Apr 13, 2022

priyawadhwa Apr 13, 2022

bobcallaway Apr 13, 2022

haydentherapper Apr 13, 2022

haydentherapper Apr 13, 2022

bobcallaway Apr 13, 2022

haydentherapper Apr 13, 2022

bobcallaway Apr 13, 2022

haydentherapper Apr 13, 2022

bobcallaway Apr 13, 2022

haydentherapper Apr 28, 2022

asraa May 6, 2022

haydentherapper Apr 13, 2022

bobcallaway Apr 13, 2022

haydentherapper Apr 13, 2022

haydentherapper Apr 13, 2022

haydentherapper Apr 13, 2022

haydentherapper Apr 13, 2022

bobcallaway Apr 13, 2022 •

edited

Loading

asraa May 6, 2022

asraa May 6, 2022

asraa May 6, 2022

initial gRPC API & message definition #771

initial gRPC API & message definition #771

Conversation

bobcallaway commented Apr 12, 2022

dlorenc commented Apr 13, 2022

bobcallaway commented Apr 13, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bobcallaway Apr 13, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bobcallaway Apr 13, 2022 •

edited

Loading