Implementation of new In-toto 0.1.0, DSSE 1.0.0, and SLSA Provenance 0.1.1 #24

Alos · 2021-09-17T00:40:34Z

Hey folks!

Please have a look at the PR. This change adds the following:

Moved existing Link implementation into legacy directory and classes are now tagged as deprecated.
Clean up and updated the pom.xml
- Removed commented dependencies
- Added new dependencies used for Java to JSON and bean validation
Added in-toto models and helper classes as well as tests that cover validation.
Added a new @UniqueSubject annotation along with a new UniqueSubjectValidator to make sure subjects are unique in the Statement
Added SLSA Provenance as a subclass of the Predicate as per the proposal Proposal: Updated implementation for the In-toto spec #23
Added DSSE models for Envelope and Signature
Added interfaces for Signer/Verifier for DSSE
Added a simple ECDSASigner and ECDSAVerifier as a PoC.
Added helper methods for users to create envelopes with attestations and a user provided Signer
Added general tests for the signing and producing the payloads and the final DSSE.
Updated the README to tell users how to use the new stuff.
Removed unused Makefile and .travis configuration
Disabled a broken legacy test (unsure what this was trying to test)
Added common .gitignore for Java
Updated the CHANGELOG.md

Sample of a DSSE generated by the library using the included test keys (pretty-printed here for clarity):

{
	"payloadType": "application/vnd.in-toto+json",
	"payload": "eyJfdHlwZSI6Imh0dHBzOi8vaW4tdG90by5pby9TdGF0ZW1lbnQvdjAuMSIsInN1YmplY3QiOlt7Im5hbWUiOiJjdXJsLTcuNzIuMC50YXIuYnoyIiwiZGlnZXN0Ijp7InNoYTI1NiI6ImQ0ZDU4OTlhMzg2OGZiYjZhZTE4NTZjM2U1NWEzMmNlMzU5MTNkZTM5NTZkMTk3M2NhY2NkMzdiZDAxNzRmYTIifX1dLCJwcmVkaWNhdGVUeXBlIjoiaHR0cHM6Ly9zbHNhLmRldi9wcm92ZW5hbmNlL3YwLjEiLCJwcmVkaWNhdGUiOnsiYnVpbGRlciI6eyJpZCI6Im1haWx0bzpwZXJzb25AZXhhbXBsZS5jb20ifSwicmVjaXBlIjp7InR5cGUiOiJodHRwczovL2V4YW1wbGUuY29tL01ha2VmaWxlIiwiZGVmaW5lZEluTWF0ZXJpYWwiOjAsImVudHJ5UG9pbnQiOiJzcmM6Zm9vIn0sIm1ldGFkYXRhIjp7ImJ1aWxkSW52b2NhdGlvbklkIjoiU29tZUJ1aWxkSWQiLCJidWlsZFN0YXJ0ZWRPbiI6IjE5ODYtMTItMThUMTU6MjA6MzArMDg6MDAiLCJidWlsZEZpbmlzaGVkT24iOiIxOTg2LTEyLTE4VDE2OjIwOjMwKzA4OjAwIiwiY29tcGxldGVuZXNzIjp7ImFyZ3VtZW50cyI6dHJ1ZSwiZW52aXJvbm1lbnQiOmZhbHNlLCJtYXRlcmlhbHMiOnRydWV9LCJyZXByb2R1Y2libGUiOmZhbHNlfSwibWF0ZXJpYWxzIjpbeyJ1cmkiOiJodHRwczovL2V4YW1wbGUuY29tL2V4YW1wbGUtMS4yLjMudGFyLmd6IiwiZGlnZXN0Ijp7InNoYTI1NiI6IjEyMzQuLi4ifX1dfX0=",
	"signatures": [{
		"sig": "MIGIAkIA1T0wx8okWtfwzLyhyy7tRNFonqDIqe9bJ4tCtBhGX+6SengcnLMxSTr0xMz1jJCufJWazl1bbpuwWzuwDcmRqKoCQgF9mmiFP6QjZGgDQqXZFre3tK+1cwNNpBpRyvJdN3HAki9hlzbkXJURcN/0KkuyspLXn2r6rfysVEzlgUzIz8wwDg==",
		"keyid": "MyKey"
	}]
}

Sample of the payload once its Base64 decoded and pretty-printed:

{
	"_type": "https://in-toto.io/Statement/v0.1",
	"subject": [{
		"name": "curl-7.72.0.tar.bz2",
		"digest": {
			"sha256": "d4d5899a3868fbb6ae1856c3e55a32ce35913de3956d1973caccd37bd0174fa2"
		}
	}],
	"predicateType": "https://slsa.dev/provenance/v0.1",
	"predicate": {
		"builder": {
			"id": "mailto:[email protected]"
		},
		"recipe": {
			"type": "https://example.com/Makefile",
			"definedInMaterial": 0,
			"entryPoint": "src:foo"
		},
		"metadata": {
			"buildInvocationId": "SomeBuildId",
			"buildStartedOn": "1986-12-18T15:20:30+08:00",
			"buildFinishedOn": "1986-12-18T16:20:30+08:00",
			"completeness": {
				"arguments": true,
				"environment": false,
				"materials": true
			},
			"reproducible": false
		},
		"materials": [{
			"uri": "https://example.com/example-1.2.3.tar.gz",
			"digest": {
				"sha256": "1234..."
			}
		}]
	}
}

This is the fist set of changes for the new implementation of in-toto spec. This includes: * Moving of existing functionality into legacy package * Creating a set of base model classes * Implementing validators * Implementing helper class to start validation process and transform to JSON * Set of tests for the above

- Updated the changelog - Removed unused Makefile - Remove unused .travis.yml

- Added SLSA models and tests. - Added DSSE implementations and simple Signers and Verifiers

* Added better description of helper methods and Signer/Verifier interface * Made sure the `pivateKey` field is final.

Make sure new classes have proper JavaDoc

Added missing JavaDoc

Changed some of the description

README.md

TomHennen

Can you provide example output of both the signed JSON DSSE as well as the JSON Statement?

I think that will make it easier to check if all the fields are named correctly, etc...

Also, have you tested this against the reference implementation (in terms of verifying/generating signed DSSEs)?

https://github.com/secure-systems-lab/dsse/blob/master/implementation/signing_spec.py

README.md

src/main/java/io/github/intoto/models/Predicate.java

src/main/java/io/github/intoto/models/Subject.java

src/main/java/io/github/intoto/models/PredicateType.java

src/main/java/io/github/slsa/models/Metadata.java

src/main/java/io/github/slsa/models/Recipe.java

MarkLodato

It would help to have test vectors showings the expected output and testing against that. It's must easier to hand-inspect a known-good JSON file than it is to verify that the code produces the right JSON. :-)

src/main/java/io/github/dsse/helpers/SimpleECDSASigner.java

MarkLodato · 2021-09-23T17:19:02Z

src/main/java/io/github/intoto/helpers/IntotoHelper.java

+   */
+  public static String createPreAuthenticationEncoding(String payloadType, String payload) {
+    return String.format(
+        "DSSEv1 %d %s %d %s", payloadType.length(), payloadType, payload.length(), payload);


Does this handle multi-byte UTF-8 characters properly? Could you add a test for that?

I think so. I added a test fort this.

MarkLodato · 2021-09-23T17:20:17Z

src/main/java/io/github/intoto/helpers/IntotoHelper.java

+   * LEN(s) = ASCII decimal encoding of the byte length of s, with no leading zeros
+   *<pre/>
+   * @param payloadType the type of payload. Fixed for in-toto Envelopes
+   * @param payload the base64 encoded Statement in JSON


This is not supposed to be base64 encoded. It's supposed to be the raw payload in bytes.

I was following https://github.com/in-toto/attestation/tree/main/spec#envelope where it says that the payload is:

Base64-encoded JSON Statement.

That is for the JSON payload field. JSON only handles unicode strings, not bytes, so all binary data must be base64-encoded. However, PAE access the raw binary (bytes) payload, before encoding.

I've made changes to match the spec now, please have a look

* Added missing getters and getters * Fixed issues with time formatting * Replaced `int` to `Integer` to allow `definedInMaterial` to be null * Added SHA256 in the sample signer and verifier. * Removed duplicated code. * Clarified and reformatted README.md * Added the ability to pass key id in the sample Signer * Removed PredicateType enum as its no longer needed.

Alos · 2021-09-27T20:54:11Z

Can you provide example output of both the signed JSON DSSE as well as the JSON Statement?

I think that will make it easier to check if all the fields are named correctly, etc...

Also, have you tested this against the reference implementation (in terms of verifying/generating signed DSSEs)?

https://github.com/secure-systems-lab/dsse/blob/master/implementation/signing_spec.py

You can see the output of various things while running the JUnit tests.

The test called produceIntotoEnvelopeAsJson_shouldCorrectlyCreateAnEnvelope_whenCompleteStatementIsPassed outputs the following DSSE JSON:

{
  "payloadType" : "application/vnd.in-toto+json",
  "payload" : "eyJfdHlwZSI6Imh0dHBzOi8vaW4tdG90by5pby9TdGF0ZW1lbnQvdjAuMSIsInN1YmplY3QiOlt7Im5hbWUiOiJjdXJsLTcuNzIuMC50YXIuYnoyIiwiZGlnZXN0Ijp7IlNIQTI1NiI6ImQ0ZDU4OTlhMzg2OGZiYjZhZTE4NTZjM2U1NWEzMmNlMzU5MTNkZTM5NTZkMTk3M2NhY2NkMzdiZDAxNzRmYTIifX1dLCJwcmVkaWNhdGVUeXBlIjoiaHR0cHM6Ly9zbHNhLmRldi9wcm92ZW5hbmNlL3YwLjEiLCJwcmVkaWNhdGUiOnsiYnVpbGRlciI6eyJpZCI6Im1haWx0bzpwZXJzb25AZXhhbXBsZS5jb20ifSwicmVjaXBlIjp7InR5cGUiOiJodHRwczovL2V4YW1wbGUuY29tL01ha2VmaWxlIiwiZGVmaW5lZEluTWF0ZXJpYWwiOjAsImVudHJ5UG9pbnQiOiJzcmM6Zm9vIn0sIm1ldGFkYXRhIjpudWxsLCJtYXRlcmlhbHMiOlt7InVyaSI6Imh0dHBzOi8vZXhhbXBsZS5jb20vZXhhbXBsZS0xLjIuMy50YXIuZ3oiLCJkaWdlc3QiOnsic2hhMjU2IjoiMTIzNC4uLiJ9fV0sInByZWRpY2F0ZVR5cGUiOiJodHRwczovL3Nsc2EuZGV2L3Byb3ZlbmFuY2UvdjAuMSJ9fQ==",
  "signatures" : [ {
    "sig" : "RFNTRXYxIDI4IGFwcGxpY2F0aW9uL3ZuZC5pbi10b3RvK2pzb24gNTQxIHsiX3R5cGUiOiJodHRwczovL2luLXRvdG8uaW8vU3RhdGVtZW50L3YwLjEiLCJzdWJqZWN0IjpbeyJuYW1lIjoiY3VybC03LjcyLjAudGFyLmJ6MiIsImRpZ2VzdCI6eyJTSEEyNTYiOiJkNGQ1ODk5YTM4NjhmYmI2YWUxODU2YzNlNTVhMzJjZTM1OTEzZGUzOTU2ZDE5NzNjYWNjZDM3YmQwMTc0ZmEyIn19XSwicHJlZGljYXRlVHlwZSI6Imh0dHBzOi8vc2xzYS5kZXYvcHJvdmVuYW5jZS92MC4xIiwicHJlZGljYXRlIjp7ImJ1aWxkZXIiOnsiaWQiOiJtYWlsdG86cGVyc29uQGV4YW1wbGUuY29tIn0sInJlY2lwZSI6eyJ0eXBlIjoiaHR0cHM6Ly9leGFtcGxlLmNvbS9NYWtlZmlsZSIsImRlZmluZWRJbk1hdGVyaWFsIjowLCJlbnRyeVBvaW50Ijoic3JjOmZvbyJ9LCJtZXRhZGF0YSI6bnVsbCwibWF0ZXJpYWxzIjpbeyJ1cmkiOiJodHRwczovL2V4YW1wbGUuY29tL2V4YW1wbGUtMS4yLjMudGFyLmd6IiwiZGlnZXN0Ijp7InNoYTI1NiI6IjEyMzQuLi4ifX1dLCJwcmVkaWNhdGVUeXBlIjoiaHR0cHM6Ly9zbHNhLmRldi9wcm92ZW5hbmNlL3YwLjEifX0=",
    "keyid" : "Fake-Signer-Key-ID"
  } ]
}

I'll see if I can get the python code working to check against it.

MarkLodato · 2021-09-28T12:46:57Z

Here's the payload, for reference:

{
  "_type": "https://in-toto.io/Statement/v0.1",
  "subject": [
    {
      "name": "curl-7.72.0.tar.bz2",
      "digest": {
        "SHA256": "d4d5899a3868fbb6ae1856c3e55a32ce35913de3956d1973caccd37bd0174fa2"
      }
    }
  ],
  "predicateType": "https://slsa.dev/provenance/v0.1",
  "predicate": {
    "builder": {
      "id": "mailto:[email protected]"
    },
    "recipe": {
      "type": "https://example.com/Makefile",
      "definedInMaterial": 0,
      "entryPoint": "src:foo"
    },
    "metadata": null,
    "materials": [
      {
        "uri": "https://example.com/example-1.2.3.tar.gz",
        "digest": {
          "sha256": "1234..."
        }
      }
    ],
    "predicateType": "https://slsa.dev/provenance/v0.1"
  }
}

Bugs:

The property name SHA256 should be lowercase inside subject.digest.
The property predicate.predicateType should be deleted, since it's already at the outer level.

* Made sure enums are lowercase when converted to JSON * Made sure that predicate inner types does not show when converted to JSON * Created some keys for testing * Made sure the Signer accepts a byte array instead of a Java String * Removed unnecessary Base64 encoding while creating the PAE * Updated tests to reflect the changes

Alos · 2021-09-28T19:29:41Z

Here's the payload, for reference:

{
  "_type": "https://in-toto.io/Statement/v0.1",
  "subject": [
    {
      "name": "curl-7.72.0.tar.bz2",
      "digest": {
        "SHA256": "d4d5899a3868fbb6ae1856c3e55a32ce35913de3956d1973caccd37bd0174fa2"
      }
    }
  ],
  "predicateType": "https://slsa.dev/provenance/v0.1",
  "predicate": {
    "builder": {
      "id": "mailto:[email protected]"
    },
    "recipe": {
      "type": "https://example.com/Makefile",
      "definedInMaterial": 0,
      "entryPoint": "src:foo"
    },
    "metadata": null,
    "materials": [
      {
        "uri": "https://example.com/example-1.2.3.tar.gz",
        "digest": {
          "sha256": "1234..."
        }
      }
    ],
    "predicateType": "https://slsa.dev/provenance/v0.1"
  }
}

Bugs:

The property name SHA256 should be lowercase inside subject.digest.
The property predicate.predicateType should be deleted, since it's already at the outer level.

Done.

Alos · 2021-09-29T22:23:37Z

Can you provide example output of both the signed JSON DSSE as well as the JSON Statement?

I think that will make it easier to check if all the fields are named correctly, etc...

Also, have you tested this against the reference implementation (in terms of verifying/generating signed DSSEs)?

https://github.com/secure-systems-lab/dsse/blob/master/implementation/signing_spec.py

I'm working with @MarkLodato to see if we can get a Python command that uses the spect to verify.

In an effort to make testing easier against other libs. The Java implementation now uses standard keys generated with the OpenSSL command.

Added bigger example with metadata.

src/main/java/io/github/intoto/models/Predicate.java

src/test/java/io/github/dsse/helpers/SimpleECDSATest.java

src/test/java/io/github/intoto/utilities/TestEnvelopeGenerator.java

src/test/resources/p8private.pem

README.md

* Updated the Statements so that they include their type by default. * Updated README to match implementation * Updated the example output to match the bundle name and extension. * Clarified the Predicate JavaDoc. * Clarified testing comments

TomHennen · 2021-10-05T13:31:58Z

README.md

+DigestSetAlgorithmType.SHA256.toString(),
+"d4d5899a3868fbb6ae1856c3e55a32ce35913de3956d1973caccd37bd0174fa2"))
+Predicate predicate=createPredicate();
+Statement statement=new Statement();


optional: create a version of the constructor that accepts the subject list and predicate as args so users don't have to set them manually.

This PR is getting big. Let's add this to a post PR merge. I also want to add the ability for the library to generate a DSSE envelope with multiple statements. This could be something we introduce after this PR

Made sure the tests are using the test keys from the files.

* Added additional test that can be used to manually verify a .intoto.jsonl file * Added JavaDoc and made sure pre tags are correctly closed in them.

SantiagoTorres

Looking good!

I found nothing major, but I do have a couple of questions...

SantiagoTorres · 2021-10-14T18:36:14Z

README.md

+DigestSetAlgorithmType.SHA256.toString(),
+"d4d5899a3868fbb6ae1856c3e55a32ce35913de3956d1973caccd37bd0174fa2"))
+Predicate predicate=createPredicate();
+Statement statement=new Statement();


pom.xml

src/main/java/io/github/intoto/helpers/IntotoHelper.java

src/main/java/io/github/intoto/models/DigestSetAlgorithmType.java

src/main/java/io/github/legacy/models/Link.java

src/main/java/io/github/slsa/models/Completeness.java

src/test/java/io/github/intoto/utilities/IntotoStubFactory.java

TomHennen · 2021-10-18T20:50:21Z

Everything else looks good to me. If Santiago & Aditya are happy them so am I.

adityasaky

I think I quite like the look of things here, I don't have any concerns. I do have a question for the Java folks here though regarding the namespacing. I don't know if it's significant in practice, but for several of the package namespaces used here, we don't actually control the domains they represent or even the corresponding GitHub usernames. I do recognize that the hyphen complicates things, and the Google Java style guide says to just concatenate words together: https://google.github.io/styleguide/javaguide.html#s5.2.1-package-names. Do we ever foresee any troubles because we don't control github.com/intoto? For that matter, should we also move the legacy package to be under intoto, because it's currently io.github.legacy? For dsse, we do have the option of using the secure-systems-lab domain as we do for the Go implementation. I imagine it'd be edu.nyu.engineering.ssl.dsse in this case. And I guess dev.slsa for that one? Please let me know if I'm overthinking this 😄

pom.xml

Alos · 2021-10-25T17:47:25Z

@adityasaky you bring up a good point on the package names. Originally I named the packages this way so that they can later be extracted and moved to their relevant repos. But since there is no warranty that this will be the case, I will make the change to match the intoto repository.

… to 0.4.0

adityasaky · 2021-10-28T21:07:20Z

Actions in action: https://github.com/Alos/in-toto-java/runs/4039918286?check_suite_focus=true

SantiagoTorres · 2021-11-01T20:49:11Z

I did another pass today, I think all outstanding issues are solved and the minor ones can be worked on smaller PRs and have been documented.

Thank you so much @Alos !

Alos added 10 commits September 10, 2021 13:46

Bumped version in the pom and javadoc fixes

02230cb

Added @deprecated to legacy classes

23ae210

Update README.md

6b0725c

Update the changelog

2f8373f

- Updated the changelog - Removed unused Makefile - Remove unused .travis.yml

Added DSSE and SLSA implementations

53e07e3

- Added SLSA models and tests. - Added DSSE implementations and simple Signers and Verifiers

Updated the README

dd964a3

* Added better description of helper methods and Signer/Verifier interface * Made sure the `pivateKey` field is final.

Added additional JavaDoc

48d02d9

Make sure new classes have proper JavaDoc

Update IntotoEnvelope.java

521e076

Added missing JavaDoc

Update README.md

251b162

Changed some of the description

TomHennen reviewed Sep 17, 2021

View reviewed changes

README.md Outdated Show resolved Hide resolved

TomHennen reviewed Sep 17, 2021

View reviewed changes

README.md Outdated Show resolved Hide resolved

TomHennen suggested changes Sep 17, 2021

View reviewed changes

MarkLodato suggested changes Sep 23, 2021

View reviewed changes

Alos marked this pull request as draft September 23, 2021 22:49

Alos marked this pull request as ready for review September 29, 2021 22:24

Alos added 2 commits October 4, 2021 12:30

Made Java implementation work with fixed keys generated by OpenSSL

cac7f15

In an effort to make testing easier against other libs. The Java implementation now uses standard keys generated with the OpenSSL command.

Updates with a bigger example

3638bfe

Added bigger example with metadata.

TomHennen suggested changes Oct 4, 2021

View reviewed changes

Additional PR updates

bb5e1e0

* Updated the Statements so that they include their type by default. * Updated README to match implementation * Updated the example output to match the bundle name and extension. * Clarified the Predicate JavaDoc. * Clarified testing comments

TomHennen suggested changes Oct 5, 2021

View reviewed changes

Update SimpleECDSATest.java

937a033

Made sure the tests are using the test keys from the files.

adityasaky mentioned this pull request Oct 6, 2021

"other_headers" field in "signatures" field in reference implementation is not part of the spec in-toto/specification#55

Closed

Updates to JavaDoc and Additional tests

4405b88

* Added additional test that can be used to manually verify a .intoto.jsonl file * Added JavaDoc and made sure pre tags are correctly closed in them.

adityasaky requested review from SantiagoTorres and adityasaky October 14, 2021 17:49

SantiagoTorres reviewed Oct 14, 2021

View reviewed changes

joshuagl mentioned this pull request Oct 19, 2021

Practical document describing how to achieve SLSA compliance with GitHub slsa-framework/slsa#185

Open

adityasaky reviewed Oct 19, 2021

View reviewed changes

pom.xml Outdated Show resolved Hide resolved

Alos and others added 5 commits October 25, 2021 10:55

refactor: Move implementation to intoto package name and bump version…

f2d770c

… to 0.4.0

Added additional tests for PAE transformation

206bb9e

Create maven.yml

a52c3a8

Update CHANGELOG.md

32fad8d

Merge branch 'master' of https://github.com/Alos/in-toto-java

5b7b69a

SantiagoTorres merged commit 0ff54b9 into in-toto:master Nov 1, 2021

adityasaky mentioned this pull request Nov 1, 2021

Proposal: Updated implementation for the In-toto spec #23

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implementation of new In-toto 0.1.0, DSSE 1.0.0, and SLSA Provenance 0.1.1 #24

Implementation of new In-toto 0.1.0, DSSE 1.0.0, and SLSA Provenance 0.1.1 #24

Alos commented Sep 17, 2021 •

edited

Loading

TomHennen left a comment

MarkLodato left a comment

MarkLodato Sep 23, 2021

Alos Sep 23, 2021

MarkLodato Sep 23, 2021

Alos Sep 23, 2021

MarkLodato Sep 23, 2021

Alos Sep 28, 2021

Alos commented Sep 27, 2021

MarkLodato commented Sep 28, 2021

Alos commented Sep 28, 2021

Alos commented Sep 29, 2021

TomHennen Oct 5, 2021

SantiagoTorres Oct 14, 2021

Alos Oct 18, 2021

SantiagoTorres left a comment

SantiagoTorres Oct 14, 2021

TomHennen commented Oct 18, 2021

adityasaky left a comment

Alos commented Oct 25, 2021

adityasaky commented Oct 28, 2021

SantiagoTorres commented Nov 1, 2021

Implementation of new In-toto 0.1.0, DSSE 1.0.0, and SLSA Provenance 0.1.1 #24

Implementation of new In-toto 0.1.0, DSSE 1.0.0, and SLSA Provenance 0.1.1 #24

Conversation

Alos commented Sep 17, 2021 • edited Loading

TomHennen left a comment

Choose a reason for hiding this comment

MarkLodato left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Alos commented Sep 27, 2021

MarkLodato commented Sep 28, 2021

Alos commented Sep 28, 2021

Alos commented Sep 29, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SantiagoTorres left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TomHennen commented Oct 18, 2021

adityasaky left a comment

Choose a reason for hiding this comment

Alos commented Oct 25, 2021

adityasaky commented Oct 28, 2021

SantiagoTorres commented Nov 1, 2021

Alos commented Sep 17, 2021 •

edited

Loading