in-toto-run capturing implementation

[shibumi]

feature request:

The golang implementation is missing functions for capturing changes right now.
This issue tries to give a small roadmap on implementing the capturing functionality.

For this I had a look on the python implementation and tried to conclude which funtions are missing in the golang implementation. I hope I understood this correct.

What we have right now:

• Reading and generating hashes for multiple files: RecordArtifacts()
• Converting errors to an exit code: WaitErrToExitCode()
• Running commands via RunCommands()
• Hashing files , running commands on them, hashing files again: InTotoRun()

What is missing?

• signing the linked metadata
• generating .link files?
• arbitrary RecordStart/RecordStop functions

Is this the actual functionality we want or did I misunderstood something?
What I didn't see in the python implementation is triggering intoto via another process (for example TravisCI triggers an intoto-run for hashing artifacts.. doing something like building artifacts.. and triggering again for retrieving hashes for artifacts + recording materials and products.
Is this covered via the RecordStart and RecordStop functionality? How are these functions different to our RecordArtifacts functions?

I am a little bit confused about this topic, I think ^^ I actually thought this would go away after doing a few PRs for the error handling etc.

[lukpueh]

Many thanks for the write-up, @shibumi! I am not sure I understand your question about TravisCI, can you try to rephrase?

Regarding in-toto-record, I suggest to take a look at our main demo, which shows how it differs from in-toto-run. I also recommend to take a look at the cli documentation, it is the most comprehensive documentation we currently have. Both in-toto-run and in-toto-record are interface tools to create signed evidence for supply chain steps, and both of them use an internal "RecordArtifacts" function to hash materials and products and add them to link metadata.

Regarding the tasks:

[ ] signing the linked metadata

I think this is the most important sub-task. We already support rsassa-pss-sha256 and ed25519 signature verification, so it would be nice to also be able to sign link metadata generated in runlib.go using those mechanisms. Let's start with simple keys loaded from a PEM file, and later we can maybe try to interface a local gpg client for signing.

[ ] generating .link files

Once we have signing, please add code to write the signed *.link files to disk. See the spec or reference implementation how signing and writing the file are related (hint: filename).

[ ] arbitrary RecordStart/RecordStop functions

RecordStart/RecordStop is not a priority but should also be easy, once we have signing and file writing support. It's basically in InTotoRun split in two parts).

[ ] a cli for the golang implementation

No, we don't need a CLI. We already have a feature rich Python CLI. in-toto-golang is for library use in go code.

[shibumi]

@lukpueh thanks for your clarification. Nevermind about the TravisCI question, it's clear now. I just thought that I need to create an API for travisCI or something, so that travisCI or any other service could create link data via CLI/Rest/whatever.

But I understood now, that we focus on a pure in-toto library implementation for use in other go code.

Signing link data is easy, I have already some experience with signing/validating signatures with ed25519 in Go.

Regarding generating link files:

Do we generate an extra *.link.sig file or do we sign the link files inline? I will have a look on how the demo is doing this.

Thank you for this writeup, this solves a lot of my questions.

Am I correct with the assumption that we create link data already? We just don't sign it and write it to disk yet?

[lukpueh]

@lukpueh thanks for your clarification.

You are very welcome! :)

Nevermind about the TravisCI question, it's clear now. I just thought that I need to create an API for travisCI or something, so that travisCI or any other service could create link data via CLI/Rest/whatever.

These sound like good ideas, but we try to keep in-toto implementations as un-opinionated as possible. An in-toto travis interface should be a separate code base that uses this (or another in-toto implementation) as dependency.

But I understood now, that we focus on a pure in-toto library implementation for use in other go code.

Exactly!

Signing link data is easy, I have already some experience with signing/validating signatures with ed25519 in Go.

Excellent! Looking forward to reviewing your code

Regarding generating link files:

Do we generate an extra *.link.sig file or do we sign the link files inline? I will have a look on how the demo is doing this.

Please do, the demo and above all the specification should answer that question. ;)

Thank you for this writeup, this solves a lot of my questions.

Am I correct with the assumption that we create link data already? We just don't sign it and write it to disk yet?

Correct, we create link metadata in the course of the verification, when we run "inspections", which is a client/verifier-side supply chain step. But since these are created and consumed in an adhoc during verification, we don't sign or persist them.

[shibumi]

I've updated the issue description. I think I will start with signing the link metadata that we already create + writing.. the record/stop part makes sense at the end.

[shibumi]

@lukpueh @SantiagoTorres Does this issue include layout generation, too? I just realized that layouts are completely missing in intoto-golang. Do we want to track this in a separate issue?

[lukpueh]

@lukpueh @SantiagoTorres Does this issue include layout generation, too?

Nope, in-toto-run functionality is completely independent of in-toto layouts.

I just realized that layouts are completely missing in intoto-golang.

This is only half-true. We do have the Layout struct (in model.go) and we can load, parse and use layouts for final product verification (see verifylib.go).

Do we want to track this in a separate issue?

We can, but it is not a high priority at the moment. Unlike link generation and final product verification, which may be incorporated into existing services written in go and thus benefit from an in-toto-golang implementation, it seems feasible to have project owners of any project use the Python tooling, or our (experimental) layout web tool for layout generation.

[lukpueh]

#56 fixes the better part of this issue.

This issue description mentions in-toto-record functionality, which is still missing. Although it might be nice to add this, it does not seem a huge priority.

[shibumi]

Hi, do we want to track the in-toto-record functionality in a separate issue and close this one?

[lukpueh]

Sounds good

[shibumi]

The in-toto record functionality is being tracked here: #69