Add runtime trace predicate type

[adityasaky]

This is an early draft of a runtime trace predicate type. We've noted some subsections / fields that need more input.

cc @pxp928 @nadgowdas @jedsalazar

[colek42]

Could we define "object" better here? Normalizing output from various trace tools will be helpful in discoverability.

I have some rough ideas of that could look like in this branch. https://github.com/testifysec/go-witness/blob/tetragon/attestation/commandrun/commandrun.go

[pxp928]

Hey Cole,

yes we have defined it in the demo example as the following. Would be great to get feedback :

          "eventType": "kprobe",
          "functionName": “__x64_sys_write/fd_install”,
          “processBinary” : “/bin/cat”
          “exitCode”: int
          "arguments": [
               "-wait_file-post_file ",
               "/tekton/run/2/out",
               "/tekton/run/3/out",
               "-termination_path",
               "/tekton/termination",
               "-step_metadata_dir",
               "/tekton/run/3/status",
               "-entrypoint",
               "/tekton/scripts/script-3-g5chp",
               "--",
          ],
          "privileged": [
                 “CAP_SYS_ADMIN”,
                 ],
          },
          {
          "eventType": "kprobe",
          "functionName": “tcp_connect”,
          “processBinary” : “”
          "arguments": [
               "tcp",
               "10.244.0.6:34965",
               "104.198.14.52:8",
          ],
          "privileged": [],
    },

we do want to add hash of files and such as that might be useful. Looking at your branch we are very much on the same page and would love to work together on it. I think there is information that you include that would be valuable to add into the predicate.

[colek42]

We will probably adopt whatever schema is defined here. The only issue we have is that the data needs to be normalized. For our use case, we would like the user to be able to define the same policy no matter what trace tool was used, i.e. tetragon, strace, etc

It looks like the schema here is from Tetragon. I think this needs to be mutated into a common format, and we should add another field(s) to define the trace provider.

[pxp928]

100% agree. We do want to make this generic that other tools can be used. Adding in fields that are missing. As for trace provider we do have:

  "predicate": {
    "monitor": {
      "id": "<URI>"
    },

which would define which tool is being used.

[pxp928]

Trace policies would also need to define how they are configured. Below is a snippet from an output from tetragon. This still needs to be re-worked and formatted to be much more generalized. Not a representation of the final format, but wanted to put an example here for discussion.

"policies": [
          {
            "Name": "connect",
            "Config": "Name: /var/lib/tetragon/bpf_generic_kprobe_v53.o, Attach: tcp_connect, Label: kprobe/generic_kprobe, PinPath: gkp-sensor-1-gkp-0-tcp_connect_prog, RetProbe: false, ErrorFatal: true, Override: false, Type: generic_kprobe, LoaderDate: {ID:0};Name: /var/lib/tetragon/bpf_generic_kprobe_v53.o, Attach: tcp_close, Label: kprobe/generic_kprobe, PinPath: gkp-sensor-1-gkp-1-tcp_close_prog, RetProbe: false, ErrorFatal: true, Override: false, Type: generic_kprobe, LoaderDate: {ID:1};Name: /var/lib/tetragon/bpf_generic_kprobe_v53.o, Attach: tcp_sendmsg, Label: kprobe/generic_kprobe, PinPath: gkp-sensor-1-gkp-2-tcp_sendmsg_prog, RetProbe: false, ErrorFatal: true, Override: false, Type: generic_kprobe, LoaderDate: {ID:2}"
          },
          {
            "Name": "sys-read-follow-prefix",
            "Config": "Name: /var/lib/tetragon/bpf_generic_kprobe_v53.o, Attach: fd_install, Label: kprobe/generic_kprobe, PinPath: gkp-sensor-4-gkp-3-fd_install_prog, RetProbe: false, ErrorFatal: true, Override: false, Type: generic_kprobe, LoaderDate: {ID:3};Name: /var/lib/tetragon/bpf_generic_kprobe_v53.o, Attach: __x64_sys_close, Label: kprobe/generic_kprobe, PinPath: gkp-sensor-4-gkp-4-__x64_sys_close_prog, RetProbe: false, ErrorFatal: true, Override: false, Type: generic_kprobe, LoaderDate: {ID:4};Name: /var/lib/tetragon/bpf_generic_kprobe_v53.o, Attach: __x64_sys_read, Label: kprobe/generic_kprobe, PinPath: gkp-sensor-4-gkp-5-__x64_sys_read_prog, RetProbe: false, ErrorFatal: true, Override: false, Type: generic_kprobe, LoaderDate: {ID:5};Name: /var/lib/tetragon/bpf_generic_retkprobe_v53.o, Attach: __x64_sys_read, Label: kprobe/generic_retkprobe, PinPath: gkp-sensor-4-gkp-5-__x64_sys_read_ret_prog, RetProbe: true, ErrorFatal: true, Override: false, Type: generic_kprobe, LoaderDate: {ID:5};Name: /var/lib/tetragon/bpf_generic_kprobe_v53.o, Attach: __x64_sys_write, Label: kprobe/generic_kprobe, PinPath: gkp-sensor-4-gkp-6-__x64_sys_write_prog, RetProbe: false, ErrorFatal: true, Override: false, Type: generic_kprobe, LoaderDate: {ID:6}"
          }
        ]
      }

[colek42]

I uploaded a sample of our schema here: https://gist.github.com/colek42/a9c82d3314f2cb109bfbb0b67c5e0f99

This is the data that was generated during the SecurityCon demo.

[itaysk]

this is very interesting, we're working in this space with tracee-action and have some comments, but will have to put my thoughts together after CNSC this week. BTW - is of the contributors of this PR in CNSC this week? happy to chat

[adityasaky]

@itaysk I'm attending the conference! I'll message you on Slack, let's find some time to chat.

[itaysk]

Good stuff! added some comments.
BTW when I say "us","we","our projects" I mean https://github.com/aquasecurity/tracee and https://github.com/aquasecurity/tracee-action

[itaysk]

I think we need to either rename the attestation to mention build, or try to generalize the all the build-specific fields (most obvious example is the field called build). currently it doesn't make sense

[adityasaky]

I've updated this to use process instead of build. WDYT?

cc @pxp928 @nadgowdas

[adityasaky]

Updated again to monitoredProcess to avoid collision with process in monitorLog.

[pxp928]

yes that looks good. Thanks @adityasaky

[itaysk]

Just a note, that for us including reads was way too verbose. instead we capture only writes

[colek42]

We want to capture reads to generate a BOM.

[itaysk]

we also do other cool things with reads, question is this needs to be included in the runtime trace attestation as well (in other words if it's related to provenance, I suppose)

[itaysk]

every other monitorLog (process, network) was abstract, this is the only one that specifies the contents of the trace. I think at this stage we should keep all of them abstract.

[colek42]

In order to create policy over the attestation we need to have a defined schema for the monitor log.

[itaysk]

oh I see. policy use case was not mentioned in this spec so I presumed the use case was to enrich provenance data.
I have a feeling that discussing the detailed logs schema will be much longer then the outline schema. If that's the intention, let's see the schemas and then we can discuss.

[itaysk]

I didn't see how materials is related to "runtime trace", so I think no, but I might have missed something

[itaysk]

IMO tracePolicy should move under monitor. Different tracers have different features that might not be available across the board. also the syntax will definitely be different between tracers and it will make most sense to keep the trace policy in it's original form. so if the tracePolicy is monitor-specific it's clearer to move it there.

[colek42]

+1

[pxp928]

Yes thats a good suggestion!

[adityasaky]

@pxp928

[adityasaky]

I think we can move all of invocation's fields into the monitor object. WDYT @pxp928?

[pxp928]

sounds good. Thanks!

[itaysk]

could you please clarify this question?

[itaysk]

Should we define what is the scope of the trace or it is up to the implementation to decide? For example, in GH Actions, each run gets a runner vm which executes the workflow. a tracer will likely be installed at the runner (vm) level, so it could trace the entire runner or be scoped to events just from the workflow.
I'm putting this note here because it could also be just part of the trace policy.

[adityasaky]

I think it's up to the implementation to decide. This spec should be about how the final trace is communicated and verified. The verification policy could check, I think, the scope and if the trace is usable for its context.

[itaysk]

process args/envs might include secrets which the user would not want to logged. since we're not specifying the schema of the process log, I guess it's an issue for the implementor and not for this attestation. I thought I'd just make the note here anyway

[colek42]

Should the spec require the implementer to specify black-listed env vars? We could also hash the variables.

[itaysk]

depending on whether this statement of mine is still true of not "since we're not specifying the schema of the process log" after #111 (comment)

[itaysk]

probably very subjective but trace might be more descriptive in this case than monitor. the attestation is also called runtime trace. and it's definition is a trace policy. so calling this monitor here is standing out for me

[adityasaky]

Fair! I think we've got a couple of different field rename suggestions to consider and implement here.

[marcelamelara]

Overall, I think exposing runtime trace events for builders will be really useful. That said, I have two concerns with this predicate as it's proposed right now.

(1) It's decently closely tied to the SLSA Provenance format; Will this predicate have to be updated in lockstep with every SLSA Provenance update? Maybe this won't be too bad if this predicate only needs updates with every major SLSA update or something like that, but it's still a consideration. So my recommendation would be to ensure this predicate is sufficiently decoupled from SLSA to minimize this.

(2) Given the example, I expect full runtime traces to be quite large in practice, generating a lot of data and large attestations. Especially if the intent is to generate a runtime trace attestation at every build. Is there a way this predicate could be more of a summary of a runtime trace that points to the full event log instead of including the full event log in the attestation? I'm not too familiar with different tracing tools, but I imagine that might make the predicate more interoperable between different tracing tools, too. I'm also thinking making heavier use of Reference- and/or SCAI-type fields might be helpful for this.

[marcelamelara]

Thanks for the updates! Left a few comments.

[marcelamelara]

It strikes me that the tracePolicy wouldn't actually change as often as the runtime traces are generated (correct me if I'm wrong), so I think it would definitely make sense to include a reference to a separate policy attestation that can be reused for any trace it applies to.

[adityasaky]

I think I was in favour of having it be a separate document but I think either @pxp928 or @nadgowdas noted that we need to unambiguously map the trace policy to the trace so that we can reason about it when writing policies for monitorLog. For a separate document, I guess the monitor would have to understand that a configured trace policy matches the corresponding attestation? Perhaps a TODO past the v0.1 spec?

[pxp928]

Using a separate document for the policy is a better solution in retrospect. I agree that it would not change that often but it would have to be published alongside the runtime attestation each time to ensure which policy was set when the trace was captured. If the policy for example is set to not capture any network logs, we cannot attest that during the build time, the pipeline did not reach out to the internet.

[adityasaky]

I think we have a number of points for v0.2. I'll try to capture them into one place we can use to iterate.

[adityasaky]

Note to reviewers: I'll clean up git history once you reckon this is good to go for v0.1. 😄

[marcelamelara]

Thanks for all the updates! I think this is ready for v0.1.

[adityasaky]

Updated after @marcelamelara's approval to update predicate type to https://in-toto.io/attestation/runtime-trace/v0.1 from https://in-toto.io/attestation/runtime-trace/v0.1.0.

[TomHennen]

LGTM for v0.1 (just a couple minor comments outstanding).

Thanks!

[pxp928]

I had some outstanding comment responses that I never submitted. Thanks, @adityasaky for updating this :)