feat(detectExecuteScan): Also scan images that are in the CPE #4678

c0d1ngm0nk3y · 2023-11-15T15:19:45Z

This has following prerequisites to jenkins-library (cherry-picked):

Fix sidecar conditionals #4672 Allow sidecars to have conditions
Add support for volume mounts #4673 Add volumeMount support to sidecars

The enhances detectExecuteScan to not only scan the workspace, but also all images that can be found in the cpe. If kanikoExecute or cnbBuild produce images, the information will be written to the cpe. In order to do these scans, a sidecar is needed. The sidecar is different for the distro that is scanned. The user can configure this via the new parameter containerDistro (default "ubuntu").

The user can can use scanImages: false to prevent this.

Changes

Tests
Documentation

c0d1ngm0nk3y · 2023-11-15T15:36:10Z

/it

cmd/detectExecuteScan.go

modulo11 · 2023-12-18T12:26:31Z

/it-go

vijayanjay · 2024-01-01T06:12:24Z

cmd/detectExecuteScan.go

+	args = append(args, "--detect.docker.passthrough.shared.dir.path.local=/opt/blackduck/blackduck-imageinspector/shared/")
+	args = append(args, "--detect.docker.passthrough.shared.dir.path.imageinspector=/opt/blackduck/blackduck-imageinspector/shared")
+	args = append(args, fmt.Sprintf("--detect.docker.passthrough.imageinspector.service.distro.default=%s", config.ContainerDistro))
+	args = append(args, "--detect.docker.passthrough.imageinspector.service.start=false")


Here it is indicating that the inspector itself will not need to be started, in that case how the image inspector is being started and exposed ?

It should be started as a sidecar container by the Piper step infrastructure. See https://github.com/SAP/jenkins-library/pull/4678/files/3fdefae8c191e82862b24148a873a3a84acbc928#diff-da165ed834ab60bb2fc7bc2fed1e93646b0f36264f16e79b160b4361781207efR596

sonarqubecloud · 2024-01-05T09:11:55Z

Quality Gate passed

Kudos, no new issues were introduced!

0 New issues
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarCloud

ghost · 2024-01-16T08:05:29Z

cmd/detectExecuteScan.go

+		return errors.Wrap(err, "Unable to read cpe")
+	}
+
+	registryUser := piperenv.GetResourceParameter(cpePath, "container", "repositoryUsername")


How repositoryUsername and repositoryPassword are created under CPE container path.
Looks like KanikoExecute only generates imageNameTags.json and registryUrl but not username/password. We referred this - https://github.com/SAP/jenkins-library/blob/master/resources/metadata/kanikoExecute.yaml#L309

I think, that should be given by the staging service step: https://github.wdf.sap.corp/ContinuousDelivery/piper-library/blob/51fe98e7f596950034475b3e4c7f3bd417f4a9dc/resources/metadata/sapCallStagingService.yaml#L274-L276

cc @anilkeshav27 Correct?

Thanks @loewenstein.
Tried with the combination of kanikoExecute and sapCallStagingService For generating repositoryUsername/Password. But we are getting failure message as Wrong action.
Could you please suggest the right configuration for generating the credentials file under CPE container path.
cc @anilkeshav27 @c0d1ngm0nk3y

cc @pbusko @modulo11 @phil9909

why do we need to read a CPE value from the file system, instead we could allow new params and have cpe references like ? https://github.com/SAP/jenkins-library/pull/4804/files#diff-da165ed834ab60bb2fc7bc2fed1e93646b0f36264f16e79b160b4361781207ef

with the new params, we also introduce the possibility to scan images that run outside the build, if the image lives in a private registry then users can bring in their registry url, password (maybe we would need a vault reference as well) and username . will be a good selling feature as well ?

@loewenstein It sounds good to extend support for external registries as well. With this enhancement, the behaviour of both Mend and DetectScan will align with each other.

I second to use another PR for any additional features

i would leave the final decision to @t-vijayan , who owns the piper step , since resourceRef is the standard way to read the CPE values

@anilkeshav27

why do we need to read a CPE value from the file system, instead we could allow new params and have cpe references like?

Because we need 4 values from the cpe: user, password, registry AND images. Since images is an array and can't be mapped (right?), it would be pointless and very confusing to expose those parameters. If we can map all 4 to the cpe, it would be preferable over reading it manually of course.

yes we should be able to map all 4 cpe the imageNameTags is the tricky bit , since there can be one image or multiple build, if it ties back to the cnbBuild / kaniko and if we fill this cpe

jenkins-library/resources/metadata/cnbBuild.yaml

Lines 369 to 370 in 4e3fa38

- name: container/imageNameTags

type: "[]string"

in all cases then it must be fine.

i know in kaniko we fill this list even in single image is built and i am assuming its the same in cnbBuild

anilkeshav27 · 2024-02-06T14:46:57Z

cmd/detectExecuteScan.go

+		return errors.Wrap(err, "Unable to read cpe")
+	}
+
+	registryUser := piperenv.GetResourceParameter(cpePath, "container", "repositoryUsername")


why do we need to read a CPE value from the file system, instead we could allow new params and have cpe references like ? https://github.com/SAP/jenkins-library/pull/4804/files#diff-da165ed834ab60bb2fc7bc2fed1e93646b0f36264f16e79b160b4361781207ef

with the new params, we also introduce the possibility to scan images that run outside the build, if the image lives in a private registry then users can bring in their registry url, password (maybe we would need a vault reference as well) and username . will be a good selling feature as well ?

cmd/detectExecuteScan.go

resources/metadata/detectExecuteScan.yaml

c0d1ngm0nk3y · 2024-02-08T11:40:37Z

We tested it with no scanContainerDistro and scanContainerDistro: ubuntu and it worked fine for us.

modulo11 · 2024-02-08T11:54:22Z

/it-go

modulo11 · 2024-02-08T13:29:35Z

/it-go

loewenstein · 2024-02-09T07:23:39Z

@anilkeshav27 @t-vijayan @vijayanjay Anything we could or should do? From our perspective, this is good to go.

c0d1ngm0nk3y · 2024-02-09T07:41:31Z

Anything we could or should do? From our perspective, this is good to go.

I still want to look at this comment - I think this is still suboptimal.

c0d1ngm0nk3y · 2024-02-09T09:13:27Z

/it-go

Co-authored-by: Johannes Dillmann <[email protected]> Signed-off-by: Ralf Pannemans <[email protected]>

Co-authored-by: Ralf Pannemans <[email protected]>

Co-authored-by: Johannes Dillmann <[email protected]>

Co-authored-by: Pavel Busko <[email protected]>

Co-authored-by: Ralf Pannemans <[email protected]> Signed-off-by: Ralf Pannemans <[email protected]> Co-authored-by: Johannes Dillmann <[email protected]>

Co-authored-by: Johannes Dillmann <[email protected]> Signed-off-by: Johannes Dillmann <[email protected]> Co-authored-by: Ralf Pannemans <[email protected]>

c0d1ngm0nk3y · 2024-02-09T11:26:57Z

/it-go

anilkeshav27

lgtm, i am not sure about the actual scan logic and the error handling , but i trust that this has been tested well enough

from the overall cpe auto wiring and side car logic it looks good to me

cc @t-vijayan @vijayanjay would also share your thoughts

Signed-off-by: Pavel Busko <[email protected]> Co-authored-by: Ralf Pannemans <[email protected]> Signed-off-by: Ralf Pannemans <[email protected]> Co-authored-by: Pavel Busko <[email protected]> Signed-off-by: Pavel Busko <[email protected]>

sonarqubecloud · 2024-02-09T12:12:51Z

Quality Gate passed

Issues
0 New issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarCloud

c0d1ngm0nk3y · 2024-02-09T12:12:53Z

/it-go

vijayanjay

@c0d1ngm0nk3y @anilkeshav27 With the latest changes able to trigger detect scan for the docker images built by kaniko and able to verify the scan results.
changes looks good to me.

c0d1ngm0nk3y · 2024-02-12T07:53:05Z

@anilkeshav27 The failing integration tests seem not to be related at the first glance. WDYT? Also the the mention https://api.github.com/repos/OliverNocon/piper-integration/releases which is a bit weird, isn't it?

feat(detectExecuteScan): Also scan images that are in the cpe Signed-off-by: Ralf Pannemans <[email protected]> Signed-off-by: Johannes Dillmann <[email protected]> Signed-off-by: Pavel Busko <[email protected]> Co-authored-by: Johannes Dillmann <[email protected]> Co-authored-by: Pavel Busko <[email protected]>

c0d1ngm0nk3y force-pushed the enhance-detectExecuteScan branch from f176377 to 9a334ae Compare November 15, 2023 15:33

pbusko force-pushed the enhance-detectExecuteScan branch 5 times, most recently from 252f89c to c402207 Compare November 23, 2023 14:31

c0d1ngm0nk3y commented Nov 23, 2023

View reviewed changes

cmd/detectExecuteScan.go Outdated Show resolved Hide resolved

pbusko force-pushed the enhance-detectExecuteScan branch 5 times, most recently from ab38479 to fe3f3b4 Compare November 27, 2023 10:22

pbusko force-pushed the enhance-detectExecuteScan branch from d0682d3 to e805329 Compare December 12, 2023 09:58

c0d1ngm0nk3y force-pushed the enhance-detectExecuteScan branch from 5d76dcf to 606a068 Compare December 13, 2023 12:47

modulo11 force-pushed the enhance-detectExecuteScan branch from 606a068 to 3fdefae Compare December 18, 2023 12:25

modulo11 marked this pull request as ready for review December 18, 2023 12:26

modulo11 requested a review from a team as a code owner December 18, 2023 12:26

vijayanjay reviewed Jan 1, 2024

View reviewed changes

pbusko force-pushed the enhance-detectExecuteScan branch from 3fdefae to 927d34b Compare January 5, 2024 09:11

ghost reviewed Jan 16, 2024

View reviewed changes

anilkeshav27 reviewed Feb 6, 2024

View reviewed changes

anilkeshav27 reviewed Feb 7, 2024

View reviewed changes

resources/metadata/detectExecuteScan.yaml Outdated Show resolved Hide resolved

modulo11 force-pushed the enhance-detectExecuteScan branch from 927d34b to 851022c Compare February 7, 2024 11:42

modulo11 requested review from loewenstein, anilkeshav27, vijayanjay and a user February 7, 2024 13:34

c0d1ngm0nk3y force-pushed the enhance-detectExecuteScan branch from 4df39b8 to 4d83e0c Compare February 8, 2024 11:42

c0d1ngm0nk3y force-pushed the enhance-detectExecuteScan branch from f171d11 to 1789f0f Compare February 8, 2024 13:03

c0d1ngm0nk3y force-pushed the enhance-detectExecuteScan branch from 5043035 to be744f3 Compare February 9, 2024 09:13

c0d1ngm0nk3y force-pushed the enhance-detectExecuteScan branch from be744f3 to fb51f13 Compare February 9, 2024 11:25

c0d1ngm0nk3y and others added 6 commits February 9, 2024 12:26

feat(detectExecuteScan): Also scan images that are in the cpe

21d711d

Co-authored-by: Johannes Dillmann <[email protected]> Signed-off-by: Ralf Pannemans <[email protected]>

make image scan opt-out feature

06db2b6

Co-authored-by: Ralf Pannemans <[email protected]>

Remove unused code related to volumeMounts

6d79e0d

Co-authored-by: Johannes Dillmann <[email protected]>

Rename volume

18c6ae6

Co-authored-by: Pavel Busko <[email protected]>

Simpify ux by combining 2 parameters

ef1ce7d

Co-authored-by: Ralf Pannemans <[email protected]> Signed-off-by: Ralf Pannemans <[email protected]> Co-authored-by: Johannes Dillmann <[email protected]>

Use parameter mapping for CPE instead of reading it directly

6e354df

Co-authored-by: Johannes Dillmann <[email protected]> Signed-off-by: Johannes Dillmann <[email protected]> Co-authored-by: Ralf Pannemans <[email protected]>

c0d1ngm0nk3y force-pushed the enhance-detectExecuteScan branch from fb51f13 to 1dd12f6 Compare February 9, 2024 11:26

anilkeshav27 approved these changes Feb 9, 2024

View reviewed changes

Improved error handling

bd957bb

Signed-off-by: Pavel Busko <[email protected]> Co-authored-by: Ralf Pannemans <[email protected]> Signed-off-by: Ralf Pannemans <[email protected]> Co-authored-by: Pavel Busko <[email protected]> Signed-off-by: Pavel Busko <[email protected]>

c0d1ngm0nk3y force-pushed the enhance-detectExecuteScan branch from 1dd12f6 to bd957bb Compare February 9, 2024 12:12

vijayanjay approved these changes Feb 12, 2024

View reviewed changes

c0d1ngm0nk3y enabled auto-merge (squash) February 12, 2024 07:49

c0d1ngm0nk3y merged commit 49b8080 into master Feb 12, 2024
12 checks passed

c0d1ngm0nk3y deleted the enhance-detectExecuteScan branch February 12, 2024 07:56

pbusko mentioned this pull request Feb 19, 2024

fix(detectExecuteScan): do not read imageNameTags directly from CPE #4833

Merged

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(detectExecuteScan): Also scan images that are in the CPE #4678

feat(detectExecuteScan): Also scan images that are in the CPE #4678

c0d1ngm0nk3y commented Nov 15, 2023 •

edited

Loading

c0d1ngm0nk3y commented Nov 15, 2023

modulo11 commented Dec 18, 2023

vijayanjay Jan 1, 2024

loewenstein Jan 1, 2024

sonarqubecloud bot commented Jan 5, 2024

ghost Jan 16, 2024

loewenstein Jan 16, 2024

ghost Jan 17, 2024

loewenstein Jan 17, 2024

anilkeshav27 Feb 6, 2024

vijayanjay Feb 7, 2024

c0d1ngm0nk3y Feb 8, 2024

anilkeshav27 Feb 8, 2024 •

edited

Loading

c0d1ngm0nk3y Feb 8, 2024

anilkeshav27 Feb 8, 2024

anilkeshav27 Feb 6, 2024

c0d1ngm0nk3y commented Feb 8, 2024

modulo11 commented Feb 8, 2024

modulo11 commented Feb 8, 2024

loewenstein commented Feb 9, 2024

c0d1ngm0nk3y commented Feb 9, 2024

c0d1ngm0nk3y commented Feb 9, 2024

c0d1ngm0nk3y commented Feb 9, 2024

anilkeshav27 left a comment •

edited

Loading

sonarqubecloud bot commented Feb 9, 2024

c0d1ngm0nk3y commented Feb 9, 2024

vijayanjay left a comment

c0d1ngm0nk3y commented Feb 12, 2024

feat(detectExecuteScan): Also scan images that are in the CPE #4678

feat(detectExecuteScan): Also scan images that are in the CPE #4678

Conversation

c0d1ngm0nk3y commented Nov 15, 2023 • edited Loading

Changes

c0d1ngm0nk3y commented Nov 15, 2023

modulo11 commented Dec 18, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sonarqubecloud bot commented Jan 5, 2024

Quality Gate passed

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

anilkeshav27 Feb 8, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

c0d1ngm0nk3y commented Feb 8, 2024

modulo11 commented Feb 8, 2024

modulo11 commented Feb 8, 2024

loewenstein commented Feb 9, 2024

c0d1ngm0nk3y commented Feb 9, 2024

c0d1ngm0nk3y commented Feb 9, 2024

c0d1ngm0nk3y commented Feb 9, 2024

anilkeshav27 left a comment • edited Loading

Choose a reason for hiding this comment

sonarqubecloud bot commented Feb 9, 2024

Quality Gate passed

c0d1ngm0nk3y commented Feb 9, 2024

vijayanjay left a comment

Choose a reason for hiding this comment

c0d1ngm0nk3y commented Feb 12, 2024

c0d1ngm0nk3y commented Nov 15, 2023 •

edited

Loading

anilkeshav27 Feb 8, 2024 •

edited

Loading

anilkeshav27 left a comment •

edited

Loading