Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Setup elasticsearch dependency monitoring with Snyk #87620

Closed
4 tasks done
breskeby opened this issue Jun 13, 2022 · 7 comments
Closed
4 tasks done

Setup elasticsearch dependency monitoring with Snyk #87620

breskeby opened this issue Jun 13, 2022 · 7 comments
Assignees
Labels
:Delivery/Build Build or test infrastructure Team:Delivery Meta label for Delivery team

Comments

@breskeby
Copy link
Contributor

breskeby commented Jun 13, 2022

We want to monitor the elasticsearch dependencies on a regular basis using Snyk to be able to recognise vulnerabilities faster and being able to proactively react on security issues within the production dependencies.

  • Setup Gradle plugin for monitoring dependencies in snyk ( #88036 )
  • Allow configuring snyk target reference and lifecycle properties ( 88220 )
  • Setup CI job for automated update of snyk dependency monitoring of versions under development ( 88522 )
  • Setup automation to update of snyk dependency monitoring of released versions
@breskeby breskeby added the :Delivery/Build Build or test infrastructure label Jun 13, 2022
@breskeby breskeby self-assigned this Jun 13, 2022
@elasticmachine elasticmachine added the Team:Delivery Meta label for Delivery team label Jun 13, 2022
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-delivery (Team:Delivery)

@breskeby
Copy link
Contributor Author

We discussed initially that all dependencies that are shipped as part of our distribution should be scanned. Ideally I think it would make sense to scan all dependencies to some degree. Even test dependencies can be a concern of security even though its less likely compared to production dependencies.

@jkakavas
Copy link
Member

Ideally I think it would make sense to scan all dependencies to some degree. Even test dependencies can be a concern of security even though its less likely compared to production dependencies.

That's an interesting point! My thoughts:

  • We are mostly interested in production dependencies as these are the ones that get shipped with our products and have the highest possibility to affect the products and our users, I'd be fine with keeping the scope to these at least to begin with.
  • We could (and plan to) reuse this effort to build SBOMs for our products, so the more complete our SBOMs are the better. In that sense test dependencies make sense to be included.
  • I'm worried about the scope and the potential impact this might have to the false positives that we get (and potentially have to deal with). For instance, context/applicability/impact is hard to convey and to consume by recipients in most cases. A medium impact vulnerability in a test dependency that can never impact the product might need x days of engineering work to upgrade to a new version. But a user is seeing "Product is affected by a medium impact vulnerability" and we get z requests to upgrade this ASAP.
  • We mostly deal with runtime dependencies in our SCA efforts so far across the company.

I'm inclined to prefer to try and capture all dependencies and try to figure out if there is a way to differentiate on presentation/impact analysis/prioritisation based on what configuration this comes from. We could have different SCA projects/organisations for test/production code, for instance.

@levinebw
Copy link

We are mostly interested in production dependencies as these are the ones that get shipped with our products and have the highest possibility to affect the products and our users,

Agree our higher priority should be to the dependencies included in the shipped products. That said, there is risk with the test-code, build-code, etc. we should be concerned with vis-a-vis 'supply-chain attacks' that target build environments specifically.

We could (and plan to) reuse this effort to build SBOMs for our products, so the more complete our SBOMs are the better. In that sense test dependencies make sense to be included.

My assumption (possibly mistaken) has been that published SBOMs would include only what is included in the released product and would not include test code.

@mark-vieira
Copy link
Contributor

My assumption (possibly mistaken) has been that published SBOMs would include only what is included in the released product and would not include test code.

This would be my understanding as well. I think publishing test dependencies would only cause confusion for folks. From my perspective test dependencies are completely internal and have nothing to do with the delivered software. I'd argue we should explicitly omit them for that reason, completely independent of the level of effort of including this information.

@breskeby
Copy link
Contributor Author

Understood. So for now we will only monitor dependencies for now that are shipped with our distro and can be a potential security thread for our customers.

breskeby added a commit that referenced this issue Jun 29, 2022
…de (#88036)

This adds the generation and upload logic of Gradle dependency graphs to snyk

We directly implemented a rest api based snyk plugin as:

the existing snyk gradle plugin delegates to the snyk command line tool the command line tool 
uses custom gradle logic by injecting a init file that is 

a) using deprecated build logic which we definitely want to avoid
b) uses gradle api we avoid like eager task creation.

Shipping this as a internal gradle plugin gives us the most flexibility as we only want to monitor 
production code for now we apply this plugin as part of the elasticsearch.build plugin, 
that usage has been for now the de-facto indicator if a project is considered a "production" project 
that ends up in our distribution or public maven repositories. This isnt yet ideal and we will revisit 
the distinction between production and non production code / projects in a separate effort.

As part of this effort we added the elasticsearch.build plugin to more projects that actually end up 
in the distribution. To unblock us on this we for now disabled a few check tasks that started failing by applying elasticsearch.build. 

Addresses  #87620
breskeby added a commit to breskeby/elasticsearch that referenced this issue Jun 30, 2022
We will use target reference to distinguish between different versions or branches of our elasticsearch project
to be able to trace vulnerable dependencies down to the version.

snyk lifecycle property allows filtering the project overview by `production` or `development`. When version
is ending with SNAPSHOT we configure the lifecycle as development. Otherwise its production.

Related to elastic#87620
breskeby added a commit that referenced this issue Jun 30, 2022
)

We will use target reference to distinguish between different versions or branches of our elasticsearch project
to be able to trace vulnerable dependencies down to the version.

snyk lifecycle property allows filtering the project overview by `production` or `development`. When version
is ending with SNAPSHOT we configure the lifecycle as development. Otherwise its production.

Related to #87620
breskeby added a commit to breskeby/elasticsearch that referenced this issue Jun 30, 2022
…de (elastic#88036)

This adds the generation and upload logic of Gradle dependency graphs to snyk

We directly implemented a rest api based snyk plugin as:

the existing snyk gradle plugin delegates to the snyk command line tool the command line tool
uses custom gradle logic by injecting a init file that is

a) using deprecated build logic which we definitely want to avoid
b) uses gradle api we avoid like eager task creation.

Shipping this as a internal gradle plugin gives us the most flexibility as we only want to monitor
production code for now we apply this plugin as part of the elasticsearch.build plugin,
that usage has been for now the de-facto indicator if a project is considered a "production" project
that ends up in our distribution or public maven repositories. This isnt yet ideal and we will revisit
the distinction between production and non production code / projects in a separate effort.

As part of this effort we added the elasticsearch.build plugin to more projects that actually end up
in the distribution. To unblock us on this we for now disabled a few check tasks that started failing by applying elasticsearch.build.

Addresses  elastic#87620
# Conflicts:
#	build-tools-internal/src/main/java/org/elasticsearch/gradle/internal/BuildPlugin.java
#	client/benchmark/build.gradle
#	client/test/build.gradle
#	distribution/tools/cli-launcher/build.gradle
#	distribution/tools/java-version-checker/build.gradle
#	distribution/tools/windows-service-cli/build.gradle
#	rest-api-spec/build.gradle
#	test/framework/build.gradle
breskeby added a commit to breskeby/elasticsearch that referenced this issue Jun 30, 2022
…stic#88220)

We will use target reference to distinguish between different versions or branches of our elasticsearch project
to be able to trace vulnerable dependencies down to the version.

snyk lifecycle property allows filtering the project overview by `production` or `development`. When version
is ending with SNAPSHOT we configure the lifecycle as development. Otherwise its production.

Related to elastic#87620
breskeby added a commit that referenced this issue Jul 15, 2022
Adds a new ci Jenkins job configuration for running snyk dependency monitoring on a daily basis. We setup a service account in snyk and resolve the api token for publishing in vault.

Related to #87620
breskeby added a commit to breskeby/elasticsearch that referenced this issue Jul 15, 2022
Adds a new ci Jenkins job configuration for running snyk dependency monitoring on a daily basis. We setup a service account in snyk and resolve the api token for publishing in vault.

Related to elastic#87620
breskeby added a commit that referenced this issue Jul 15, 2022
Adds a new ci Jenkins job configuration for running snyk dependency monitoring on a daily basis. We setup a service account in snyk and resolve the api token for publishing in vault.

Related to #87620
breskeby added a commit to breskeby/elasticsearch that referenced this issue Jul 15, 2022
…de (elastic#88036)

This adds the generation and upload logic of Gradle dependency graphs to snyk

We directly implemented a rest api based snyk plugin as:

the existing snyk gradle plugin delegates to the snyk command line tool the command line tool
uses custom gradle logic by injecting a init file that is

a) using deprecated build logic which we definitely want to avoid
b) uses gradle api we avoid like eager task creation.

Shipping this as a internal gradle plugin gives us the most flexibility as we only want to monitor
production code for now we apply this plugin as part of the elasticsearch.build plugin,
that usage has been for now the de-facto indicator if a project is considered a "production" project
that ends up in our distribution or public maven repositories. This isnt yet ideal and we will revisit
the distinction between production and non production code / projects in a separate effort.

As part of this effort we added the elasticsearch.build plugin to more projects that actually end up
in the distribution. To unblock us on this we for now disabled a few check tasks that started failing by applying elasticsearch.build.

Addresses  elastic#87620
# Conflicts:
#	test/framework/build.gradle
breskeby added a commit that referenced this issue Jul 15, 2022
…ion code (#88036) (#88566)

* Setup elasticsearch dependency monitoring with Snyk for production code (#88036)

This adds the generation and upload logic of Gradle dependency graphs to snyk

We directly implemented a rest api based snyk plugin as:

the existing snyk gradle plugin delegates to the snyk command line tool the command line tool
uses custom gradle logic by injecting a init file that is

a) using deprecated build logic which we definitely want to avoid
b) uses gradle api we avoid like eager task creation.

Shipping this as a internal gradle plugin gives us the most flexibility as we only want to monitor
production code for now we apply this plugin as part of the elasticsearch.build plugin,
that usage has been for now the de-facto indicator if a project is considered a "production" project
that ends up in our distribution or public maven repositories. This isnt yet ideal and we will revisit
the distinction between production and non production code / projects in a separate effort.

As part of this effort we added the elasticsearch.build plugin to more projects that actually end up
in the distribution. To unblock us on this we for now disabled a few check tasks that started failing by applying elasticsearch.build.

Addresses  #87620
# Conflicts:
#	test/framework/build.gradle

* Fix license report when backporting
breskeby added a commit to breskeby/elasticsearch that referenced this issue Jul 19, 2022
Adds a new ci Jenkins job configuration for running snyk dependency monitoring on a daily basis. We setup a service account in snyk and resolve the api token for publishing in vault.

Related to elastic#87620
breskeby added a commit that referenced this issue Jul 19, 2022
Adds a new ci Jenkins job configuration for running snyk dependency monitoring on a daily basis. We setup a service account in snyk and resolve the api token for publishing in vault.

Related to #87620
@breskeby
Copy link
Contributor Author

We've integrated the snyk monitoring data publishing of released versions as part of the elasticsearch finalize release step

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Delivery/Build Build or test infrastructure Team:Delivery Meta label for Delivery team
Projects
None yet
Development

No branches or pull requests

5 participants