Docker node.js image comes with many vulnerabilities on Debian

[dgonzalez]

Hi, I am part of the security working group (cc @nodejs/security-wg) and I also work as a DevSecOps engineer and one of the challenges that I find with Node.js official docker images is that scanning them with Clair (Debian version, not Alpine) I get a high number of vulnerabilities (I will attach a report in the morning if required) which is not necessarily bad because the majority of them are marked as NO-DSA from the Debian security board which means that either do not affect Debian or in general, it has been decided not to fix.

I am wondering if it would make sense to also release a CentOS (or other) version with focus on low vulnerability count (or even 0) as, today, it is really hard to demonstrate that my images are vulnerability free.

I opened this for discussion and I offer my help to work on the outcomes if required.

[chorrell]

I don't know if switching to CentOS or maybe Ubuntu would get you the outcome you are looking for. We rely on the upstream Docker team to update the base images periodically. Not sure about CentOS, I see a lot of flagged (but maybe false positive?) vulnerabilities in the latest CentOS 7 image:

https://hub.docker.com/r/library/centos/tags/7/

https://cl.ly/0W423w1o0i44

Ubuntu Bionic seems better:

https://hub.docker.com/r/library/ubuntu/tags/bionic/

https://cl.ly/3t2s3F0B373q

(note you need to be logged into the docker hub to see the vulnerability scan)

I honestly don't have a good answer for this. The "This image has vulnerabilities!" topic comes up pretty regularly (see for example #764 and the other linked issues).

I guess one option would be to do an apt-get update; apt-get upgrade to pull in the latest security updates, but that would affect the image sizes.

[dgonzalez]

Yeah... it is a hairy one. I mentioned CentOS because it is pretty much a "industry standard" but... not a strong push for it.

I think this is being pushed more into the "evangelisation" part of the equation.

[tianon]

See https://github.com/docker-library/faq#why-does-my-security-scanner-show-that-an-image-has-so-many-cves for some more detail about where we (official images curators) stand on this issue.

[tianon]

Just for kicks, I did a Clair scan of centos:7, and there are a number of fixable RHSA issues in it (my Clair reports don't ever report anything that doesn't have a package update, since they're not really actionable for me in any way), so switching to CentOS really wouldn't mitigate this much (if any).

[LaurentGoderre]

In my opinion, the Alpine version is the one to use to reduce vulnerability. SInce you have to explicitly specify what you include it makes it way safer.

[lirantal]

Completely agree with @dgonzalez here - if we can impact the node image it will definitely help us resolve concerns of security vulnerabilities for teams.

I tested in a project of mine, which is a classic example of Dockerfiles for teams:

FROM node:10

RUN mkdir /usr/src/goof
RUN mkdir /tmp/extracted_files
COPY . /usr/src/goof
WORKDIR /usr/src/goof

RUN npm update
RUN npm install
EXPOSE 3001
EXPOSE 9229
ENTRYPOINT ["npm", "start"]

And a snyk scan snyk test --docker node:10 --file=Dockerfile revealed:

Not so nice to see so many vulns and I assume that many users are just using the default node image, and not many consider this security concern to try out different images to reduce the risk.

[StefanSchoof]

Maybe someone explains what the numbers, that snyk is reporting and not giving thumbs down. There is also a blog post from snyk about the vulnerabilities: https://snyk.io/blog/top-ten-most-popular-docker-images-each-contain-at-least-30-vulnerabilities/
This can spread some uncertainty.

This issue comes first if I google "node docker known vunabilties" trying to find some background information about the snyk post.

[sam-github]

@tianon I read https://github.com/docker-library/faq#why-does-my-security-scanner-show-that-an-image-has-so-many-cves and don't understand where the maintainers stand.

It describes why things are as they are, but doesn't describe why Node.js doesn't switch to an image base, like Alpine, that scans with less CVEs.

I'm trying to understand why this isn't considered a problem, but I don't yet.

[tianon]

There is an Alpine variant of every currently supported Node version.

What that FAQ entry is trying to convey is that the presence of a vulnerability is not necessarily a true red flag. There are a lot more factors, so it would be interesting to see a good deep dive into the current list and whether any are actively applicable to how the image is used.

[sam-github]

A deep dive is possible, we just need a volunteer.... :-)

I look at this like compiler warnings: we try to rewrite code that causes them so the warnings are silenced, even if the warning was invalid, so that no one coming along after has to look at those warnings and take time to evaluate them.

In this case, if it was possible with mild changes to eliminate many of the possibly-spurious warnings, then we wouldn't have to deep dive into them. And users wouldn't have to wonder if they are applicable, because they aren't there.

I received a suggestion that

a lot of it would probably go away Node's default base image was switched from buildpack-deps:jessie to buildpack-deps:stretch

Is that reasonable?

[tianon]

IMO absolutely not only reasonable but necessary and desirable (and the whole point of adding a -jessie suffix to the existing image tags to help communicate that transition), although I'm not the image maintainer here 😅

[sam-github]

@tianon I thought when you wrote "we (official images curators)" it meant you were! Sorry. you curate some other images?

[tianon]

I'm a maintainer on https://github.com/docker-library/official-images, which is one level up from this repo (so I help review changes to the node image, but I don't directly maintain the image). If you look at https://github.com/docker-library/official-images/blob/master/library/node, it lists the "Node.js Docker Team" as the maintainers of the node official image. So in this context, I'm more of an advisor than a decision maker with regards to changes to the image.

[tianon]

(See https://github.com/nodejs/docker-node/blob/master/ACTIVE_MAINTAINERS for the current list of declared-active maintainers of this repo. 👍)

[mhdawson]

In my mind, there is some responsibility on those raising/publicizing the issue to have confidence that the issues being reported to have an impact. We know that a lot of scanning can have false positives. I don't know what the case is here though. @dgonzalez, @lirantal do you have an assessment of the issues that are being reported in terms of severity/importance.

It seems to me that the problem is that the base images have a bunch of vulns being reported, but that is a bigger problem outside of Node.js. I do agree that if there are some base images which are better than others then we might want to expand coverage to include them.

[apolloclark]

I built it:
https://github.com/apolloclark/packer-docker#support-matrix

The project uses: Gradle, Packer, Ansible, Serverspec, to rebuild base images (including node.js), in a fully automated and parallelized way, which includes pulling down security updates. I run it once every 7 days on Sunday. It supports: Ubuntu 18.04, Ubuntu 16.04, Debian 10, Debian 9, RHEL 7 UBI, CentOS 7 UBI, and Amazon Linux 2. I run into the issue of supporting a specific base OS in Docker images all the time with my consulting clients. Generally they'll have been using Amazon or Azure for the past 5+ years, and their entire Ops, Dev, QA, and Security teams have standardized on a single distro; often either Ubuntu 16.04 / 18.04, or RHEL / CentOS, or Amazon Linux. Then some of their dev teams want to use Docker, and there's no singular consistent base image:
https://gist.github.com/apolloclark/8f339de76182d5fcebff7e5b3a24159c#summary

For the security aspects, I work as a "DevSecOps consultant," so I'll throw out my experiences. Not all CVE's are equal. What matters is "critical" and "high" rated CVE's. Those vulns include things like buffer overflows, which allow remote code execution, privilege escalation (spectre), denial of service (http slowloris), and information disclosure (heartbleed). If a company falls under PCI (credit card data), MARS-E / HIPAA (medical data), or FIPS / FedRAMP (government), then they are required to fix "critical" and "high" vulns within 30 days of a fix being available. A fix could be a simple config change, or may require a service upgrade (ex: apache), or an upgrade and restart (kernel modules). About 80% of the time, a given critical / high CVE will be announced along with a fix. The other 20% of the time, the critical / high CVE is announced, while a fix is being tested, and is released within about 7 days. Ideally every Official Docker Image would be rebuilt with security updates every 7 days.

[lirantal]

@mhdawson

In my mind, there is some responsibility on those raising/publicizing the issue to have confidence that the issues being reported to have an impact. We know that a lot of scanning can have false positives.

The concern I think isn't about false positives. At least, not as far as I'm concerned with the Snyk findings. It is however the notion that even if say you have a 20 gcc high severity vulnerability in a container image, then the likelihood of someone being able to exploit it from interacting with the app is presumably very low.

@apolloclark nice project! With regards to:

Not all CVE's are equal. What matters is "critical" and "high" rated CVE's

I will challenge this to add that it's not just the CVSS that is of matter but also whether there's an exploit in the wild, is it just a PoC, or more? and other factors that come to play.

To add to @apolloclark's "Ideally every Official Docker Image would be rebuilt with security updates every 7 days." - we found related stats in our internal tests and a report we published around this that 20% of docker image vulnerabilities can be remediated if you re-build the image. This is of course depending on how the image is built to begin with, but the notion behind this is that if you're doing a RUN apt-get update ... in a Dockerfile, then if you re-build the image, you'll be getting new (security) updates, and therefore chances increase of lowering the footprint of vulns in an image.

I'll be happy to help work on this and I wonder where do others stand here and if anyone from the Node.js Docker team would be happy to collaborate on the effort of getting to a better state of the Node.js docker images vulns.

[nschonni]

@lirantal we have no ability to address any of these security issues here. Since this issue was opened we've documented this over in https://github.com/nodejs/docker-node/blob/master/SECURITY.md

If Snyk is flagging issues with the image, they need to raise them with the upstream official Debian/Alpine images, or the nodejs project. There is nothing that we add here except the binaries, NPM, and Yarn. All other build deps are cleaned back to the base image during the build