Support custom cert store

[andrewvc]

We should support some simple way of supporting custom certs for playwright. See microsoft/playwright#1799 (comment) for more info.

[a03nikki]

As a temporary work around, build a custom container based on the official one. The private certificate authority (CA) file needs to be copied to /usr/share/pki/ca-trust-source/anchors/ and then the /usr/bin/update-ca-trust command needs to be ran as root, all before Heartbeat starts.

[andrewvc]

We've documented @a03nikki 's approach here: https://github.com/elastic/synthetics/blob/master/examples/todos/Dockerfile#L20 .

Ideally there'd be a way to do this in a more streamlined way, just running the update-ca-trust command automatically, only requiring the user to share the file into the anchors folder.

It doesn't seem like there's any way to adjust the CA store just for the browser rather than for the whole system. @vigneshshanmugam @jahtalab do you know of any way to do this without requiring root?

[philippkahr]

Since we changed to Ubuntu as a basis for the docker images, the command changed from update-ca-trust to update-ca-certificates. Additionally the folder changed to /usr/local/share/ca-certificates/

And ubuntu needs a crt file. Leaving a few openssl commands here:

Converting a base64 file (often known as PEM)
openssl x509 -in foo.pem -inform PEM -out foo.crt

maybe we could add some magic to the docker image that it tries to detect what format it is and we could convert it on the fly?

[philippkahr]

It seems that changing to Ubuntu invokes a bunch of other problems in this context.

Updating the system cert store does not update the cert store of the chromium browser?

There seems to be a difference between the behaviour of Ubuntu and Centos in that case. Changing back to 7.16.3 and adding the certificate to the docker image, works:

COPY elasticsearch-ca.pem /usr/share/pki/ca-trust-source/anchors/
USER root
RUN /usr/bin/update-ca-trust
USER heartbeat

[beantoast]

This workaround work for me on ubuntu image:

Login as root to run:
apt update && apt install libnss3-tools

Exit and login as elastic-agent and run
certutil -A -n "<cert_nick_name>" -d ~/.pki/nssdb -t C,, -a -i <cert_location>.crt

[paulb-elastic]

Linking #717

[paulb-elastic]

A recent discussion also led to one idea being to include custom CAs when pushing configurations.

[Jaraxal]

+1

Does this issue incorporate the ability for Synthetics to support certificate-based authentication, or would that be a separate issue?

[andrewvc]

@Jaraxal for lightweight monitors that is very readily supported. For browser monitors with playwright it's a bit tricky

[graphaelli]

Will this approach change again with the move to wofli images?

[wortmanb]

Clusters on air-gapped networks with PKI are going to need this badly.

[felix-lessoer]

Playwright is supporting this option in their version 1.46 -- https://playwright.dev/docs/next/api/class-browser#browser-new-context-option-client-certificates

Which version are we running and when can we expect to make it available in Elastic Synthetics?

[shahzad31]

@felix-lessoer
we are on =1.45.1 at the moment, we can consider upgrading to 1.46.0 , what do you think @vigneshshanmugam @emilioalvap ??

[emilioalvap]

Which version are we running and when can we expect to make it available in Elastic Synthetics?

@felix-lessoer In-memory certificates will land on a upcomingPW release, we might be able to extend Elastic Synthetics to support it then. cc @drewpost @andrewvc @vigneshshanmugam @graphaelli

Will this approach change again with the move to wofli images?

@graphaelli It shouldn't, Chromium should use nss store regardless of the distro. The tools required to manually edit the storage do vary across distros.

[shahzad31]

shipped with https://github.com/elastic/synthetics/releases/tag/v1.13.0

[vigneshshanmugam]

Keeping this as open till we have the necessary docs in place which should cover both Elastic managed locations and also private locations.

[vigneshshanmugam]

elastic/observability-docs#4351