Temporary release problems following certificate expiry

[RichardHaselgrove]

I tried downloading and building under VS2019 for the first time today. The client built - with warnings, but it's currently running.

But the Manager failed to build. I can provide error messages tomorrow, but that's not important.

There's a suggestion that we could release a composite installer, with an updated client and ca-certificate bundle, but an older Manager. Could that keep the science on track, and buy time to sort out what's wrong with the Manager build?

[AenBleidd]

@RichardHaselgrove, it would be nice if you provide build errors, because I build it with no issues.
And I suggest not to build client and Manager with different compilers since it's not necessary for now, and in any case building current master will require an extensive testing for client also

[RichardHaselgrove]

Happy to do that, but the build machine is dual-boot Win 10 and Win 7 - I switch in separate storage devices, so the two OSs are completely separate. Currently, it's doing science under Win 7.

I just want to keep our clients - the science projects - serviced as soon as possible. If you can build the Manager OK, why isn't there a new release in testing already? What's the hold-up? The silence is deafening.

[AenBleidd]

If you can build the Manager OK, why isn't there a new release in testing already? What's the hold-up? The silence is deafening.

'I can build Manager' doesn't mean it's ready for release ;) And there is a blocker I'm working on currently: #4073

Since it will be done - I believe we could think about new major release for alpha testing.

[RichardHaselgrove]

I don't think I'd call that a blocker, compared with all Windows clients being unable to return completed science to multiple projects.

[AenBleidd]

It's a blocker for next major release.
Also, I don't think we should release hotfix with anything but certificate (and 2 more tiny fixes) since it will require a full testing cycle, and currently users can fix their env by manual certificate copy.
Release of untested build made in rush is much-much worse

[RichardHaselgrove]

The whole principle of the GitHub workflow is that the master branch is maintained in constant readiness, just in case of an eventuality like this. Blockers should be isolated in WIP branches until solved and merged.

I made a temporary working certificate bundle available to active participants on 30 September. Set-and-forget participants need something stronger.

Closing comms for tonight.

[CharlieFenton]

The whole principle of the GitHub workflow is that the master branch is maintained in constant readiness, just in case of an eventuality like this. Blockers should be isolated in WIP branches until solved and merged.

Hopefully the master branch is always ready for building a release candidate ready to go through extensive alpha testing before becoming a public release. This is different from being ready for immediate public release, even in an emergency. The latter would require every PR to go through full alpha testing before merging into master.

I agree with @AenBleidd that the correct way to do this is to add the minimal necessary changes to the exiting release. However, due to the terrible pollution of the current release branch client_release/7/7.16 due to people mistakenly merging master into it repeatedly, that will be difficult.

I created a less polluted release branch for Mac releases: client_release/7.16/7.16.12_branch by branching off the client_release/7/7.16 branch at a point before much of the damage was done. You might want to create a new branch from my client_release/7.16/7.16.12_branch as a starting point. Please do not merge anything into my client_release/7.16/7.16.12_branch!

[AenBleidd]

I already made a branch directly from 7.16.11 release tag that was the last Windows release

[CharlieFenton]

I already made a branch directly from 7.16.11 release tag that was the last Windows release

That works too ;=). But you may want to look at the cherry picks I merged into my client_release/7.16/7.16.12_branch from master as I believe some of them are bug fixes that apply to MS Windows and Linux as well.

[AenBleidd]

@CharlieFenton, we want this hotfix release as small as possible to avoid full cycle of testing.
We can test other fixes with major release (hopefully soon)

[CharlieFenton]

I want to expand on my earlier comment about the use of the release branch rather than master. BOINC versions numbers are of the form major.minor.bugfix, and all releases with the same major and minor version numbers are normally built from the same release branch. The BOINC Client Release Process says:

only important bug fixes should be merged into the release branch.

This implies that all releases from a release branch must be built with the same versions of all the dependent libraries, unless using a different library version is the only way to fix a serious bug. This is especially true of an emergency fix like this one. In this case, it is sufficient to include an updated ca-bundle.crt, since this is a short-term fix and there should be a new major release very soon which can contain an updated OpenSSl library.

One reason I want to mention the importance of using the same dependent library versions throughout all releases from the same release branch (i.e., with the same BOINC major and minor version numbers) is my concern about the use of vcpkg. Additional care may be required to ensure that newer builds from the same release branch do not inadvertently use newer library versions.

I tried to express this concern in my comments in PR #4385 but unfortunately I said it poorly so it was not understood.

[AenBleidd]

One reason I want to mention the importance of using the same dependent library versions throughout all releases from the same release branch (i.e., with the same BOINC major and minor version numbers) is my concern about the use of vcpkg. Additional care may be required to ensure that newer builds from the same release branch do not inadvertently use newer library versions.

This will be true for the next release.
vcpkg now supports manifest mode https://vcpkg.readthedocs.io/en/latest/users/manifests/ and it provides you a way to use some particular vesion of any dependency.
I plan to implement this for the next major release.

[CharlieFenton]

@AenBleidd I agree about minimizing the changes for this hotfix release. You had mentioned 2 more tiny fixes so I thought I would bring to your attention some possible other important bug fixes. When I cherry-picked commits from master into my client_release/7.16/7.16.12_branch back on 2 June, I spent a great deal of time examining every commit in master to select only important bug fixes. These have been in the field since my release of 7.16.17 in early June, so they have all been throroughly tested.

But I will leave it up to you to decide whether any are important enough for this hotfix release.

[AenBleidd]

@CharlieFenton, is ca-bundle.crt used on OSX?

[RichardHaselgrove]

Build reports - they look much the same as yesterday's run. I concentrate on boincmd (which builds the client as well) and boincmgr - as an end user, I don't need all the other special builds for daily use. Software listed in VS 2019 install was downloaded yesterday.

VS 2019 install.txt
boinccmd output.txt
boinccmd error list.txt
boincmgr output.txt
boincmgr error list.txt

[AenBleidd]

Damn.

@RichardHaselgrove, 7.16.20 branch should be built with VS2010 only (you can use VS2013 on your own risk).
VS2019 is for current master only, since on 7.16.20 branch it's outdated and not planned to be fixed there because there is no need in this.

[RichardHaselgrove]

Oops. I did build v7.16.20 from a zip download on Monday morning, using my usual VS2013 under Windows 7. The machine I deployed it on is still running fine, so I'd be happy with that.

I might try a VS2019 build from master later, purely for fun - it's not germane to this issue.

[CharlieFenton]

is ca-bundle.crt used on OSX?

Yes, but the Mac build has not had this problem because it has used newer versions of OpenSSL for both the build of curl and for the client for several years. It has used openssl-1.1.0l (the last character is lower case L) since BOINC 7.16.17.

Please note that curl is not the only place that BOINC uses OpenSSL; the BOINC client code itself also uses OpenSSL extensively. So I suspect that your PR #4545 may not fully eliminate the need for OpenSSL on MS Windows.

[AenBleidd]

Please note that curl is not the only place that BOINC uses OpenSSL; the BOINC client code itself also uses OpenSSL extensively. So I suspect that your PR #4545 may not fully eliminate the need for OpenSSL on MS Windows.

OpenSSL is kept as a dependency. But when curl is build without openssl, it doesn't use ca-bundle.crt file, that's actually a goal of that PR.

[AenBleidd]

@CharlieFenton,

Yes, but the Mac build has not had this problem because it has used newer versions of OpenSSL for both the build of curl and for the client for several years.

Is there a way to get rid of it and use OSX system certificate store (whatever it's called?).
What advantages our own 'ca-bundle.crt' gives us?

[CharlieFenton]

What advantages our own 'ca-bundle.crt' gives us?

This answer applies to MSWindows as well as OS X, and perhaps to Linux. As I have mentioned elsewhere, it is common for people to repurpose their old obsolete computers to run BOINC when they upgrade. Some of those computers are running a version of the OS that is no longer supported, such as Windows 7 or MacOS 10.13, which BOINC still supports, even with our new releases.
In fact, it is not unusual for some BOINC volunteers to run these past-support operating systems on their current computers.

I suspect that the OS developers (e.g., Microsoft or Apple) no longer update the system-provided certificate for operating systems they no longer support, so using the system-provided certificate may create a worse situation than we have now using OpenSSL with a BOINC-provided certificate. Manually upgrading a system-provided certificate is likely beyond the understanding of most users.

A better solution might be to do the following three things:
[1] Implement an automatic or semi-automatic process that periodically updates curl/ca-bundle.crt in GIT master so each new release of BOINC has the most current available certificate set.

[2] BOINC should alert the user in clear language when the certificate is expired or near expiration, with a link to instructions how to fix the problem. Ideally, this alert should be more noticeable than an entry in the Event Log or even a Notice in the Manager since most users probably just set and forget BOINC; at minimum it should be an actual alert dialog that is visible even when BOINC Manager is hidden.

[3] Create a tiny installer for each platform (MS Windows, MacOS and Linux) that fetches the latest ca-bundle.crt either from GIT master or directly from Mozilla and updates it in the BOINC user's BOINC Data directory. An even better alternative would be to add code to the BOINC client itself to automatically download a new certificate before the current one expires; this would eliminate the need for item [2].

This third item assumes that curl is configured to look for the certificate in the BOINC Data directory at run time. I believe that is already done by client/http_curl.cpp lines 489 - 539.

[AenBleidd]

1 - This is easy to achieve
2 - This means we should implement som parsing mechanism to retrieve this information from the ca-bundle.crt file. Also, if some of the certificates is expired, that doesn't mean that it will affect any Project since we don't know which certificates are used by any Project.
3 - BOINC client might have no permissions to write to installation directory, so automatic update usually not possible.

[RichardHaselgrove]

I'm one of those users who still runs Windows 7 on several machines. As I noted yesterday, the 'Without SSL' build is running successfully - but I'll do more extensive tests on the projects known to be having connection problems currently. I'll also check the system certificates on a sample machine, to see (a) when they were last updated, and (b) when the next expiry is due. But at least it should be free of the OpenSSL v1.0 bug, where a bad certificate blocks a perfectly good one being chosen.

3 - BOINC client might have no permissions to write ...

But a mini-installer could be built to request an admin rights upgrade, and could work.

[CharlieFenton]

2 - This means we should implement som parsing mechanism to retrieve this information from the ca-bundle.crt file.

I don't understand why, other than to determine when a certificate expires. We would simply download and replace the entire ca-bundle.crt file in the BOINC Data directory (not the install directory.) As I understand it, we get the entire certificate bundle as a unit from Mozilla. the file says:
This is a bundle of X.509 certificates of public Certificate Authorities (CA). These were automatically extracted from Mozilla's root certificates file (certdata.txt). This file can be found in the mozilla source tree: https://hg.mozilla.org/releases/mozilla-release/raw-file/default/security/nss/lib/ckfw/builtins/certdata.txt

But you raise an interesting point. Examining the contents of the ca-bundle.crt file, I don't know how we can determine the expiration date for each separate certificate in the bundle. There is information on using OpenSSL to determine whether a certificate in PEM format will expire within a certain number of days here and for determining the expiration date of ca-bundle.crt here. When I use the latter on my Mac I get this:
`$ openssl x509 -enddate -noout -in /Library/Application\ Support/BOINC\ Data/ca-bundle.crt

notAfter=Jan 28 12:00:00 2028 GMT`
Does that date look correct to you?

Also, if some of the certificates is expired, that doesn't mean that it will affect any Project since we don't know which certificates are used by any Project.

Do the project servers not use the same certificate bundle as the client?

3 - BOINC client might have no permissions to write to installation directory, so automatic update usually not possible.

ca-bundle.crt is in the BOINC Data directory, not the installation directory. Almost everything in the BOINC Data directory is written by the BOINC client.

[RichardHaselgrove]

Useful links at

https://www.michev.info/Blog/Post/1435/windows-certificate-stores
https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.security/about/about_certificate_provider?view=powershell-7

Probably too much information, but the second does include a PowerShell example for finding certificates due to expire in the near future.

[AenBleidd]

ca-bundle.crt is in the BOINC Data directory, not the installation directory.

@RichardHaselgrove, you're not right:

Examining the contents of the ca-bundle.crt file

I'm not sure we even need it.

I'm checking whether this solution still works on Windows Vista (that was definitely not updated last 5 years for sure), and if yes - I believe we're good to go.

[RichardHaselgrove]

But you raise an interesting point. Examining the contents of the ca-bundle.crt file, I don't know how we can determine the expiration date for each separate certificate in the bundle.

I've developed a way (manual, tedious) of doing that. But it could be automated.

The ca-bundle file is simply a concatenation of the individual certificates, in plain text format. I have a small code snippet which splits the bundle into individual single-certificate files: any certificate handler - such as Windows built-in tools - can process each single file and find the expiry date.

[CharlieFenton]

you're not right

I think you meant that for me, not @RichardHaselgrove. On the Mac, it is in the BOINC Data directory (and apparently also on Linux.) I don't know why it is in the install directory on MS Windows, but it could easily be put in the BOINC Data directory instead. It would only require a simple change to client/http_curl.sh to use the code currently in lines 533-538 for MS Windows as well as the other platforms (MacOS and Linux.)

[AenBleidd]

I think you meant that for me
@CharlieFenton, yes, sorry, I tagged a wrong person.
Yes, this could be a solution. I need to think a little bit about it.
Thanks for the ideas!

[CharlieFenton]

Examining the contents of the ca-bundle.crt file

I'm not sure we even need it.
I'm checking whether this solution still works on Windows Vista (that was definitely not updated last 5 years for sure), and if yes - I believe we're good to go.

But unfortunately only until the next time. This is not the first time we've had to scramble to fix an expired ca-bundle.crt. I'm hoping we can come up with a more permanent solution so we hopefully never need to worry about this again.

[RichardHaselgrove]

I think the Linux location is dependant on the distro package maintainers.

On my Ubuntu-derivative machine, ca-bundle.crt appears in the data directory, but it's actually a link to

/etc/ssl/certs/ca-certificates.crt

[CharlieFenton]

@RichardHaselgrove I wonder if /etc/ssl/certs/ca-certificates.crt is kept current by the OS. If so, then we may not need to worry about this issue for those systems. On the other hand, if only some Linux distros use that and others use the ca-bundle.crt we provide, that complicates things a bit. The BOINC client would need to determine whether or not it is using a BOINC-provided certificate bundle and act accordingly.

[CharlieFenton]

Of course, if the Linux OS is keeping the certificate up to date, then the BOINC client code would never detect one about to expire so perhaps BOINC would not need to determine whether or not it is using a BOINC-provided certificate bundle.

[RichardHaselgrove]

The ca-certificate.crt file is dated Sat 06 Feb 2021 09:54:15 GMT - which is about when I updated this machine with a clean install of Linux Mint 20.2 - so that tells us nothing.

But my system maintenance utility says that there's a security update waiting to be installed, which starts with

ca-certificates (20210119~20.04.2) focal-security; urgency=medium

[ Dimitri John Ledkov ]

• mozilla/blacklist.txt: blacklist expired "DST Root CA X3".
  (LP: #1944481)

Installing it changes the datestamp to Thu 07 Oct 2021 10:42:18 BST - so it is a system responsibility to keep it updated, not a BOINC responsibility.

I would assume it's also a system responsibility to keep the SSL program itself free of those nasty OpenSSL v1.0 bugs. Though all I can see in Synaptic is

ssl-cert
simple debconf wrapper for OpenSSL

[CharlieFenton]

In PR #4545, @Ageless93 wrote:

By the way, I also found https://boinc.berkeley.edu/trac/wiki/AdminReleaseManagement which stipulates:

Check for a new CA bundle from ​https://curl.haxx.se/docs/caextract.html; copy to curl/ca-bundle.crt

That link shows some potentially useful ways of automatically updating ca-bundle.crt.
curl --remote-name --time-cond cacert.pem https://curl.se/ca/cacert.pem downloads a file cacert.pem which upon a quick inspection appears identical in format and content to ca-bundle.crt. Also potentially useful is the mk-ca-bundle utility described here as:

mk-ca-bundle - convert mozilla's certdata.txt to PEM format

and

mk-ca-bundle is a command line tool that is shipped as part of every curl and libcurl release

However, we may not want to have every BOINC client download the certificates from curl.se because the web page implies their servers could get overloaded:

We don't mind you downloading the PEM file from us in an automated fashion, but please don't do it more often than once per day. It is only updated once every few months anyway.