Core Add boost

[roigcarlo]

📝 Description
Due to constant failure when downloading boost, we decided to try to add it into the core.

This PR attempts to add the minimal subset of libraries used, from version 1.80. Current weight is 27MB.

This does not mean you can't use your own boost. It will mean that there is a default one for example if you want to compile with support with vexcl, which needs a broader subset of headers that falls outside the scope of this PR.

I mark this as a draft as I will be making additions when I go through the app, but I need CI feedback on systems I cannot test locally.

This also should help to improve the creation of dockers that were depending on boost when image was created/updated.

I will upload the script and instructions to generate/update the headers once its working.

🆕 Changelog

• Adding boost as part of the core

[roigcarlo]

Ping @KratosMultiphysics/altair as I don't have access to your sources, chances are that I will miss something at the end.

I use:
bin.v2/tools/bcp/gcc-11/release/link-static/bcp --scan **** extractdir

To extract the list from scanning all files in our repo, if you get a different subset pleas tell me or just add the missing files directly (but wait until I parse all our apps, currently only the core deps are auploaded)

[loumalouomega]

I don't like this...

[roigcarlo]

I am open to other solutions

[roigcarlo]

To give some more context:

The problem is that boost fails to download on a regular basis. This impacts the stability of the CI.

For linux it does not suppose much problem as we can build images with boost already on them, and use them as the building images of the CI, but for windows this is not possible to use those custom images (which already exists), so we are forced to download the libs.

This was not a problem in the past as boost was preinstalled but since actions/runner-images#2667 that's no longer the case. As you can see, official recommended solution is what we have in place right now, but that is causing problems.

As other alternatives:

• Host boost ourselves: We have full control. I doubt we have the resources to serve all builds we launch.
• Download directly from boost git repo: Should be always available and use the same actions as kratos Headers are separated into subrepos and we will have to reconstruct the "release" dir struct every compilation
• Upload the libs ourselves: Integrated, Consumes a lot of space, we will have to update them regularly.
• Launch our docker container inside GHA container: I have not idea on the implications of this (or if will even work)

I am not specially keen on this solution either, but honestly I don't know how to pull off the others or come up with another solution :/

[matekelemen]

Any links/leads on why the download fails? I imagine other projects relying on boost have the same issue.

[RiccardoRossi]

to give my 2 cents i think that extracting from boost what we need and providing it together with the kratos is a good thing.

we ultimoately want to reduce our dependency on boost, and doing it this way we can do it quite effectively. Also we could ensure that no one is adding unnoticed new dependencies on a given Pr

also this way we can easily giarantee that q minimum version is ised and is employed by everyone

also th

[roigcarlo]

Any links/leads on why the download fails? I imagine other projects relying on boost have the same issue.

I suspect Sourceforge may have some restrictions on the number of download per hour. Also is a commercial project, and a good chunk of they revenue is via ads, so I would be not surprised if they are not putting all the resources in the world to a services that basically skips ads entirely.

If you look at the issue i linked, many projects are having the problem. Is not just us.

I've been reviewing all Kratos code, and I suspect we are including a lot of headers that are either outdated, not used at all, or swapable by std features. Let me create a draft removing most of there to see if we can further reduce the size.

[roigcarlo]

@ddemidov Vexcl makes extensive use of boost, have you encountered any problem in windows builds lately? (or have you any hint or projects having the same issues?)

[ddemidov]

I haven't touched vexcl for some time; for linux tests it is still not converted from travis-ci.org. When it used to work there, it used caching mechansm for boost (travis-ci allowed to put downloads into cache):
https://github.com/ddemidov/vexcl/blob/8e7312a26865d2cebdbaa1a4caeb8494a1796585/.travis.yml#L53-L55

For windows builds it used appveyour, which provided build images with boost out of the box:
https://github.com/ddemidov/vexcl/blob/8e7312a26865d2cebdbaa1a4caeb8494a1796585/.appveyor.yml#L5

May be you could have a look at the code of some boost libraries on github, as each library uses a separate repo and also has to download the whole of boost for CI tests.

[oberbichler]

Download directly from boost git repo: Should be always available and use the same actions as kratos Headers are separated into subrepos and we will have to reconstruct the "release" dir struct every compilation

Could be done by iterating over the subdirs in CMake.

but for windows this is not possible

I have always used the packages from Anaconda to build Kratos (Boost, MKL, HDF5, etc.). I know you guys are not conda fans but if it is just about installing packages for the CI this might be a solution.

[roigcarlo]

Could be done by iterating over the subdirs in CMake.

mm I am afraid is not so easy, they do some preprocessing and stuff to generate the headers. I don't fully comprehend the process yet but is surely something I am looking into

I have always used the packages from Anaconda to build Kratos

We could use that, but then we also have to download conda :/

In the meantime o made #10154. Seems to be more reliable. Let's see...

[matekelemen]

In the meantime o made #10154. Seems to be more reliable. Let's see...

Oh I was under the impression that we've already been using the artifactory link because that's in the ci.yml. Are we running both configs? Does this one fail occasionally too?

[roigcarlo]

We changed it in #10051 but I realized it was not changed on nightlies. Let's see if this solves the problem

[roigcarlo]

I will close for now and reopen if the new mirror fails. In the mean time I will try to clean up as much as possible boost usages.

[oberbichler]

mm I am afraid is not so easy, they do some preprocessing and stuff to generate the headers. I don't fully comprehend the process yet but is surely something I am looking into

At least it was so simple some time ago, anyway....

We could use that, but then we also have to download conda :/

It is preinstalled. Perhaps a(nother) package manager would suit Kratos better, too.

[roigcarlo]

Oh nice, did not know. If it keeps failing I will try that next 👍

Perhaps a(nother) package manager would suit Kratos better, too.

Not in this topic but we are considering conda to distribute the intel version with MKL an TBB, if we finally go for it we will most likely extend to normal builds as well