Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Entire set of Boost Libraries (Work in Progress) #1280

Closed
wants to merge 29 commits into from

Conversation

Attempt3035
Copy link
Contributor

@Attempt3035 Attempt3035 commented Jan 7, 2024

Hi all! I'm working on adding all the boost libraries, still got a bit to do before my mess will be useful to anyone else to help with, but had a few people asking so wanted to get this up here so others can start to look through. No action required right now, will update as I progress!

Note:
Check out boost.rules.tools which is where all the common utilities used by all the modules is, plus some super handy maintenance tools. Probably not super helpful until I finish the readme though, sorry!

Credit to https://github.com/nelhage/rules_boost and https://github.com/Vertexwahn/rules_boost/tree/add-bzlmod-support which both provided inspiration and help with parts of the code.

Fixes #292 and Fixes #787

Initial copy across from previous workings to test with entire central registry.
Currently working on adding boost tests for every module
….assert//:assert". Changes were in boost.rules.tools but needed extra changes here
… main targets, as it wasn't included and causes build failures for boost as boost libraries aren't expecting to prefix the headers with "lib/"
@Attempt3035
Copy link
Contributor Author

Hey just wondering what the best way to point source urls to git tags is? I've done for example "https://github.com/boostorg/safe_numerics/archive/refs/tags/boost-1.83.0.tar.gz", but looks like that errors out as an unstable url. Looking at how other modules are done, it uses urls like "https://github.com/aspect-build/bazel-lib/releases/download/v1.34.2/bazel-lib-v1.34.2.tar.gz". I don't think we can get these for the boost libraries though, as they have tags, but not releases?

@fmeum
Copy link
Contributor

fmeum commented Jan 8, 2024

Hey just wondering what the best way to point source urls to git tags is? I've done for example "https://github.com/boostorg/safe_numerics/archive/refs/tags/boost-1.83.0.tar.gz", but looks like that errors out as an unstable url. Looking at how other modules are done, it uses urls like "https://github.com/aspect-build/bazel-lib/releases/download/v1.34.2/bazel-lib-v1.34.2.tar.gz". I don't think we can get these for the boost libraries though, as they have tags, but not releases?

There isn't, unfortunately GitHub doesn't provide a way to get a stable source archive without manual maintainer effort. This is a widespread issue and we will have to find a better solution to this, but for now all we can do is apply a, unfortunately very common, exception (see the label I added).

@fmeum fmeum added presubmit-auto-run Presubmit jobs will be triggered for new changes automatically without reviewer's approval skip-url-stability-check Skip the URL stability check for the PR labels Jan 8, 2024
@Attempt3035
Copy link
Contributor Author

CI/CD pipeline is happy to build now! Looks like we've got 13 modules not building, that's what I was getting in my local testing too. They are the ones I haven't gotten around to yet, mostly the tricky ones like beast and log. I'll hopefully sort out the tests so the ones that are building have matching test suites and can be verified working, then will get onto the last few that aren't at build stage yet.

@Attempt3035
Copy link
Contributor Author

Been working on boost.rules.tools and boost.rules to solve a few requests and make it super easy to contribute to maintaining or working on these modules. For anyone wanting to help out with this PR, check it out!

See This Thread for some discussion about the benefits of this system.

Added xz
Deleted bazel_test folders and tests from rules_boost - not needed as we are using native boost tests
Added build files to all test folders
Updated presubmit.yml to run tests
@Attempt3035
Copy link
Contributor Author

Attempt3035 commented Jan 21, 2024

Task List

Fix Compiling issues

  • boost.test
  • boost.align
  • boost.log
  • boost.property_map.parallel
  • boost.mpi
  • boost.beast
  • boost.wave
  • boost.iostreams
  • boost.numeric.odeint
  • boost.graph.parallel
  • boost.locale
  • boost.python
  • boost.parameter_python
  • boost.local_function
  • The rest are compiling!!

Potentially needs work (reference their cmakelists.txt and compare to what BUILD.bazel file is doing)

  • Add asio SSL!
  • atomic
  • filesystem
  • gil (+ extra optional deps)
  • hof
  • math
  • callable_traits
  • chrono
  • compute
  • hana
  • iostreams
  • json
  • nowide
  • stacktrace
  • url

Random TODO:

  • Make WINAPI boost only visibility
  • make config_settings boost only visibility (same package list as winapi)
  • iostreams: Other external deps
  • Work out what flags to provide based on compiler abilities. See results of boost.config test to see what works or doesn't in your compiler. How do we provide the right flags per compiler? Maybe this should be run automatically as a macro/dep of every target??
  • (boost.accumulators) p_square_quantile_extended test should use extended_p_square_quantile.hpp ??
  • Remove default visibilities - they are done in boost_library
  • Enforce cxx minimum versions on particular libs. (Should we enforce a higher like 20 when extra features are available with that version?) As Bazel 17 defaults to c++17, (the highest minimum required by any boost lib) we won't enforce any versions. The user can choose to build with higher (eg c++20) if they desire.
  • boost.geometry --cxxopt='-std=c++14'
  • boost.hana --cxxopt='-std=c++14'
  • boost.stl_interfaces --cxxopt='-std=c++14'
  • boost.histogram --cxxopt='-std=c++14'
  • boost.pfr --cxxopt='-std=c++17’
  • boost.spirit --cxxopt='-std=c++17’

Tests Passing (hence fully operational and ready to be used)

  • bazel test @boost.accumulators//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.algorithm//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.align//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.any//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.array//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.asio//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.assert//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.assign//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.atomic//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.beast//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.bimap//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.bind//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.callable_traits//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.chrono//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.circular_buffer//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.compute//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.concept_check//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.config//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.container//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.container_hash//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.context//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.contract//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.conversion//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.convert//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.core//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.coroutine//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.coroutine2//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.crc//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.date_time//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.describe//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.detail//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.dll//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.dynamic_bitset//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.endian//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.exception//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.fiber//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.filesystem//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.flyweight//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.foreach//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.format//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.function//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.function_types//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.functional//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.fusion//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.geometry//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.gil//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.graph//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.graph.parallel//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.hana//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.heap//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.histogram//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.hof//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.icl//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.integer//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.interprocess//test:tests --test_verbose_timeout_warnings -k
    --------------- Negative tests and test deps not added yet (hence will likely fail on test) below this line ---------------
  • bazel test @boost.intrusive//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.io//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.iostreams//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.iterator//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.json//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.lambda//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.lambda2//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.leaf//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.lexical_cast//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.local_function//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.locale//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.lockfree//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.log//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.logic//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.math//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.metaparse//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.move//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.mp11//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.mpi//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.mpl//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.msm//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.multi_array//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.multi_index//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.multiprecision//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.nowide//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.numeric.conversion//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.numeric.interval//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.numeric.odeint//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.numeric.ublas//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.optional//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.outcome//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.parameter//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.parameter_python//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.pfr//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.phoenix//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.poly_collection//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.polygon//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.pool//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.predef//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.preprocessor//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.process//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.program_options//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.property_map//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.property_map.parallel//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.property_tree//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.proto//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.ptr_container//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.python//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.qvm//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.random//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.range//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.ratio//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.rational//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.regex//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.safe_numerics//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.scope_exit//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.serialization//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.signals2//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.smart_ptr//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.sort//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.spirit//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.stacktrace//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.statechart//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.static_assert//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.static_string//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.stl_interfaces//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.system//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.test//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.thread//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.throw_exception//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.timer//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.tokenizer//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.tti//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.tuple//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.type_erasure//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.type_index//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.type_traits//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.typeof//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.units//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.unordered//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.url//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.utility//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.uuid//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.variant//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.variant2//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.vmd//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.wave//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.winapi//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.xpressive//test:tests --test_verbose_timeout_warnings -k
  • bazel test @boost.yap//test:tests --test_verbose_timeout_warnings -k

Tests Not Needed:

  • boost.compatibility

Test Notes

  • Test results here are on my local machine (Apple Arm M2 pro, Ventura). Some may fail in CI/CD when being built cross platform, we will have to re-check after the tests are run in the pipeline.
  • A lot of the tests are failing due to boost.test not linking properly. That needs some work in the BUILD.bazel file.

Testing Process

I've been iteratively doing the cycle of:

  • Run test for a module
  • Fix issues
  • Update patches using interactive script in boost.rules.tools
  • bazel clean --expunge
  • (Repeat)

I've also been doing this with a .bazelversion file with 6.4.0 as bazel 7.0.1 has not been clearing caches for me properly. I literally have not found a way (even manually deleting cache files) for it to actually realise the hash and module has been updated, even though the module version from the local repository is the same. Weird and highly annoying, I would've though bazel clean --expunge would get rid of everything, but apparently not with my config🤷‍♀️
Lockfiles were definitely doing their job properly! Note to self - use --lockfile_mode=off when developing modules!

…t building due to unrelated modules in error
Pivoted test system to auto-find all tests and assume they are a positive test unless in negative_test_names section. Reduces the need for a ton of lines specifying test files
@fmeum
Copy link
Contributor

fmeum commented Jan 21, 2024

You need to set --lockfile_mode=off in addition to using bazel clean --expunge if you use a local registry.

Since Bazel 7 raised the C++ default to C++17, setting flags on individual libraries may not be necessary anymore. Fixed flags could lead to differing C++ standards across the build.

@lalten lalten mentioned this pull request Jan 25, 2024
@Vertexwahn
Copy link
Contributor

Vertexwahn commented Feb 4, 2024

Since Bazel 7 raised the C++ default to C++17, setting flags on individual libraries may not be necessary anymore. Fixed flags could lead to differing C++ standards across the build.

@fmeum Just for the records: If I use Bazel 7.0.2 the default C++ standard is C++14. If I use 8.0.0-pre.20240108.6 the default C++ Standard is C++17 (tested on Ubuntu 22.04, gcc)

@bazel-io
Copy link
Member

bazel-io commented Mar 4, 2024

Hello @Vertexwahn, modules you maintain (xz) have been updated in this PR. Please review the changes.

@bazel-io
Copy link
Member

bazel-io commented Mar 4, 2024

Hello @bazelbuild/bcr-maintainers, modules without existing maintainers (boost.accumulators, boost.algorithm, boost.align, boost.any, boost.array, boost.asio, boost.assert, boost.assign, boost.atomic, boost.beast, boost.bimap, boost.bind, boost.callable_traits, boost.chrono, boost.circular_buffer, boost.compatibility, boost.compute, boost.concept_check, boost.config, boost.container, boost.container_hash, boost.context, boost.contract, boost.conversion, boost.convert, boost.core, boost.coroutine, boost.coroutine2, boost.crc, boost.date_time, boost.describe, boost.detail, boost.dll, boost.dynamic_bitset, boost.endian, boost.exception, boost.fiber, boost.filesystem, boost.flyweight, boost.foreach, boost.format, boost.function, boost.function_types, boost.functional, boost.fusion, boost.geometry, boost.gil, boost.graph.parallel, boost.graph, boost.hana, boost.heap, boost.histogram, boost.hof, boost.icl, boost.integer, boost.interprocess, boost.intrusive, boost.io, boost.iostreams, boost.iterator, boost.json, boost.lambda, boost.lambda2, boost.leaf, boost.lexical_cast, boost.local_function, boost.locale, boost.lockfree, boost.log, boost.logic, boost.math, boost.metaparse, boost.move, boost.mp11, boost.mpi, boost.mpl, boost.msm, boost.multi_array, boost.multi_index, boost.multiprecision, boost.nowide, boost.numeric.conversion, boost.numeric.interval, boost.numeric.odeint, boost.numeric.ublas, boost.optional, boost.outcome, boost.parameter, boost.parameter_python, boost.pfr, boost.phoenix, boost.poly_collection, boost.polygon, boost.pool, boost.predef, boost.preprocessor, boost.process, boost.program_options, boost.property_map.parallel, boost.property_map, boost.property_tree, boost.proto, boost.ptr_container, boost.python, boost.qvm, boost.random, boost.range, boost.ratio, boost.rational, boost.regex, boost.rules.tools, boost.safe_numerics, boost.scope_exit, boost.serialization, boost.signals2, boost.smart_ptr, boost.sort, boost.spirit, boost.stacktrace, boost.statechart, boost.static_assert, boost.static_string, boost.stl_interfaces, boost.system, boost.test, boost.thread, boost.throw_exception, boost.timer, boost.tokenizer, boost.tti, boost.tuple, boost.type_erasure, boost.type_index, boost.type_traits, boost.typeof, boost.units, boost.unordered, boost.url, boost.utility, boost.uuid, boost.variant, boost.variant2, boost.vmd, boost.wave, boost.winapi, boost.xpressive, boost.yap, boost, zstd) have been updated in this PR. Please review the changes.

@Attempt3035
Copy link
Contributor Author

Oh oops. Sorry @Vertexwahn, ignore this plz

@Attempt3035
Copy link
Contributor Author

Attempt3035 commented Mar 4, 2024

Closing this PR for the moment due to reorganising branches on my repo.

CONTINUED AT #1577

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
presubmit-auto-run Presubmit jobs will be triggered for new changes automatically without reviewer's approval skip-url-stability-check Skip the URL stability check for the PR
Projects
None yet
Development

Successfully merging this pull request may close these issues.

wanted: boostorg/boost wanted: nelhage/rules_boost
4 participants