Migrate to CMake buildsystem

[dnakamura]

Long running issue to track progress on migrating openj9 from UMA to CMake/

• Platform Support:

  • xLinux
  • pLinux

Timeline

Nov 23

• Integrate VMCPTool and NLSTool
  • CMake: Add support for NLS and CP tools #3675

Dec 17

• JDK 11 Extension changes
  • Remaining changes are ready, just need to open PRs. All fairly trivial, should be merged quickly
  • class library natives
  • redirector
  • CMake: Build libffi as a static library #3659
  • CMake: Add support for NLS and CP tools #3675
  • CMake: Don't append version suffix for jextractnatives #4041
  • CMake: add JDK11 redirector #4042
  • CMake: add j9vm_jdk11 #4043

Dec 21

• osx
  • Use NASM assembler on x86-64 Linux #3293
  • Update CMake code to handle NASM eclipse-omr/omr#3124

March 1

• DDR support for non-windows platforms
  • DDR: fix ddrgen bugs involving empty structs eclipse-omr/omr#3209
  • CMake: Add initial support for ddrgen eclipse-omr/omr#3308

March 1

• stop staging the source files
  • Rework extensible enums eclipse-omr/omr#3189
  • Rename extensible header files in the the compiler eclipse-omr/omr#3213
  • A number of compiler sources assume that omr is a subdirectory of openj9 #3725
  • Use renamed headers in the compiler #4928
  • Stop prepending includes with "omr/" where unnecessary #3726
  • CMake: Properly add nlsgen dependencies to omrutil #5341
  • Stop staging sources for the cmake build ibmruntimes/openj9-openjdk-jdk11#68

Milestone: First pass of linux x86-64 cmake build modulo any bugs

March 20

• Windows support (un-optimized bytecode interpreter)
  • CMake: Search for windows message compiler eclipse-omr/omr#3677
  • CMake: Add OMR_EXE_LAUNCHER variable eclipse-omr/omr#3669
  • Fix use after free bug in Port::omrfile_stat eclipse-omr/omr#3670
  • CMake: Pass VERBATIM option to add_custom_command #5227
  • CMake: Add windows support for m4 generated assembly #5228
  • CMake: specify RUNTIME DESTINATION when installing shared libraries #5240
  • CMake: Enable ASM_MASM on windows #5267
  • CMake: rework platform handling in port library #5268
  • CMake: Update libffi to support windows #5269
  • CMake: add windows support to j9util #5270
  • CMake: Add windows support to the vm #5271
  • CMake: On windows link j9realtime against winmm.lib #5280
  • CMake: Add export info eclipse-omr/omr#3705
  • CMake: add export symbols to shared libraries #5308

March 22

• Optimized bytecode interpreter on windows

Milestone: First pass of win x86-64 cmake build modulo any bugs

April 12

• AIX Support
• All supported platforms work on cmake
• Turn off UMA

Validation

• TODO: how to validate the cmake build is equivalent to the autotools build?
• Test equivilent build flags between uma and cmake builds
• Testing
• Perf

Misc TODOS

• Copyright info
• Documentation describing how the cmake build is structured

[DanHeidinga]

Thanks @dnakamura. Can you outline the plan (including the great details in PR #78) for how we'll transition to cmake?

[dnakamura]

Yeah working on fleshing out some details now

[jbduncan]

I'd completely understand if the decision to use CMake is set in stone now, but I wondered if Bazel was at all considered as an option? By my understanding, it is made specifically for building and testing large code bases quickly, which may be beneficial for OpenJ9.

[dnakamura]

@jbduncan I don't recall all the details but I do remember looking into Bazel somewhat. IIRC it didn't really get us anything that CMake can't do. Also it has some drawbacks compared with CMake, namely lack of platform support (I don't think it can handle AIX or z/OS)

[jbduncan]

@dnakamura Okay, that's totally fair! Many thanks for explaining a bit of the history behind the decision to use CMake for me. :)

[DanHeidinga]

This won't complete in 0.12.0. Moving to 0.14.0. @dnakamura Can you update the timeline for when this will finish?

[dnakamura]

DDR has proven to be a bigger issue than anticipated. Have WIP pr open at omr. Plan to get finishing touches done today and move on (although I suspect the PR will spend considerable time in review)

[dnakamura]

DDR PRs pending review, as is one or two changes regarding staging the source files.
Only one other changre required in the jdk extensions repo, but leaving that for now as I am on vacation this week, and the chances of a build breakage is high

[dnakamura]

Things running behind due to some old PRs that fell on the back burner and got forgotten about, as well as having to deal with some unexpected flooding at my house. WIll work through the weekend to see if we can get things back on track

[DanHeidinga]

some unexpected flooding at my house

Good luck with the cleanup! And hopefully the rising temps don't lead to more.

[fjeremic]

FYI a bunch of z/OS CMake issues are fixed in [1]. I think we'll need all of them for OpenJ9 as well.

[1] eclipse-omr/omr#3693

[dnakamura]

Status update:
Just polishing off a large PR which significantly refactors the way cmake handles ddr, will open the PR before end of day. (largely a matter of isolating the changes from my current dev tree).

My current monolthic working tree can now successfully run a full make all on the openjdk extensions as well as test matterial /wconfigure --enable-ddr

Looking at making sanity tests pass in order to run cmake builds/testing as part of nightly builds. Out of sanity test set, there are currently 20 failing tests. Will look at resolving once DDR related prs are open

FAILED test targets:
        cmdLineTester_jvmtitests_5
        cmdLineTester_jvmtitests_7
        cmdLineTester_jvmtitests_8
        cmdLineTester_jvmtitests_hcr_7
        cmdLineTester_jvmtitests_hcr_8
        cmdLineTester_jvmtitests_hcr_10
        cmdLineTester_jvmtitests_debug_3
        cmdLineTester_jvmtitests_debug_5
        cmdLineTester_jvmtitests_debug_7
        cmdLineTester_jvmtitests_debug_8
        cmdLineTester_jvmtitests_debug_9
        cmdLineTester_jvmtitests_debug_11
        cmdLineTester_decompilationTests_1
        cmdLineTester_CryptoTest_0
        cmdLineTester_callsitedbgddrext_openj9_0
        J9vmTest_2
        J9vmTest_4
        J9vmTest_5
        SharedCPEntryInvokerTests_0
        SharedCPEntryInvokerTests_1

[fjeremic]

@dnakamura FYI I just completed full migration of OMR configure/build/test to CMake in eclipse-omr/omr#3693. I have DDR disabled at the moment in the pipeline file (search of -DOMR_DDR=0). I hope my changes play nice with yours!

[DanHeidinga]

Discussed with @dnakamura:

• We are a week to a week+half away from a completed xlinux jdk11 cmake build.
• Timeline: 6 weeks after xlinux for all of JDK 11 to be complete