Migration to Bazel

@regisd is attempting to migrate the build of JFlex from Maven to Bazel.

Motivation

Disclaimer very opinionated, you are free to think otherwise.

JFlex is currently built with the Maven build system. Maven had good promises (central repository of jars, dependency management, conventional build steps, well defined build, etc.). And the central repository has been great value to the whole Java ecosystem.

However, Maven isn't satisfactory in many other ways.

Maven didn't deliver its promises

However, Maven failed short in many areas:

• As long as the project is simple, everything goes well. But if when a project can't follow the convention, it's a long fight with the configuration of the plugins.
• The promise of having a reproducible build is a lie. The Maven plugins become incompatible with every upgrade of Java or Maven.
• The XML based configuration is extremely verbose.
• The documentation was extremely lacking or outdated.
• Incremental builds are inexistant or broken

Running the testsuite is slow

If we ask to run the test suite, Maven will run all tests are executed again, even if the code is unchanged. And the testsuite takes time — around 10 minutes on Travis.

Bazel offers faster build and testing

Bazel which was open-sourced by Google in 2015, after being used internally at scale for years.

Bazel manages the graph dependency very well. If a test in added, or the documentation is modified, the other tests run in a couple seconds because their execution is cached. That's why @regisd is attempting to migrate the everything to Bazel.

Migration Phases

1. Maven is the main build system. Migration in progress.
   • Add Bazel BUILD on JFlex modules, one by one. Status ✔ JFlex can be compiled with bazel.
   • Build Unicode properties with Maven. Status [WIP]
   • Migrate test suite to Bazel Status [WIP]
2. Bazel main build system.
   • Bazel is the main build system used by developers. Maven is kept as a backup.
   • People who want to simply use jflex, can download the targz distribution. The targz doesn't contain java sources.
   • People who want to build clone the git repo and use bazel.
3. Bazel unique build system.
   • Only the Maveb plugin is built with Maven.
   • POM is removed.
   • testcases are removed
   • maven-uniocde-plugin is removed
4. Optional: reorganise directory structure.

Tasks

See project Bazel

Continuous integration

• Initial attempt of using Travis PR #395
• Currently using Cirrus

Bazel Rule

difficulty: very easy

Status ✔ PR #401

See bazel_rules

jflex-maven-plugin

difficulty: very hard

status: not started, probably the only piece that will stay as a Maven artifact.

Ideally, we need to infer the deps to generate a pom.xml

Examples

difficulty: easy

Status ✔ on "simple". PR #401

CUP

Replace cup-maven-plugin by Bazel rule

difficulty: Very easy

Status ✔ PR #442

Jflex itself

difficulty: easy (once cup rule and jflex rule are done)

Status ✔ PR #444

testsuite

difficulty: hard status: WIP (More below)

• custom test runner
• custom format to define test cases (.test)
• a lot of convention on the input name, the golden file name, etc.
• handling generation failure or compilation failures as expected

jflex-testsuite-maven-plugin

status: obsolete, won't do

uberjar

status: native behavior of Bazel for java_bin.

Target directory structure

Source code, phases 2-3

Anyone who wants to build from source clones the git repo and uses Maven.

The source code directory structure looks like this:

├── bazel-*                           [REMOVED for targz]
├── cup-maven-plugin                  [OBSOLETE, removed]
├── bin                               [MOVED from jflex/bin]
├── docs                              [OK]
├── examples                          [MOVED from jflex/examples]
├── jflex                             [OK]
│   ├── common-testing                [REMOVED for targz]   
│   └── src                           [OK]
├── jflex-maven-plugin                [REMOVED for targz]
├── jflex-unicode-maven-plugin        [REMOVED for targz]
├── report-module                     [OBSOLETE, removed]
├── scripts                           [REMOVED for targz]
├── src                               [OBSOLETE, removed]
├── testsuite                         [REMOVED for targz]
│   ├── bzltestsuite                  [NEW, replaces jflex-testsuite-maven-plugin]
│   ├── jflex-testsuite-maven-plugin  [OBSOLETE, removed]
│   └── testcases
├── lib                               [UNCHANGED]
├── third_party                       [NEW]
└── tools                             [REMOVED for targz]

targz distribution

As a result, the targz has the following strucuture:

Distribution targz

├── bin
│   ├── jflex.bat
│   └── jflex.sh
├── docs                              [COPIED from bazel-bin]
│   ├── manual.html
│   └── manual.pdf
├── examples
├── jflex
│   └── src                           [OK]
├── lib                               [UNCHANGED; add jars copied from bazel-bin]
├── third_party 
│   ├── cup
│   ├── com
│   └── org                           [etc.]
└── README.md LICENSE etc.

Note: for the distribution to avoid the bootstrap question, we can also copy relevant files from bazel-genfiles:

bazel-genfiles
├── examples
│   └── simple
│       └── Yylex.java
└── jflex
    ├── LexParse.java
    ├── LexScan.java
    └── sym.java

Sources structure, phase 4 (optional)

This is the flattest structure that works well with Maven and makes releasing the targz easy.

├── bin
│   └── jflex.bat
├── docs
│   ├── grammar.md
│   ├── intro.md
│   └── template.tex
├── java
│   └── jflex // consider splitting de.flex.* for tools and jflex.* for core generator
│       ├── LexGenerator.java
│       ├── gui
│       ├── lexparse.flex
│       ├── lexscan.cup
│       ├── maven
│       │   └── JflexMojo.java
│       ├── testing
│       │   └── testsuite
│       │       └── TestRunner.java
│       └── unicode
│           └── UnicodeProperies.java
├── javatests
│   └── jflex
│       └── testsuite
│           └── apiprivate
│               ├── ApiPrivateTest.java
│               └── private.lex
├── scripts
│   ├── release.sh
│   └── travis.sh
└── third_party
    ├── com
    │   └── google
    │       └── guava
    └── java
        └── java_cup
            ├── java_cup-11b.jar
            └── runtime
                ├── ComplexSymbolFactory.java
                └── lr_parser.java