Migration to Bazel

@regisd is attempting to migrate the build of JFlex from Maven to Bazel.

Motivation

Disclaimer very opinionated, you are free to think otherwise.

JFlex is currently built with the Maven build system. Maven had good promises (central repository of jars, dependency management, conventional build steps, well defined build, etc.). And the central repository has been great value to the whole Java ecosystem.

However, Maven isn't satisfactory in many other ways.

Maven didn't deliver its promises

However, Maven failed short in many areas:

As long as the project is simple, everything goes well. But if when a project can't follow the convention, it's a long fight with the configuration of the plugins.
The promise of having a reproducible build is a lie. The Maven plugins become incompatible with every upgrade of Java or Maven.
The XML based configuration is extremely verbose.
The documentation was extremely lacking or outdated.
Incremental builds are inexistant or broken

Running the testsuite is slow

If we ask to run the test suite, Maven will run all tests are executed again, even if the code is unchanged. And the testsuite takes time — around 10 minutes on Travis.

Bazel offers faster build and testing

Bazel which was open-sourced by Google in 2015, after being used internally at scale for years.

Bazel manages the graph dependency very well. If a test in added, or the documentation is modified, the other tests run in a couple seconds because their execution is cached. That's why @regisd is attempting to migrate the everything to Bazel.

Migration Phases

Maven is the main build system. Migration in progress.
- Add Bazel BUILD on JFlex modules, one by one. Status ✔ JFlex can be compiled with bazel.
- Build Unicode properties with Maven. Status [WIP]
- Migrate test suite to Bazel Status [WIP]
Bazel main build system.
- Bazel is the main build system used by developers. Maven is kept as a backup.
- People who want to simply use jflex, can download the targz distribution. The targz doesn't contain java sources.
- People who want to build clone the git repo and use bazel.
Bazel unique build system.
- Only the Maveb plugin is built with Maven.
- POM is removed.
- testcases are removed
- maven-uniocde-plugin is removed
Optional: reorganise directory structure.

Tasks

See project Bazel

Continuous integration

Initial attempt of using Travis PR #395
Currently using Cirrus

Bazel Rule

difficulty: very easy

Status ✔ PR #401

See bazel_rules

jflex-maven-plugin

difficulty: very hard

status: not started, probably the only piece that will stay as a Maven artifact.

Ideally, we need to infer the deps to generate a pom.xml

Examples

difficulty: easy

Status ✔ on "simple". PR #401

CUP

Replace cup-maven-plugin by Bazel rule

difficulty: Very easy

Status ✔ PR #442

Jflex itself

difficulty: easy (once cup rule and jflex rule are done)

Status ✔ PR #444

Migrate testsuite

Replace jflex-testsuite-maven-plugin and the test case by a testsuite made of java_test

status: WIP (More below)

some tests have been migrated already
tests of golden input/output can be migrated automatically with migration tool
tests that assert that an exception in the jflex generation can use a simple @JflexTestRunner

Migrate a golden test

If you want to contribute, here is how to migrate a test from maven-based testsuite/testcases to bazel-based javatests/jflex/testcase.

The migration tool can help you migrate a test case, still using input and output golden files.

For instance, to migrate "caseless-jflex":

Use migration tool

# You need to indicate the absolute path of the maven-based case:
bazel run java/jflex/migration:migrator -- ~/Projects/jflex/testsuite/testcases/src/test/cases/caseless-jflex

Copy the files, as the tool tells you, e.g.

# cp -r /tmp/caseless_jflex ~/Projects/jflex/javatests/jflex/testcase

Commit as is
Review the changes, update the javadoc if it points to an old bug in sourceforge, and test that the case passes
```
bazel test //javatests/jflex/testcase/...
```
Commit and send for review

Migrate a test to a unit test

There are a few more steps to migrate a test to a unit test not relying on golden files, and PR #592 on testcase/caseless is a simple example of how the GoldenTest was transformed into a unit test.

Generate a scanner from the flex file with the jflex rule, by creating a build target (this is done automatically by the migrator).

Example for bol.flex:

jflex(
    name = "gen_bol_scanner",        # follow the convention gen_<filename>_scanner
    srcs = ["bol.flex"],             # the source itself
    jflex_bin = "//jflex:jflex_bin", # Important!
    outputs = ["BolScanner.java"],   # as defined byy the class
)

Important Be sure to point to the current jflex_bin, otherwise the test will run against the (previous) released version.
Also, define the java package corresponding to the directory of this test
```
package jflex.testcase.bol;
%%
// rest of the grammar
```

Define a scanner state / token
- With the jflex-testsuite-maven-plugin, the test was defined with an input file and an expected output on System.out where each lexer action was using a System.println statement. A test should be on the API, not on the console debug output, so we will migrate to unit tests.
- Update the flex definition to output a state instead.
```
%type State
```
- Create a State.java enum with the states
```
public enum State {
  HELLO_AT_BOL_AND_EOL,
  HELLO_AT_EOL,
}
```
- Modify the flex grammar to return this states. Replace
```
^"hello"$  { System.out.println("hello at BOL and EOL"); }
"hello"$   { System.out.println("hello at BOL"); }
```
  by the newly defined states
```
^"hello"$  { return State.HELLO_AT_BOL_AND_EOL; }
"hello"$   { return State.HELLO_AT_EOL; }
```
- Optionally, also add an EOF state in the grammar:
```
<<EOF>>    { return State.END_OF_FILE; }
```

Use unit tests to verify the scanner.

Add a standard jUnit class, e.g. BolTest.java.

Create tests from the old .input and .output files, as deemed appropriate. For instance, the line

␣␣hello

with expected output

line: 2 col: 1 char: 6 match: -- --
action [27] { System.out.println( "\" \"" ); }
" "
line: 2 col: 2 char: 7 match: -- --
action [27] { System.out.println( "\" \"" ); }
" "
line: 2 col: 3 char: 8 match: --hello--
action [21] { System.out.println("hello at EOL"); }
hello at EOL
line: 2 col: 8 char: 13 match: --\u000A--
action [25] { System.out.println("\\n"); }
\n

could be migrated to a single test

  @Test
  public void eol() throws Exception {
    scanner = createScanner("  hello\n");
    assertThat(scanner.yylex()).isEqualTo(State.SPACE);
    assertThat(scanner.yylex()).isEqualTo(State.SPACE);
    assertThat(scanner.yylex()).isEqualTo(State.HELLO_AT_EOL);
    assertThat(scanner.yylex()).isEqualTo(State.LINE_FEED);
  }

Please use Google truth to write the assertions.

Add test target (the migration tool created a "GoldenTest", just rename)

   java_test(
       name = "BolTest",
       srcs = [
           "BolTest.java",
           "State.java",
           ":bol_scanner",
       ],
       deps = [
           "//third_party/com/google/truth",
       ],
   )

Migrating a test where the scanner generation fails

Of course, this cannot be applied if the generation of the scanner is expected to fail: We can't break the build to make a test pass.

In that case, you can use a custom https://github.com/jflex-de/jflex/blob/master/java/jflex/testing/testsuite/JFlexTestRunner.java and write a test in just a few lines of codes.

Example: EofPipeActionTest.

Tests Not migrated

ccl_pre with JDK variants. ccl-pre/ccl2.test was not migrated. Bazel makes it hard to change the runtime.
cup2private JFlex has a %cup2 directive, but I don't know where to get cup2 itself. There is apparently no release.
encoding. Bazel makes it hard to change the encoding used by javac. It's UTF-8 by default. And the use of --encoding changes both the input .flex and the generated .java files.
filename (#399) because Bazel doesn't allow \ anyways

BUILD:8:1: //javatests/jflex/testcase/filename:FilenameTest: invalid label 'filename\u000AFILE_NAMES_MUST_BE_ESCAPED\u000A.flex' in element 0 of attribute 'data' in 'java_test' rule: invalid target name 'filename\u000AFILE_NAMES_MUST_BE_ESCAPED\u000A.flex': target names may not contain ''
java because the tests are large integration tests, and it's unclear to me what is their benefit.

uberjar

status: native behavior of Bazel for java_bin.

Target directory structure

Source code, migration phases

Anyone who wants to build from source clones the git repo and uses Maven.

The source code directory structure looks like this:

├── bazel-*                           [REMOVED for targz]
├── cup-maven-plugin                  [OBSOLETE, removed]
├── bin                               [MOVED from jflex/bin]
├── docs                              [OK]
├── examples                          [MOVED from jflex/examples]
├── jflex                             [OK]
│   ├── common-testing                [REMOVED for targz]   
│   └── src                           [OK]
├── jflex-maven-plugin                [REMOVED for targz]
├── jflex-unicode-maven-plugin        [REMOVED for targz]
├── report-module                     [OBSOLETE, removed]
├── scripts                           [REMOVED for targz]
├── src                               [OBSOLETE, removed]
├── testsuite                         [REMOVED for targz]
│   ├── bzltestsuite                  [NEW, replaces jflex-testsuite-maven-plugin]
│   ├── jflex-testsuite-maven-plugin  [OBSOLETE, removed]
│   └── testcases
├── lib                               [UNCHANGED]
├── third_party                       [NEW]
└── tools                             [REMOVED for targz]

targz distribution

As a result, the targz has the following strucuture:

Distribution targz

├── bin
│   ├── jflex.bat
│   └── jflex.sh
├── docs                              [COPIED from bazel-bin]
│   ├── manual.html
│   └── manual.pdf
├── examples
├── jflex
│   └── src                           [OK]
├── lib                               [UNCHANGED; add jars copied from bazel-bin]
├── third_party 
│   ├── cup
│   ├── com
│   └── org                           [etc.]
└── README.md LICENSE etc.

Note: for the distribution to avoid the bootstrap question, we can also copy relevant files from bazel-genfiles:

bazel-genfiles
├── examples
│   └── simple
│       └── Yylex.java
└── jflex
    ├── LexParse.java
    ├── LexScan.java
    └── sym.java

Sources structure, reorganized after migration

This is the flattest structure that works well with Maven and makes releasing the targz easy.

├── bin
│   └── jflex.bat
├── docs
│   ├── grammar.md
│   ├── intro.md
│   └── template.tex
├── java
│   └── jflex // consider splitting de.flex.* for tools and jflex.* for core generator
│       ├── LexGenerator.java
│       ├── gui
│       ├── lexparse.flex
│       ├── lexscan.cup
│       ├── maven
│       │   └── JflexMojo.java
│       ├── testing
│       │   └── testsuite
│       │       └── TestRunner.java
│       └── unicode
│           └── UnicodeProperies.java
├── javatests
│   └── jflex
│       └── testsuite
│           └── apiprivate
│               ├── ApiPrivateTest.java
│               └── private.lex
├── scripts
│   ├── release.sh
│   └── travis.sh
└── third_party
    ├── com
    │   └── google
    │       └── guava
    └── java
        └── java_cup
            ├── java_cup-11b.jar
            └── runtime
                ├── ComplexSymbolFactory.java
                └── lr_parser.java

JFlex Wiki Home

Wiki Home
JFlex website User manual Grammars Download
Issues

Provide feedback

Saved searches

Use saved searches to filter your results more quickly