[RFC] POSIX-compliance of scan.l and bootstrapping

[jannick0]

@westes Here some thoughts and questions on the bootstrap feature:

1. Non-cross-compilation:

• The release tarball ships scan.c so that a bootstrap should not be necessary.
• What is the reason to have the bootstrap feature other than having flex being able to compile itself, even when compiling from the git repo?
• Shouldn't scan.l be compliant with a minimal standard, e.g. POSIX, such that any POSIX-compliant version of lex should be able to compile flex? (E.g. 4b5111d introduced the non-POSIX feature %top{...} to scan.l which would imply that some other lex version might not be able to digest scan.l) Why would it be necessary to have a two-step approach (i.e. bootstrap) then at all?

=> Preliminary conclusions:

• The bootstrap feature could be dropped for non-cross-compilation. A one-step approach is sufficient. The check that flex reproduces scan.c (either shipped or generated using a (f)lex version on the system) could be added as test to the test suite.
• scan.l should be POSIX-compliant. (Otherwise flex would be insulated with other flex-compliant lex versions (do they exist?) and independent from any other lex implementation.)

2. Cross-compilation:

• The current cross-compilation implementation is based on the flawed assumption that config.h generated using the host compiler works for the build compiler, too. For a proper approach configure && make should be run twice, first for the build compiler to produce a flex version executable on the system - if not yet available - , then for the host compiler ... something like so:

mkdir b-build
cd b-build
../configure --host=${build_triplet} && make && make check
cd ..

mkdir b-host
cd b-host
../configure --host=${host_triplet} LEX='../b-build/src/flex' 
make # necessarily a one-step approach whenever host executables cannot be run on the system (which is the standard case)
cd ..

=> Preliminary conclusions:

• The bootstrap feature must necessarily not apply, i.e. it should be dropped.
• The first configure run (build) could be dropped and the second configure run (host) only is needed, if scan.l was POSIX-compliant a POSIX-compliant lex version was available on the system.

All-in-all here my suggestions:

1. apply a one-step approach to compile flex
2. (optional) scan.l should be POSIX-compliant, otherwise it can be compiled by flex and other flex-compliant lex versions (not sure if there were any requests about this at all out there in the past).
3. drop the implemented cross-compile feature and use for cross compilation the approach along the lines given above (this could straighten the issue about how to use rpl_malloc and rpl_realloc which I believe currently is not correctly implemented (LIBOBJS if non-void does not capture object files in the lib dir)).

Thoughts? Any missing points? Any plans where flex is headed to?

Thanks.

[Explorer09]

1. Ask @westes for the bootstrap decision. I have been personally against that in the past, but ultimately I can't change that. Instead I pushed options like --disable-bootstrap (Build system tweaks and '--disable-bootstrap' configure option #129) to let builders to workaround bootstrap problems if any. The main reason for bootstrap was to ensure the internal scanner of Flex is generated by the exact version of Flex to be built/released.
2. For cross-compilation problems, I have been trying to fix that before. Could you review build: Deterministic scan.c and bootstrap byte comparison check #372 and build: config_for_build.h for cross, bootstrap config. #373 for me so that we won't make duplicated effort for what looks like the same problem?
3. I can't comment on the POSIX-compliance of Flex's internal scan.l. It's a good idea for scan.l to be POSIX-compliant but I don't think it's a high priority goal for Flex as far as I can tell (there are compromises between features and POSIX-compliance, you know).

[Mightyjo]

Re: POSIX compliance. Flex is not lex. Flex has a POSIX-compliant mode that we should make sure works as advertised but there's no reason flex should limit itself to using that mode.

I was thinking about the bootstrap process. Like @Explorer09 said, flex used to emit A LOT of C that isn't in the skeleton. We had to bootstrap from one version to the next in order to replace all that emitted C code in scan.c, the compiled skeletons, etc. That's been dramatically reduced, but there's still some.

[Explorer09]

My personal opinions about bootstrapping:

1. The scanner generated by Flex (including the skeleton code) should be compatible with standard C, and it has been so for long.
2. There's no need for Flex's internal scanner (scan.l) to be limited to POSIX lex features.
3. scan.l bootstrapping is a low concern - as long as it can be built with the previous release of Flex, we are fine. The scanner C code scan.c are pre-generated in every Flex release, so technically you don't need POSIX lex at all to bootstrap Flex. The bootstrap process would look like this: (a) Get the latest release tarball of Flex, and build it. (b) Checkout the master branch if you want to try the development version, and bootstrap with the Flex version you just built from the release tarball. That's it. There's no need for POSIX lex.

[westes]

bootstrapping of (f)lex is not a posix feature of lex.