Makefile skips generating certain files when building smlnj-mlton from scratch

[shwestrick]

I'm on a fresh install of Ubuntu 22.04 and am attempting to build smlnj-mlton from scratch. I have SML/NJ version 110.99.4.

By "from scratch" I mean that I don't have easy access to an existing mlton binary on this machine, nor do I have access to MLton's mllex and mlyacc. This is because, AFAIK, at the moment unfortunately there is not a mlton package available for Ubuntu 22.04. I've tried a couple mlton binaries from a prior release, but none of these have worked out-of-the-box for me, yet.

So, I'm trying to build smlnj-mlton using only what is available from SML/NJ. In particular, instead of mllex and mlyacc, I'm trying to use SML/NJ's ml-lex and ml-yacc.

I eventually managed to get this to succeed, but only with a workaround.

TL;DR: the Makefile seems to not automatically generate files such as lib/stubs/mlton-stubs/pre-mlton.sml, mlton/control/version.sml, etc. My workaround was essentially to make these explicitly before doing make smlnj-mlton.

Error details

I tried this command:

$ make RUN_MLLEX=ml-lex RUN_MLYACC=ml-yacc smlnj-mlton 

We soon hit an error:

$ make RUN_MLLEX=ml-lex RUN_MLYACC=ml-yacc smlnj-mlton 
Makefile.config:197: 'mlton' command not found: mlton
make dirs
...
... omitted: lots of output ...
... `make dirs` and `make runtime` both appear to build successfully ...
...
make[1]: Entering directory '/home/shwestrick/installs/mlton-master/mlton'
CM_VERBOSE=false ml-makedepend -DSMLNJ_PATCH_VERSION=4 -n -f mlton-compile-smlnj.amd64-linux.d mlton-smlnj.cm mlton-compile-smlnj.amd64-linux.d || (rm -rf mlton-compile-smlnj.amd64-linux.d ; exit 1)
Standard ML of New Jersey (64-bit) v110.99.4 [built: Sun Oct 29 13:34:49 2023]
[opening 386338-mlscript.sml]
../lib/stubs/mlton-stubs/pre-sources.cm:15.1-15.14 Error: Io: openIn failed on "../lib/stubs/mlton-stubs/pre-mlton.sml", No such file or directory
/home/shwestrick/installs/smlnj-110.99.4/bin/ml-makedepend: CM dependency analysis failed
(									\
	echo 'local';							\
	echo 'fun loop 0 = () | loop n = (CM.Server.start {cmd = (CommandLine.name (), ["@CMslave"]), name = "server" ^ (Int.toString n), pathtrans = NONE, pref = 0}; loop (n - 1));'; \
	echo 'in';							\
	echo 'val _ = loop 0;';			\
	echo 'end;';							\
	echo 'if (CM.make "mlton-smlnj.cm") handle _ => false';		\
	echo '   then ()';						\
	echo '   else OS.Process.exit OS.Process.failure;'; 		\
	echo 'SMLofNJ.exportFn("mlton-compile-smlnj.amd64-linux",Main.main);'			\
) | CM_VERBOSE=false CONTROL_POLY_EQ_WARN=false "sml" -DSMLNJ_PATCH_VERSION=4
Standard ML of New Jersey (64-bit) v110.99.4 [built: Sun Oct 29 13:34:49 2023]
- ../lib/stubs/mlton-stubs/pre-sources.cm:15.1-15.14 Error: Io: openIn failed on "../lib/stubs/mlton-stubs/pre-mlton.sml", No such file or directory
make[1]: *** [Makefile:206: mlton-compile-smlnj.amd64-linux] Error 1
make[1]: Leaving directory '/home/shwestrick/installs/mlton-master/mlton'
make: *** [Makefile:274: smlnj-mlton] Error 2

Both dirs and runtime targets appeared to build successfully. The error is this:

../lib/stubs/mlton-stubs/pre-sources.cm:15.1-15.14 Error: Io: openIn failed on "../lib/stubs/mlton-stubs/pre-mlton.sml", No such file or directory

And, this is not the only file missing. We're also missing control/version.sml and various front-end/... files.

I tried to figure out how to modify the Makefile to fix this, but I'm not much of a Makefile expert and got a bit lost in the weeds.

Workaround

The following sequence seems to work. The idea is to just build the list of generated files explicitly before getting to the smlnj-mlton target.

# these commands work fine
make dirs
make runtime

# next, need to explicitly make generated files (is this the correct list...?)
make RUN_MLLEX=ml-lex RUN_MLYACC=ml-yacc -C mlton \
  ../lib/stubs/mlton-stubs/pre-mlton.sml \
  control/version.sml \
  front-end/ml.lex.sml \
  front-end/ml.grm.sig \
  front-end/ml.grm.sml \
  front-end/mlb.lex.sml \
  front-end/mlb.grm.sig \
  front-end/mlb.grm.sml

# .boot files got overwritten??? restore them.
git restore mlton/front-end/*.boot

# finally, this works! :)
make smlnj-mlton

One funky thing that took me a while to figure out is that we need to do git restore mlton/front-end/*.boot after building the front-end lex/grm stuff. These targets appear to overwrite the .boot files, which in turn causes a build error somewhere deeper within make smlnj-mlton.

(Specifically, if we don't git restore these files, we get the error front-end/ml.lex.sml:10829.20-10829.37 Error: unbound structure: Unsafe ...)

I don't know what the .boot files are, but restoring them to their unmodified contents seemed to work... 😅

[MatthewFluet]

There probably are some missing dependencies for the non-MLton bootstrap builds.

The .boot files are copies of the mllex- and mlyacc-generated files that can be used to create the front-end/*.{lex,grm}.{sig,sml} files in the absence of mllex and mlyacc. So, you shouldn't need to specify MLLEX=ml-lex MLYACC=ml-yacc. Restoring the .boot files (which will then have a timestamp newer than the ml-lex and ml-yacc generated files) makes the subsequent make smlnj-mlton succeed, because the front-end files are recreated from the .boot files.

[MatthewFluet]

Actually, I think it suffices to add $(MLTON_GEN_SOURCES) as a dependency of $(MLTON_OUTPUT_SMLNJ_HEAP).d on line 218 of mlton/Makefile. The $(MLTON_GEN_SOURCES) does accumulate all of the source files that need to be generated.

[shwestrick]

The .boot files are copies of the mllex- and mlyacc-generated files that can be used to create the front-end/*.{lex,grm}.{sig,sml} files in the absence of mllex and mlyacc. So, you shouldn't need to specify MLLEX=ml-lex MLYACC=ml-yacc.

Interesting. Am I understanding correctly that none of {mllex, mlyacc, ml-lex, ml-yacc} are actually required to build from source, then? What are you thoughts on removing these as dependencies? It seems like it would be a win to always just use the .boot files... just one less thing that might go wrong.

Actually, I think it suffices to add $(MLTON_GEN_SOURCES) as a dependency of $(MLTON_OUTPUT_SMLNJ_HEAP).d on line 218 of mlton/Makefile. The $(MLTON_GEN_SOURCES) does accumulate all of the source files that need to be generated.

Great! Happy to test this later on my machine.

[MatthewFluet]

The .boot files are copies of the mllex- and mlyacc-generated files that can be used to create the front-end/*.{lex,grm}.{sig,sml} files in the absence of mllex and mlyacc. So, you shouldn't need to specify MLLEX=ml-lex MLYACC=ml-yacc.

Interesting. Am I understanding correctly that none of {mllex, mlyacc, ml-lex, ml-yacc} are actually required to build from source, then? What are you thoughts on removing these as dependencies? It seems like it would be a win to always just use the .boot files... just one less thing that might go wrong.

Yes, in the absence of mllex and mlyacc, the Makefile will generate the files from the .boot files. We still want to generate the files (and update the .boot files) automatically when there are changes to the .lex and .grm files, so the Makefile still handles both situations.

By "removing these as dependencies", you mean to remove their mention from the top-level README.adoc? Yes, that would probably be fine.

[shwestrick]

Just tested #520 and make smlnj-mlton works now, thanks Matthew!

By "removing these as dependencies", you mean to remove their mention from the top-level README.adoc?

I think that would be helpful, or at least, it would have been helpful for me in this case. I assumed I needed to use RUN_MLLEX=ml-lex and RUN_MLYACC=ml-yacc, which turned out to be a bit problematic.

By the way, even after #520, the command make smlnj-mlton RUN_MLLEX=ml-lex RUN_MLYACC=ml-yacc doesn't work; I get the following error:

front-end/ml.lex.sml:10829.20-10829.37 Error: unbound structure: Unsafe in path Unsafe.Vector.sub
front-end/ml.lex.sml:10845.36-10845.57 Error: unbound structure: Unsafe in path Unsafe.CharVector.sub
front-end/ml.lex.sml:10846.18-10846.35 Error: unbound structure: Unsafe in path Unsafe.Vector.sub

It seems like perhaps there's some divergence between ml-lex and mllex?

[MatthewFluet]

Yes, MLton's mllex and SML/NJ's ml-lex have some small differences, documented at the top of the lexgen.sml file.