- Start Date: 2024-03-01
- RFC PR: lf-lang/rfcs#8
- Tracking Issue(s): lf-lang/lingua-franca#2203
This RFC proposes multiple changes which when combined can address a single problem, which is conflicts between generated directories and directories that may have been created by the user.
- Make surface-level UX improvements to address the pitfall described in this issue. This is not inherently a breaking change but is partially dependent on item 3, which is breaking.
- Make the locations of generated directories configurable. This is also not inherently a breaking change.
- Change the default locations of some generated directories. This is a breaking change.
Currently, lfc will silently delete directories that have the same name as generated directories without any warning to the user. For example, if the user is already maintaining a directory called include that contains source files, then the current behavior will cause that directory to be overwritten or even deleted, which in the worst case can result in lost work.
Furthermore, the only way to address such a directory naming conflict is for the user to change the names of the directories that they maintains.
In the following discussion, the "project root" associated with a given LF file with absolute canonical path f is the parent of the most recent ancestor of f that is named src. The project root is also the location of the applicable Lingo.toml file, if such a file exists. Having a directory called src inside the root src directory is undefined behavior. A Lingo.toml file is not applicable to an LF file whose absolute canonical path is not a child of a directory named src that is a sibling of that Lingo.toml file.
High-level context: In this RFC, there is a bizarre two-way interaction/dependency between lfc and lingo. This is because of the history of our project: Instead of tolerating a complex-to-use CLI for lfc with the understanding that we would later make it easy to use by providing a package manager like lingo, we chose to design everything so that lfc would work properly even when run with very few arguments. For example, this is why build configurations currently live inside LF files. Now, we have users who invoke lfc directly, and so it may not be acceptable to add functionality to lingo without easing the transition by replicating some of it in lfc.
- Detect, on a best-effort basis, whether a directory has been generated by
lfcbefore overwriting it. This may involve generating a file in the directory that marks it as having been generated bylfc. If the directory has not been generated bylfc, warn the user and request confirmation to overwrite the directory. Note that this is much more difficult to do when the directory is only indirectly generated bylfc. For example, iflfcinvokes CMake and CMake generates various directories in the project root, as is the current behavior in the C++ target, then in order to work properly our tools would need to make assumptions about the behavior of CMake in order to recognize these directories as being indirectly generated bylfc. This problem should be resolved by requiring directories generated by CMake to be placed in aninstalldirectory that is generated bylfc.
- Users may (unwisely) add their own files inside generated directories. We can use timestamps (provided by the operating system) to detect this scenario and warn the user about this.
- Add a
lingo cleanfunctionality that reads Lingo.toml and deletes all directories generated bylfc. This makes it practical to use multiple generated directories, such asfed-genandinclude-gen, that are siblings ofsrc-gen, without making the process of deleting those directories excessively burdensome to the user.
Introduce optional properties to be included in Lingo.toml:
src-gen: The location for code that is generated bylfcinstall: The location for any directories generated by CMake that are not insrc-gen, and for thebindirectory regardless of whether thebindirectory is generated by CMakefed-gen: The location for code that is generated for individual federatesinclude-gen: The location for user-facing C/C++ header files that are generated bylfc
Additionally, add the following property:
bin: Override the location of the generatedbindirectory so that it is not inside theinstalldirectory.
Any relative paths shall be resolved relative to the project root.
The generated directory include which currently appears in the project root for the C target should be renamed to include-gen by default. It is possible that for many users, this will not be a breaking change because we will generate CMake code such that include-gen is on the user's include path.
The files generated by CMake in the case of the C++ target should be moved to a directory called install. As explained above (in the "UX improvements" section), we should not make it possible for the user to configure their Lingo.toml such that these directories stay in the project root, where they are now.
The generated directory bin should be moved to be inside whatever the install directory is.
To mitigate the problems that could result from these breaking changes, the release notes should include a link to a short Markdown file that presents and discusses an example Lingo.toml file that will minimize the amount of change in behavior.
A drawback that applies to this entire RFC is that it will be complex to implement the RFC in such a way that the desired behavior is unambiguous. If the behavior is different depending on whether the user is invoking lfc or lingo, then we will have to enforce that lfc will not be directly invoked without lingo when a Lingo.toml file is present. This may be difficult to enforce without some hack, such as an environment variable that says whether lfc is being invoked by lingo.
This proposal consists of a large amount of work, not all of which is strictly necessary in the near term; however, it could be awkward for users to adapt to the changes in this RFC if the RFC is only partially implemented before the next release.
This RFC introduces complexity which users would have to learn about. It may be too early to introduce such complexity because we may realize later that we need to change the behavior again.
Any request for user confirmation will cause problems in headless runs of lfc.
This is a breaking change.
For a competing alternative, see RFC #7. Note that placing the generated headers inside the src-gen directory is not compatible in any obvious way with the idea of making the location of generated headers configurable. This is because it will be complex and error-prone for us to allow users to configure distinct generated directories to be children of each other.
A potential drawback of this alternative is that users may not be fully aware of the user-facing include directory, or it may not be as clear that the interfaces defined there are considered stable and safe to depend on. Putting both user-facing and non-user-facing files in src-gen could blur the lines between the two kinds of files.
Should lingo clean read the contents of LF files, or should we forbid generated directory locations from being specified in LF files? The latter is preferable because we wish to encourage users to migrate to lingo and because in order to implement the former without having to maintain two distinct LF parsers, we would need lingo clean to invoke lfc to obtain an AST, which will incur a JVM startup time delay even for users that do not use this target property.
How can the proposed UX improvements be made available to users who have not yet migrated to lfc? Should we introduce lingo as a dependency of lfc, and have lfc invoke lingo? Or should we introduce wasmtime as a dependency of lfc, and pack a WASM bin of lingo into lfc? Or should we ignore the problem and accept that people who do not use lingo will not benefit from these changes?
Is it adequate to require all paths to be either absolute, or relative to the project root? Or, will it be necessary to introduce CMake-like variables in Lingo.toml, such as ${LF_PROJECT_ROOT} and ${LF_INSTALL_DIR}?
To what extent is the location of the bin directory the responsibility of lingo versus the responsibility of CMake? It is complex to implement the installation of binaries and appealing to let CMake take care of it because CMake is a mature tool that can address all of the corner cases that users are likely to encounter. However, it is also easy to implement a solution in lingo that is convenient to use and that works most of the time, and it is necessary for our tools (including our test framework and our VS Code extension) to know where binaries are installed; otherwise they cannot work properly. Besides, we generate executables, such as bash scripts, which are not managed by CMake. It is possible that we can have the best of both worlds by letting our tools query CMake in order to find out where executables will be found and then copy the executables to other locations as needed.
Let lingo automatically add generated directories to a .gitignore.