Files in include Folder Removed After Compiling with lfc

[Depetrol]

Issue Description

I encountered this issue when attempting to run the CARLA simulator with Lingua Franca following this instruction. When compiling carla_sync.lf which references include/carla_client.py, carla_client.py within the include folder has been deleted.

This leads to test failures in the PR and should not be the expected behavior.

Temporary Fix

Move files in the include folder elsewhere, and change the file references. In the case of CARLA simulator, move include/carla_client.py to the parent/workspace folder and change carla_sync.lf :

target Python {
  files: carla_client.py
}

Proposed Fix

Ensure files in the include folder are retained after compilation, or issue a warning during the compilation process to notify the user that files in the include folder might be removed.

[lhstrh]

Thanks for the bug report, @Depetrol. We will work on this and push a fix.

[petervdonovan]

I think that this is probably because we generate some headers and keep them in a user-facing (not in src-gen) directory. When the headers are re-generated, we probably delete the whole directory.

Probably the right way to solve this is to rename the directory for the generated headers from include to include-gen to reduce the chance of getting confused about whether the user is supposed to edit the files in that directory. This should have been done when I introduced the mechanism for generating headers because the potential for it to conflict with the user's source files is clear.

[lhstrh]

I like the include-gen solution you propose.

[cmnrd]

Oh, so the problem is that this example has an include directory next to the src directory, which violates our assumption that all sources are located in src.

Renaming "our" include directory is not easy. It is generated automatically by cmake when installing the runtime lib (and potentially other libs and reactor headers). We could try to rename this (I am sure there is a hack for that), but the bin, lib, include, share directory structure is established on POSIX systems and we would break with that.

I see two solutions:

1. Clearly state that all source files should be located within src and try to enforce this in our validation as far as we can.
2. Introduce an install directory. Alternatively, we could also use the build-type and use a debug and release directory, which some build-tool or IDEs do. Then we would let cmake install within this directory. But this would change the location of our binaries, which might cause problems elsewhere.

[petervdonovan]

This is complicated by the fact that the C target does not work in the same was as the C++ target. In the C target, some headers are generated by the code generator lfc and not by CMake, so actually we do have control over where those headers go. Those headers are intended to be used in the files that appear in src, which is probably pretty different from what the include directory is for in the case of the C++ target.

I'm not sure if it makes sense to try to unify the behavior of the two targets.

[cmnrd]

Ok, those files should go into src-gen. We can just create a nested include directory and make this part of the include path. The C++ also copies an LF specific header file into src-gen.

Those headers are intended to be used in the files that appear in src, which is probably pretty different from what the include directory is for in the case of the C++ target.

Yes, in the C++ target the include directory is part of the compilation artifacts (like the bin directory). I thought the C target actually does the same, but we should investigate more closely.

I'm not sure if it makes sense to try to unify the behavior of the two targets.

I think we should, it's just unclear what the best solution would be.

[petervdonovan]

Ok, those files should go into src-gen. We can just create a nested include directory and make this part of the include path. The C++ also copies an LF specific header file into src-gen.

This works for me. I'll go along with whichever directory layout we can get everyone on board with.

[lhstrh]

lf-lang/rfcs#11 discusses project layout/directory structure.

[cmnrd]

Yes, we also discussed the overlap between this issue and the new RFC in the last developer meeting. However, we resolved to focus lf-lang/rfcs#11 on where we place LF code, and not where and which directories we generate. This is a different concern that should be handled in another RFC.