Changing board configuration leaves behind unnecessary files, even after "lbuild clean"

[WasabiFan]

Say I have a project which extends modm:disco-f469ni. Note that it has a significant number of files, some of which aren't included by other targets. Now, I change my project to instead extend modm:disco-f429zi. I run lbuild clean (optional) and lbuild build again. While the core board source files have been updated, the README.md and miscellaneous support files from the other board were left behind.

The output from lbuild clean:

...
Removing modm\src\modm\architecture\interface\spi_master.hpp
Removing modm\src\modm\architecture\utils.hpp
Removing modm\src\modm\board.hpp
Removing modm\src\modm\board\board.cpp
Removing modm\src\modm\board\board.hpp
Removing modm\src\modm\driver\inertial\l3gd20.hpp
Removing modm\src\modm\driver\inertial\l3gd20_impl.hpp
...

It seems strange to me in general that lbuild never seems to remove many of the files in the build tree. I have a diff here which shows the changes made by lbuild clean. What am I missing? Does lbuild only remove particular files, e.g. files that were generated via template?

While the left-over files might have technical ramifications if you are using a custom build configuration, for me this caused a bit of confusion when I was trying to make a custom board configuration. I had originally intended to copy the files from the GitHub repo, but this document suggested copying them out of the build tree after running it through lbuild. If I did so after changing my board configuration I'd be left with extra files that weren't needed and said they were for a different board.

[salkinium]

lbuild clean uses the project.xml.log file to find out what files to remove. If that file is not found, it simulates a build and removes these files. Of course if you updated your config a few times and lbuild build a few times, then project.xml.log is going to be overwritten with the latest build, thus lbuild clean misses a lot of files.

The DISCO-F469NI board in particular pulls in a lot of modules, because it uses a lot more hardware than the DISCO-F429ZI. And thus the file diff you see is quite a lot.
This is a known issue modm-io/lbuild#33, with a (relatively simple) solution, it just hasn't been implemented yet.

(There's also the __pycache__ spam, which is a whole other problem, because Python thought it to be a good idea to fill the entire file system with its caching, instead of using a central caching mechanism that the OS provides (and cleans).)

If I did so after changing my board configuration I'd be left with extra files that weren't needed and said they were for a different board.

Vigerously delete the generated modm/ folder in that case!

[WasabiFan]

Of course if you updated your config a few times and lbuild build a few times, then project.xml.log is going to be overwritten with the latest build, thus lbuild clean misses a lot of files.

That seems like a strange design to me. With most build tools, running clean will delete everything the build tool had touched: often simply deleting the output artifact directory. Why does lbuild only delete a subset of those files? Is there a downside to not deleting all of them?

It sounds to me like you're trying to support a scenario in which the user is modifying files within the build output tree, but I don't understand that use case yet.

Vigerously delete the generated modm/ folder in that case!

That is a solution I can confirm works 😁 But doesn't it seem odd that the user has to do that to truly clean the build output?

[WasabiFan]

Perhaps stated differently: as far as I can tell, lbuild has no way to restore the working tree to a known-good consistent state. Even running clean leaves files behind which could cause problems in the future.

[salkinium]

It sounds to me like you're trying to support a scenario in which the user is modifying files within the build output tree, but I don't understand that use case yet.

That's exactly right. Once you've settled on a reasonable configuration, the clean generated code should be committed into your repo and then the required changes can be tracked in another commit. When you want to upgrade modm, you can diff the changes in the generated files, as well as re-applying the patches you made.

This allows to upgrade long-running projects to the latest modm release with a reasonable effort (I need to actually create a release with changelog again for this to work tho -> #44).

[rleh]

The user may also have to clean up the build systems (scons) build path manually.
I usually use a command like this from my bash history and it's horrible:

rm -rf modm/ project.xml.log SConstruct ../some/../build/../directory/ ../modm/ext/modm-devices.cache && lbuild build && scons -j8

A tooling for actually utilized use cases would be nice!

[rleh]

This allows to upgrade long-running projects to the latest modm release with a reasonable effort

How many users does this actually affect?

[salkinium]

How many users does this actually affect?

Everyone at the DLR, aka. the people that lbuild was originally designed for 😉. Although they use the OUTPOST repo not modm, but same issues.

[WasabiFan]

Ah, interesting! I suppose my mindset is wrong then. I keep trying to imagine modm as a library but it's truly designed as a generator for HAL code which you then adopt into your project as a peer. It seems foreign to be to bring in an external module and then modify the code it generates, but I suppose there are use cases for that.

In that case, are there any available projects which you can point me to which consume modm as an external party? e.g., a GitHub repo where someone added a modm submodule, configured a project, checked in the generated output, and added code which uses it? That would be very helpful as a reference for the expected structure of my own project.

Thanks for all the quick responses!

[salkinium]

A tooling for actually utilized use cases would be nice!

Yeah, Niklas, just do it right from the beginning!1!! Why couldn't you have forseen all the problems beforehand?2?? Are you even a real programmer??? 😉

I know, no, I feel these issues too, I've been refactoring this stuff for the last 5 years…

I keep trying to imagine modm as a library but it's truly designed as a generator for HAL code which you then adopt into your project as a peer.

Yeah, that's cos it isn't well documented either. The constant code committing is annoying with a very dynamic project that changes a lot or even modifies modm itself a lot. The @roboterclubaachen is/was the main user of modm and lbuild didn't integrate very well into the existing workflow. (we didn't commit any generated code there either).

It's difficult to solve these things particularly without getting inspiration from other projects, however, there aren't a lot (any?) projects that use code generation so extensively. All projects that do use some code generation, commit their generated code into their own repo, that's of course not possible with modm.

In that case, are there any available projects which you can point me to which consume modm as an external party?

Nothing public, sorry! The invensense repo uses modm (via lbuild init, don't ask), but without committing code. Maybe @dergraaf has some projects?

[salkinium]

Also keep in mind, that lbuild doesn't deal with dependencies on a file level, like actual build tools do, only at a module level. We don't want to write a build tool, since they already exist (hence the modm:build modules), but it's very hard to not write a build tool it seems, if only for the dependency tracking 😉

[salkinium]

One project where I did commit code was the ELVA project, but I cannot upgrade that since we're still using C++14 for that code, and modm is using a few C++17 features very, very extensively (specifically constexpr if for the GPIO code). So, can't say that was a success story…

[salkinium]

Regarding upgrading use-cases: roboterclubaachen/xpcc#160
Last release was from 1.5y ago, not good: https://github.com/modm-io/modm/blob/develop/docs/CHANGELOG.md

[WasabiFan]

Some backstory: I myself am on a competitive robotics team and we are looking into using modm for our main controller code. I really, really want to get it working and integrated with a reasonable development experience, and think it would dramatically improve our quality-of-life over the workflows and tools we currently use. The hardest part is just figuring out how to use it 😁

If I am able to get my team set up with modm and it works well for our use cases, I'd be happy to contribute some time to improving things as I go. Most of it would probably be documentation, but I'd also contribute code where I am competent enough to do so.

---

While I'm going along and figuring things out, I'd really like to help improve this experience for future users. It was striking to me that this repo had only about 200 start despite having such an apparently clean API surface, and I think improving the docs and the "getting started" experience would help a lot with that. There are a lot of API usage examples (and that is excellent!) but:

• There are no examples of how you set up your own project, as a user.
• Getting modm integrated into your project currently requires diving into the internals of modm, and especially lbuild configuration, in a way that might dissuade users.
• I haven't found much information on tooling: what editors have people gotten working? What debuggers work well?

Anyway, getting back to the main point of this issue (some of this is better-suited to #280), I think clarifying how lbuild and its output can be integrated into a user's codebase would go a long way to improving the setup experience. Heck, why not just create a modm-io/modm-template repo which has a basic project with modm included as a submodule and built/configured as you'd suggest? It would just be a basic "blinky" app, but set up to be forked and worked off of.

Admittedly, it sounds like you aren't 100% sure on how modm is best integrated into other projects, making it hard to settle on a single suggestion to give to others. But if there is a specific setup that you think would work well (e.g., have modm as a submodule, commit the generated output to your repo, then upgrade modm intermittently), please do document it! And really, I am serious about having a template repo to start from; that would be extremely helpful. I would even be willing to make one if you tell me how you think it should be structured. And I can always set up my project, try it out for a bit, and give feedback on the experience.

And speaking of all this, as you mentioned, somewhat consistent releases would be very helpful for signalling when people should upgrade. It seems a bit scary to work on a rather arbitrary commit, and have no single tag where users are actually testing.

[WasabiFan]

Sorry for the long rambling comment only tangentially related to the original issue 😆

[salkinium]

Heck, why not just create a modm-io/modm-template repo which has a basic project with modm included as a submodule and built/configured as you'd suggest? It would just be a basic "blinky" app, but set up to be forked and worked off of.

Yes, I've been thinking about this ever since GitHub introduced template repositories some months ago.

[salkinium]

Started a template repo here: https://github.com/modm-io/modm-template
Currently it does not copy the git submodule correctly, not sure if that's a GitHub bug or my incompetence.

[salkinium]

GitHub Developer Support answer came back:

Submodules not transferring is a known limitation of the template feature I'm afraid. It is something we have a bug report opened for however.

It isn't possible to use submodules and templates currently, but I've made a note to get back to you when we have a fix in place.

Hm, well… why did you even ship this feature?

[salkinium]

Alternatively we can also commit normally onto the develop branch of the template repo and have a CI job squash the commits and force push them into the master branch, and then people can clone it normally. Or we just wait until GitHub fixes this issue, and until then people just have to deal without repo history. Meh.

[WasabiFan]

Hmph, that's frustrating. What's the scenario you want to support -- maintaining history of the repo template? Or avoiding history of the repo template?

[salkinium]

I'll just ignore the template feature for now, and just make people fork it. That means, they'll have the full history of it. If it's less than a year or two until they fix it, the history won't be that large anyways.