Allowing mbed_app.json files to be discovered for tests.

[bridadan]

Description

Before, mbed_app.json files were explicitly ignored when building tests.
This was mostly because you could have multiple mbed_app.json files in the
tree (for instance, in test case folders) and the behavior would be
undefined. Now the tools explicitly ensure that there aren't multiple
mbed_app.json files in your source files. So auto discovery of
mbed_app.json for testing is being reintroduced.

Status

ON HOLD (pending review)

Migrations

If this PR changes any APIs or behaviors, give a short description of what API users should do when this PR is merged.

YES

If you relied on the behavior of mbed_app.json being ignored when building tests, this is a breaking change.

Todos

• Tests
• Review by @mazimkhan (Does this affect uVisor CI testing?)
• Review by @AlessandroA (Does this affect uVisor local testing?)

FYI @BlackstoneEngineering and @screamerbg

[mazimkhan]

Does not impact uVisor testing as uvisor tests are implemented in separate repo uvisor-tests and compiled like an app.

[sg-]

/morph test

[sg-]

This means you don't have a reproducible testing environment anymore given local config can modify the build used for executing the tests. @screamerbg

[bridadan]

@sg- This is very true. In CI this obviously isn't as big of a deal since we control the environment and what files are present very carefully, however when testing locally this may create complications/differing experiences depending on what files are left over in a workspace. It'd be very easy to cause testing issues with this if you aren't vigilant about what config files you have laying around. The current method of explicitly setting the --app-config flag for testing means you have to be aware of what settings/config you're using.

@screamerbg We did discuss at one point just printing what config files are being used, but I still think that won't be enough. People will only see the test results, which happens way after the build. So the log message about the config may be lost way before the test failures occur.

[mbed-bot]

Result: SUCCESS

Your command has finished executing! Here's what you wrote!

/morph test

Output

mbed Build Number: 1079

All builds and test passed!

[bridadan]

@sg- Had a talk with @screamerbg. Basically the issue at hand is the following:

For testing, do we (1) break with what we do for applications and default to ignoring all app config unless explicitly specified. OR, (2) do we stay consistent with applications and pick up app config automatically?

For consistency, I think we should go with (2), with the added option of providing a way of disabling automatic detection of the mbed_app.json. So something like --app-config none or --disable-app-config. This option to ignore app config would also propigate to compiling applications, which would probably be a useful feature. It would fit into the similar strategy we use with ignoring source files by using .mbedignore files: Auto detect and use with the option to ignore vs explicit inclusion of every file.

@sg- thoughts?

[screamerbg]

Ping @sg-

[sg-]

LGTM 👍

[sg-]

@marhil01 @jupe feedback on pr-head CI