Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmake creates too many build artifacts #5009

Closed
andrewboie opened this issue Nov 15, 2017 · 4 comments
Closed

cmake creates too many build artifacts #5009

andrewboie opened this issue Nov 15, 2017 · 4 comments
Assignees
Labels
area: Build System bug The issue is a bug, or the PR is fixing a bug priority: low Low impact/importance bug

Comments

@andrewboie
Copy link
Contributor

As a baseline, any Zephyr build needs to produce a .elf file for the kernel.
Depending on the target hardware requirements, this .elf file may need to be additionally converted into another binary type to support device provisioning.

Examples:

  • Nios II Altera MAX10 needs the kernel converted to IHEX format
  • Many targets (but not the QEMU targets) need the elf stripped and turned into BIN format
  • RiscV32 needs the ELF turned into a S19 format file
  • IIRC, ESP32 and all the QEMU targets take the ELF file directly and don't need additional build artifacts

What CMake seems to be doing is generating all these different output file types unnecessarily for all board targets.

What we need is at a board or SOC level, have some Kconfig which tells the build system what output artifact types are necessary for that board.
For targets like QEMU targets, no additional artifacts need to be created at all; the ELF is the final build product.

The scripts for "make flash" for our boards should provide clues on what particular build artifacts are needed for any given board.

@andrewboie andrewboie added area: Build System bug The issue is a bug, or the PR is fixing a bug labels Nov 15, 2017
@lpereira
Copy link
Collaborator

Just a clarification: ESP32 takes the striped ELF and turns it into a format that's specific to ESP32 (it's the format that the first stage bootloader, part of the SoC mask ROM, understands). The .bin file generated in the output directory is not used by the flasher tool; the flashing script will overwrite the BIN format with this proprietary format prior to flashing.

@SebastianBoe
Copy link
Collaborator

Modifying the build scripts is pretty straightforward.
SebastianBoe@4da2f76

Creating a user-friendly Kconfig is a bit more tricky, I'll give it a go later on unless someone has any ideas themselves. I imagine we might want a menu for the configuring the build system. Perhaps the menu "Compiler Options" could be sub-menu in this "Build system options" menu.

@linkmeyer linkmeyer added the priority: low Low impact/importance bug label Nov 20, 2017
@nashif
Copy link
Member

nashif commented Nov 22, 2017

@SebastianBoe can you submit this as a PR, I am looking into making this configurable using the board yaml files which will be able to specify which output binaries they need and will support. This will be done in a layered approach, so we can specify this for an architecture or a group of boards as well.

@SebastianBoe
Copy link
Collaborator

Done: #5112

nashif added a commit to nashif/zephyr that referenced this issue Nov 22, 2017
Not all boards require the various binary formats zephyr generates. So
be selective based on the arch, SoC or board and only geenrate the
binaries actually needed.

Fixes zephyrproject-rtos#5009

Signed-off-by: Anas Nashif <[email protected]>
nashif added a commit that referenced this issue Nov 23, 2017
Not all boards require the various binary formats zephyr generates. So
be selective based on the arch, SoC or board and only geenrate the
binaries actually needed.

Fixes #5009

Signed-off-by: Anas Nashif <[email protected]>
marc-hb added a commit to marc-hb/zephyr that referenced this issue Nov 4, 2022
Temporary bugs, corner cases and obsolete toolchains aside, the Zephyr
build is reproducible most of the time: zephyrproject-rtos#50205 and zephyrproject-rtos#14593

This means two different build machines using the same toolchain will
always produce the same binary output. The previous, one-line commit made
it trivial to verify that binary outputs are indeed the same by adding this
single line in the buid logs:

```
[16/16] Linking C executable zephyr/zephyr.elf
Memory region         Used Size  Region Size  %age Used
             RAM:       53280 B         3 MB      1.69%
        IDT_LIST:          0 GB         2 KB      0.00%
fdd2ddf2ad7d5da5bbd79b41cef8d7...1a896b989a8281111d8e205  zephyr.strip
```

This commit enables that feature by default because build
reproducibility matters for (at least) two important reasons:

- Security / supply chain attacks, see https://www.cisa.gov/sbom, zephyrproject-rtos#50205,
  https://reproducible-builds.org/ and many others.
- Making sure build configurations are strictly identical when trying to
  reproduce elusive issues or when issuing releases.

It was of course already possible to _manually_ make this Kconfig change
and manually compute this checksum. However this can be impossible when
dealing with an automated build system that does not archive all
_intermediate_ (zephyrproject-rtos#5009) files like `zephyr.elf`. Tweaking the build
configuration can also be difficult and error-prone for people who are
not Zephyr developers.

Most automated CI systems preserve build logs by default.

Displaying the reproducible checksum by default accelerates the
discovery of reproducibility bugs like zephyrproject-rtos#48195.

When measured with `west build -p -b qemu_x86 samples/hello_world/`, the
additional `build/zephyr/zephyr.strip` disk space required is 43
kilobytes compared to a total of 11 Megabytes. Measuring a more
realistic SOF example, `zephyr.strip` weighed 690 kb which was about
0.1% of a total `build/` directory weighing 65M.

To measure the build time cost I ran `west build -p -b qemu_x86
samples/hello_world/` many times in a loop with and without this PR on
my Linux workstation. Stripping and checksumming made literally no time
difference compared to the "noise" observed when building the same
configuration. This is not surprising considering how small
`zephyr.strip`: so the extra cost is most likely dominated by process
creation and the total number of processes created during a Zephyr build
dwarfs the few extra processes required by this feature.

More surprisingly, I measured incremental builds by running `touch
kernel/timer.c; west build ...` in a loop and I could not observe any
visible time difference either.

Signed-off-by: Marc Herbert <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area: Build System bug The issue is a bug, or the PR is fixing a bug priority: low Low impact/importance bug
Projects
None yet
Development

No branches or pull requests

5 participants