Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[STABLE v2.4] cherry-pick build reproducibility fixes #6944

Closed
wants to merge 5 commits into from

Conversation

marc-hb
Copy link
Collaborator

@marc-hb marc-hb commented Jan 12, 2023

Only clean git cherry-pick -x of well tested commits from main.

Better late than never? For 2.4.2...

Windows fixes are omitted.

marc-hb and others added 5 commits January 11, 2023 17:05
This reverts commit ce28e09.

This fixes Zephyr's git describe command and build reproducibility.

I tried fairly hard various git fetch options like --shallow-exclude
and --shallow-since but they did not save that much download (200MB at
best), required some hardcoding and most importantly they make complete
clones shallow again when invoked unconditionally. Not worth the
effort, build reproducibility is more important.

Signed-off-by: Marc Herbert <[email protected]>
(cherry picked from commit 09386bc)
Make stripped ELF files compiled by XCC identical across different
machines.

Some Xtensa compilers (ab?)use the .ident / .comment
section and append the typically absolute and not
reproducible /path/to/the.c file after the usual
compiler ID.

https://sourceware.org/binutils/docs/as/Ident.html

strip --strip-all does not remove the .comment section.
Remove ourselves like some gcc test scripts do:

 https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=c7046906c3ae

Signed-off-by: Marc Herbert <[email protected]>
(cherry picked from commit 501204c)
Make sure any forward slashes in strings are immediately converted to
OS-dependent directory separators.

Signed-off-by: Marc Herbert <[email protected]>
(cherry picked from commit bc39491)
.config is not deterministic because it has absolute paths in comments
and its order seems hard to predict. configs.c has the same information
generated in a determistic way.

Signed-off-by: Marc Herbert <[email protected]>
(cherry picked from commit 906508b)
During testing of Windows CI workflow on windows-latest turned out
that github action console has issues rendering python
anytree.RenderTree graph with default style of
anytree.render.ContStyle (continuous vertical and horizontal lines).
Tests on local machines in cmd, pwershell and msys shells
proves, that all shells can render the graph (while powershell
does not do this correctly and graph is malformed).
Changing graph style to ASCII resolves the issue with Github actions.

Signed-off-by: Andrey Borisovich <[email protected]>
(cherry picked from commit b34cb8b)
@marc-hb marc-hb changed the title [STABLE v2.4] cherry-pick build reproducibility [STABLE v2.4] cherry-pick build reproducibility fixes Jan 12, 2023
@marc-hb
Copy link
Collaborator Author

marc-hb commented Jan 12, 2023

@wszypelt another empty failure in QB build/11213128:

17:28:17,070 INFO  - Executing post-execute action...
17:28:17,070 ERROR - Step 'master>SCANS>TESTS>TRIGGER PYTHON SMOKE TESTS' is failed: Step is failed since the triggered build is failed, cancelled, or timed out.

@marc-hb marc-hb marked this pull request as ready for review January 12, 2023 01:44
@marc-hb marc-hb requested a review from aborisovich as a code owner January 12, 2023 01:44
@marc-hb
Copy link
Collaborator Author

marc-hb commented Jan 12, 2023

One unrelated alsabat failure and 2 unavailable models in https://sof-ci.01.org/sofpr/PR6944/build3343/devicetest/index.html . Totally unrelated

@marc-hb marc-hb requested a review from lgirdwood January 18, 2023 04:59
@lgirdwood
Copy link
Member

@marc-hb we are only going to do a v2.4.2 if there are critical runtime errors. This is still a opt-in release on Intel platforms.

@marc-hb
Copy link
Collaborator Author

marc-hb commented Jan 19, 2023

If a 2.4.2 ever happens then these zero-risk, pure build fixes could be very useful, especially if 2.4.2 is "critical" and with more important and/or urgent things to focus on. They only affect the quantity and quality of the information in the build logs, they don't change the actual firmware.

If 2.4.2 never happens then these will obviously not make any difference :-)

marc-hb added a commit to marc-hb/sof-bin that referenced this pull request Jan 24, 2023
No ADL (RPL) signatures.

intel-signed binaries from Liam, 2.4.1 2023WW03 archive, internal
build ID 11237573.

Community binaries uploaded by Mengdong to the v2.4.1/ folder from
storage folder v2.4.1/e047ccd992ab_release-2/. Test run ID 20002,
build ID sof_generic_build_template/492

Tests currently failing:

- ERROR:root:MISMATCH:	sofz-tgl.ri	different
   in v2.4.x/sof-v2.4.1/intel-signed vs  v2.4.x/sof-v2.4.1/community

These can't be matched yet because
thesofproject/sof#6944 did not make v2.4.1

Signed-off-by: Marc Herbert <[email protected]>
lgirdwood pushed a commit to thesofproject/sof-bin that referenced this pull request Jan 24, 2023
No ADL (RPL) signatures.

intel-signed binaries from Liam, 2.4.1 2023WW03 archive, internal
build ID 11237573.

Community binaries uploaded by Mengdong to the v2.4.1/ folder from
storage folder v2.4.1/e047ccd992ab_release-2/. Test run ID 20002,
build ID sof_generic_build_template/492

Tests currently failing:

- ERROR:root:MISMATCH:	sofz-tgl.ri	different
   in v2.4.x/sof-v2.4.1/intel-signed vs  v2.4.x/sof-v2.4.1/community

These can't be matched yet because
thesofproject/sof#6944 did not make v2.4.1

Signed-off-by: Marc Herbert <[email protected]>
@marc-hb
Copy link
Collaborator Author

marc-hb commented Mar 2, 2023

The 2017 XCC toolchain used for TGL has a bug where the length of west topdir affects the object code:

@marc-hb marc-hb closed this Mar 2, 2023
@marc-hb marc-hb deleted the deterministic-2.4 branch March 24, 2023 19:32
@marc-hb marc-hb restored the deterministic-2.4 branch March 24, 2023 23:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants