Add recipe for Perfetto

[quadpixels]

Hello,

This is the rough shape of a BitBake recipe for the Perfetto performance profiler/visulizer. Perfetto already works for a few OSes on a few architectures such as X86, ARM32 and ARM64, and we recently tested it with OpenBMC (which is mostly ARM).

I'm not super fluent with BitBake, and had done a few ugly steps to make the compilation process pass (I have only tested for OpenBMC-based systems:)

I would like to have some comments on:

• I need to execute the tools/install-build-deps script during a Yocto stage that allows network connection
  • Need to make the curl tool available to Bitbake
  • Need to make the CA bundle available to Bitbake so curl can access the HTTPS URL
• Because tools/install-build-deps is run from the Git working directory, I had to override the unpack step too
• Perfetto builds by generating Ninja scripts with gn, and I manually edited the Ninja scripts with a few shell commands, because gn only generates certain combinations (linux/macos/windows/solaris, etc + x86/arm/arm64, etc)
• I overrode the package_qa task with a no-op to silence the GNU_HASH error

Any comments would be greatly appreciated.
Thanks!

[quadpixels]

Modified the recipe a bit so it works with devtool extract and devtool modify -n. The main point is to use ${S} for the configuration step so Bitbake enters the source directory for configuration (it must enter the source directory to configure Perfetto due to the way Perfetto works now).
Also change the binary file from the un-stripped one (which is ~52MB in size) to the stripped one (~1MB in size).

Please take another look. Thanks!

[kraj]

also getting patching errors

ERROR: perfetto-1.0-r0 do_patch: Applying patch '0001-Remove-check_build_deps-build-steps.patch' on target directory '/mnt/b/yoe/master/build/tmp/work/cortexa15t2hf-neon-yoe-linux-gnueabi/perfetto/1.0-r0/perfetto-1.0'
CmdError('quilt --quiltrc /mnt/b/yoe/master/build/tmp/work/cortexa15t2hf-neon-yoe-linux-gnueabi/perfetto/1.0-r0/recipe-sysroot-native/etc/quiltrc push', 0, 'stdout: Applying patch 0001-Remove-check_build_deps-build-steps.patch
can\'t find file to patch at input line 17
Perhaps you used the wrong -p or --strip option?
The text leading up to this was:
--------------------------
|From ec414dc44702c71db3d57d69b27a388bc3426cad Mon Sep 17 00:00:00 2001
|From: Sui Chen <[email protected]>
|Date: Mon, 13 Jun 2022 17:46:49 +0000
|Subject: [PATCH] Remove "check_build_deps" build steps
|
|When building with Bitbake, we want Bitbake to manage our dependencies,
|so we remove the "check_build_deps" steps from Perfetto.
|---
| gn/BUILD.gn            |  1 -
| gn/standalone/BUILD.gn | 12 ++++++------
| 2 files changed, 6 insertions(+), 7 deletions(-)
|
|diff --git a/gn/BUILD.gn b/gn/BUILD.gn
|index 8a7ca72a8..e4a2d39f0 100644
|--- a/gn/BUILD.gn
|+++ b/gn/BUILD.gn
--------------------------
No file to patch.  Skipping patch.
1 out of 1 hunk ignored
can\'t find file to patch at input line 29
Perhaps you used the wrong -p or --strip option?
The text leading up to this was:
--------------------------
|diff --git a/gn/standalone/BUILD.gn b/gn/standalone/BUILD.gn
|index 33fd7aab2..6235044c7 100644
|--- a/gn/standalone/BUILD.gn
|+++ b/gn/standalone/BUILD.gn
--------------------------
No file to patch.  Skipping patch.
1 out of 1 hunk ignored
Patch 0001-Remove-check_build_deps-build-steps.patch does not apply (enforce with -f)

[quadpixels]

Sorry for the confusion about the build error, I didn't check Yocto's source and build folders very closely, and ran into some side-effects of stuck environment variables. I did some more detailed, clean tests for all 8 possible combinations. It seems I need to set both:

1. Not setting S or B, without devtool override -n perfetto /tmp/perfetto:
   B="/opt/build/tmp/work/armv7a-openbmc-linux-gnueabi/perfetto/1.0-r0/perfetto-1.0"
S="/opt/build/tmp/work/armv7a-openbmc-linux-gnueabi/perfetto/1.0-r0/perfetto-1.0"
   --> "patch" task will fail (because source codes are still checked out to ${WORKDIR}/git)

2. Not setting S or B, with override
   B="/tmp/perfetto"
S="/tmp/perfetto"
   --> "configure" task will fail (because Perfetto doesn't allow the build folder to be the same as the source folder)

3. Only setting S = "${WORKDIR}/git", without override
   B="/opt/build/tmp/work/armv7a-openbmc-linux-gnueabi/perfetto/1.0-r0/git"
S="/opt/build/tmp/work/armv7a-openbmc-linux-gnueabi/perfetto/1.0-r0/git"
   --> "configure" task will fail (same as above)

4. Only specifying S, with override
   B="/tmp/perfetto"
S="/tmp/perfetto"
   --> "configure" task will fail (same as above)

5. Only specifying B = "${WORKDIR}/build", without override
   B="/opt/build/tmp/work/armv7a-openbmc-linux-gnueabi/perfetto/1.0-r0/build"
S="/opt/build/tmp/work/armv7a-openbmc-linux-gnueabi/perfetto/1.0-r0/perfetto-1.0"
   --> "patch" task will fail (same as above)

6. Only specifying B, with override
   B="/tmp/perfetto"
S="/tmp/perfetto"
   --> "configure" task will fail (same as above)

7. Specifying both S and B, no override
   B="/opt/build/tmp/work/armv7a-openbmc-linux-gnueabi/perfetto/1.0-r0/build"
S="/opt/build/tmp/work/armv7a-openbmc-linux-gnueabi/perfetto/1.0-r0/git"
   --> works

8. Specifying both S and B, with override
   B="/opt/build/tmp/work/armv7a-openbmc-linux-gnueabi/perfetto/1.0-r0/perfetto-1.0/"
S="/tmp/perfetto"
   --> works

So with both S and B variables set in the recipe, the recipe works both with and without a devtool modify -n override.

[quadpixels]

I updated the script and tested with qemux86 and qemux86-64 targets. Both can generate the build files fine. Other architectures, such as mips, do not appear to be supported by Perfetto yet.
I need to force the is_cross_compiling flag to true, otherwise the build system would attempt to execute Bitbake-generated intermediate tools like protoc. Those generated files generated by Yocto would not work on a regular host due to the version of the C run time. So setting is_cross_compiling to true such that the host-side tools are generated using the native toolchain rather than the Bitbake-generated one.

[quadpixels]

The referenced files have been updated per request.

[quadpixels]

Friendly ping, thanks!

[kraj]

Friendly ping, thanks!

I am busy traveling next week. I will try to get to it this week ( provided my machine recovers )

[quadpixels]

Friendly ping for review --- thanks!

[kraj]

Added Signed-off-By to the patch file. Tested building for qemuarm64 locally. Are there any other issues that need to be resolved? Thanks!

I will give it a go on Autobuilder again.

fails with clang compiler this time

https://errors.yoctoproject.org/Errors/Details/668126/

We should be using OE built clang. It seems to be poking at build host

[quadpixels]

I was able to reproduce the build error by downloading and configuring a Yoe Distribution.

The linker in clang++ generated by the Yoe seems to only look into system directories, while that of the clang++ generated by OpenBMC looks into Yocto-generated library paths:

clang++ (version 15) generated by Yoe:

$ git clone --recurse-submodules -j8 -b master https://github.com/YoeDistro/yoe-distro.git yoe
$ yoe_setup
$ bitbake perfetto -c devshell
$ /home/nitroglycerine/yoe/build/tmp/work/cortexa57-yoe-linux/perfetto/1.0-r0/build# /home/nitroglycerine/yoe/build/tmp/work/cortexa57-yoe-linux/perfetto/1.0-r0/recipe-sysroot-native/usr/bin/clang++ /home/nitroglycerine/yoe/a.cpp -v
...
"/home/nitroglycerine/yoe/build/tmp/hosttools/ld" -pie --hash-style=both --eh-frame-hdr -m elf_x86_64 -dynamic-linker /lib64/ld-linux-x86-64.so.2 -o a.out /usr/lib/x86_64-linux-gnu/Scrt1.o /usr/lib/x86_64-linux-gnu/crti.o /usr/lib/gcc/x86_64-linux-gnu/10/crtbeginS.o -L/usr/lib/gcc/x86_64-linux-gnu/10 -L/lib/x86_64-linux-gnu -L/lib/../lib64 -L/usr/lib/x86_64-linux-gnu -L/lib -L/usr/lib /tmp/a-4acb41.o -lc++ -lm /usr/lib/clang/15.0.0/lib/linux/libclang_rt.builtins-x86_64.a --as-needed -lunwind --no-as-needed -lc /usr/lib/clang/15.0.0/lib/linux/libclang_rt.builtins-x86_64.a --as-needed -lunwind --no-as-needed /usr/lib/gcc/x86_64-linux-gnu/10/crtendS.o /usr/lib/x86_64-linux-gnu/crtn.o

clang++ (version 14) generated by OpenBMC distribution:

$ bitbake perfetto -c devshell
$ arm-openbmc-linux-gnueabi-clang++ /tmp/a.cpp -v
...
"/opt/evb-ast2500/tmp/work/arm1176jzs-openbmc-linux-gnueabi/perfetto/1.0-r0/recipe-sysroot-native/usr/bin/arm-openbmc-linux-gnueabi/arm-openbmc-linux-gnueabi-ld" -pie -EL -z relro -X --hash-style=gnu --eh-frame-hdr -m armelf_linux_eabi -dynamic-linker /lib/ld-linux.so.3 -o a.out Scrt1.o crti.o crtbeginS.o -L/opt/evb-ast2500/tmp/work/arm1176jzs-openbmc-linux-gnueabi/perfetto/1.0-r0/recipe-sysroot-native/usr/bin/../lib/arm-openbmc-linux-gnueabi -L//lib/../lib -L//usr/lib/../lib -L/opt/evb-ast2500/tmp/work/arm1176jzs-openbmc-linux-gnueabi/perfetto/1.0-r0/recipe-sysroot-native/usr/bin/../lib -L//lib -L//usr/lib /tmp/a-18b282.o -lstdc++ -lm -lgcc_s -lgcc -lc -lgcc_s -lgcc crtendS.o crtn.o

For the Yoe distribution, I followed the Yoe tutorial, the only modification I made is to copy meta-openembedded/meta-oe/recipes-devtools/perfetto to ~/yoe/sources/meta-openembedded/meta-oe/recipes-devtools. Is this a close resemblance of the build environment used by the Auto Builder?

[quadpixels]

The above error is with the native clang generated by Yocto. It happens with the clang-15 bundled with the Yoe distribution, and does not appear in the clang-14 bundled with OpenBMC.

A yet simpler way to reproduce the error:

$ cat a.cpp
#include <stdio.h>
int main() { printf("Hey!\n"); }

clang-15 from the Yoe distribution:

$ ~/yoe/build/tmp/work/x86_64-linux/clang-native/15.0.0-r0/build/bin/clang++ -nostdlib++ /tmp/a.cpp 
/usr/bin/ld: cannot find /usr/lib/clang/15.0.0/lib/linux/libclang_rt.builtins-x86_64.a: No such file or directory
/usr/bin/ld: cannot find -lunwind

clang-14 from OpenBMC:

$ /opt/evb-ast2500/tmp/work/x86_64-linux/clang-native/14.0.0-r0/build/bin/clang++ /tmp/a.cpp 
(successfully completes)

Trying to figure out what happened in the native clang build.

===================================

Edit: Adding --rtlib=compiler-rt to clang-14 can also trigger this error.

[quadpixels]

Forcing clang's compiler runtime to "libgcc" seems to fix the problem with the Yoe build.

BB_CLANGXX="${RECIPE_SYSROOT_NATIVE}/usr/bin/clang++ --rtlib=libgcc --unwindlib=libgcc"

Please take another look if this is okay or if there is a better solution. Thanks!

[quadpixels]

Followed the reply in this TMPDIR post and updated the recipe to inherit the clang-native class such that it picks up the BUILD_LDFLAGS specified in the class that sets clang's runtime to gcc, and makes the build pass.

[kraj]

It failed on mips see
https://errors.yoctoproject.org/Errors/Details/672086/

also on musl see
https://errors.yoctoproject.org/Errors/Details/672091/

IMO its best to limit the compatibility to the architectures it supports, its fine to ignore mips.

COMPATIBLE_HOST = "(i.86|x86_64|powerpc|powerpc64|aarch64|arm).*-linux*"`

or further cut this list to down to where it is tested to work.

I think the musl failure seems should be fixed.

[quadpixels]

Added COMPATIBLE_HOST so that only supported platforms are included.
For the musl failure, the cause seems to be:

1. strtoull_l and strtoll_l are not supported in musl.
2. The BUILD_CXX compiler (aarch64-yoe-linux-musl-clang++) is used for certain host-side libraries.

Fixes:

1. Substitute the two with the ones without the _l suffix, namely, strtoull and strtoll. This is done by running sed in the configure step.
2. Use clang++ located in STAGING_DIR_NATIVE.

[quadpixels]

Found a problem myself

/home/.../yoe/build/tmp/work/cortexa15t2hf-neon-yoe-linux-musleabi/perfetto/1.0-r0/recipe-sysroot-native/usr/bin/arm-yoe-linux-mus
leabi/arm-yoe-linux-musleabi-ld.gold: error: obj/src/traced/probes/ftrace/format_parser/format_parser.format_parser.o uses VFP register argumen
ts, output does not 

This happens with qemuarm.

qemuarm64, qemux86, qemux86-64 are fine.

[kraj]

I am seeing following errors when not using meta-clang

Parsing recipes...ERROR: ParseError at /home/pokybuild/yocto-worker/meta-oe/build/meta-openembedded/meta-oe/recipes-devtools/perfetto/perfetto.bb:31: Could not inherit file classes/clang-native.bbclass

I think you should be using dynamic-layers technique to enable it only when when meta-clang is enabled.

See https://github.com/openembedded/meta-openembedded/blob/master/meta-oe/conf/layer.conf#L28

add clang-layer there

and then define the recipe under meta-oe/dynamic-layers/clang-layer/recipes-devtools

[quadpixels]

I following errors when not using meta-clang

Parsing recipes...ERROR: ParseError at /home/pokybuild/yocto-worker/meta-oe/build/meta-openembedded/meta-oe/recipes-devtools/perfetto/perfetto.bb:31: Could not inherit file classes/clang-native.bbclass

Yes, I noted that error as well, when I was building for an OpenBMC platform without meta-clang enabled.
It seems removing inherits clang-native from this recipe fixes this error.

and then define the recipe under meta-oe/dynamic-layers/clang-layer/recipes-devtools

Perfetto supports both building with clang and without clang, and the original purpose of inherit clang-native was to make sure the native clang++ compiler is available, so lines 112~121 of the patch can work normally.
It seems to me when the meta-clang layer is enabled, it is implied that clang-native is present on the system and the native clang++ compiler can be found in ${STAGING_DIR_NATIVE}/usr/bin/clang++, and the above lines would work correctly, thus there it seems we don't need to inherit clang-native.

[kraj]

slowly getting better, there are warnings seen

https://autobuilder.yoctoproject.org/typhoon/#/builders/88/builds/2106

I think you need to specify branch= also in SRS_URI

[quadpixels]

Added branch= to the three repositories: "master" for perfetto, "main" for libcxx, libcxxabi, and libunwind.

There is only one warning left: WARNING: perfetto-1.0-r0 do_package_qa: QA Issue: File /usr/bin/.debug/tracebox in package perfetto-dbg contains reference to TMPDIR [buildpaths].

Trying to find a fix ...

[quadpixels]

Was able to quench the "reference to TMPDIR" warning with INHIBIT_PACKAGE_DEBUG_SPLIT = "1", not sure if this is the best approach though.

[kraj]

Was able to quench the "reference to TMPDIR" warning with INHIBIT_PACKAGE_DEBUG_SPLIT = "1", not sure if this is the best approach though.

I am fine to live with a warning and not silence it like this. Because the root cause is that some path is leaking into compilation and that needs to be fixed

[quadpixels]

Thanks for the suggestions, I removed the INHIBIT_PACKAGE_DEBUG_SPLIT line.

I opened the debug symbol file with "Cutter", it showed 3 strings that contain the TMPDIR. Those strings are in the .debug_str section of the binary. The "Cutter" decompiler does seem to work very well with that debug symbol file though, perhaps because it wasn't a proper executable.

I also tried setting -fcoverage-prefix-map, -fdebug-prefix-map, -ffile-prefix-map and -fmacro-prefix-map to empty values, this did not fix the QA warning.

[kraj]

Search the absolute path of your workspace in the output of strings command that might give some clue.

[kraj]

@quadpixels thanks for sticking with me and addressing all review feedback, it has been accepted into master with 8073ec2