How to speed up meta-sca

[gomme600]

In my CI environment (gitlab-ci), I keep the sstate-cache directory and reuse it for multiple builds. The tmp directory is deleted for each build. This causes meta-sca to rerun everytime (and takes almost 4 hours!).

I can't keep the tmp directory because I am building different projects (with a common base). Is there anything similar to the sstate-cache for meta-sca which could be shared across projects ?

Also, is there some kind of whitelist for layers instead of a blacklist ? I am currently using SCA_SPARE_LAYER and I don't want to inherit sca on each layer individually if possible.

Thanks

[priv-kweihmann]

Hmmm, meta-sca should be sstate-safe, so you wouldn't require a tmp directory for the build.
Regarding the allow-list - I think that would be an interesting enhancement, could you please open up a dedicated ticket for that?

In the past I used this layer in a way to always include everything unless explicitly excluded, but I get your point.
BTW does your build without meta-sca fully support the sstate cache? meaning that there won't be a rerun of it when nothing got modified

[gomme600]

I'm pretty sure it does fully support sstate-cache, I will check again.

When rerunning a build without removing tmp it takes 2 minutes.
When removing tmp it does everything meta-sca related again.

[gomme600]

I have opened a ticket for the whitelist here. Thanks for looking into it!

[priv-kweihmann]

@gomme600 it seems you were right, as a few host specific paths slipped into the task hash signatures - although they shouldn't be there - I fixed that with #9492.
Could you please give it a try and report back if sstate caches are now working for you

[gomme600]

Is that included in the latest dunfell build now ?

Edit : Just checked and seen that it is, am trying now !

[gomme600]

For some reason I am getting this error in my CI environment but not on my local build :
Parsing recipes...ERROR: ParseError at /app/gitlab-runner/builds/c89cebdd/1/BS_PAMELA/manifests/repo_project_dir/sources/meta-sca/dynamic-layers/meta-oe/php_%.bbappend:1: unparsed line: 'PACKAGECONFIG:class-native = "mbstring openssl zip"'

Edit : That's the error I was getting here , looks like I need to update my CI environnement.

[gomme600]

I just got this error :

ERROR: base-files-3.0.14-r89 do_sca_get_sources: Execution of '/pamela/pcm-imx6quad-10191/tmp/work/pcm_imx6quad-actia-linux-gnueabi/base-files/3.0.14-r89/temp/run.do_sca_get_sources.5406' failed with exit code 1
ERROR: Logfile of failure stored in: /pamela/pcm-imx6quad-10191/tmp/work/pcm_imx6quad-actia-linux-gnueabi/base-files/3.0.14-r89/temp/log.do_sca_get_sources.5406
Log data follows:
| DEBUG: Executing shell function do_sca_get_sources
| find: ‘/pamela/pcm-imx6quad-10191/tmp/work/pcm_imx6quad-actia-linux-gnueabi/base-files/3.0.14-r89/sstate-build-deploy_source_date_epoch’: No such file or directory
| WARNING: /pamela/pcm-imx6quad-10191/tmp/work/pcm_imx6quad-actia-linux-gnueabi/base-files/3.0.14-r89/temp/run.do_sca_get_sources.5406:1 exit 1 from 'find /pamela/pcm-imx6quad-10191/tmp/work/pcm_imx6quad-actia-linux-gnueabi/base-files/3.0.14-r89 -type f -exec md5sum {} ; >> /pamela/pcm-imx6quad-10191/tmp/work/pcm_imx6quad-actia-linux-gnueabi/base-files/3.0.14-r89/temp/sca_seen_sources.txt'
| ERROR: Execution of '/pamela/pcm-imx6quad-10191/tmp/work/pcm_imx6quad-actia-linux-gnueabi/base-files/3.0.14-r89/temp/run.do_sca_get_sources.5406' failed with exit code 1
NOTE: recipe base-files-3.0.14-r89: task do_sca_get_sources: Failed
ERROR: Task (/pamela/sources/poky/meta/recipes-core/base-files/base-files_3.0.14.bb:do_sca_get_sources) failed with exit code '1'

[gomme600]

And I got this error on my local build :

ERROR: gobject-introspection-1.62.0-r0 do_sca_tracefiles: Error executing a python function in exec_func_python() autogenerated:

The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_func_python() autogenerated', lineno: 2, function:
0001:
*** 0002:do_sca_tracefiles(d)
0003:
File: '/home/slucas/meta-sca/repo3/sources/meta-sca/classes/sca-tracefiles.bbclass', lineno: 82, function: do_sca_tracefiles
0078:
0079:python do_sca_tracefiles() {
0080: import json
0081: with open(d.expand("${SCA_TRACEFILES_LIST}"), "w") as o:
*** 0082: json.dump(sca_get_trace_files(d), o)
0083:}
0084:
0085:addtask do_sca_tracefiles after do_install before do_package
0086:
File: '/home/slucas/meta-sca/repo3/sources/meta-sca/classes/sca-tracefiles.bbclass', lineno: 55, function: sca_get_trace_files
0051: _res = {}
0052: _pkg_files = sca_get_pkg_files(d)
0053: for k, v in _pkg_files.items():
0054: try:
*** 0055: cmd_out = subprocess.check_output(["tracefiles", d.getVar("D"), d.getVar("SCA_SOURCES_DIR")] + v)
0056: except subprocess.CalledProcessError as e:
0057: cmd_out = e.output or ""
0058:
0059: if not isinstance(cmd_out, str):
File: '/usr/lib/python3.6/subprocess.py', lineno: 356, function: check_output
0352: # empty string. That is maintained here for backwards compatibility.
0353: kwargs['input'] = '' if kwargs.get('universal_newlines', False) else b''
0354:
0355: return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
*** 0356: **kwargs).stdout
0357:
0358:
0359:class CompletedProcess(object):
0360: """A process that has finished running.
File: '/usr/lib/python3.6/subprocess.py', lineno: 423, function: run
0419: if 'stdin' in kwargs:
0420: raise ValueError('stdin and input arguments may not both be used.')
0421: kwargs['stdin'] = PIPE
0422:
*** 0423: with Popen(*popenargs, **kwargs) as process:
0424: try:
0425: stdout, stderr = process.communicate(input, timeout=timeout)
0426: except TimeoutExpired:
0427: process.kill()
File: '/usr/lib/python3.6/subprocess.py', lineno: 729, function: init
0725: startupinfo, creationflags, shell,
0726: p2cread, p2cwrite,
0727: c2pread, c2pwrite,
0728: errread, errwrite,
*** 0729: restore_signals, start_new_session)
0730: except:
0731: # Cleanup if the child failed starting.
0732: for f in filter(None, (self.stdin, self.stdout, self.stderr)):
0733: try:
File: '/usr/lib/python3.6/subprocess.py', lineno: 1364, function: _execute_child
1360: if errno_num != 0:
1361: err_msg = os.strerror(errno_num)
1362: if errno_num == errno.ENOENT:
1363: err_msg += ': ' + repr(err_filename)
*** 1364: raise child_exception_type(errno_num, err_msg, err_filename)
1365: raise child_exception_type(err_msg)
1366:
1367:
1368: def _handle_exitstatus(self, sts, _WIFSIGNALED=os.WIFSIGNALED,
Exception: FileNotFoundError: [Errno 2] No such file or directory: 'tracefiles': 'tracefiles'

ERROR: Logfile of failure stored in: /home/slucas/meta-sca/repo3/build-imx_imx-distro_security_sota_test_xwayland/tmp/work/cortexa9t2hf-neon-actia-linux-gnueabi/gobject-introspection/1.62.0-r0/temp/log.do_sca_tracefiles.215
ERROR: Task (/home/slucas/meta-sca/repo3/sources/poky/meta/recipes-gnome/gobject-introspection/gobject-introspection_1.62.0.bb:do_sca_tracefiles) failed with exit code '1'