Python exception when running buildstream twice in the same command

[johngavingraham]

When I run bst show to show a diff between two projects (e.g. to do diff <(bst show ...) <(bst show ...)) I sometimes get exceptions - the stack trace of one is below.

I was using -C to switch to a different buildstream project, if that makes any difference.

I installed recently in a venv, through pip:

$ pip show buildstream
Name: BuildStream
Version: 2.3.0
Summary: A framework for modelling build pipelines in YAML
Home-page: https://buildstream.build
Author: The Apache Software Foundation
Author-email: [email protected]
License: Apache License Version 2.0
Location: /home/john/src/project/.venv/lib64/python3.12/site-packages
Requires: Click, grpcio, Jinja2, pluginbase, protobuf, psutil, pyroaring, ruamel.yaml, ruamel.yaml.clib, setuptools, ujson
Required-by: 

Traceback (modified only to hide project name):

Traceback (most recent call last):
  File "/home/john/src/project/.venv/bin/bst", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/home/john/src/project/.venv/lib64/python3.12/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/john/src/project/.venv/lib64/python3.12/site-packages/buildstream/_frontend/cli.py", line 272, in override_main
    original_main(self, args=args, prog_name=prog_name, complete_var=None, standalone_mode=standalone_mode, **extra)
  File "/home/john/src/project/.venv/lib64/python3.12/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/john/src/project/.venv/lib64/python3.12/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/john/src/project/.venv/lib64/python3.12/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/john/src/project/.venv/lib64/python3.12/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/john/src/project/.venv/lib64/python3.12/site-packages/click/decorators.py", line 45, in new_func
    return f(get_current_context().obj, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/john/src/project/.venv/lib64/python3.12/site-packages/buildstream/_frontend/cli.py", line 631, in show
    with app.initialized():
         ^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/home/john/src/project/.venv/lib64/python3.12/site-packages/buildstream/_frontend/app.py", line 250, in initialized
    self.context.artifactcache.preflight()
    ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/john/src/project/.venv/lib64/python3.12/site-packages/buildstream/_context.py", line 482, in artifactcache
    self._artifactcache = ArtifactCache(self)
                          ^^^^^^^^^^^^^^^^^^^
  File "/home/john/src/project/.venv/lib64/python3.12/site-packages/buildstream/_artifactcache.py", line 36, in __init__
    super().__init__(context)
  File "/home/john/src/project/.venv/lib64/python3.12/site-packages/buildstream/_assetcache.py", line 306, in __init__
    self.cas: CASCache = context.get_cascache()
                         ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/john/src/project/.venv/lib64/python3.12/site-packages/buildstream/_context.py", line 709, in get_cascache
    self._cascache = CASCache(self.cachedir, casd=self.get_casd(), remote_cache=bool(self.remote_cache_spec))
                                                  ^^^^^^^^^^^^^^^
  File "/home/john/src/project/.venv/lib64/python3.12/site-packages/buildstream/_context.py", line 696, in get_casd
    self._casd = CASDProcessManager(
                 ^^^^^^^^^^^^^^^^^^^
  File "/home/john/src/project/.venv/lib64/python3.12/site-packages/buildstream/_cas/casdprocessmanager.py", line 109, in __init__
    self._logfile = self._rotate_and_get_next_logfile()
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/john/src/project/.venv/lib64/python3.12/site-packages/buildstream/_cas/casdprocessmanager.py", line 277, in _rotate_and_get_next_logfile
    os.remove(os.path.join(self._log_dir, logfile_to_delete))
FileNotFoundError: [Errno 2] No such file or directory: '/home/john/.cache/buildstream/logs/_casd/1727856548.523361.log'

[abderrahim]

diff <(bst show ...) <(bst show ...)

This may be running two buildstream instances in parallel, starting at the same time, which is likely the cause of the error.

buildstream doesn't support having more than one instance at a time, even though it mostly works. But such a race is probably worse.