InteliSeness & Debugging does't work with python namespace splitting

[JeroenSchmidt]

Description of the bug:

Hi all,
The bazel pycharm plugin fails to pickup the correct imports when namespace splitting is used in a python monorepo and where incompatible_default_to_explicit_init_py is set. This impacts both debugging and intelisense.

Suppose I have two python subpackages occupying the base namespace my_package.aaa.bbb and each of these subpackages occupy their own distinct bazel target //lib/packageZ & //lib/packageY

lib
  |- packageZ/
  |        |- BUILD
  |        |- src/my_package/aaa/bbb/zzz/codeZ.py
  |        |- src/tests/...
  |- packageY/
  |        |- BUILD
  |        |- src/my_package/aaa/bbb/yyyy/codeY.py
  |        |- src/tests/...

1) InteliSense Example

In codeY.py you will see the following working:

from my_package.aaa.bbb.zzz import codeZ # inteliSense Works

but this will fail:

from my_package.aaa.bbb import zzz # <- Intelisense error: "Cannot find reference 'zzz' in 'imported module lib.packageZ.src.my_package.aaa.bbb"

2) pytest Debug Example

Suppose I have a test file called test_myfunc.py within lib/packageY/src/tests/.
Running //lib/packageY:test will work. Clicking on the "run icon" on one of test tests in test_myfunc.py will invoke //lib/packageY:test and that will also work. Things fail when a debug run is attempted.

Running debug within the file will produce the following error:

/Users/USER1/.pyenv/versions/pdm-sdk-311/bin/python3 "/Users/USER1/Applications/PyCharm Professional Edition.app/Contents/plugins/python-ce/helpers/pydev/pydevd.py" 
--multiprocess --qt-support=auto --client 127.0.0.1 --port 59342 
--file /private/var/tmp/_bazel_USER1/bdf93a2b33b4c731566cb9fa9e8f36ec/execroot/__main__/bazel-out/darwin_arm64-fastbuild-ST-791d5382ff72/bin/lib/packageY/test \
    lib/packageY/src/tests/test_myfunc.py -vv --cov my_package.aaa.bbb.yyyy --cov-fail-under 100 --cov-report term-missing:skip-covered --no-cov-on-fail

Connected to pydev debugger (build 242.23339.19)
============================= test session starts ==============================
platform darwin -- Python 3.10.14, pytest-8.3.1, pluggy-1.5.0 -- /private/var/tmp/_bazel_USER1/bdf93a2b33b4c731566cb9fa9e8f36ec/execroot/__main__/bazel-out/darwin_arm64-fastbuild-ST-791d5382ff72/bin/lib/packageY/test.runfiles/python_3_10_aarch64-apple-darwin/bin/python3
cachedir: .pytest_cache
rootdir: /
plugins: cov-5.0.0
collecting ... collected 0 items

=============================== warnings summary ===============================
private/var/tmp/_bazel_USER1/bdf93a2b33b4c731566cb9fa9e8f36ec/execroot/__main__/bazel-out/darwin_arm64-fastbuild-ST-791d5382ff72/bin/lib/packageY/test.runfiles/my_python_deps_pytest/site-packages/_pytest/cacheprovider.py:477
  /private/var/tmp/_bazel_USER1/bdf93a2b33b4c731566cb9fa9e8f36ec/execroot/__main__/bazel-out/darwin_arm64-fastbuild-ST-791d5382ff72/bin/lib/packageY/test.runfiles/my_python_deps_pytest/site-packages/_pytest/cacheprovider.py:477: PytestCacheWarning: could not create cache path /.pytest_cache/v/cache/nodeids: [Errno 30] Read-only file system: '/pytest-cache-files-5fvlzawg'
    config.cache.set("cache/nodeids", sorted(self.cached_nodeids))

private/var/tmp/_bazel_USER/bdf93a2b33b4c731566cb9fa9e8f36ec/execroot/__main__/bazel-out/darwin_arm64-fastbuild-ST-791d5382ff72/bin/lib/packageY/_test.runfiles/my_python_deps_pytest/site-packages/_pytest/stepwise.py:51
  /private/var/tmp/_bazel_USER/bdf93a2b33b4c731566cb9fa9e8f36ec/execroot/__main__/bazel-out/darwin_arm64-fastbuild-ST-791d5382ff72/bin/lib/packageY/test.runfiles/my_python_deps_pytest/site-packages/_pytest/stepwise.py:51: PytestCacheWarning: could not create cache path /.pytest_cache/v/cache/stepwise: [Errno 30] Read-only file system: '/pytest-cache-files-cbnw1heh'
    session.config.cache.set(STEPWISE_CACHE_DIR, [])

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
============================= 2 warnings in 0.00s ==============================

ERROR: file or directory not found: `lib/packageY/src/tests/test_myfunc.py`

When lib/packageY/BUILD is something like this:

py_library(
    name = "packageY",
    srcs = glob(["src/**/*.py"]),
    imports = ["src"],
    deps = [
        "//lib/packageZ:packageZ",
    ],
    visibility = ["//visibility:public"],
)

py_test(
    name = "test",
    srcs = glob(["tests/**/*.py"]),
    coverage = "my_package.aaa.bbb.yyyy",
    deps = [
        ":packageY",
    ],
)

Which category does this issue belong to?

PyCharm

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

See the description above.

Which Intellij IDE are you using? Please provide the specific version.

pycharm 2024.2.3 Profesional Edition

What programming languages and tools are you using? Please provide specific versions.

bazel==7.3.1, rules_python==0.36.0

What Bazel plugin version are you using?

2024.09.10.0.1-api-version-242

Have you found anything relevant by searching the web?

I think these items might be related:

• Plugin doesn't recognise python source roots #350
• Correct handling of imports rule attribute on py... rules #6599
• For Python debugging to work, a specific Python toolchain MUST be set #4745

Any other information, logs, or outputs that you want to share?

Toolchain Usage

I saw in an old github issue #4745 that an old version of the plugin struggled with python toolchains. In case it matters, I use a multi_toolchain.

default_python_version = "3.9"

python_register_multi_toolchains(
    name = "python",
    default_version = default_python_version,
    python_versions = [
        "3.9",
        "3.10",
    ],
    register_coverage_tool = True,
)

bazelrc config within my repo

always --enable_bzlmod=false
test --test_output=errors

build --nolegacy_external_runfiles
build --incompatible_default_to_explicit_init_py

common --enable_platform_specific_config
build:macos --action_env BAZEL_DO_NOT_DETECT_CPP_TOOLCHAIN=0

[JeroenSchmidt]

After some more investigation I found the problem. It has nothing to do with namespace splitting. There appears that the Intelij plugin is using the wrong working directory when debug calls a test target that uses python wrapper script that calls pytest inside it. (Identical to the example here provided in this raised issue (comment)

We have a thin python script that wraps pytest, this python script is passed to the main arg within py_test.
The working directory is in the root of the sandbox folder when running it via CLI (default behaivor from bazel) however the working directory within the wrapper script appears to / and not the root of the sandbox; this is why we were getting ERROR: file or directory not found: