Windows: Use of strip_include_prefix causes "missing dependency declarations" errors

[mumbleskates]

Description of the problem / feature request:

On Windows, inclusion of any header file from a rule with strip_include_prefix set will cause a "missing dependency declaration" error to be thrown, breaking the build.

Feature requests: what underlying problem are you trying to solve with this feature?

I'm trying to use Bazel as it is documented and as it works in Linux. strip_include_prefix is an essential tool for describing the build processes of third-party repositories that are not natively set up in Bazel (i.e., most of them). The main alternative is to modify every file that includes another file by forking or by rewriting it with genrules, both of which levy a major maintenance burden on the developer on top of already-tedious BUILD-writing.

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Make the following directory tree:

bazel_bug_reproduction
  |-  WORKSPACE
  |-  BUILD
  \-  dir
       |-  a.cc
       |-  a.h
       \-  b.h

WORKSPACE, a.h, b.h:
(empty files)

BUILD:

cc_library(
    name = "a",
    srcs = ["dir/a.cc"],
    hdrs = ["dir/a.h"],
    strip_include_prefix = "dir",
    deps = [
        ":b",
    ],
)

cc_library(
    name = "b",
    hdrs = ["dir/b.h"],
    strip_include_prefix = "dir",
)

a.cc:

#include "a.h"
#include "b.h"

In the directory, run bazel build .... On Linux, this will probably succeed. On Windows, you are likely to receive an error like:

ERROR: O:/docs/repos/bazel-test/BUILD:1:1: undeclared inclusion(s) in rule '//:a':
this rule is missing dependency declarations for the following files included by 'dir/a.cc':
  'dir/a.h'
  'dir/b.h'

What operating system are you running Bazel on?

Windows 10 build 1803

What's the output of bazel info release?

release 0.17.2

Have you found anything relevant by searching the web?

Yes, more issues describing the same thing, that have gone unanswered for a long time: #3828

Any other information, logs, or outputs that you want to share?

This still works fine in Linux by comparison:

mumbles@ubuntu ~/repos $ mkdir -p bazel-test/dir
mumbles@ubuntu ~/repos $ cd bazel-test/
mumbles@ubuntu ~/repos/bazel-test $ touch WORKSPACE
mumbles@ubuntu ~/repos/bazel-test $ cat > BUILD << EOF
> cc_library(
>     name = "a",
>     srcs = ["dir/a.cc"],
>     hdrs = ["dir/a.h"],
>     strip_include_prefix = "dir",
>     deps = [
>         ":b",
>     ],
> )
> 
> cc_library(
>     name = "b",
>     hdrs = ["dir/b.h"],
>     strip_include_prefix = "dir",
> )
> EOF
mumbles@ubuntu ~/repos/bazel-test $ touch dir/a.h
mumbles@ubuntu ~/repos/bazel-test $ touch dir/b.h
mumbles@ubuntu ~/repos/bazel-test $ cat > dir/a.cc << EOF
> #include "a.h"
> #include "b.h"
> EOF
mumbles@ubuntu ~/repos/bazel-test $ ls
BUILD  dir  WORKSPACE
mumbles@ubuntu ~/repos/bazel-test $ bazel build ...
Starting local Bazel server and connecting to it...
INFO: Analysed 2 targets (7 packages loaded).
INFO: Found 2 targets...
INFO: Elapsed time: 3.693s, Critical Path: 0.24s
INFO: 3 processes: 3 linux-sandbox.
INFO: Build completed successfully, 8 total actions
mumbles@ubuntu ~/repos/bazel-test $ bazel info release
release 0.17.2

[Nurdok]

@hlopko this is still an issue with 1.0.0. Can you please take a look? Is there a workaround available? This feature is critical for running on Windows, IMO this should be P1 if there's no workaround.

[skeptic-monkey]

I'am trying to make bazel rules for google/glog compatible with windows and I face the same issue.
This is annoying, as the only workaround I found is to use the includes argument of cc_library. But it kinda pollutes the compilation flags of all projects depending on it, which is not something we should do.

[drigz]

Workaround: add a separate cc_library that doesn't use strip_include_prefix, eg:

    cc_library(
        name = "windows_glog_headers",
        hdrs = glob(["src/windows/glog/*.h"]),
        strip_include_prefix = "src/windows",
        deps = [":strip_include_prefix_hack"],
    )

    # Workaround https://github.com/bazelbuild/bazel/issues/6337 by declaring
    # the dependencies without strip_include_prefix.
    cc_library(
        name = "strip_include_prefix_hack",
        hdrs = glob(["src/windows/*.h"]),
    )

google/glog@1bb0133#diff-71fb65f6a0f5906fd8a4c156e6f77a7dR130

[hlopko]

I'm afraid I won't be able to work on this any time soon. Maybe @meteorcloudy will have a free afternoon? :)

[meteorcloudy]

Hmm. this is another "include" issue that's caused by not having sandbox on Windows. Same root cause as #5640

To summarize, the reason is a.h, b.h, a.cc are all in the same dir. During compiling time, a.h and b.h will be visible without sandbox. So the compiler always finds them first before a.h and b.h in the virtual directory we created to strip the prefix.

You can reproduce the same issue by disabling sandbox on Linux:

pcloudy@pcloudy:~/workspace/my_tests/debug-6337
$ bazel build //...
Starting local Bazel server and connecting to it...
INFO: Analyzed 2 targets (13 packages loaded, 49 targets configured).
INFO: Found 2 targets...
INFO: Elapsed time: 2.713s, Critical Path: 0.22s
INFO: 3 processes: 3 linux-sandbox.
INFO: Build completed successfully, 8 total actions
pcloudy@pcloudy:~/workspace/my_tests/debug-6337
$ bazel clean 
INFO: Starting clean (this may take a while). Consider using --async if the clean takes more than several minutes.
pcloudy@pcloudy:~/workspace/my_tests/debug-6337
$ bazel build //... --spawn_strategy=standalone
INFO: Analyzed 2 targets (13 packages loaded, 49 targets configured).
INFO: Found 2 targets...
ERROR: /usr/local/google/home/pcloudy/workspace/my_tests/debug-6337/BUILD:1:1: undeclared inclusion(s) in rule '//:a':
this rule is missing dependency declarations for the following files included by 'dir/a.cc':
  'dir/a.h'
  'dir/b.h'
INFO: Elapsed time: 0.379s, Critical Path: 0.03s
INFO: 0 processes.
FAILED: Build did NOT complete successfully

There's really no good fix from Bazel side except implementing sandbox on Windows, which is not currently a priority.

But users should be able to workaround by not having the headers they want to strip in the same directory as the source files.

[mumbleskates]

"not having the headers they want to strip in the same directory as the source files" is a poor solution that does not work for many use cases, because much of the time the entire reason for using strip_include_prefixes is because the code is from a third party, such as a submodule.

I'm not sure what makes implementing the entire sandbox easier than detecting this aliasing between the file in . vs the file in the virtual // path, but complexity of the fix doesn't make it less of a blocker for using bazel cross-platform.

[meteorcloudy]

I'm not sure what makes implementing the entire sandbox easier than detecting this aliasing between the file in . vs the file in the virtual // path

It's because even if you don't specify /I., the compiler will still look into the . directory. So we have to remove dir/a.h and dir/b.h to allow compiler to pick headers from the virtual path.

pcloudy@PCLOUDY0-W C:\src\tmp\xrc6ifik\execroot\__main__
$ cl.exe /Ibazel-out/x64_windows-fastbuild/bin /Ibazel-out/x64_windows-fastbuild/bin/_virtual_includes/a /Ibazel-out/x64_windows-fastbuild/bin/_virtual_includes/b /showIncludes /Fobazel-out/x64_windows-fastbuild/bin/_objs/a/a.obj /c dir/a.cc
Microsoft (R) C/C++ Optimizing Compiler Version 19.00.24234.1 for x64
Copyright (C) Microsoft Corporation.  All rights reserved.

a.cc
Note: including file: c:\src\tmp\xrc6ifik\execroot\__main__\dir\a.h
Note: including file: c:\src\tmp\xrc6ifik\execroot\__main__\dir\b.h

pcloudy@PCLOUDY0-W C:\src\tmp\xrc6ifik\execroot\__main__
$ del c:\src\tmp\xrc6ifik\execroot\__main__\dir\a.h

pcloudy@PCLOUDY0-W C:\src\tmp\xrc6ifik\execroot\__main__
$ del c:\src\tmp\xrc6ifik\execroot\__main__\dir\b.h

pcloudy@PCLOUDY0-W C:\src\tmp\xrc6ifik\execroot\__main__
$ cl.exe /Ibazel-out/x64_windows-fastbuild/bin /Ibazel-out/x64_windows-fastbuild/bin/_virtual_includes/a /Ibazel-out/x64_windows-fastbuild/bin/_virtual_includes/b /showIncludes /Fobazel-out/x64_windows-fastbuild/bin/_objs/a/a.obj /c dir/a.cc
Microsoft (R) C/C++ Optimizing Compiler Version 19.00.24234.1 for x64
Copyright (C) Microsoft Corporation.  All rights reserved.

a.cc
Note: including file: bazel-out/x64_windows-fastbuild/bin/_virtual_includes/a\a.h
Note: including file: bazel-out/x64_windows-fastbuild/bin/_virtual_includes/b\b.h

But perhaps we can fix this issue by allowing including headers in the source file's directory.

[jbms]

I can confirm that the workaround suggested by @drigz works well. I've packaged it up as a macro so that you can just use cc_library to cc_library_with_strip_include_prefix in place of cc_library:

https://github.com/google/tensorstore/blob/53ca393768bd9f47937b7a78b33429f3ab1891d5/utils.bzl#L125

# Workaround https://github.com/bazelbuild/bazel/issues/6337 by declaring
# the dependencies without strip_include_prefix.
def cc_library_with_strip_include_prefix(name, hdrs, deps = None, **kwargs):
    strip_include_prefix_name = name + "_strip_include_prefix_hack"
    if deps == None:
        deps = []
    deps = deps + [":" + strip_include_prefix_name]
    native.cc_library(
        name = strip_include_prefix_name,
        hdrs = hdrs,
        visibility = ["//visibility:private"],
    )
    native.cc_library(
        name = name,
        hdrs = hdrs,
        deps = deps,
        **kwargs
    )

However, it would be ideal if Bazel could somehow just make cc_library already have the behavior of this workaround, which presumably wouldn't require any work on a Windows sandbox, etc.

Is there a reason that would be difficult to do?

[mumbleskates]

It's not a complete fix either, iirc, since sometimes the stripped prefix is needed to get the files inside the folder to include each other correctly.

[fmeum-ci]

@scentini Not sure why this didn't happen automatically, but shouldn't e3b7e17 have closed this issue?

[scentini]

Indeed, thanks for flagging. @meteorcloudy would you do the honors?