incompatible_applicable_licenses: Add global attribute "applicable_licenses"

[aiuto]

This flag enables a new attribute applicable_licenses to be globally available on any rule. It also enables a package() level attribute default_applicable_licenses.

Flag: --incompatible_applicable_licenses

Motivation: #7444 - we are pulling the existing licenses rule out of Bazel and creating a Starlark based rule set that adds more flexibility. Also see:
License Checking in Bazel

Migration
If your rules define an attribute applicable_licenses, you must rename that.
We expect to flip this flag in Bazel 4.x

[keith]

Is this actively being worked on?

[aiuto]

Yes. I'm trying to prove out working code at Google before releasing things that are I flux. That said, when I am back in the office later this week I can do the paperwork to create the rules-repo repository to hold the macros to.make use of it. The core Bazel changes should be all done, if you want to try with the flag flipped.

…

On Sun, Mar 8, 2020, 1:52 AM Keith Smiley ***@***.***> wrote: Is this actively being worked on? — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#10687?email_source=notifications&email_token=AAXHHHEO7T3VATZW5SLKJZLRGM6D7A5CNFSM4KN6OP52YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEOEOEEA#issuecomment-596173328>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAXHHHAN7FOZ7CM3R55JXPLRGM6D7ANCNFSM4KN6OP5Q> .

[keith]

Great thanks for the update!

[keith]

@aiuto any updated estimate on when the macros repo will be open sourced?

[aiuto]

It is on my plate for April. The Covid response has caused some slowdowns and reprioritization in the last few weeks, so it is hard for me to promise, but that is still my target.

…

On Mon, Mar 30, 2020 at 5:56 PM Keith Smiley ***@***.***> wrote: @aiuto <https://github.com/aiuto> any updated estimate on when the macros repo will be open sourced? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#10687 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAXHHHBIXTMUZT2EXHPRDNTRKEIP3ANCNFSM4KN6OP5Q> .

[ulfjack]

We needed to handle licenses in preparation for our first public release, and since this wasn't ready, I had to design my own license handling infrastructure. These are our requirements so far:

• Generate a list of projects and license URLs.
• Generate a directory or zip of all applicable licenses.
• Also handle notice files from downstream projects.
• Handle compile-time dependencies (both host and target configuration).

The documentation currently primarily documents what your proposed rules look like, but I'm more interested in what the intended interface between Bazel and the licensing rules looks like. From what I've seen so far, I'm not sure the Google rules will work for us, and I expect to continue to use our own.

[aiuto]

Hi Ulf: FYI, I am days away from releasing the work so far to bazelbuild/rules_license. Just waiting for the last launch bit to flip. Once that happens, I can more quickly iterate on the examples there and canonical license kinds.

[aiuto]

It's longish to explain that interface here. I hope the examples in the public repo explain it. But roughly. Someone publishing software can specify it is available under license - with a specific text of their license - and that it is of a well known kind, like GPL2. The well known kinds are based on SPDX identifiers. Every organization that cares about this will mirror in the list of well known identifiers, but they are expected to set their own license *conditions* for what that license identifier implies. We provide canonical tools that - gather transitive license declarations for a top level target - provide the package names, license texts, conditions, copyright notices, ... as a big JSON blob for you to process into things like Android resource files or an audit report - check that the conditions used matches the set of conditions allowed by a license_policy. Each organization will write their own license_policy rules. In the typical case there is at least a policy each for things like mobile_app, internal_server, server_to_delivered_in_binary, private_test_app. So. The rules provide a framework, but each organization must write their own policy.

…

On Wed, Apr 15, 2020 at 8:29 AM Ulf Adams ***@***.***> wrote: We needed to handle licenses in preparation for our first public release, and since this wasn't ready, I had to design my own license handling infrastructure. These are our requirements so far: - Generate a list of projects and license URLs. - Generate a directory or zip of all applicable licenses. - Also handle notice files from downstream projects. - Handle compile-time dependencies (both host and target configuration). The documentation currently primarily documents what your proposed rules look like, but I'm more interested in what the intended interface between Bazel and the licensing rules looks like. From what I've seen so far, I'm not sure the Google rules will work for us, and I expect to continue to use our own. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#10687 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAXHHHHYOZJA7M4SLO2Q2UDRMWSDHANCNFSM4KN6OP5Q> .

[aiuto]

On Wed, Jun 10, 2020 at 2:13 PM Keith Smiley ***@***.***> wrote: @aiuto <https://github.com/aiuto> any update on the release of these rules?

https://github.com/bazelbuild/rules_license This is mostly identical to what Google has internally. We have some extra warts to try to account for our company specific compliance rules. Mind you that is still just a beginning. We have a lot of room to play with ways to write check_license, and if it can be a lightweight starklark rule. Would you want to review some PRs where I try to bring the two versions closer together? —

…

You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#10687 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAXHHHGIORAKPJX742KROJDRV7ELFANCNFSM4KN6OP5Q> . [image: image.gif]

[keith]

After just a brief look I don't completely understand how the pieces fit together, but I'm happy to look at some changes to try and get a better sense

[aiuto]

Have you read the design doc? That gives a lot of background.

…

On Wed, Jun 10, 2020 at 6:55 PM Keith Smiley ***@***.***> wrote: After just a brief look I don't completely understand how the pieces fit together, but I'm happy to look at some changes to try and get a better sense — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#10687 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAXHHHHETO5BZLMSECRI7GLRWAFPZANCNFSM4KN6OP5Q> .

[jheaff1]

Any update on this?

[adam-azarchs]

It isn't clear from this issue, the design doc, or the rules_license repository what the current status is. I think it might be safe to start building stuff around the rules (e.g. our own list of license_kinds) but I'd rather not put a lot of effort into this if the public rules are way out of sync with the internal version so I'd have to overhaul everything for some future update.

Another thing that is unclear from the design document is whether this is ever expected to handle non-vendored third-party dependencies. In particular, what is going to be badly needed to make this useful for people who don't just vendor everything into their monorepo are updates to various repository fetching rules. npm_install would be a particularly important and complex example use case. If there is ever going to be built-in support from the official rules_nodejs for this, I would want to make sure that whatever I build for us internally stays at least mostly compatible with it. Other examples would include e.g. go_respository, pip_install, maven_install, and so on. Obviously can be done (with varying degrees of ease of use and reliability) but not clear whether there's plans to actually do it.

[aiuto]

Sorry for the late response.

On Wed, Aug 18, 2021 at 9:30 PM Adam Azarchs ***@***.***> wrote: It isn't clear from this issue, the design doc, or the rules_license repository what the current status is. I *think* it might be safe to start building stuff around the rules (e.g. our own list of license_kinds) but I'd rather not put a lot of effort into this if the public rules are way out of sync with the internal version so I'd have to overhaul everything for some future update.

The answer is yes, you could build on it. We've been a little slower in rolling it out than I would have liked. We've been trying to work out some issues in Bazel by using Google as a test bed for a large migration. I intend to start updating rules_license with what we learned. The usage is virtually the same as described in the design.

Another thing that is unclear from the design document is whether this is ever expected to handle non-vendored third-party dependencies. In particular, what is going to be badly needed to make this useful for people who don't just vendor everything into their monorepo are updates to various repository fetching rules. npm_install would be a particularly important and complex example use case. If there is ever going to be built-in support from the official rules_nodejs for this, I would want to make sure that whatever I build for us internally stays at least mostly compatible with it. Other examples would include e.g. go_respository, pip_install, maven_install, and so on. Obviously *can* be done (with varying degrees of ease of use and reliability) but not clear whether there's plans to actually do it.

We have that problem in Bazel, I intend to - If the dep has Bazel support already, try to upstream license usage. I expect that will be cost on early adopters and the Bazel team until there is more widespread adoption. - If that is not feasible, or if the module does not have Bazel support, we'll have to splice in the license attributions in the BUILD file. That is slightly easier if there is no upstream BUILD file, because you will have created your own. If it does, they will have to splice together. I don't have a solution for that yet, but it is not that hard. What we have no plans for is automatic license classification based on reading the text and deciding what it means. There are projects that do that, and perhaps someone could work that into repository rules. But that is definitively beyond rules_license's ambitions for the foreseeable future. Like the rest of the Bazel project, we are focused on precise specification and repeatable mechanisms. Autoclassification can return results which are at some confidence level <100%, and worse, change each time the classification corpus changes. I'll let the experts in that domain work on that problem.

…

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#10687 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAXHHHBTQZYLWMKA5G2BOJDT5RNEDANCNFSM4KN6OP5Q> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email> .

[adam-azarchs]

Like the rest of the Bazel project, we are focused on precise specification and repeatable mechanisms. Autoclassification can return results which are at some confidence level <100%, and worse, change each time the classification corpus changes. I'll let the experts in that domain work on that problem.

Sorry, I of course understand that license classification is outside the scope of rules_license per se. If a repository rule fetches a single package covered by a single license (usually the case for e.g. http_archive), splicing in a license specifier into the repo is not unreasonable and I wouldn't really expect an automated solution in those cases.

What I'm asking about is somewhat more limited in scope: for repository rules which are wrapping some other package ecosystem that already has a well-established and adhered-to mechanism for precisely specifying the license for a package (e.g. npm or maven), would there be plans to extend the corresponding repository rules (e.g. npm_install or maven_install) to surface that information, at least in the cases where there was no uncertainty about the license that was involved (that is the license name is an exact match for a SPDX identifier)? This would be particularly important for rules like npm_install where the fetched repository actually consists of potentially several thousand individual packages, each with their own license.

[ulfjack]

Questions:

• Should an alias rule have a license?
• Is there a way to disable inheriting the package-level default_applicable_licenses attribute for license rules? Right now, if you use default_applicable_licenses on a package that contains the license rule that it refers to, it creates a cyclical dependency because license rules also inherit the attribute.

[ulfjack]

Another problem I have is that the collection is unreliable. For example, the example rules only have a small list of propagating attributes, like srcs and deps, which is potentially missing a lot of dependencies, e.g., through resources. Also, if a rule pulls in an output file of another rule directly, the example aspect doesn't propagate to the generating rule (this can be fixed with apply_to_generating_rules = True).

[aiuto]

On Thu, Oct 14, 2021 at 2:56 AM Adam Azarchs ***@***.***> wrote: Like the rest of the Bazel project, we are focused on precise specification and repeatable mechanisms. Autoclassification can return results which are at some confidence level <100%, and worse, change each time the classification corpus changes. I'll let the experts in that domain work on that problem. Sorry, I of course understand that license classification is outside the scope of rules_license *per se*. If a repository rule fetches a single package covered by a single license (usually the case for e.g. http_archive), splicing in a license specifier into the repo is not unreasonable and I wouldn't really expect an automated solution in those cases. What I'm asking about is somewhat more limited in scope: for repository rules which are wrapping some other package ecosystem that already has a well-established and adhered-to mechanism for precisely specifying the license for a package (e.g. npm <https://docs.npmjs.com/cli/v7/configuring-npm/package-json#license> or maven <https://maven.apache.org/pom.html#Licenses>), would there be plans to extend the corresponding repository rules (e.g. npm_install <https://bazelbuild.github.io/rules_nodejs/dependencies.html#using-bazel-managed-dependencies> or maven_install <https://github.com/bazelbuild/rules_jvm_external/blob/master/docs/api.md#maven_install>) to surface that information, at least in the cases where there was no uncertainty about the license that was involved (that is the license name is an exact match for a SPDX identifier)? This would be particularly important for rules like npm_install where the fetched repository actually consists of potentially several thousand individual packages, each with their own license.

That is a fine plan. If we can match the SPDX ID, we can certainly auto-assign. The only issue is staffing to build the glue.

[aiuto]

On Mon, Oct 25, 2021 at 6:23 PM Ulf Adams ***@***.***> wrote: Questions: - Should an alias rule have a license? Off the top of my head, probably not. It should inherit that of the

target. My hunch is that you are asking that because alias is not behaving the expected way w.r.t. license. I'm not surprised, so many things are broken around alias.

- Is there a way to disable inheriting the package-level default_applicable_licenses attribute for license rules? Right now, if you use default_applicable_licenses on a package that contains the license rule that it refers to, it creates a cyclical dependency because license rules also inherit the attribute. That sounds broken. I'll have to look into that, it works right in Blaze.

If you have a reproduction please file an issue.

…

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#10687 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAXHHHG4627V6GFPEEPNHKLUIXKG7ANCNFSM4KN6OP5Q> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[aiuto]

On Mon, Oct 25, 2021 at 7:04 PM Ulf Adams ***@***.***> wrote: Another problem I have is that the collection is unreliable. For example, the example rules only have a small list of propagating attributes, like srcs and deps, which is potentially missing a lot of dependencies, e.g., through resources. Also, if a rule pulls in an output file of another rule directly, the example aspect doesn't propagate to the generating rule (this can be fixed with apply_to_generating_rules = True).

That is a start. Internally, we have a huge aspect which does some important pruning. I want to sanitize that and export the good parts to rules_license.

[ulfjack]

Also, there are different use cases requiring different attributes for licenses. In many cases, it's sufficient to know the type of license (like bsd-3 or apache-2), but we also have cases where we have to generate a 'software bill of materials'-style list with full license files, notices, and URLs. We are able to automatically (using heuristics) extract this information from npm sources.

[ulfjack]

The reason why we get the cyclical dependencies is that our license rule doesn't have a license_kinds attribute, as required by Bazel (and not documented anywhere?). See RuleClass.java:

  public boolean isBazelLicense() {
    return name.equals("_license") && hasAttr("license_kinds", BuildType.LABEL_LIST);
  }

[ulfjack]

It is apparently documented, so my fault for not seeing that earlier.

[ulfjack]

And I'm not able to get it to work. The https://github.com/bazelbuild/rules_license/blob/main/rules/license.bzl#L100 uses a macro to explicitly override applicable_licenses with an empty list, and that works, but I wasn't able to get it to work just by naming the license rule _license and adding the license_kinds attribute.

[adam-azarchs]

Somewhat related to the propagation of the aspect across attributes is that there are different kinds of dependencies with different license propagation requirements. For example, cmake has a license, and you'd want to know about that license if you ended up packaging parts of cmake in your final artifacts, but if you were just using it for an invocation of rules_foreign_cc then you (probably) wouldn't need to propagate its license into the output of that invocation. The license for Intel MKL is another important weird case (I'm not a lawyer, so I'm not going to say anything about why). It would be helpful if at least some of that nuance could be captured in the rules, though of course be expected to cover every scenario like that (for example g++, where its license should propagate if you use -static-libstdc++ but not if you don't).

[fmeum]

@aiuto I tried to adopt rules_license in a ruleset, but encountered two blockers that I listed in bazelbuild/rules_license#9 that are fixed by bazelbuild/rules_license#7 and bazelbuild/rules_license#8. If these were fixed, rulesets could start to use rules_license to add license annotations.

[adam-azarchs]

According to the design doc, it's expected that every organization will fork rules_license. However, generally-applicable changes like the above would be nice to get into the "official" version to keep things from diverging too far. Certainly the private visibility for the SPDX license set makes them useless (since there's nothing else in that package). For license rules in I think the vast majority of cases you'd want the rule to be visible to at least sub-packages - there's a decent argument to be made for the macro to set the visibility to :__subpackages__ by default.

Related to that, as I've been rolling these rules out I've found it would be convenient if default_applicable_licenses were available on workspace as well as package, since in a majority of cases (outside of Google, anyway), a bazel workspace corresponds to a git repository, which usually has a single license file in the root which applies to everything in the repo (possibly with a few exceptions here and there). I do however recognize that that might be tricky to get working because as far as I can tell the content of WORKSPACE files is more or less ignored for external repositories (though maybe bzlmod will be changing the situation there? It would be nice if bzlmod had some affordances for doing that, though last I heard that wasn't part of the plan).

[fmeum]

According to the design doc, it's expected that every organization will fork rules_license.

This point in the doc might warrant further clarification. As I understand it, many organizations will fork or vendor rules_license. However, rulesets and other open-source dependencies would have to be able to refer to a common set of license_kind definitions.

I do however recognize that that might be tricky to get working because as far as I can tell the content of WORKSPACE files is more or less ignored for external repositories (though maybe bzlmod will be changing the situation there? It would be nice if bzlmod had some affordances for doing that, though last I heard that wasn't part of the plan).

I think that bzlmod is flexible enough to support this use case as is: If rules_license defines appropriate module extensions, modules could use those to declare their default licenses and rules_license would be able to associate them with the modules as it will have visibility into the entire module dependency graph.

[aiuto]

I had forgotten this issue, since it was about the flag to add the feature. The feature was added and the flag removed long ago, so I am going to close this issue. Let's discuss more of what rules_license does in that repository rather than this thread.

To answer some of the points from above.

• many of the issues/PR mentioned are now done
• I am working on a unified mechanism to splice license and package information from upstream repos like Maven into the BUILD graph. I have a working prototype, but bzlmod breaks it, so more engineering is needed.
• SBOM work is changing the whole face.