LICENSE file in the wheel package references files that are not included in the wheel

[mattip]

🐛 Describe the bug

PR #50745 automatically creates a third_party/LICENSES_BUNDLED.txt which has text like

The Pytorch repository and source distributions bundle several libraries that are 
compatibly licensed.  We list these here.

Name: FP16
License: MIT
Files: third_party/FP16
  For details, see third_party/FP16/LICENSE

Name: FP16-source
License: MIT
Files: third_party/XNNPACK/build/FP16-source
  For details, see third_party/XNNPACK/build/FP16-source/LICENSE

but the third_party directory is not included in the wheel/sdist so the license text is incomplete. This came up in conda-forge/pytorch-cpu-feedstock#121. There should be some file with all the LICENSEs involved.

xref @rgommers

Versions

HEAD and any release since PR #50745

cc @ezyang @seemethere @malfet

[mattip]

Possible solutions could be

• include all the LICENSE files in the wheel/sdist, either by adding them directly to the third_party/LICENSES_BUNDLED.txt or as separate files
• have links to the project licenses

[rgommers]

Probably including all individual files in the .dist-info directory is the way to go, since it's most readable and it is what PEP 639 is going to make the standard soon. setuptools already does this by default, with the license_files keyword to setup() (see docs). We're just not using that in PyTorch yet.

I believe LICENSES_bundled.txt is still very useful as an overview of all included code with different licenses.

Maybe try to add license_files in setup.py and see if it does the right thing out of the box?

[mattip]

Adding license_files to the setup() call in setup.py fails because of pypa/setuptools#2739. It turns out setuptools finds the files, but wheel requires an additional configuration option to add them. Even then, the files cannot be in a heirarchy, as mentioned in the wheel documentation. If it is desirable to keep the files separate, we could rename them as third_party_XNNPACK_build_FP16-source_LICENSE and then add all the names to the wheel via a setup.cfg file.

In light of the complications, perhaps it would be better to concatenate all the licenses into the LICENSE_bundled.txt after the current summary, as

Summary
-------
<current LICENSE_bundled.txt contents>

third_party/FP16/LICENSE
------------------------
<file contents>

third_party/XNNPACK/build/FP16-source/LICENSE
---------------------------------------------
<file contents>

[rgommers]

Always more bugs to run into:(

In light of the complications, perhaps it would be better to concatenate all the licenses into the LICENSE_bundled.txt after the current summary,

I agree

[mattip]

hmm. The test in test/test_license.py to make sure the third_party/LICENCES_BUNDLED.txt is up to date is failing...