This will facilitate the implementation of other configuration formats
(such as pyproject.toml as initially defined by PEP 621)

We can split the process of interpreting configuration files into 2 steps:

1. The parsing the file contents from strings to value objects
   that can be understand by Python (for example a string with a comma
   separated list of keywords into an actual Python list of strings).

2. The expansion (or post-processing) of these values according to the
   semantics ``setuptools`` assign to them (for example a configuration field
   with the ``file:`` directive should be expanded from a list of file paths to
   a single string with the contents of those files concatenated)

The idea of this change is to extract the functions responsible for (2.)
into a new module, so they can be reused between different config file
formats.

… from config to expand

… from config to expand

This eventually will allow reading project metadata directly from
`pyproject.toml`

In order to minimise dependencies, `validate-pyproject` has the ability
to "dump" only the code necessary to run the validations to a given
directory. This special strategy is used instead of the default
`pip install -t`.

The idea of using JSONSchema for validation was suggested in pypa#2671,
and the rationale for that approach is further discussed in
https://github.com/abravalheri/validate-pyproject/blob/main/docs/faq.rst

Using a library such as `validate-pyproject` has the advantage of
incentive sing reuse and collaboration with other projects.

Currently `validate-pyproject` ships a JSONSchema for the proposed
use of `pyproject.toml` as means of configuration for setuptools.
In the future, if there is interest, setuptools could also ship its own
schema and just use the shared infrastructure of `validate-pyproject`
(by advertising the schemas via entry-points).

This is the first step towards making setuptools understand
`pyproject.toml` as a configuration file.

The implementation deliberately allows splitting the act of loading the
configuration from a file in 2 stages: the reading of the file itself
and the expansion of directives (and other derived information).

The user might specify dynamic `entry-points` via a `file:`
directive (a similar feature for `setup.cfg` is documented in
[declarative config]).

The changes introduced here add the ability to expand them
when reading the configuration from `pyproject.toml`.

[declarative config]: https://setuptools.pypa.io/en/latest/userguide/declarative_config.html

The `setuptools.metadata` module contains functions that allows applying
metadata-related configuration read from a `pyproject.toml` file into an
existing `dist` object. It also allows comparing metadata obtained from
different places to check if they are equivalent or not.

The `setuptools.options` module contains functions that allow applying
non-metadata related configuration that comes from a `pyproject.toml`
file into an existing `dist` object. Similarly to `setuptools.metadata`,
comparing options is also allowed.

This change allows comparing and applying command options stored under
the `[tool.distutils.<COMMAND NAME>]` table in `pyproject.toml`.

In pypa#2685 the plan that the community seems to agree on is to always
automatically translate `setup.cfg` into a `pyproject.toml` equivalent
and then proceed reading/parsing the configuration.

The objective of this change is to open space so we can implement this
way of reading `setup.cfg` in a new `setuptools.config.setupcfg` module
while keeping the legacy way of handling `setup.cfg` around.

The rationale behind keeping the legacy way around is that, to avoid
breaking existing packages during a transition period, we can compare
the old and the new way of parsing the configuration (e.g. via
`setuptools.{metadata,options}.compare`) and in the case they are
conflicting, use the old way and emit a warning asking the user to
report the error.

Some `ini2toml` modules are removed, since they are not necessary for
setuptools use case and might require extra dependencies (e.g. CLI
usage, writing TOML files, format preserving).

--

The automatic conversion of `setup.cfg` into `pyproject.toml` as the
*one true way* of doing declarative builds was first suggested in pypa#1688.

--

There are advantages and disadvantages in using an external tool such
as `ini2toml` for the automatic conversion.

The main advantage is that users can use the exact same tool for
converting their old packages and it will work in the same way
setuptools works. This means that on top of doing an automatic
conversion, the users are also offered an alternative to explicitly
update their package configuration.

The main disadvantage is that `ini2toml` works in a way that is a bit
more complex than simply parsing the `setup.cfg` file and dumping it as
`pyproject.toml` (we can think about `ini2toml` as if it was
transforming an "AST-equivalent" representation of the INI and TOML
formats -- this is necessary for other use cases covered by the library).
In order to minimise this complexity, some parts of the
package are stripped out during the vendoring process.

Also note that currently `ini2toml` ships a "built-in" plugin for
setuptools. In the future, if there is interest, there is also the
possibility of moving this plugin directly under setuptools repository.

This is the initial implementation of the "configuration driver"
that indirectly reads `setup.cfg` by first converting it to a data
structure corresponding to `pyproject.toml` and then expanding it.

This idea is based on the approach defined in pypa#2685.

LIMITATION: Differently from the `legacy_setupcfg` "configuration driver",
`setupcfg` does not support reading other distutils file.
The `find_others` flag is removed because of that.

Instead we can used `importlib` to load modules directly from a specific
path.

(This also seems to prevent some order-dependent errors in the tests?)

This make it easier to re-use the `_apply` function.

The `_apply` function can be used to apply a `setup.cfg`
file into an existing dist object, and then obtain the metadata and
options using the `setuptools.{metadata,options}` modules.

This will be useful to display a message to the user
if the configuration resulting from reading `setup.cfg` via
config.legacy_setupcfg or via config.setupcfg differs.

Knowing this diff also facilitates debugging.

Additionally, deprecate the old `setup.cfg`-only API.

…uptools

The `run_setup` function from `distutils.core` allows retrieving the
distribution object before the configuration is read.
Then using the new functions from the `setuptools.config` package
we can apply the TOML-style configuration (either from a
`pyproject.toml` file or by an automatically converted `setup.cfg`).

To ensure backward compatibility, in the case of discrepancies in the
conversion of the `setup.cfg` files, the implementation falls back to
the legacy configuration procedure.

Once we have access to the distribution object, it is no longer
necessary to fully run the setup script.
Instead the backend can have more control over the process.

This change build on top of the `_get_dist` function and the
`distutils.core.run_commands` to trigger commands on the distribution
object.

For the legacy backend the setup script still runs traditionally
(but the corner cases are handled directly by `distutils.core.run_setup`)

Now that the dist object is available for the backend, we can simplify
the workarounds to obtain the `setup_requires` (just reading the
equivalent attribute in the dist object should suffice)

After the changes in build_meta, some checks in the associated test
suite are required.

It seems that some assertions are not really needed, but it is important
to clarify that. For the time being, some skips are added until further
clarification.

The `setuptools.config` module needs at least 3 vendored dependencies to
work, but it seems that this might cause some bootstrapping problems.

This change implements a workaround for that (and add better debugging
messages).

The `tmp_src` fixture copies the setuptools directory to prevent errors
that appear when running tests concurrently. However it seems that is
copying everything including the `.git` directory (and possibly others
like `.tox`). These directories can be quite heavy and error prone to
copy.

The changes introduced here prevent copying these unnecessary
folders/files. As a side effect, the tests should run slightly faster.

There seems to be an opinion in the community that
`include_package_data=True` would be a better default and a quality of
life improvement.

This change relies on the fact that `ini2toml` will automatically
backfill `include_package_data=False` for converted `setup.cfg` files
to implement this change in a backward compatible way.

…nstalls

in _distutils/dist/_parse_command_opts a issubclass check is done
to verify the given command is a subclass of distutils.cmd.Command.
This check will fail if the command is mocked (the mock object is not a
class object and will generate a TypeError).

For some reason it is not easy to replicate the cygwin failures pointed out
by the CI in the local environment, this commit just adds some extra
debugging statements so we can have more information about the problem

As pointed out in
pypa#2924 (comment),
license files are likely to be required for vendored packages.
This change manually adds the missing license files for the dependencies
introduced in the PR 2924.