Remove [project] section from `pyproject.toml` #34014

Taragolis · 2023-09-01T14:44:28Z

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

potiuk · 2023-09-01T14:47:30Z

Very match that

Taragolis · 2023-09-01T15:22:36Z

I suspect that we did not enable this field before due to this 🤣

Processing /dist/apache_airflow-2.8.0.dev0-py3-none-any.whl
ERROR: Cannot install apache-airflow 2.8.0.dev0 (from /dist/apache_airflow-2.8.0.dev0-py3-none-any.whl) and apache-airflow==2.4.0 because these package versions have conflicting dependencies.

The conflict is caused by:
    The user requested apache-airflow 2.8.0.dev0 (from /dist/apache_airflow-2.8.0.dev0-py3-none-any.whl)
    The user requested apache-airflow==2.4.0

Taragolis · 2023-09-02T14:32:06Z

My investigation.

After I add version and name into the pyproject.toml it break our breeze release-management prepare-provider-packages --version-suffix-for-pypi dev0 --package-format wheel and maybe other, in this case we only build apache-airflow package which include provider content within it

And also we have extra warnings / errors (under the spoiler)

########## Prepare provider package for 'airbyte' ##########
Changing directory to /opt/airflow/provider_packages
Building provider package: airbyte in format wheel
Executing command: 'python3 setup.py build --build-temp /tmp/tmpu10af4vn egg_info --tag-build dev0 bdist_wheel --bdist-dir /tmp/tmpod49imhc'
/usr/local/lib/python3.8/site-packages/setuptools/config/_apply_pyprojecttoml.py:103: _WouldIgnoreField: 'description' defined outside of `pyproject.toml` would be ignored.
    !!


    ##########################################################################
    # configuration would be ignored/result in error due to `pyproject.toml` #
    ##########################################################################

    The following seems to be defined outside of `pyproject.toml`:

    `description = 'Provider for Apache Airflow. Implements apache-airflow-providers-influxdb package'`

    According to the spec (see the link below), however, setuptools CANNOT
    consider this value unless 'description' is listed as `dynamic`.

    https://packaging.python.org/en/latest/specifications/declaring-project-metadata/

    For the time being, `setuptools` will still consider the given value (as a
    **transitional** measure), but please note that future releases of setuptools will
    follow strictly the standard.

    To prevent this warning, you can list 'description' under `dynamic` or alternatively
    remove the `[project]` table from your file and rely entirely on other means of
    configuration.
    

!!
    
  warnings.warn(msg, _WouldIgnoreField)
/usr/local/lib/python3.8/site-packages/setuptools/config/_apply_pyprojecttoml.py:103: _WouldIgnoreField: 'readme' defined outside of `pyproject.toml` would be ignored.
    !!


    ##########################################################################
    # configuration would be ignored/result in error due to `pyproject.toml` #
    ##########################################################################

    The following seems to be defined outside of `pyproject.toml`:

    `readme = '\n.. Licensed to the Apache Software Foundation (ASF) under one\n   or more contributor license agreements.  See the NOTICE file\n   distributed with this work for additional information\n   regarding copyright ownership.  The ASF licenses this file\n   to you under the Apache License, Version 2.0 (the\n   "License"); you may not use this file except in compliance\n   with the License.  You may obtain a copy of the License at\n\n..   http://www.apache.org/licenses/LICENSE-2.0\n\n.. Unless required by applicable law or agreed to in writing,\n   software distributed under the License is distributed on an\n   "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n   KIND, either express or implied.  See the License for the\n   specific language governing permissions and limitations\n   under the License.\n\n .. Licensed to the Apache Software Foundation (ASF) under one\n    or more contributor license agreements.  See the NOTICE file\n    distributed with this work for additional information\n    regarding copyright ownership.  The ASF licenses this file\n    to you under the Apache License, Version 2.0 (the\n    "License"); you may not use this file except in compliance\n    with the License.  You may obtain a copy of the License at\n\n ..   http://www.apache.org/licenses/LICENSE-2.0\n\n .. Unless required by applicable law or agreed to in writing,\n    software distributed under the License is distributed on an\n    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n    KIND, either express or implied.  See the License for the\n    specific language governing permissions and limitations\n    under the License.\n\n\nPackage ``apache-airflow-providers-influxdb``\n\nRelease: ``2.2.2dev0``\n\n\n`InfluxDB <https://www.influxdata.com/>`__\n\n\nProvider package\n----------------\n\nThis is a provider package for ``influxdb`` provider. All classes for this provider package\nare in ``airflow.providers.influxdb`` python package.\n\nYou can find package information and changelog for the provider\nin the `documentation <https://airflow.apache.org/docs/apache-airflow-providers-influxdb/2.2.2/>`_.\n\n\nInstallation\n------------\n\nYou can install this package on top of an existing Airflow 2 installation (see ``Requirements`` below\nfor the minimum Airflow version supported) via\n``pip install apache-airflow-providers-influxdb``\n\nThe package supports the following python versions: 3.8,3.9,3.10,3.11\n\nRequirements\n------------\n\n===================  ==================\nPIP package          Version required\n===================  ==================\n``apache-airflow``   ``>=2.4.0``\n``influxdb-client``  ``>=1.19.0``\n``requests``         ``>=2.26.0``\n===================  ==================\n\nThe changelog for the provider package can be found in the\n`changelog <https://airflow.apache.org/docs/apache-airflow-providers-influxdb/2.2.2/changelog.html>`_.'`

    According to the spec (see the link below), however, setuptools CANNOT
    consider this value unless 'readme' is listed as `dynamic`.

    https://packaging.python.org/en/latest/specifications/declaring-project-metadata/

    For the time being, `setuptools` will still consider the given value (as a
    **transitional** measure), but please note that future releases of setuptools will
    follow strictly the standard.

    To prevent this warning, you can list 'readme' under `dynamic` or alternatively
    remove the `[project]` table from your file and rely entirely on other means of
    configuration.
    

!!
    
  warnings.warn(msg, _WouldIgnoreField)
/usr/local/lib/python3.8/site-packages/setuptools/config/_apply_pyprojecttoml.py:103: _WouldIgnoreField: 'license' defined outside of `pyproject.toml` would be ignored.
    !!


    ##########################################################################
    # configuration would be ignored/result in error due to `pyproject.toml` #
    ##########################################################################

    The following seems to be defined outside of `pyproject.toml`:

    `license = 'Apache License 2.0'`

    According to the spec (see the link below), however, setuptools CANNOT
    consider this value unless 'license' is listed as `dynamic`.

    https://packaging.python.org/en/latest/specifications/declaring-project-metadata/

    For the time being, `setuptools` will still consider the given value (as a
    **transitional** measure), but please note that future releases of setuptools will
    follow strictly the standard.

    To prevent this warning, you can list 'license' under `dynamic` or alternatively
    remove the `[project]` table from your file and rely entirely on other means of
    configuration.
    

!!
    
  warnings.warn(msg, _WouldIgnoreField)
/usr/local/lib/python3.8/site-packages/setuptools/config/_apply_pyprojecttoml.py:103: _WouldIgnoreField: 'authors' defined outside of `pyproject.toml` would be ignored.
    !!


    ##########################################################################
    # configuration would be ignored/result in error due to `pyproject.toml` #
    ##########################################################################

    The following seems to be defined outside of `pyproject.toml`:

    `authors = 'Apache Software Foundation'`

    According to the spec (see the link below), however, setuptools CANNOT
    consider this value unless 'authors' is listed as `dynamic`.

    https://packaging.python.org/en/latest/specifications/declaring-project-metadata/

    For the time being, `setuptools` will still consider the given value (as a
    **transitional** measure), but please note that future releases of setuptools will
    follow strictly the standard.

    To prevent this warning, you can list 'authors' under `dynamic` or alternatively
    remove the `[project]` table from your file and rely entirely on other means of
    configuration.
    

!!
    
  warnings.warn(msg, _WouldIgnoreField)
/usr/local/lib/python3.8/site-packages/setuptools/config/_apply_pyprojecttoml.py:103: _WouldIgnoreField: 'classifiers' defined outside of `pyproject.toml` would be ignored.
    !!


    ##########################################################################
    # configuration would be ignored/result in error due to `pyproject.toml` #
    ##########################################################################

    The following seems to be defined outside of `pyproject.toml`:

    `classifiers = ['Development Status :: 5 - Production/Stable', 'Environment :: Console', 'Environment :: Web Environment', 'Intended Audience :: Developers', 'Intended Audience :: System Administrators', 'Framework :: Apache Airflow', 'Framework :: Apache Airflow :: Provider', 'License :: OSI Approved :: Apache Software License', 'Programming Language :: Python :: 3.8', 'Programming Language :: Python :: 3.9', 'Programming Language :: Python :: 3.10', 'Programming Language :: Python :: 3.11', 'Topic :: System :: Monitoring']`

    According to the spec (see the link below), however, setuptools CANNOT
    consider this value unless 'classifiers' is listed as `dynamic`.

    https://packaging.python.org/en/latest/specifications/declaring-project-metadata/

    For the time being, `setuptools` will still consider the given value (as a
    **transitional** measure), but please note that future releases of setuptools will
    follow strictly the standard.

    To prevent this warning, you can list 'classifiers' under `dynamic` or alternatively
    remove the `[project]` table from your file and rely entirely on other means of
    configuration.
    

!!
    
  warnings.warn(msg, _WouldIgnoreField)
/usr/local/lib/python3.8/site-packages/setuptools/config/_apply_pyprojecttoml.py:103: _WouldIgnoreField: 'urls' defined outside of `pyproject.toml` would be ignored.
    !!


    ##########################################################################
    # configuration would be ignored/result in error due to `pyproject.toml` #
    ##########################################################################

    The following seems to be defined outside of `pyproject.toml`:

    `urls = {'Documentation': 'https://airflow.apache.org/docs/apache-airflow-providers-influxdb/2.2.2/', 'Changelog': 'https://airflow.apache.org/docs/apache-airflow-providers-influxdb/2.2.2/changelog.html', 'Bug Tracker': 'https://github.com/apache/airflow/issues', 'Source Code': 'https://github.com/apache/airflow', 'Slack Chat': 'https://s.apache.org/airflow-slack', 'Twitter': 'https://twitter.com/ApacheAirflow', 'YouTube': 'https://www.youtube.com/channel/UCSXwxpWZQ7XZ1WL3wqevChA/'}`

    According to the spec (see the link below), however, setuptools CANNOT
    consider this value unless 'urls' is listed as `dynamic`.

    https://packaging.python.org/en/latest/specifications/declaring-project-metadata/

    For the time being, `setuptools` will still consider the given value (as a
    **transitional** measure), but please note that future releases of setuptools will
    follow strictly the standard.

    To prevent this warning, you can list 'urls' under `dynamic` or alternatively
    remove the `[project]` table from your file and rely entirely on other means of
    configuration.
    

!!
    
  warnings.warn(msg, _WouldIgnoreField)
/usr/local/lib/python3.8/site-packages/setuptools/config/_apply_pyprojecttoml.py:103: _WouldIgnoreField: 'entry-points' defined outside of `pyproject.toml` would be ignored.
    !!


    ##########################################################################
    # configuration would be ignored/result in error due to `pyproject.toml` #
    ##########################################################################

    The following seems to be defined outside of `pyproject.toml`:

    `entry-points = {'apache_airflow_provider': ['provider_info=airflow.providers.influxdb.get_provider_info:get_provider_info']}`

    According to the spec (see the link below), however, setuptools CANNOT
    consider this value unless 'entry-points' is listed as `dynamic`.

    https://packaging.python.org/en/latest/specifications/declaring-project-metadata/

    For the time being, `setuptools` will still consider the given value (as a
    **transitional** measure), but please note that future releases of setuptools will
    follow strictly the standard.

    To prevent this warning, you can list 'entry-points' under `dynamic` or alternatively
    remove the `[project]` table from your file and rely entirely on other means of
    configuration.
    

!!
    
  warnings.warn(msg, _WouldIgnoreField)
/usr/local/lib/python3.8/site-packages/setuptools/config/_apply_pyprojecttoml.py:103: _WouldIgnoreField: 'optional-dependencies' defined outside of `pyproject.toml` would be ignored.
    !!


    ##########################################################################
    # configuration would be ignored/result in error due to `pyproject.toml` #
    ##########################################################################

    The following seems to be defined outside of `pyproject.toml`:

    `optional-dependencies = {'http': ['apache-airflow-providers-http']}`

    According to the spec (see the link below), however, setuptools CANNOT
    consider this value unless 'optional-dependencies' is listed as `dynamic`.

    https://packaging.python.org/en/latest/specifications/declaring-project-metadata/

    For the time being, `setuptools` will still consider the given value (as a
    **transitional** measure), but please note that future releases of setuptools will
    follow strictly the standard.

    To prevent this warning, you can list 'optional-dependencies' under `dynamic` or alternatively
    remove the `[project]` table from your file and rely entirely on other means of
    configuration.
    

!!
    
  warnings.warn(msg, _WouldIgnoreField)
warning: no files found matching 'CHANGELOG.rst'
warning: no previously-included files matching '__pycache__' found anywhere in distribution
warning: no previously-included files matching '*.pyc' found anywhere in distribution
warning: build_py: byte-compiling is disabled, skipping.

/usr/local/lib/python3.8/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
warning: install_lib: 'build/lib' does not exist -- no Python modules to install

error: [Errno 2] No such file or directory: 'apache_airflow.egg-info'
The command returned an error %s Command '['python3', 'setup.py', 'build', '--build-temp', '/tmp/tmpu10af4vn', 'egg_info', '--tag-build', 'dev0', 'bdist_wheel', '--bdist-dir', '/tmp/tmpod49imhc']' returned non-zero exit status 1.
Error when preparing airbyte package
########## Generate setup files for 'alibaba' ##########

With current (main branch) configuration everything fine except

Warnings about invalid pyproject.toml

Executing command: 'python3 setup.py build --build-temp /tmp/tmp63irt8no egg_info --tag-build dev0 bdist_wheel --bdist-dir /tmp/tmp1p0sj695'
/usr/local/lib/python3.8/site-packages/setuptools/config/pyprojecttoml.py:129: _InvalidFile: The given `pyproject.toml` file is invalid and would be ignored.
    !!


    ############################
    # Invalid `pyproject.toml` #
    ############################

    Any configurations in `pyproject.toml` will be ignored.
    Please note that future releases of setuptools will halt the build process
    if an invalid file is given.

    To prevent setuptools from considering `pyproject.toml` please
    DO NOT include the `[project]` or `[tool.setuptools]` tables in your file.
    

!!
    
  if _skip_bad_config(project_table, orig_setuptools_table, dist):
warning: no files found matching 'CHANGELOG.rst'
warning: no previously-included files matching '__pycache__' found anywhere in distribution
warning: no previously-included files matching '*.pyc' found anywhere in distribution
warning: build_py: byte-compiling is disabled, skipping.

If I remove entirely [project] section from pyproject.toml then we only got warning about legacy python -m build setup.py command

Generated regular package setup files for airbyte
########## Prepare provider package for 'airbyte' ##########
Changing directory to /opt/airflow/provider_packages
Building provider package: airbyte in format wheel
Executing command: 'python3 setup.py build --build-temp /tmp/tmp5j_17a3a egg_info --tag-build dev0 bdist_wheel --bdist-dir /tmp/tmpndt3z9s_'
warning: no files found matching 'CHANGELOG.rst'
warning: no previously-included files matching '__pycache__' found anywhere in distribution
warning: no previously-included files matching '*.pyc' found anywhere in distribution
warning: build_py: byte-compiling is disabled, skipping.

/usr/local/lib/python3.8/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
warning: install_lib: byte-compiling is disabled, skipping.

Prepared provider package airbyte in format wheel

So I think that we need remove completely [project] section, until we could write correct values in it. It was added during remove Python 3.7 support (#30963)

dismiss previous implementation approve

potiuk · 2023-09-03T23:25:06Z

I think we should also adda python_requires='>3.8.0' in setup( in setup.py in order to replace the requires-python

https://packaging.python.org/en/latest/guides/distributing-packages-using-setuptools/#python-requires

Taragolis · 2023-09-04T07:47:59Z

I think we should also adda python_requires='>3.8.0' in setup( in setup.py in order to replace the requires-python

I'm not 100% sure but I guess it would be enough to have it in setup.cfg, https://setuptools.pypa.io/en/latest/userguide/dependency_management.html#python-requirement

airflow/setup.cfg

Line 58 in 6c649ae

python_requires = ~=3.8

potiuk

Ah yeah. Missed that one.

* Add name and version of project into 'pyproject.toml' * Remove [project] section from `pyproject.toml` * Remove regex for update pyproject [project] version * Fight agains static checks (cherry picked from commit ba8ee90)

Taragolis requested review from potiuk, ashb and jedcunningham as code owners September 1, 2023 14:44

boring-cyborg bot added the area:dev-tools label Sep 1, 2023

potiuk previously approved these changes Sep 1, 2023

View reviewed changes

Lee-W approved these changes Sep 1, 2023

View reviewed changes

Taragolis added 2 commits September 2, 2023 18:29

Add name and version of project into 'pyproject.toml'

460b196

Remove [project] section from pyproject.toml

724179e

Taragolis force-pushed the fix-validate-pyproject branch from 8b063de to 724179e Compare September 2, 2023 14:32

Remove regex for update pyproject [project] version

d4ec44d

Taragolis changed the title ~~Add name and version of project into 'pyproject.toml'~~ Remove [project] section from pyproject.toml Sep 2, 2023

Taragolis requested a review from potiuk September 2, 2023 14:34

Fight agains static checks

c91497e

uranusjr approved these changes Sep 4, 2023

View reviewed changes

potiuk approved these changes Sep 4, 2023

View reviewed changes

potiuk merged commit ba8ee90 into apache:main Sep 4, 2023
64 checks passed

Taragolis deleted the fix-validate-pyproject branch September 11, 2023 15:58

ephraimbuddy added this to the Airflow 2.7.2 milestone Oct 3, 2023

ephraimbuddy added the type:misc/internal Changelog: Misc changes that should appear in change log label Oct 3, 2023

ephraimbuddy mentioned this pull request Oct 9, 2023

Status of testing of Apache Airflow 2.7.2rc1 #34830

Closed

99 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remove [project] section from `pyproject.toml` #34014

Remove [project] section from `pyproject.toml` #34014

Taragolis commented Sep 1, 2023

potiuk commented Sep 1, 2023

Taragolis commented Sep 1, 2023

Taragolis commented Sep 2, 2023 •

edited

Loading

potiuk commented Sep 3, 2023 •

edited

Loading

Taragolis commented Sep 4, 2023 •

edited

Loading

potiuk left a comment

Remove [project] section from pyproject.toml #34014

Remove [project] section from pyproject.toml #34014

Conversation

Taragolis commented Sep 1, 2023

potiuk commented Sep 1, 2023

Taragolis commented Sep 1, 2023

Taragolis commented Sep 2, 2023 • edited Loading

potiuk commented Sep 3, 2023 • edited Loading

Taragolis commented Sep 4, 2023 • edited Loading

potiuk left a comment

Choose a reason for hiding this comment

Remove [project] section from `pyproject.toml` #34014

Remove [project] section from `pyproject.toml` #34014

Taragolis commented Sep 2, 2023 •

edited

Loading

potiuk commented Sep 3, 2023 •

edited

Loading

Taragolis commented Sep 4, 2023 •

edited

Loading