[python-package] drop Python 3.6 support, use dataclasses

[jameslamb]

Contributes to #3756.
Contributes to #3867.
Inspired by discussions in #5672 (thanks @IdoKendo !).

This PR proposes the following:

• replacing two uses of collections.namedtuple in the Python package with dataclasses.dataclass
• dropping support for Python 3.6

Why use dataclasses?

dataclasses are preferable to namedtuple in the following ways:

• easier to set type annotations + default values than named tuples
  • makes it easier for both human developers and and tools like mypy to understand the expected types of properties
• possible to attach other methods or computed properties with @property (just like normal classes)

Why drop support for Python 3.6?

dataclasses has been a part of the Python standard library since Python 3.7 (https://docs.python.org/3/library/dataclasses.html).

Python 3.6 reached end-of-life (stopped receiving even security fixes) in December 2021 (https://devguide.python.org/versions/).

Most of lightgbm's dependencies dropped Python 3.6 a long time ago, for example:

• pandas in August 2020: Drop Python 3.6 support pandas-dev/pandas#35214
• dask and distributed in December 2020: NEP 29 - Dropping Python version support dask/community#66
• numpy in December 2020: MAINT: Remove remaining uses of Python 3.6. numpy/numpy#17898

[jameslamb]

See https://mypy.readthedocs.io/en/stable/runtime_troubles.html#import-cycles for an explanation of this.

Adding this pattern to avoid cyclical imports... engine.CVBooster is needed for the type annotation on CallbackEnv.model, but engine.py also tries to import all of callback's contents.

LightGBM/python-package/lightgbm/engine.py

Line 12 in 709ea4c

from . import callback

[jmoralez]

Even though some dependencies have dropped support for python 3.6, (I think) we don't rely on any new functionality from them, so if the drop is only for using dataclasses I don't think it's worth it, mainly because of the reasoning described in #5006 (comment), I know a couple of companies that are still stuck with py36 and since 4.0.0 will have many new features I don't think users should miss out on those just for the sake of type hints on a couple of data structures.

I'm not hardly against it though, if you feel it's time and think it's worth it I won't oppose.

[jameslamb]

Thanks @jmoralez ! I think you're right, I shouldn't have coupled the "drop Python 3.6" decision together with this attempt to get slightly clearer type annotations.

I'm going to close this and open some new PRs proposing replacing these namedtuple with regular-old Python classes.

[github-actions]

This pull request has been automatically locked since there has not been any recent activity since it was closed.
To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues
including a reference to this.