[python-package][R-package] adapt to scikit-learn 1.6 testing changes, pin more packages in R 3.6 CI jobs

[jameslamb]

Fixes #6717, adapting lightgbm to the following changes in scikit-learn (currently visible in scikit-learn 1.6 nightlies).

from scikit-learn/scikit-learn#30122:

• estimators must populate tags.estimator_type in the tags returns by __sklearn_tags__()
• regressors must populate tags.regressor_tags with a non-None values
• classifiers must populate tags.classifier_tags with a non-None values

from scikit-learn/scikit-learn#30149:

• @parametrize_with_checks() ignores _xfail_checks in _more_tags() / __sklearn_tags__() on estimators... projects must now pass a function that generates a dictionary of xfailed checks to @parametrize_with_checks() in test code

Fixes #6719, pinning all the versions of dependencies installed in R 3.6 CI jobs to make those jobs more stable (see previous issues: #6442 (comment), #6434).

[jameslamb]

All of these hard-coded versions are the latest for which there's a package at https://cran.r-project.org/src/contrib/Archive. That provides a set of packages that were all working together as of a few days ago.

We shouldn't need to actively manage this list... this should be able to remain untouched (I hope) until we drop R 3 support and delete this script entirely.

[jmoralez]

Why are these defined again here?

[jameslamb]

Because if an earlier ImportError happens, then this part above would never get evaluated:

    except ImportError:
        _sklearn_ClassifierTags = None
        _sklearn_RegressorTags = None

And then the from .compat imprt _sklearn_ClassifierTags in sklearn.py would fail. This is not new, just following the pattern that's existed in LightGBM for a long time (see all the other {something} = None above it).

Happy to consider something else if you have a recommendation for improving this!

[jmoralez]

Oh yeah, sorry.

The first one is defined for scikit-learn<1.6 and the second when scikit-learn isn't installed.

[jameslamb]

yeah exactly. No need to be sorry, it's confusing!

[StrikerRUS]

Thank you very much for your hard exploration!
Just two minor suggestions.