[python-package] record_evaluation callback doesn't work with cv function

[jmoralez]

Description

The record_evaluation callback fails when trying to use it with the cv function. Since some integrations that log metrics may use that callback to save the training information and the cv function is very useful to do hyperparameter tuning I believe these two should work together.

Reproducible example

import lightgbm as lgb
import numpy as np

X = np.random.rand(100, 2)
y = np.random.rand(100)
ds = lgb.Dataset(X, y)
params = {'objective': 'l2', 'num_leaves': 5, 'verbose': -1}
eval_result = {}
callbacks = [lgb.record_evaluation(eval_result)]
bst = lgb.train(params, ds, num_boost_round=2, valid_sets=[ds], callbacks=callbacks)
print(eval_result)
# {'training': OrderedDict([('l2', [0.08664710410521784, 0.08580684200793261])])}

eval_result = {}
callbacks = [lgb.record_evaluation(eval_result)]
cv_hist = lgb.cv(
    params, ds, num_boost_round=2, stratified=False, callbacks=callbacks, eval_train_metric=True
)
# Traceback (most recent call last):
#   File "record_eval.py", line 14, in <module>
#     cv_hist = lgb.cv(params, ds, stratified=False, callbacks=callbacks)
#   File "/hdd/github/LightGBM/python-package/lightgbm/engine.py", line 582, in cv
#     cb(callback.CallbackEnv(model=cvfolds,
#   File "/hdd/github/LightGBM/python-package/lightgbm/callback.py", line 140, in _callback
#     _init(env)
#   File "/hdd/github/LightGBM/python-package/lightgbm/callback.py", line 134, in _init
#     for data_name, eval_name, _, _ in env.evaluation_result_list:
# ValueError: too many values to unpack (expected 4)

This can be easily fixed by capturing the extra content from env.evaluation_result_list with *_ here:

LightGBM/python-package/lightgbm/callback.py

Line 134 in 4aaeb22

for data_name, eval_name, _, _ in env.evaluation_result_list:

and here:

LightGBM/python-package/lightgbm/callback.py

Line 141 in 4aaeb22

for data_name, eval_name, result, _ in env.evaluation_result_list:

which would yield:
{'cv_agg': OrderedDict([('train l2', [0.06319766819578873, 0.06253713563921684]), ('valid l2', [0.06679350623951755, 0.06694167044391186])])}

Environment info

LightGBM version or commit hash: db045f4

Additional Comments

I'm happy to make the described changes if other maintainers agree with my solution.

[StrikerRUS]

Ah, I remember we had a similar problem in early_stopping callback: #2209 and had to split that string.

LightGBM/python-package/lightgbm/callback.py

Lines 281 to 282 in 4aaeb22

	# split is needed for "<dataset type> <metric>" case (e.g. "train l1")
	first_metric = env.evaluation_result_list[0][1].split(" ")[-1]

LightGBM/python-package/lightgbm/callback.py

Lines 322 to 329 in 4aaeb22

	# split is needed for "<dataset type> <metric>" case (e.g. "train l1")
	eval_name_splitted = env.evaluation_result_list[i][1].split(" ")
	if first_metric_only and first_metric != eval_name_splitted[-1]:
	continue # use only the first metric for early stopping
	if ((env.evaluation_result_list[i][0] == "cv_agg" and eval_name_splitted[0] == "train"
	or env.evaluation_result_list[i][0] == env.model._train_data_name)):
	_final_iteration_check(env, eval_name_splitted, i)
	continue # train data for lgb.cv or sklearn wrapper (underlying lgb.train)

I believe we should preserve the nested structure of resulting dictionery in which we have dataset names at the first level and metric names at the second one, e.g.

{
  'cv_agg':
    {
      'train':
        {
          'l2': [0.06319766819578873, 0.06253713563921684]
        },
      'valid':
        {
          'l2': [0.06679350623951755, 0.06694167044391186]
        }
    }
}

[jmoralez]

I agree with preserving the structure but I think the cv_agg key doesn't provide much value, maybe we could drop it?

[StrikerRUS]

... maybe we could drop it?

Yeah, for sure!

[github-actions]

This issue has been automatically locked since there has not been any recent activity since it was closed. To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues including a reference to this.