[python-package] remove 'fobj' in favor of passing custom objective function in params

[TremaMiguel]

Context

This PR closes contributes to #3244

Changes

• Any custom evaluation passed to train through params['metric'] or feval` is accepted.
• Allow passing custome objective functions through params['objective']. In the case that params['objective'] and fobj are both callable the first one is taken into account. fobj argument is removed.

Test

• Functional tests with both classification and regression cases.

[shiyu1994]

@TremaMiguel Thanks for working on this! Could you please check my comments above?

[TremaMiguel]

@shiyu1994 thanks for the time for reviewing, I've provided some answer to open threads.

[shiyu1994]

Close and reopen to retriever CI.

[shiyu1994]

Thank you! The changes LGTM now. I'm not sure if other reviewers would like to take a look before this is merging. So gently ping @StrikerRUS @jameslamb @jmoralez @guolinke.

[StrikerRUS]

@TremaMiguel Thanks for your work!

Since scikit-learn wrapper internally calls train() function,

LightGBM/python-package/lightgbm/sklearn.py

Lines 753 to 764 in 9a4e706

	self._Booster = train(
	params=params,
	train_set=train_set,
	num_boost_round=self.n_estimators,
	valid_sets=valid_sets,
	valid_names=eval_names,
	fobj=self._fobj,
	feval=eval_metrics_callable,
	init_model=init_model,
	feature_name=feature_name,
	callbacks=callbacks
	)

we can remove duplicated code from sklearn.py that is in engine.py now, right?

LightGBM/python-package/lightgbm/sklearn.py

Lines 668 to 675 in 9a4e706

	# Separate built-in from callable evaluation metrics
	eval_metrics_callable = [_EvalFunctionWrapper(f) for f in eval_metric_list if callable(f)]
	eval_metrics_builtin = [m for m in eval_metric_list if isinstance(m, str)]
	# concatenate metric from params (or default if not provided in params) and eval_metric
	params['metric'] = [params['metric']] if isinstance(params['metric'], (str, type(None))) else params['metric']
	params['metric'] = [e for e in eval_metrics_builtin if e not in params['metric']] + params['metric']
	params['metric'] = [metric for metric in params['metric'] if metric is not None]

[StrikerRUS]

Also, please make the same changes for cv() function.

[StrikerRUS]

These changes are quite complicated and require more work that it seems from the first glance. So I suggest to split this PR into two ones with changes related to custom objective and metric respectively.

[StrikerRUS]

OK, I see now.
Seems that LightGBM checks the consistency for objective and number of classes even during predicting.

LightGBM/src/io/config.cpp

Line 261 in 6b56a90

void Config::CheckParamConflict() {

Full logs:

==================================================================================================== FAILURES ====================================================================================================
________________________________________________________________________________________ test_multiclass_custom_objective ________________________________________________________________________________________

    def test_multiclass_custom_objective():
        centers = [[-4, -4], [4, 4], [-4, 4]]
        X, y = make_blobs(n_samples=1_000, centers=centers, random_state=42)
        params = {'n_estimators': 10, 'num_leaves': 7}
        builtin_obj_model = lgb.LGBMClassifier(**params)
        builtin_obj_model.fit(X, y)
        builtin_obj_preds = builtin_obj_model.predict_proba(X)

        custom_obj_model = lgb.LGBMClassifier(objective=sklearn_multiclass_custom_objective, **params)
        custom_obj_model.fit(X, y)
>       custom_obj_preds = softmax(custom_obj_model.predict(X, raw_score=True))

test_sklearn.py:1299:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
d:\miniconda3\lib\site-packages\lightgbm\sklearn.py:1050: in predict
    result = self.predict_proba(X, raw_score, start_iteration, num_iteration,
d:\miniconda3\lib\site-packages\lightgbm\sklearn.py:1063: in predict_proba
    result = super().predict(X, raw_score, start_iteration, num_iteration, pred_leaf, pred_contrib, **kwargs)
d:\miniconda3\lib\site-packages\lightgbm\sklearn.py:813: in predict
    return self._Booster.predict(X, raw_score=raw_score, start_iteration=start_iteration, num_iteration=num_iteration,
d:\miniconda3\lib\site-packages\lightgbm\basic.py:3538: in predict
    return predictor.predict(data, start_iteration, num_iteration,
d:\miniconda3\lib\site-packages\lightgbm\basic.py:813: in predict
    preds, nrow = self.__pred_for_np2d(data, start_iteration, num_iteration, predict_type)
d:\miniconda3\lib\site-packages\lightgbm\basic.py:903: in __pred_for_np2d
    return inner_predict(mat, start_iteration, num_iteration, predict_type)
d:\miniconda3\lib\site-packages\lightgbm\basic.py:873: in inner_predict
    _safe_call(_LIB.LGBM_BoosterPredictForMat(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

ret = -1

    def _safe_call(ret: int) -> None:
        """Check the return value from C API call.

        Parameters
        ----------
        ret : int
            The return value from C API calls.
        """
        if ret != 0:
>           raise LightGBMError(_LIB.LGBM_GetLastError().decode('utf-8'))
E           lightgbm.basic.LightGBMError: Number of classes must be 1 for non-multiclass training

d:\miniconda3\lib\site-packages\lightgbm\basic.py:142: LightGBMError
---------------------------------------------------------------------------------------------- Captured stdout call ----------------------------------------------------------------------------------------------
[LightGBM] [Warning] Auto-choosing col-wise multi-threading, the overhead of testing was 0.000075 seconds.
You can set `force_col_wise=true` to remove the overhead.
[LightGBM] [Info] Total Bins 510
[LightGBM] [Info] Number of data points in the train set: 1000, number of used features: 2
[LightGBM] [Info] Start training from score -1.096614
[LightGBM] [Info] Start training from score -1.099613
[LightGBM] [Info] Start training from score -1.099613
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] Using self-defined objective function
[LightGBM] [Warning] Auto-choosing col-wise multi-threading, the overhead of testing was 0.000071 seconds.
You can set `force_col_wise=true` to remove the overhead.
[LightGBM] [Info] Total Bins 510
[LightGBM] [Info] Number of data points in the train set: 1000, number of used features: 2
[LightGBM] [Warning] Using self-defined objective function
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
---------------------------------------------------------------------------------------------- Captured stderr call ----------------------------------------------------------------------------------------------
[LightGBM] [Fatal] Number of classes must be 1 for non-multiclass training

Let's then pass 'None' for the custom object during predict phase.

[StrikerRUS]

Some more minor suggestions for tests.

[TremaMiguel]

@StrikerRUS thanks for the time taken to review, I've addressed your comments. Let me know if something else is missing.

[StrikerRUS]

Thank you so much for this hard work with API and test changes! LGTM!

[StrikerRUS]

@jameslamb @shiyu1994 @jmoralez @guolinke Please help with another pair of eyes.

[TremaMiguel]

Perhaps a helping hand here @jameslamb ?

[jameslamb]

I reviewed the changes and threads tonight, I have no additional comments. Thanks very much for all the hard work on this @TremaMiguel , and as always for the very thorough review @StrikerRUS .

[jameslamb]

I changed the PR title to explicitly include fobj, so it's easier to find for people looking through the release notes trying to understand why their code is broken.

[TremaMiguel]

@jameslamb @StrikerRUS @shiyu1994 thank you very much for reviewing this. 🙌

[StrikerRUS]

@TremaMiguel Let's go further with feval according to your plan! 😃

[bfan1256]

Is there a specific way to use this with .train() I've tried to specifically set params['objective'] but get Unknown type parameter:objective got:function

And I'm simply passing a function reference

[jmoralez]

Hey @bfan1256, you can find an example on how to use it here. If that doesn't work for you could you please open a new issue?

[github-actions]

This pull request has been automatically locked since there has not been any recent activity since it was closed. To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues including a reference to this.