Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[7.x][ML] Hyperparameter importance #1649

Merged
merged 1 commit into from
Jan 8, 2021

Conversation

valeriy42
Copy link
Contributor

@valeriy42 valeriy42 commented Jan 8, 2021

This PR implements hyperparameter importance calculation using ANOVA decomposition of the Gaussian process model used in the Bayesian optimization routine. The results is recorded in the model metadata structure. For convenience, I record the anova variance as an absolute value as well as a fraction of the total variation. This "relative importance" simplifies the comparison between different parameters.

Additionally, I record the best value for the individual hyperparameter s.t. it can be used in a follow-up job configuration and doesn't have to be looked up separately.

Example output:

"hyperparameter_importance": [
        {
            "name": "downsample_factor",
            "value": 0.35355339059327386,
            "absolute_importance": 0.123,
            "relative_importance": 0.9123
        },
        {
            "name": "alpha",
            "value": 0.010766199582740555,
            "absolute_importance": 0.0123,
            "relative_importance": 0.0321
        }
    ]

In order to verify the analytical formulas, I implemented Sobol sequence random number generation, which leads to lower variance for Monte-Carlo estimations.

Backport of #1627

This PR implements hyperparameter importance calculation using ANOVA decomposition of the Gaussian process model used in the Bayesian optimization routine. The results is recorded in the model metadata structure. For convenience, I record the anova variance as an absolute value as well as a fraction of the total variation. This "relative importance" simplifies the comparison between different parameters.

Additionally, I record the best value for the individual hyperparameter s.t. it can be used in a follow-up job configuration and doesn't have to be looked up separately.

Example output:

"hyperparameter_importance": [
        {
            "name": "downsample_factor",
            "value": 0.35355339059327386,
            "absolute_importance": 0.123,
            "relative_importance": 0.9123
        },
        {
            "name": "alpha",
            "value": 0.010766199582740555,
            "absolute_importance": 0.0123,
            "relative_importance": 0.0321
        }
    ]

In order to verify the analytical formulas, I implemented Sobol sequence random number generation, which leads to lower variance for Monte-Carlo estimations.
@valeriy42 valeriy42 merged commit 5e2e2ff into elastic:7.x Jan 8, 2021
@valeriy42 valeriy42 deleted the backport-pr1627 branch January 8, 2021 15:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant