From 213bbc816a6d5785435a0c3156d1032fb4e5d087 Mon Sep 17 00:00:00 2001 From: Teytaud Date: Tue, 22 Mar 2022 08:25:06 +0100 Subject: [PATCH] Add tests for the suggest method (#1316) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * testing_suggest * fix * fix * fix * fix * fix * Update nevergrad/optimization/test_optimizerlib.py Co-authored-by: Jérémy Rapin * Update nevergrad/optimization/test_optimizerlib.py Co-authored-by: Jérémy Rapin * Cleaning and rebasing "suggest" (#1380) * Add architecture search for Gym (#1315) * as * black * add_Test * type_fix * typing * fix * fix_bug * fix_weird * wtf * fix * fix * fix * black * switch_to_choice * mypy * fix * fix * fix * fix * fix * fix_type * Adding optimization specifically for RL (#1303) * Adding optimization specifically for RL * fix * fix * Import benchmarks online for fixing dependencies (#1310) * fix_dependencies * fix_tests * : missing * fix_lint * fix * fix * fix * XPs with sparsity on Open AI Gym (#1319) * XPs with sparsity on Open AI Gym * Update gymexperiments.py * Adding state of the art results on OpenAI Gym results (#1318) * Update plotting.py * fix * ok_good * fix * Adding example for OpenAI Gym (#1320) * testing more algorithms for permutations (#1321) * testing more algorithms for permutations * fix * Adding yahdlbbbob and simplifying stuff (#1145) * Update experiments.py * black * fix_case_1 * fix * Update experiments.py * Add Olympus benchmark (#1190) * Add olympus function and benchmark * remove surfaces with error * Add noise and use average of noise free surfaces for evaluation_finction * fix static tests * rename olympus to olympussurfaces * fix test_core olympus * replace olympus with olymp in requirements * fix olymp version in requirements * black reformatting * Add discrete and GaussianMixture surfaces * Add discrete and GaussianMixture surfaces * Move surfaces in a class variable * no_need_for_42_because_noiseless * nowindowsforcarraz * fix * fix * fix * fix * fix * fix * fix * fix * bettertest * Add olympus emulators * add BayesNeuralNet * reformat code * minor fix * minor fix * minor fix * add missing package and correct seed problem * remove unused packages * fix silence_tensorflow version * try after merge * fix * gamma_fix * fix * fix * fix_naming * fix * fix * fix * fix * fix * fix * fix * fix * Update requirements/bench.txt Co-authored-by: Jérémy Rapin * YEEEEEEEES_it_works Co-authored-by: ncarraz Co-authored-by: Jérémy Rapin * Documentation for permutations (#1322) * change sparsity in gym (#1328) * change sparsity in gym * fix * Unstack ngopt (#1330) * fix plot (#1332) * fix plot * black * Fix gym: thread safety (#1338) * fixgym * fix * fix * fix * Avoid bool check on params (#1349) * Keep only final optimizer in portfolio (#1350) * Fix nan in cma (#1347) * fixnan * fixnan * fixnan * fo * Keep data in cache for CMA (#1351) * Add installation instruction for MuJoCo (#1352) * adding installation instruction for MuJoCo * raise error * better error handling Co-authored-by: Mathurin Videau * enable_pickling for NGOpt (#1356) * Bugfix in scrambled progressive optimization. (#1357) * Add anisotropic progressive optimization (#1344) * Adding anisotropic progressive optimization * black * Finish adding enable_pickling for all optimizers (Rescaled) (#1358) * Residual controllers in RL. (#1359) * Residual controllers in RL. It's not exactly residual, it's initializing with something closer to identity. It works pretty well. * Update multigym.py * fix * fix * fix * black * fix * seed * Update test_core.py * fix * fix * fix * Bump version to 0.4.3.post10 (#1364) * Removing an incomplete sentence from the doc (#1367) * Fix broken CI (#1370) * docs: add GH button in support of Ukraine (#1369) * Add the FAO crop model (#1343) * aquacrop * fix * fix * fix * Update ac.py * black * Update experiments.py (#1361) * fix * Update bench.txt * fix * fix * fix * tentative_pip3 * yet_another_tentative_fi * yet_another_tentative_fi * fix * fix_suffering * desperate_try * desperate_try * desperate_try * desperate_try * fix * desperate_try * desperate_try * desperate_try * desperate_try * fix * Update config.yml * fix * Update setup.py * Update main.txt * fix * Use up-to-date headers (#1371) * Add NLOPT as a solver (#1340) * Update version and changelog to 0.5.0 (#1372) * Deactivate mutation test in CI (#1374) * Reduce noise in gym (#1333) * Reduce noise in gym * Update multigym.py * Add comment * NaN robustness in Gym (#1289) * NaN robustness in Gym * Update nevergrad/functions/gym/multigym.py Co-authored-by: Jérémy Rapin * fix * fix Co-authored-by: Jérémy Rapin * Add more models for Gym control (#1346) * more models for Gym control * Update gymexperiments.py * Update multigym.py * fix * Update gymexperiments.py * update moremodels (#1378) * Bump version to 0.4.3.post10 (#1364) * Removing an incomplete sentence from the doc (#1367) * Fix broken CI (#1370) * docs: add GH button in support of Ukraine (#1369) * Add the FAO crop model (#1343) * aquacrop * fix * fix * fix * Update ac.py * black * Update experiments.py (#1361) * fix * Update bench.txt * fix * fix * fix * tentative_pip3 * yet_another_tentative_fi * yet_another_tentative_fi * fix * fix_suffering * desperate_try * desperate_try * desperate_try * desperate_try * fix * desperate_try * desperate_try * desperate_try * desperate_try * fix * Update config.yml * fix * Update setup.py * Update main.txt * fix * Use up-to-date headers (#1371) * Add NLOPT as a solver (#1340) * Update version and changelog to 0.5.0 (#1372) * Deactivate mutation test in CI (#1374) * Reduce noise in gym (#1333) * Reduce noise in gym * Update multigym.py * Add comment Co-authored-by: Jérémy Rapin Co-authored-by: Dmitry Vinnik * fix * fix * im_lost * fix * fix * fix Co-authored-by: Jérémy Rapin Co-authored-by: Dmitry Vinnik * Add a conformant GP experiment (#1337) * Adding a conformant GP experiment Because sometimes conformant planning, in spite of being super simple, performs incredibly well. * fix * High-speed differential evolution (#1366) * High-speed differential evolution * Update base.py * Update optimizerlib.py * fix * fix * clean * clean * Update differentialevolution.py * Update nevergrad/optimization/differentialevolution.py Co-authored-by: Jérémy Rapin * Update nevergrad/optimization/differentialevolution.py Co-authored-by: Jérémy Rapin * fi * fix * fix * fix Co-authored-by: Jérémy Rapin * cleaning * cleaning * cleaning Co-authored-by: ncarraz Co-authored-by: Jérémy Rapin Co-authored-by: mathuvu <56407945+mathuvu@users.noreply.github.com> Co-authored-by: Mathurin Videau Co-authored-by: Jeremy Reizenstein <669761+bottler@users.noreply.github.com> Co-authored-by: Dmitry Vinnik * fix * Update nevergrad/optimization/test_optimizerlib.py Co-authored-by: Jérémy Rapin * Update test_optimizerlib.py * fix * Update test_optimizerlib.py * Update test_optimizerlib.py * fix * Update test_suggest.py * fix Co-authored-by: Jérémy Rapin Co-authored-by: ncarraz Co-authored-by: mathuvu <56407945+mathuvu@users.noreply.github.com> Co-authored-by: Mathurin Videau Co-authored-by: Jeremy Reizenstein <669761+bottler@users.noreply.github.com> Co-authored-by: Dmitry Vinnik --- nevergrad/optimization/test_optimizerlib.py | 15 --- nevergrad/optimization/test_suggest.py | 114 ++++++++++++++++++++ 2 files changed, 114 insertions(+), 15 deletions(-) create mode 100644 nevergrad/optimization/test_suggest.py diff --git a/nevergrad/optimization/test_optimizerlib.py b/nevergrad/optimization/test_optimizerlib.py index 04d5d4308..6d1f4f3e4 100644 --- a/nevergrad/optimization/test_optimizerlib.py +++ b/nevergrad/optimization/test_optimizerlib.py @@ -254,21 +254,6 @@ def recomkeeper() -> tp.Generator[RecommendationKeeper, None, None]: keeper.save() -@testing.suppress_nevergrad_warnings() -@pytest.mark.parametrize("name", registry) # type: ignore -def test_optimizers_suggest(name: str) -> None: # pylint: disable=redefined-outer-name - optimizer = registry[name](parametrization=4, budget=2) - optimizer.suggest(np.array([12.0] * 4)) - candidate = optimizer.ask() - try: - optimizer.tell(candidate, 12) - # The optimizer should recommend its suggestion, except for a few optimization methods: - if name not in ["SPSA", "TBPSA", "StupidRandom"]: - np.testing.assert_array_almost_equal(optimizer.provide_recommendation().value, [12.0] * 4) - except base.errors.TellNotAskedNotSupportedError: - pass - - # pylint: disable=redefined-outer-name @pytest.mark.parametrize("name", registry) # type: ignore def test_optimizers_recommendation(name: str, recomkeeper: RecommendationKeeper) -> None: diff --git a/nevergrad/optimization/test_suggest.py b/nevergrad/optimization/test_suggest.py new file mode 100644 index 000000000..aa0dae87d --- /dev/null +++ b/nevergrad/optimization/test_suggest.py @@ -0,0 +1,114 @@ +# Copyright (c) Meta Platforms, Inc. and affiliates. +# +# This source code is licensed under the MIT license found in the +# LICENSE file in the root directory of this source tree. + +import pytest +import numpy as np +import sys +import nevergrad as ng +import nevergrad.common.typing as tp +from nevergrad.common import testing +from . import base +from .optimizerlib import registry + + +# decorators to be used when testing on Windows is unecessary +# or cumbersome +skip_win_perf = pytest.mark.skipif( + sys.platform == "win32", reason="Slow, and no need to test performance on all platforms" +) + + +def suggestable(name: str) -> bool: + # Some methods are not good with suggestions. + keywords = ["TBPSA", "BO", "EMNA", "EDA", "BO", "Stupid", "Pymoo"] + return not any(x in name for x in keywords) + + +def suggestion_testing( + name: str, + instrumentation: ng.p.Array, + suggestion: np.ndarray, + budget: int, + objective_function: tp.Callable[..., tp.Any], + optimum: tp.Optional[np.ndarray] = None, + threshold: tp.Optional[float] = None, +): + optimizer_cls = registry[name] + optim = optimizer_cls(instrumentation, budget) + if optimum is None: + optimum = suggestion + optim.suggest(suggestion) + optim.minimize(objective_function) + if threshold is not None: + assert ( + objective_function(optim.recommend().value) < threshold + ), "{name} proposes {optim.recommend().value} instead of {optimum} (threshold={threshold})" + return + assert np.all( + optim.recommend().value == optimum + ), "{name} proposes {optim.recommend().value} instead of {optimum}" + + +@skip_win_perf # type: ignore +@pytest.mark.parametrize("name", [r for r in registry if suggestable(r)]) # type: ignore +def test_suggest_optimizers(name: str) -> None: + """Checks that each optimizer is able to converge when optimum is given""" + + instrum = ng.p.Array(shape=(100,)).set_bounds(0.0, 1.0) + instrum.set_integer_casting() + suggestion = np.asarray([0] * 17 + [1] * 17 + [0] * 66) # The optimum is the suggestion. + target = lambda x: 0 if np.all(np.asarray(x, dtype=int) == suggestion) else 1 + suggestion_testing(name, instrum, suggestion, 7, target) + + +def good_at_suggest(name: str) -> bool: + keywords = [ + "Noisy", + "Optimistic", + "Multi", + "Anisotropic", + "BSO", + "Sparse", + "Recombining", + "PortfolioDiscreteOne", + ] + return not any(k in name for k in keywords) + + +@skip_win_perf # type: ignore +@pytest.mark.parametrize("name", [r for r in registry if "iscre" in r and good_at_suggest(r)]) # type: ignore +def test_harder_suggest_optimizers(name: str) -> None: + """Checks that discrete optimizers are good when a suggestion is nearby.""" + instrum = ng.p.Array(shape=(100,)).set_bounds(0.0, 1.0) + instrum.set_integer_casting() + optimum = np.asarray([0] * 17 + [1] * 17 + [0] * 66) + target = lambda x: min(3, np.sum((np.asarray(x, dtype=int) - optimum) ** 2)) + suggestion = np.asarray([0] * 17 + [1] * 16 + [0] * 67) + suggestion_testing(name, instrum, suggestion, 1500, target, optimum) + + +@skip_win_perf # type: ignore +def test_harder_continuous_suggest_optimizers() -> None: + """Checks that somes optimizer can converge when provided with a good suggestion.""" + instrum = ng.p.Array(shape=(100,)).set_bounds(0.0, 1.0) + optimum = np.asarray([0] * 17 + [1] * 17 + [0] * 66) + target = lambda x: min(2.0, np.sum((x - optimum) ** 2)) + suggestion = np.asarray([0] * 17 + [1] * 16 + [0] * 67) + suggestion_testing("NGOpt", instrum, suggestion, 1500, target, optimum, threshold=0.9) + + +@testing.suppress_nevergrad_warnings() +@pytest.mark.parametrize("name", registry) # type: ignore +def test_optimizers_suggest(name: str) -> None: # pylint: disable=redefined-outer-name + optimizer = registry[name](parametrization=4, budget=2) + optimizer.suggest(np.array([12.0] * 4)) + candidate = optimizer.ask() + try: + optimizer.tell(candidate, 12) + # The optimizer should recommend its suggestion, except for a few optimization methods: + if name not in ["SPSA", "TBPSA", "StupidRandom"]: + np.testing.assert_array_almost_equal(optimizer.provide_recommendation().value, [12.0] * 4) + except base.errors.TellNotAskedNotSupportedError: + pass