Nano HPO Roadmap

[shane-huang]

Design of HPO is described in issue #3712 (tensorflow) and issue #3925 (pytorch).

Below table summarized major functions and corresponding PR.

category	pr
Common	global hpo settings[#4486, #4488], space[#4458], backend[#4461], callcache[#4461], decorators[#4464], search functions(#4467), utils[#4461,#4464,#4486]
Tensorflow	objective[#4467], mixin[#4467],keras.Model[#4499], keras.Sequential[#4499],tf.keras.layers[#4486], tf.keras.activations[#4486], tf.keras.Input[#4486], tf.* functions(e.g. tf.cast[#4486]), tf.keras.optimizers[#4608],learning rate tuning[#4608,#4610], batch size tuning[#4612], allow fit without search[#4566]
TF Use Cases	sequetial[#4499], functional[#4499], subclassing model[#4499], transfer_learning[TODO]), quickstart notebook[on-going]
PyTorch	objective[#4559], hposearcher[#4559], pl.Trainer[#4560], learning rate tuning[#4631], batch size tuning[#4631]
PyTorch Use Cases	subclassing pl.module[#4559], transfer_learning[TODO]
Parallel Search	refactor[#4680, #4718], pytorch[#4709], tf custom[#4725], tf functional[#4726], tf sequential[#4728], example[TODO]
Resume Search	fix a resume flag behavior[#4733], quickstart notebook[#4632]
Visualization	quickstart notebook[#4632]
Miscellaneous	docs style[#4524], deps refactor[#4560,#4570], user-friendly error reporting, documentation[#4639,#4666,#4734,#4735], ci/cd(PR validation jobs[Done], move pr job to github[TODO], bug fixes[#4564, #4572, #4623, #4646, #4648]

A list of more detailed functions and future TODOs.

• Basic building blocks
  • Search spaces
  • Decorators to create auto objects (class, func, model)
  • search functions (search, search_summary, etc.)
  • Searcher backend (Optuna utils)
  • [TF][PyTorch] Objective classes
  • [TF] Mixins for Sequential/Model and AutoModel (add search, etc. to class)
  • [TF] Sequential class
  • [TF] Model class
  • [PyTorch] HPOSearcher (similar architecture as Tuner)
  • [PyTorch] Trainer.search/search_summary
• Basic Worflow Enabling
  • [TF] Sequential API work flow
  • [TF] Functional API workflow (lazily build computation graphs, etc.)
  • [TF] Custom Model API workflow (subclassing tf.keras.Model)
  • [TF] auto layers, functions, etc. (nano.tf.layers.*)
  • [PyTorch] Custom Model (subclassing pl.LightningModule)
  • transfer learning/ fine tune use case Nano HPO: support transfer learning use case  #4866
  • a flag to enable and disable hpo
• RecSys pipeline support
  • Functional API
  • layers: Embedding, Concatenate, Dense, Activation, etc.
  • keras.Input,
  • tensor operations (i.e. slicing)
  • special functions: tf.cast
  • activations functions: sigmoid
  • enable tuning learning rate
  • support fit (with fixed hyper params) without search
  • result tuning
  • demo
• Enhancements, Usability & Performance
  • performance tuning
  • usability: pytorch parallel does not support threads parallel search. pending
  • usability: pytorch cloud pickle cannot find modules defined/imported from another file. Nano HPO: cloud pickle cannot find module defined in multiple files (parallel search issue) #4868
  • enhancement: [PyTorch] support nn.module tuning
  • enhancement: support more functions in tf (reference) Nano HPO: support more functions in tf.* #4869
  • enhancement: add other tf layers other than tf.keras module (e.g. tf.keras.experimental, etc.)
  • enhancement: through tests on resume search (may only work only when study dir is provided)
  • usability: allow more options of target metrics (e.g. max of all epochs, last epoch, or custom) to be reported in objective. Nano HPO: provide more user options to obtain the target metric value in objective #4870
  • usability: user-friendly error msg (e.g. detect un supported tf operations at early stage, check all objects in defined model are decorated objects before search starts and report errors early etc.) Nano HPO: user-friendly error reporting #4877
  • usability: visualization tools
  • usability: do not expose optuna configurations to end-user (e.g. in search/search_summary)
  • validity: generate layer names for HP prefixes if not specified by user (e.g. If there're two Dense layers in the same model, the search space of the same argument will override each other. ) Nano HPO: automatically generate layer names for user as hyper parameter prefix #4874
  • enhancement: [PyTorch] allow different params (e.g. batch size, max_epochs) for search and fit.
  • usablity: should allow tuning normal keras model (e.g. using layers from tf.keras.layers, etc.) if user do not want to search model architecture. Nano HPO: allow normal keras model tuning #4871
  • usability: support search space in non key-word arguments (i.e. in args but not in kwargs) Nano HPO: support search space in non key-word arguments (i.e. in args instead of kwargs) #4872
  • usability: enable model.summary after compile (build with a default or random sample of search space) Nano HPO: enable model.summary before model.search w/ search space defined #4873
  • enhancement: [TF] allow search spaces in fit arguments (e.g. batch size
  • usability: use the same decorator for tensorflow and pytorch model "@model" instead of "@tfmodel" "@plmodel"
  • usability: allow using bigdl.nano.tf.keras.layers.xxx in either hpo disable or enable scenario. (so that user doesn't have to change imports each time they disable/enable hpo)
  • enhancement: [TF] allow search spaces in compile arguments (e.g. learning rate in optimizer)
  • enhancement: infer learning rate and batch size search spaces by default
  • validity: tolerate both string and objects in params in metrics and target_metrics (match won't throw error) Nano HPO: allow string and objects in metrics and target_metric #4875
  • validity: add unit tests to verify model is actually using the params specified in trial params. (e.g. optuna hp_name vs. arg_name mapping)
  • validity: Functional API various use case validation & fix
    • sequential call
    • multi-input/multi-output
    • layer used in mutliple models - e.g. autoencoder
    • residual connections
    • shared layers
• Bug
  • max_epoch in search does not work in pytorch Nano HPO: max_epoch in Trainer.search does not take effect (pytorch)  #4876
  • resume=False should not reuse the study but create a new one.
• Examples & Tests
  • [TF] Sequantial example
  • [TF] Custom Model example (using decorator)
  • [TF] Functional API example
  • [TF] no-hpo at all
  • [PyTorch] lightning module example
• Miscellaneous
  • fix doc string
  • add automl tests to PR validation jobs
  • move optional lib imports to deps
  • add unit tests for various components.
  • clean up the decorator code
  • refactor code to let tensorflow and pytorch hpo share common codes
  • refactor OptunaBackend to map better map config space to optuna trial suggest
  • merge updates of nano.tf.keras.layers
  • merge nano.automl.tf.keras.Sequantial/Model to nano.tf.keras.Sequantial/Model (PENDING)
  • automatically forward method calls to internal model (use proxy_methods)