Iteration Plan for Sept 2019

[suiguoxin]

Endgame

Sept 13 : Feature freeze for the endgame
Sept 20 24 27 : Code freeze for the endgame
Sept 25 29 Oct 16 Oct 22 : Endgame done
Release manager : @liuzhe-lz

Bug bash

• Add NAS Search Space Converter (branch: dev-nas-tuner) Ref tests in annotations and add NAS Search Space Converter #1432 dev owner: ultmaster ; bug bash owner: @QuanluZhang
  • cifar10 failed
• Add cifar10 example for NAS (branch: dev-nas-tuner) add cifar10 example for NAS #1476 dev owner: QuanluZhang; bug bash owner: @ultmaster
• _(review support specifying gpus for tuner and advisor #1556 )_Assign dedicated resources (such as GPU) to Tuner and Advisor Assign dedicated resources (such as GPU) to Tuner #874 dev owner: QuanluZhang ; bug bash owner: @liuzhe-lz
• [External NNI Example Repo] Supporting EfficientNet in NNI [External NNI Example Repo] Supporting EfficientNet in NNI #1487; bug bash owner: @suiguoxin
• Web UI - @liuzhe-lz @lvybriage @SparkSnail @lan-tang - Please pay especial attention to:
  • optional columns in trail detail page, including fields of search space
  • trial comparing (should show the trial no.)
  • help button in the banner
  • performance of large experiment
• Support automatic model compression on NNI Support automatic model compression on NNI #876 dev owner: liuzhe-lz, LeonardoWang; bug bash owner: @lan-tang @chicm-ms
• Support experiment view #1524 Provide command to show experiment only Provide command to show experiment only. #1414 dev owner: SparkSnail; bug bash owner: @leckie-chn

---

Issues found

• Trial display issue when all trials are failed. @liuzhe-lz
• Tuner UT failure work around fix version: hyperopt 0.1.2 @ultmaster
• Windows pip package can not be built on pipeline @SparkSnail
• TensorFlow 2.0 and hyperopt 0.2 support

Tuners and Assessors

• PPO tuner algorithm for NAS (branch: dev-nas-tuner) PPO tuner for NAS, supports NNI's NAS interface #1380 @QuanluZhang
• Add NAS Search Space Converter (branch: dev-nas-tuner) Ref tests in annotations and add NAS Search Space Converter #1432 @ultmaster
• Add cifar10 example for NAS (branch: dev-nas-tuner) add cifar10 example for NAS #1476 @QuanluZhang

Training Service

• _(review support specifying gpus for tuner and advisor #1556 )_Assign dedicated resources (such as GPU) to Tuner and Advisor Assign dedicated resources (such as GPU) to Tuner #874 @QuanluZhang

Examples

• [External NNI Example Repo] Supporting EfficientNet in NNI [External NNI Example Repo] Supporting EfficientNet in NNI #1487

NNI Mananger

• Support experiment view #1524 Provide command to show experiment only Provide command to show experiment only. #1414 @SparkSnail;

Others

• Pipeline for dev branch @chicm-ms 8.26

Web Portal & User Experience

• Webui not support netsted params visualization WebUI not support nested params visualization #1110 @lvybriage @ultmaster
• Show startTime and endTime as table additory columns startTime and endTime: Integer in experiment parameters file #1492 @lvybriage
• Add Trial No. message in compare modal table Add Trial No. message in compare modal table #1466 @lvybriage
• Metric data file too large Improve web UI performance for large experiment #1490 NNI breaks down when there are too many intermediate results #907 @lvybriage @liuzhe-lz
• Improve WebUI loading performance Improve WebUI loading performance (esp. for large data size experiment) #1425 @lvybriage
• Show parameters' key as table additory column Show parameters' key as table additory column #1209 Some improvements on WebUI #1537 @lvybriage
• Add a button similar to help, link directive related documents Add a button similar to help, link directive related documents #1282 @lvybriage @rabbit008 @yangou1988

Automatic model compression

• Support automatic model compression on NNI Support automatic model compression on NNI #876 @liuzhe-lz @LeonardoWang

Docs, Blogs, Samples, Articles

• Evolution tuner: add a description about population_size Relationship between maxTrialNum and population_size #1481 @xuehui1991
• Update the roadmap @rabbit008
• Tutorial for automatically tuning SPTAG on nni Tutorial for automatically tuning SPTAG on nni #1059 @ chenqi
• GPTuner does not support string choices GPTuner does not support string choices #1546 @suiguoxin

Bug Fix

• Solve the problem of webui semantic inconsistency (use # Intermediate Result for both) Some improvements on WebUI #1537 @lvybriage
• (waiting customer confirm) Multiphase job hangs when search space is exhuasted with NoMoreTrialError raised Multiphase job hangs when search space is exhuasted with NoMoreTrialError raised #1204 @chicm-ms
• It cannot work with nni-0.9 version but work with nni-0.7 version It cannot work with nni-0.9 version but work with nni-0.7 version #1375 @suiguoxin
• Fix nnictl log trial error KeyError: 'logPath' while running remote experiment #1548 @SparkSnail

---

Defer Candidates

NNI Manager

• (in progress) Support Resubmit/Clone/Customized trial in WebUI Customized trial #1498 @liuzhe-lz @Ivybriage @yangou1988
• Add NAS's doc Add blogs of popular NAS paper  #1160 @QuanluZhang