[VS Code] version 0.2.0 (#2442)

* add installation guide for VS code extension (#2223)

* add installation guide for VS code extension

* [VS Code] view container (#2301)

* add a dashboard in grafana to list all tasks in node (#2197)

* Fix format in issue templates (#2233)

Fix format in issue templates:
- remove trailing spaces
- change chinese colon into english

* Fix auto retries when out of memory. (#1108)

* Distinguish cgroup OOM from dmesg.

* Remove cgroup OOM detection

Make all OOM cause exiting by 5

* Exit 55 when OOM

* Refine homepage for new users (#2155)

Updated first level bullets, to add more content for administrators and users, who is first time touch OpenPAI, or computing platform.

* Fix yarn container failed when docker container exited quickly. (#2256)

* REST server: remove expires in JWT payload of unit test (#2263)

* Deploy: add explicit config field in webportal  plugin (#2251)

* Deploy: add explicit config field in webportal  plugin

* Fix json.dumps

* t

* fix

* Update PLUGINS.md

* Update webportal.md

* alert on unhealthy gpu (#2209)

* Pylon: fix double start query in yarn redirect (#2258)

* Pylon: fix double start query in yarn redirect

* Hide debug info in docker-compose.yaml

* adapt user transfer script to new config (#2266)

* Webportal: add pai-version attribute to <pai-plugin> (#2245)

* Webportal: add pai-version attribute to <pai-plugin>

* Use preprocess to apply window.PAI_VERSION

* set version in layout.html

* Fix ib drivers bug (#2269)

* FIx ib installation script bug (#2271)

* [BUG] Fix hadoop ai build path (#2262)

* fix hadoop ai build bugs

* refine

* Web portal submit job: support init json from sessionStorage. (#2253)

* YARN and HDFS log persistence  (#2244)

* rm log persist

* change log dir to host

* persist nm log to host

* resolve conflict

* persist namenode log

* persist data node log

* add comments

* move log path to common pai storage

* use twisted in yarn-exporter (#2273)

* [Job Debugging] Basic Implement Of Job Debugging. (#2272)

* Refine document for new user to submit job (#2278)

1. add new guidance to submit job for beginners.
2. refine homepage to connect with new guidance.
3. reorganize content of troubleshooting for next refactoring.
4. fix links in md files.

* [Drivers] Fix the issue when installing IB drivers.  (#2275)

* fix can not report zombie process using gpu error (#2279)

* fix external process error

* add debug log

* fix short ID and long ID do not match

* use time based atomic ref to exchange info between threads

* add test case for AtomicRef

* fix bug in file remove (#2288)

* fix hadoop build error (#2296)

* export vc/node related metrics from yarn (#2289)

* 720

* open hdfs explorer in view container
enable tslint rule "ordered-imports"

* add tslint rule for indent

* add home button to hdfs explorer's navigation;
adjust octicon's color

* fix lint error

* [VS Code] Add job list (#2160)

* add job list view to pai extension

* [VS Code] joblist fix (#2185)

* eager load recent jobs when job submitted

* avoid eager getChildren, and let vscode treeview.reveal do it implicitly

* fix lint error

* [VS Code] default to generate jsonc job config file  (#2368)

* 720

* open hdfs explorer in view container
enable tslint rule "ordered-imports"

* add tslint rule for indent

* add home button to hdfs explorer's navigation;
adjust octicon's color

* fix lint error

* [VS Code] Add job list (#2160)

* add job list view to pai extension

* [VS Code] joblist fix (#2185)

* eager load recent jobs when job submitted

* avoid eager getChildren, and let vscode treeview.reveal do it implicitly

* default to generate jsonc job config file

* [VS Code] Refine error messages; Fix Cluster Explorer's bug

* [VS Code] changelog and readme (#2429)

* 539

* 712

* 452

* [VS Code] v0.11 compatible issue (#2457)

* 536

* 600

* [VS Code] fix cluster explorer's right-click menu (#2463)

contrib/pai_vscode/CHANGELOG.md

@@ -4,8 +4,17 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
-## [0.0.1] - 2018-01-09
+## [0.1.0] - 2019-01
 ### Added
 - Open job pages and dashboard pages in VS Code
 - Submit job to PAI cluster from VS Code
 - Open PAI's hdfs as a workspace folder
+
+## [0.2.0] - 2019-03
+### Added
+- Generate jsonc job config by default
+- Add a PAI view container (sidebar), includes
+  - Job list view
+    - Auto refresh enabled
+  - HDFS explorer
+    - You can choose where hdfs explorer will be shown (view container or workspace folder)

contrib/pai_vscode/README.md

@@ -41,6 +41,21 @@ The extension has a useful feature called "Simulate Job Running". It enables the
 
 ![](https://raw.githubusercontent.com/Microsoft/pai/master/contrib/pai_vscode/assets/simulate-job.gif)
 
+## Sidebar
+HDFS Explorer and Job List view will be shown in the extension's view container (sidebar).
+
+### HDFS Explorer
+1. You are able to connect to an OpenPAI cluster's HDFS by double click its corresponding tree node.
+2. The HDFS directory structure will be expended in the tree view. 
+3. Right click the folder or file node to perform file system operations.
+
+### Job List
+1. Browse latest jobs in the OpenPAI cluster in job list tree view.
+2. Browse recent submitted jobs by local (extension) in job list tree view.
+3. Job list will auto refresh every 10 seconds.
+4. Double click job node to browse Job details in external browser.
+
+![](https://raw.githubusercontent.com/Microsoft/pai/master/contrib/pai_vscode/assets/job-list.png)
 
 ## Commands
 ### Command Pallete
@@ -61,7 +76,7 @@ The extension has a useful feature called "Simulate Job Running". It enables the
 |Submit Job...|Submit job to selected cluster|
 |Simulate Job Running...|Generate Dockerfile to simulate PAI job running|
 |Edit Configuration...|Edit cluster configuration|
-|Open HDFS...|Open selected cluster's HDFS as a VS Code workspace folder|
+|Open HDFS...|Open HDFS explorer of selected cluster|
 
 ## Settings
 |ID|Description|
@@ -70,6 +85,10 @@ The extension has a useful feature called "Simulate Job Running". It enables the
 |pai.job.upload.exclude|Glob pattern for excluding files and folders|
 |pai.job.upload.include|Glob pattern for including files and folders|
 |pai.job.generateJobName.enabled|Controls whether the extension will add a random suffix to your job name when submitting job|
+|pai.job.jobList.recentJobsLength|Controls the number of recently submitted jobs to keep in history for each PAI cluster|
+|pai.job.jobList.allJobsPageSize|Controls the page size of list when listing jobs for each PAI cluster|
+|pai.job.jobList.refreshInterval|Controls the refresh interval of job list (in seconds)|
+|pai.hdfs.location|Location where hdfs explorer will be shown|
 
 ## Requirements
 PAI Cluster Version >= 0.8.0

contrib/pai_vscode/VSCodeExt.md

@@ -1,23 +1,22 @@
 # OpenPAI VS Code Client
 
-OpenPAI VS Code Client can submit AI jobs, simulate job running locally, manage HDFS files, and etc. It's an extension of Visual Studio Code.
-
-Visual Studio Code is a popular, free, lightweight but powerful source code editor which runs on your desktop and is available for Windows, macOS and Linux.
-
 ## Installation
 
-1. Download and install [Visual Studio Code](https://code.visualstudio.com/) by several clicks.
+Visual Studio Code is a lightweight but powerful source code editor which runs on your desktop and is available for Windows, macOS and Linux.
+
+More information please refer to the [Visual Studio Code Official Site](https://code.visualstudio.com/).
 
-2. Install **OpenPAI Client**.
+OpenPAI Client is a VS Code extension to connect PAI clusters, submit AI jobs, and manage files on HDFS, etc. You need to install the extension in VS code before using it.
 
-    1. Launch VS Code.
-    2. Click the "Extensions" icon in Activity Bar or press **Ctrl+Shift+X** to bring up the Extensions view.
-    3. Input **openpai** in the text box, the OpenPAI VS Code Client will appear in the result list.
-    4. Click the **Install** button. The extension will be installed.
-    5. After a successful installation, you will see an introduction page. Follow the instructions there and try the PAI client.
+To install the OpenPAI Client:
+1. Launch VS Code.
+2. Click the "Extensions" icon in Activity Bar or press **Ctrl+Shift+X** to bring up the Extensions view.
+3. Input **openpai** in the text box, the OpenPAI VS Code Client will appear in the result list.
+4. Click the **Install** button. The extension will be installed.
+5. After a successful installation, you will see an introduction page. Follow the instructions there and try the PAI client.
 
-    ![Extension](./assets/ext-install-1.png)
+![Extension](./assets/ext-install-1.png)
 
-## Next step
+## Next steps
 
 Learn how to [use OpenPAI VS Code Client](./README.md)

contrib/pai_vscode/i18n/common.json

@@ -23,10 +23,18 @@
         "treeview.node.edit": "Edit Configuration...",
         "treeview.node.openhdfs": "Open HDFS...",
         "treeview.node.openPortal": "Open Web Portal...",
-        "treeview.node.listjob": "List Jobs...",
+        "treeview.node.listjob": "List Jobs Externally...",
         "treeview.node.create-config": "Create Job Config...",
         "treeview.node.submitjob": "Submit Job...",
         "treeview.node.simulate": "Simulate Job Running...",
+        "treeview.hdfs.select-cluster.label": "Double click to connect to a PAI cluster's HDFS...",
+        "treeview.joblist.recent": "Recent Submitted Jobs from VS Code",
+        "treeview.joblist.all": "All Jobs",
+        "treeview.joblist.view": "View Job Detail",
+        "treeview.joblist.more": "View More...",
+        "treeview.joblist.error": "Failed to load job list: {0}",
+        "container.hdfs.mkdir.prompt": "Please enter a folder name",
+        "container.hdfs.mkdir.cancelled": "Cancelled creating new folder",
         "hdfs.workspace.title": "HDFS Explorer - {0}",
         "hdfs.progress": "Transfering file - {0}% ({1} bytes / {2} bytes)",
         "hdfs.downloading": "Downloading {0}",
@@ -47,6 +55,7 @@
         "job.prepare.status": "PAI: Preparing for job submission",
         "job.prepare.cluster.cancelled": "No cluster selected, job submission cancelled.",
         "job.prepare.config.prompt": "Please select a PAI job config json file",
+        "job.prepare.config.invalid": "Invalid job config json file, job submission cancelled.",
         "job.prepare.config.cancelled": "No job config selected, job submission cancelled.",
         "job.prepare.upload.prompt": "Enable auto uploading of code?",
         "job.prepare.upload.yes.detail": "The extension will upload your project files to PAI job config's code dir automatically.",
@@ -57,6 +66,7 @@
         "job.upload.status": "PAI: Uploading code",
         "job.upload.progress": "PAI: Uploading code - {0} / {1}",
         "job.upload.error": "Error occurred while uploading code: {0}",
+        "job.upload.invalid-code-dir": "Auto uploading doesn't support code dir with url scheme hdfs:// or webhdfs://. Please use environment variable $PAI_DEFAULT_FS_URI instead.",
         "job.request.status": "PAI: Submitting job",
         "job.submission.error": "Error occurred while submitting job: {0}",
         "job.submission.success": "Successfully submitted job.",
@@ -93,10 +103,18 @@
         "treeview.node.edit": "编辑配置...",
         "treeview.node.openhdfs": "打开 HDFS...",
         "treeview.node.openPortal": "打开 OpenPAI 门户...",
-        "treeview.node.listjob": "打开任务列表...",
+        "treeview.node.listjob": "在浏览器里打开任务列表...",
         "treeview.node.create-config": "创建任务配置文件...",
         "treeview.node.submitjob": "提交任务...",
         "treeview.node.simulate": "模拟任务执行...",
+        "treeview.hdfs.select-cluster.label": "双击以连接到 PAI 集群的 HDFS...",
+        "treeview.joblist.recent": "近期从 VS Code 提交的任务",
+        "treeview.joblist.all": "所有任务",
+        "treeview.joblist.view": "查看任务详情",
+        "treeview.joblist.more": "显示更多...",
+        "treeview.joblist.error": "载入任务列表时发生错误：{0}",
+        "container.hdfs.mkdir.prompt": "请输入文件夹名",
+        "container.hdfs.mkdir.cancelled": "新建文件夹操作已取消",
         "hdfs.workspace.title": "HDFS 浏览器 - {0}",
         "hdfs.progress": "正在传输 - {0}% ({1} 字节 / {2} 字节)",
         "hdfs.downloading": "正在下载 {0}",
@@ -118,6 +136,7 @@
         "job.prepare.status": "PAI: 正在准备提交任务",
         "job.prepare.cluster.cancelled": "未选择集群，任务提交已被取消。",
         "job.prepare.config.prompt": "请选择一个 PAI 任务配置 JSON",
+        "job.prepare.config.invalid": "任务配置文件不合法，任务提交已被取消。",
         "job.prepare.config.cancelled": "未选择任务配置文件，任务提交已被取消。",
         "job.prepare.upload.prompt": "是否启用代码自动上传功能？",
         "job.prepare.upload.yes.detail": "插件将会自动上传你的项目文件至 PAI 任务配置中的 code dir",
@@ -128,6 +147,7 @@
         "job.upload.status": "PAI: 正在上传代码",
         "job.upload.progress": "PAI: 代码上传 - {0} / {1}",
         "job.upload.error": "代码上传时发生错误：{0}",
+        "job.upload.invalid-code-dir": "自动上传不支持 hdfs:// 及 webhdfs:// 形式的 code dir, 请使用环境变量 $PAI_DEFAULT_FS_URI",
         "job.submission.name-exist": "提交失败，已存在同名任务，是否启用自动生成任务名称后缀功能?",
         "job.submission.name-exist.enable": "启用并重新提交",
         "job.submission.error": "提交任务时发生错误：{0}",