From 492e05cd51ab55605ab06816ea883ea6c4ed3e3b Mon Sep 17 00:00:00 2001 From: YingzhePeng <78460824+ForJadeForest@users.noreply.github.com> Date: Tue, 20 Sep 2022 15:33:11 +0800 Subject: [PATCH] Remove rl_pong example from Jenkins example test (#5796) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * Enable FL Server in SGX (#5396) * add assembly * Enable FL Sever in SGX * Change the url in tutorial.md * Add start-fl-server.py to Docker Image * rm ppml-assebly.xml * Add start-python-fl-server-sgx.sh to Docker Image * Put start-fl-server.py into assembly * Update 3.1.2 Start FL Server in SGX in pytorch-nn-lr-tutorial.md * Change the location of start-fl-server.py * Correct the path in start-python-fl-server-sgx.sh * Delete -r in cp command Co-authored-by: Jiaming * fix tf inc metric format (#5127) * fix tf inc metric format * re-run actions * Apply suggestions from code review Co-authored-by: Zhentao Chen <90437536+zhentaocc@users.noreply.github.com> * Update Co-authored-by: Zhentao Chen <90437536+zhentaocc@users.noreply.github.com> * Orca: update known-issues. (#5505) * fix: update known-issues. * fix: fix wording. * Chronos: update js, css and md files to add filter function in tutorial page. (#5506) * Chronos: PR validation job to Github action (#5349) Migrate current jenkins CI/CD jobs to github action, this is chronos PR validation-Spark2.4-ray job * Chronos: an error message is added to AutoformerForecaster.predict() (#5483) When the input data to AutoformerForecaster.predict() is incorrect, the error message is confusing. Thus, a new error message is added. * [Nano] Openvino quantization notebooks with nano (#5491) * notebook with nano * load with nano * modify readme * clear all outputs and enable benchmark_app * fix typo * Correct the path of start-fl-server.py in assembly.xml (#5509) * add assembly * Enable FL Sever in SGX * Add start-fl-server.py to Docker Image * rm ppml-assebly.xml * Add start-python-fl-server-sgx.sh to Docker Image * Put start-fl-server.py into assembly Co-authored-by: Jiaming * fix compile issue for spark2.3 (#5501) * Nano: update ipex_bf16_inference_model (#5470) * rollback requirement-doc * Update * Update * Update * Update * Fix ipex with amp error * Fix unit test * Update * Remove redundant method * Fix error * remove redundant import * Update ut to conver cases * Fix the problem of cannot find the directory. (#5518) * add assembly * Enable FL Sever in SGX * Add start-fl-server.py to Docker Image * rm ppml-assebly.xml * Add start-python-fl-server-sgx.sh to Docker Image * Put start-fl-server.py into assembly * Fix the target directory Co-authored-by: Jiaming * [Orca] Replace pyarrow dependency in tf2 ray estimator (#5465) * remove pyarrow dependency in tf2 ray backend * update ut to compare prediction results * CICD: re-organize existing workflows into two workflows: nightly_build and nightly_test (#5511) * reorg workflows * remove duplicate yamls * add dependencies * PPML add pytorch incremental training and tutorial (#5379) * Orca: add save and load model doc to PyTorch Estimator quick-start. (#5504) * feat: add save and load model doc to pytorch estimator quickstart. * fix: fix typo. * fix: fix typo * fix: fix typo. * feat: add doc to ray backend quickstart * fix: fix typo. * feat: add doc to ray backend quickstart * Orca: add shutdown API to tf2 pyspark estimator. (#5499) * feat: add shutdown API to tf2 pyspark estimator. * fix: replace raise with log4Error. * refactor: refactor the stop log server function. * fix: fix code style issue. * update nightly Scala build (#5523) * Orca: refactor HDFS operation in estimator (#5325) * fix: use default value if no hdfs port specified. * fix: add default hdfs port when saving pkl * refactor: replace pyarrow deps in estimator. * fix: add process wait() * fix: resolve unnecessary change * Update nano training ipex bf16 (#5333) * Update * Fix code style * re-run action * Fix code style * re-run action * Update * Fix code style * support bf16 multi training * Update * Update * Update * Update * Update * Update * Update * Update * Update * Update * Update * Update * Update * Update * Update * Update * reduce ut time and re-run action * track avx512 * Update lite bf16 training * Update * Update * Update bf16 api * Update * Fix typo * Orca: add get remote file to local decorator in orca file utils. (#5508) * feat: add get remote file to local decorator in orca file utils. * feat: add unit-test for local and s3 FS. * fix: update func name and comments. * fix: fix code style. * fix: fix code style. * fix: fix typo. * fix: fix import. * PPML FL add PSI support for fl nn server (#5434) * [Orca] Refactor tf2 pyspark estimator save model (#5425) * fix save model in pyspark est * use a function to replace * use orca.data to replace dllib * Add Xshards test to github actions (#5512) * Add Xshards test to github actions * add file path * fix yml * fix csv path * fix csv path * fix yml * add wget * fic typo * fix typo * change csv file path * update pwd * add new line * delete a line * add time Co-authored-by: yansu * Update some packages' edition (#5527) * add assembly * Enable FL Sever in SGX * Add start-fl-server.py to Docker Image * rm ppml-assebly.xml * Add start-python-fl-server-sgx.sh to Docker Image * Put start-fl-server.py into assembly * Change some packages' edition * Fix the problem of fl/start-fl-server.py Co-authored-by: Jiaming * Use tracker.py to build xgboost4j_2.12-1.1.2.jar (#5513) Reason to change: tracker.py in ${BIGDL_HOME}/jars/bigdl-dllib-spark_${SPARK_VERSION}-${BIGDL_VERSION}.jar is not functional. Instead of using tracker.py to build ${BIGDL_HOME}/jars/bigdl-dllib-spark_${SPARK_VERSION}-${BIGDL_VERSION}.jar, we now use it to build xgboost4j_2.12-1.1.2.jar, which can be reached when executing xgboost examples. * Create nightly-build-example-tests-ppml.yaml (#5517) * Create nightly-build-example-tests-ppml.yaml * Update nightly-build-example-tests-ppml.yaml Triggered by schedule * Update nightly-build-example-tests-ppml.yaml * Chronos: add tune for lstm and s2s (#5521) * set use_hpo=True and add ut * PPML: add k8s to Gramine Docker (#5428) * feat: add k8s related files * fix: change secure-argv to secure_argv in Gramine * fix: check if secured_argv exists with -f * fix: exit when not ready in init.sh * fix: change path of encalve-key in k8s template * feat: copy k8s directory to Gramine directory * mod: version in development * mod: enclave-key path * del: remove tmp build action * feat: Simplify deployment operations for users * doc: update README.md * doc: correct a typo * doc: correct typo * add spark pi on k8s scripts * ppml python psi hashing (#5531) * cicd: refactor the github actions workflows (#5529) * add composite actions * add manually build * Nano: Enhance optimizer pipeline (#5490) * refactor code * enable get_best_model * basic impl finnish * add license * clean openvino output * remove one line * remove partial output of openvino and inc * add trace and quantize * update based on comments * ingore errors * update based on comment * update based on comment * add type hint for return * fix issues (#5539) * Chronos: set kernel of Autoformer to odd number (#5534) * set kernel to odd * [PPML] Change Occlum tmp_dir to use hostfs (#5520) * change_tmp_dir_hostfs，but xgboost example will crash (Need to find the reason). * add a switch to use tmp hostfs * [PPML] Implement verifyQuote in QuoteVerifier.scala (#5495) * Add SGXDCAMQuoteVerifierImpl in for QuoteVerifier * Add unit test for SGXDCAPQuoteVerifierImpl * Move attestation.Attestation to dcap.Attestation * [ORCA] Add xshards example (#5542) * update sh * add data example 2 * add data example 2 * fix style * fix time * fix time * fix time Co-authored-by: yansu * add psi in nn tutorial (#5532) * [PPML] refine eHSN-KMS according to user experience (#5544) * Add python hint to bigdl.friesian (#5430) * friesian.feature.table * script refactor * doc * [PPML] Fix XGBoost example in docker-graphene/README.md (#5510) * Fix the error in docker-graphene/README.md * Add instructions on how to process Boston_Housing.csv * Change to use the correct path for xgboost examples * [ORCA] Add Xshrads tutorial test (#5548) * add tutorial test * fix typo * fix name * remove shards_regressionExample.py * update yml * update yml * update yml * update yml * update yml * add readme * update yml * update yml Co-authored-by: yansu * Change the environment settings in Dockerfile for FL Server. (#5554) * add assembly * Enable FL Sever in SGX * Add start-fl-server.py to Docker Image * rm ppml-assebly.xml * Add start-python-fl-server-sgx.sh to Docker Image * Put start-fl-server.py into assembly * Improve the Dockerfile to create right environment for FL Server * Fix the problem of pytorch-nn-lr-tutorial.md Co-authored-by: Jiaming * Reduce XShards tutorial test time (#5555) * Update README of XShards tutorials (#5557) * update readme * minor * add orca ray python test to github action (#5550) * add orca ray python test to github action * update * update * update * update * update * update * [PPML] Remove sgxsdk quote verify in ut (#5553) * Remove sgxsdk quote verify in ut * Use Env SGXSDK for UT * Refine * [PPML] Refine get RABIT_TRACKER_IP env in XGBoost examples (#5528) * The original code will lead to keyError if "RABIT_TRACKER_IP" is not set. The ideal result should be return None. * Change the Dockerfile for better. (#5559) * add assembly * Enable FL Sever in SGX * Add start-fl-server.py to Docker Image * rm ppml-assebly.xml * Add start-python-fl-server-sgx.sh to Docker Image * Put start-fl-server.py into assembly * Change the Dockerfile for better Co-authored-by: Jiaming * cicd: add a new workflow called PR_Validation (#5546) * add PR_Validation workflow * test * test * test * test * test * test * test * add friesian ut to PR validation * revert changes * add dllib ut * test * revert * [PPML] refine EHSM-KMS and modify annotations (#5563) * add spark sql examples (#5535) * add spark sql local mode scripts * update local ip value Co-authored-by: baishaojie * fix prometheus-client version for ray dashboard (#5549) * Fix bug that string_embed function Only replace one columns when replace=True. (#5497) * add for loop to replace all Columns. * add replace test for string_embds function * Orca: Add orca ray ctx example to run on self hosted (#5484) * Add orca ray ctx example in github workflows * Install cmake 3.16.3 with pip * Increase the CPU limits * Add newline at EOF * Align with the new format * Add the pull_request trigger * Add missing property * Uninstall spark first * Uninstall spark3 * Remove test triggers * Add tests to PR_validation * Fix a bug * Test the validation trigger * Undo the change * Chronos: Add `from_tsdataset` method, BaseTF2Forecaster can input a tsdataset (#5064) * add from_tsdataset for tfForecaster * fix known issues * fix known issues again * Add attestation guide to README.md in ppml/docker-graphene (#5568) * Add attestation guide to README.md in ppml/docker-graphene * Refine * Refine * Refine * Refine * Refine * Remove hard coded spark mode in xgboost examples (#5579) * kms-utils README fix (#5582) * rename inference ppl test to move to tests-ipex (#5581) * [PPML] Fix ppml code style (#5584) * Add BigDL-core dep and scope * Fix ppml code style * Chronos : add [how to tune a forecaster model] notebook (#5399) * add how_to_tune_a_forecaster_model notebook * modify step1: data preparation * update based how to train * modify a bug of summary * update based on comment * move to howto * fix typo * add file * update based on comments * rename the tile * Fixprocessing for several orca tutorials so it can use the original dataset download from website (#5572) * [PPML] Modify FTP_URI for sgxdcap quote verifier ut (#5591) * Modify FTP_URI for sgxdcap quote verifier ut * Fix * Fix * Add shutdown api (#5592) * feat: add shutdown api * test: add unit test for shutdown API. * add orca py37 UT to github action (#5578) * add orca py37 UT to github action * update pull_request path * update build python whl * update bigdl-core version * remove pull_request * remove scala app test for nightly test (#5595) * Fix yaml load (#5597) * PPML FL FGBoost save/load server model (#5577) * remove hardcode credentials (#5598) * PPML FL protobuf and generated update (#5604) * PPML FL Pytorch NN tutorial refine (#5545) * Update powered-by.md * Update powered-by.md * optim get schema and get class name in shards into 1 rdd operation (#5573) * update output file to hdfs path (#5593) * Chronos: Remove UTs for all `Deprecated` API and `tf1` (#5530) * delete deprecated UTs * remove tf1 UTs * Add spark auth secret env to occlum (#5602) * Add Azure KMS support for PPML python api (#5547) * add azure kms for python * update ppml doc * update ppml tutorial readme * update kms-utils entrypoint.sh * update entrypoint.sh Co-authored-by: Zhou * [Orca] Refactor `model_dir` as an option in tf2 pyspark estimator (#5541) * refactor model_dir as an option * modify ut to test non model_dir * update coding format * update return results * add support in load api * move model_dir from ut * Pytorch VFL NN refactor, refine and example test (#5607) * [Documentation] Add Extension nbsphinx-link (#5596) * Add extension nbsphinx-link: link notebook out of the sphinx root folder to the sphinx project * Link Notebook custom.ipynb directly from python folder to the sphinx project * Link Notebook seq_and_func.ipynb directly from python folder to the sphinx project * Small title fix * Make interactive graphs visible in readthedocs webpage for seq_and_func.ipynb * Regenerate outputs for custom.ipynb and remove parts of them * add five components to tpp (#5614) Co-authored-by: Jin Hanyu <476099001@qq.com> * Chronos: change get_latency to a static method (#5618) * Implement get quote from ehsm (#5621) * Implement getQuoteFromServer in EHSMAttestationService * Fix * Remove Base64 lib Co-authored-by: xiangyuT * Chronos : add regression functions for forecast metrics to replace torchmetrics (#5402) * add regression functions * delete unexpected import * add function api * remove eval function * revised the problems mentioned * Modify files imported TORCHMETRICS_REGRESSION_MAP * fix bug caused by numpy function * fix pep8 Co-authored-by: Xinyi Zhang Co-authored-by: theaperdeng * Fix Friesian READMEs (#5626) * Chronos: installation panel enhancement (#5588) * Update:installation panel enhancement Left a bug to fix and install CMD to revise. * Bugfix: Fixed a bug when using Firefox browser to use the install guide form. Fixed the bug that the table's background color may be changed by mistake on Firefox browser. * Update: update panel as comment required. * Bugfix: fixed a bug in js file that will show wrong cmd when clicking 'windows' or 'docker'. * Bugfix: fixed the bug that 'cmd' shows incorrectly when clicking 'prophet' and 'ARIMA'. * support convert rdd of pandas df to spark df with arrow (#5385) * Doc: update hadoop document. (#5623) * doc: update hadoop document. * fix: fix wording. * fix: fix wording. * fix: fix wording. * Nano: Add pytorch-lightning bf16 training example (#5526) * rollback requirement-doc * Add Nano BF16 Training example * Update * Update * Chronos: numpy and python version in pr validation Github Action is fixed (#5613) * [Nano] Nano How-to Guides: Format & PyTorch Inference (#5480) * Create doc tree index for Nano How-to Guides * Add How to guide for PyTorch Inference using ONNXRuntime * Add How to guide for PyTorch Inference using OpenVINO * Update How to guide for PyTorch Inference using OpenVINO/ONNXRuntime * Change current notebook to md and revise contents to be more concentrated * Add How-to Guide: Install BigDL-Nano in Google Colab (need further update) * Revise words in How-to Guide for PyTorch Inference using OpenVINO/ONNXRuntime * Add How-To Guide: Quantize PyTorch Model for Inference using Intel Neural Compressor * Add How-To Guide: Quantize PyTorch Model for Inference using Post-training Quantization Tools * Add API doc links and small revision * Test: syncronization through marks in py files * Test: syncronization through notebook with cells hidden from rendering in doc * Remove test commits for runnable example <-> guides synchronization * Enable rendering notebook from location out of sphinx source root * Update guide "How to accelerate a PyTorch inference pipeline through OpenVINO" to notebook under python folder * Update guide "How to quantize your PyTorch model for inference using Intel Neural Compressor" to notebook under python folder * Fix bug that markdown will be ignored inside html tags for nbconvert, and notebook revise * Update guide 'How to quantize your PyTorch model for inference using Post-training Optimization Tools' to notebook under python folder * Small updates to index and current guides * Revision based on Junwei's comments * Update how-to guides: How to install BigDL-Nano in Google Colab, and update index page * Small typo fix * Fix kms-utils README * Fix enroll app command in README * split assembly (#5564) * split assembly * update * update * [PPML] Refine occlum readme and config (#5625) * Refine occlum readme and config * Update docker sql config * Reduce instance to 1 * [PPML] Add user doc for how to install python dependencies (#5494) * Add user doc in docker-graphene/README.md * Add instructions on how to add egg dependencies * [PPML] Fix file_name too long issue (#5628) Test xgboost and tpch example successfully. * [PPML] Refine xgboost docker example config (#5631) * Chronos : Penalize underestimation example (#5627) * add example * remove valid dataset * fix typo * modify parameters and graph * update based on comment * modify index link * modify js * update * Chronos: Sync SparkXShards changes(TCMFForecaster) (#5630) * Fix xshards tensorflow (#5633) * fix * fix * fix * fix * Chronos: Delete existing tags and replace with new Installation options (#5165) * remove skipif tf2 * delete skip_onnxrt tags * fix synatx error * add installation options * remove invalid options * fix known issues * fix known issues * rollback tf1 uts * rollback tf1 UTs * rollback invalid changes Co-authored-by: fuckyouNoob * Force update bigdl-core jar during build (#5569) * Add -U in dllib release * Add -U to orca and friesian * Chronos: make `roll=True` the default value for `to_torch_data_loader` (#5605) * Chronos: modify examples of `.evaluate` (#5622) * fix nano UT (#5650) * [PPML]Remove simple KMS entroll command (#5624) * fix: simple enroll * Update DeepRec README.md (#5652) * fix dllib pom scope and typo in examples (#5575) * fix dlliv typo in examples * [PPML] Fix bash expected unary operator (#5653) * [PPML] Implement bi-attestation in AttestationCLI.scala (#5648) * Implement bi-attestation in AttestationCLI.scala * Refine * Refine * Refine * Refine * Fix problems according to comments Co-authored-by: xiangyuT * PPML FL change clientID from String to Int (#5647) * support convert spark df to shards with arrow (#5558) * Chronos: syntax error in Evaluator API documentation is fixed (#5660) * Chronos: how to create a forecaster (#5172) * how to guide from_tsdataset * improve md * remove redundant message * fix known issues * add rst * move create-forecaster to new line * fix known issue * fix syntax error * fix syntax error again * add open-colab icon * fix known issues * fix some words * fix typo * fix syntax error * add some update Co-authored-by: theaperdeng * Nano : ResNet Demo for InferenceOptmizer (#5580) * add resnet demo * add basic readme * add inference result * update based on comment * update based on comment * fix style * fix typos and update based on comment * update numpy version * update for faster demo * modify some number * [PPML] Add SGX plugin device readme (#5663) * add sgx plugin device readme * fix tutorial xshards env (#5661) * fix tutorial xshards env * add protobuf * add torch * add pyarrow * add python requirements * Nano : fix and enhancement of InferenceOptimizer (#5611) * fix accuracy-related bug and add model.eval() * automatically print optimization results * dynamic control latency_sample_num * add ut for no optimize * fix accuracy * update ut * update based on comment * remove statue restore * Nano: Update Nano PyTorch Lightning tutorial test script (#5612) * Update * Add test for lightning bf16 * Update * Fix shell * remove redundant code in resnet18 example (#5665) * support init_orca_context with spark.master=local[*, F]/local[N, F] (#5651) * support format local[N,F] & local[*,F] * add comment and UT Co-authored-by: Zhou * Chronos: how to speedup inference on one node (onnx, openvino) (#5556) * how to speedup inference on one node * index added * fixed * fixed * split to 2 guides * add build_onnx and build_openvino * add note border * fix * fix colab conflict Co-authored-by: binbin * Nano: Add nano pytorch example (#5570) * rollback requirement-doc * Add example * Update * Update * Update * Update * Add comments and remove testing loop * Add comments about linear scale and warmup * remove .keep file * add create-job-status-badge action (#5676) * [Nano] add __call__ method for OpenVINOModel (#5540) * [Nano] support __call__ method for OpenVINOModel * remove unnecessary kwargs in __call__ method of OpenVINOModel * specify which __call__ method to call in KerasOpenVINOModel * remove whitespace * make KerasOpenVINOModel and PytorchOpenVINOModel not inherit from OpenVINOModel * fix docstring of OpenVINOModel._save_model * remove extra import * fix _save_model * change_save_model * simplify _model_exists_or_error * Support decimal for Spark DataFrame (#5671) * support decimal * remove * Fix yaml load (#5655) * [PPML] Fix EHSM 0.3.0 create enclave failure (#5683) Co-authored-by: glorysdj * fix issue 4642, fix DBFS file path problem on Dataricks (#5679) * fix issue 4642 * parse model_dir Co-authored-by: Zhou * Change Analytics Zoo related to BigDL in doc (#5685) * change az to bigdl * minor * minor * Add Ray arguments in init_orca_context to Doc page (#5686) * add ray docs * fix * fix * fix style * minor * scala lightGBM classifier and regressor[WIP] (#5447) * scala lightGBM classifier and regressor * Fix Orca Example styles (#5692) * part1 * part2 * part3 * part4 * exclude and finish * Chronos: fix spark 3.1 bug in xshards unscale (#5689) * Add selfHostedRunner input (#5701) * add create-job-status-badge action * add selfHostedRunner input * fix: update redis port random range. (#5702) * [Nano] Add Github Action Tests for How-To Guides: Inference PyTorch (#5668) * Disable other nano action tests temporarily * Add github action tests for Nano How-To Guides: Inference > PyTorch * Add dependency install for pytest and nbmake * Separate tests based on openvino/onnx dependencies * Tighten the scope for sed to change * Fix [AttributeError: 'OutStream' object has no attribute 'watch_fd_thread'] when testing for OpenVINO related notebook Downgrade ipykernal to solve this * Edit trigger conditions for how-to gudies tests * Enable other nano tests again * Style fix: remove extra tabs and add missing comments * Add how-to guides tests trigger limitations for push on main branch * fix integration test for dllib examples (#5707) * Chronos: train tcn model on gpu and speed up inference on cpu (#5594) * add nano_gpu * simplify the code * simplify the code * modify the code * add some updates * add document Co-authored-by: theaperdeng * bigdl all in one jar (#5703) * bigdl all in one jar * update * cicd: scan dependency tree to help debug if fossa scan failed (#5698) * scan dependency * if always * update * filter dependency tree if failure * revert * [PPML] EHSM 0.3.0 AS Support for Gramine (#5644) * [PPML]correct the command and add explanation in kms-utils README (#5664) * fix command argument in README * explain file structure * typo * diff between file and directory * typo * commeng location * make comment and variable understandable * fix issue-702 (#5708) * Change the environment of container and fix the problem of cannot compare two variables. (#5620) * add assembly * Enable FL Sever in SGX * Add start-fl-server.py to Docker Image * rm ppml-assebly.xml * Add start-python-fl-server-sgx.sh to Docker Image * Put start-fl-server.py into assembly * Change the environment of container and fix the problem of cannot compare two variables * Change the location of command "rm protobuf" in Dockerfile * Add explicit cast to avoid thread being syspended. * Change docker-gramine/Dockerfile to fit FL's environment requirements. Co-authored-by: Jiaming * Chronos: PR validation-spark3.1-ray job to Github Action (#5681) * pytest ray-part1 * pytest ray-part1+part2 Co-authored-by: binbin * Add Friesian README (#5709) * Create README.md * Update README.md * Update README.md * Add create dynamic badge job (#5706) * add create-job-status-badge action * add selfHostedRunner input * add create badge job * change isSelfHostedRunner to is-self-hosted-runner * Nano : migrate trace and quantize from Trainer to InferenceOptimizer (#5667) * basic migration * add load_xxx_model * modify init * Chronos: install openvino-dev in Github action (#5704) * install openvino * fix * fix * numpy==1.19.5 Co-authored-by: binbin * Chronos: Fix `LazyImport` known issue (#5502) * fix known issues * fix known isses * Add nano tensorflow examples (#5463) * add tensorflow multi instance training * add tensorflow sparse embedding example * Chronos: Add support for `id_sensitive`=True to Forecaster.from_tsdataset (#5551) * supports id_sensitive=True * new params is_predict * rollback is_predict * fix known issues * fix code style * Chronos: how to guide page cleaning (#5657) * update * add nblink * windows guide fix * add some updated * update chronos * add update image * add new image * add updates * add update * add new image * fix typo * fix notebook location * add new index * update * fix some doc * update tree * fix typo * Fix the problem of indents. (#5714) * add assembly * Enable FL Sever in SGX * Add start-fl-server.py to Docker Image * rm ppml-assebly.xml * Add start-python-fl-server-sgx.sh to Docker Image * Put start-fl-server.py into assembly * Fix the problem of indents. Co-authored-by: Jiaming * Change the editions of torch and torchvision. (#5717) * add assembly * Enable FL Sever in SGX * Add start-fl-server.py to Docker Image * rm ppml-assebly.xml * Add start-python-fl-server-sgx.sh to Docker Image * Put start-fl-server.py into assembly * Change the editions of torch and torchvision Co-authored-by: Jiaming * Pentest Issue: remove hardcode credentials (#5649) * remove hardcode credentials * update * add orca ut to PR_validation.yml * use redis_paasword in extra_params otherwise use the default password in ray * refine code * update * typo * remove orca ut from pr validation * update * update * Modify two errors in create-job-badge and license-scan workflow. (#5718) * add create-job-status-badge action * add selfHostedRunner input * add create badge job * change isSelfHostedRunner to is-self-hosted-runner * update create-job-status-badge * adjust indent * Update create aks script on Graphene (#5566) * update create aks * update * update * fix create sp * fix * make executable for az scripts * install command Co-authored-by: root * Add long descriptions to pypi releases (#5711) * long desc * style * meet comment * Update PR_validation.yml (#5722) * add orca horovod tf2 test to github actions (#5601) * add orca horovod tf2 test to github actions * update pull_request * update * add keras * update python version * update horovod * update cmake * install bigdl-orca-spark3 * update * update mxnet * update * update numpy * update * update ray version * unintsall intel-tensorflow * add conda env * install horovod * remove conda env * install bigdl-orca-spark3 * split conda env * source conda * split conda env * update * update env * remove env * update * update * uninstall requirements * update * update install whl * update * add remove env * install bigdl-orca-spark3 * update * Add conditional judgment in scala-style-check and python-style-check (#5721) * add create-job-status-badge action * add selfHostedRunner input * add create badge job * change isSelfHostedRunner to is-self-hosted-runner * update create-job-status-badge * adjust indent * update python-style-check and scala-style-check * Update scala-style-check.yml * Update python-style-check.yml * [PPML] Initiate README.md for AttestationCLI (#5688) * Init * Refine * Refine * Refine according to comments * Chronos: install openvino in Chronos-PRVN-Spark3.1-ray check (#5719) Co-authored-by: binbin * Chronos: PR validation-spark3.1-onnxrt job to Github Action (#5725) Co-authored-by: binbin * Chronos: test onnxrt==1.11.1 in PR validation-spark2.4-onnxrt job (#5677) * onnxrt==1.11.1 * fix Co-authored-by: binbin * [PPML] change KMS appkey to apikey (#5658) * change appkey to apikey * change APiKEY to APIKEY Co-authored-by: Heyang Sun <60865256+Uxito-Ada@users.noreply.github.com> * [PPML]delete unused argument in kms-utils README(#5733) * [PPML] Fix format error of quote from attestation service (#5732) * Fix format error of quote fom attestatioon service * Refine * Refine * skip dummy verify Co-authored-by: xiangyuT * release fat jars (#5736) * release fat jars * format * [DLlib] GBT CriteoClickLogsDataset example (#5723) * init gbt class * remove something unimportant * add readme * change xgb to gbt * use overwrite to save * add text to fix message=Header does not match expected text line=1 * Chronos: 'forecaster.predict_with_openvino' supports quantization API (#5690) * add openvino quantization * add some comments * fix code style * update assembly pom (#5737) Co-authored-by: Zhou * Add more data types in convert DataFrame to numpy (#5680) * add more datatype * update unit test * update ut in test_estimator_pyspark_backend * update ut in test_estimator_pyspark_backend * update ut in test_estimator_ray_backend * update ut in test_estimator_ray_backend * add array decimal type * update array data type Co-authored-by: Zhou * copy config (#5731) * Nano : Enhancement for output format of InferenceOptimizer (#5705) * first commit * improve output of optimize * fix method_type for method lack of dependency * update openvino model for thread num * add thread num for trace and quantize * update based on comment: modify output, add progress bar and remove method type * modify latency to latency(ms) * add thread_num for Trainer.trace and Trainer.quantize * fix bug of openvino cpu_num * [PPML]doc: change appkey to apikey in kms README (#5747) * cicd: add debug info to license-scan workflow (#5739) * add debug info * debug * echo * update * update * Add nightly-test job badge (#5750) * add create-job-status-badge action * add selfHostedRunner input * add create badge job * change isSelfHostedRunner to is-self-hosted-runner * update create-job-status-badge * adjust indent * update python-style-check and scala-style-check * Update scala-style-check.yml * Update python-style-check.yml * Update nightly_test.yml * refactor assmbly deploy (#5753) * Fix702 (#5727) add lightgbm scope * [PPML] Implement VerificationCLI to verify Attestation Service (#5742) * Implement VerificationCLI and a bash to use VerificationCLI * Refine Co-authored-by: xiangyuT * Add ipython extension for rendering code blocks in excuted notebooks properly in our readthedocs project (#5757) * [Orca] Refacor `model_dir` as optional in pytorch pyspark estimator (#5738) * refactor model_dir as optional in pytorch estimator * update ut to test non model_dir * support standard scaler for shards (#5716) * support standard scaler * support param map for lightGBM (#5712) * Support load state dict in TorchNano (#5687) * support load state dict in TorchNano * fix UT * add http prometheus client (#5743) * [PPML] Add occlum gbt readme and fix file path (#5756) * change path examples to example * add run_spark_gbt_criteo.sh * update readme and timeout config * some improvement (#5765) * add conda env to orca test actions (#5735) * add conda env to orca test actions * add pull_request * update env * update * update * update * update * update * update cmake * update ut * update ut * update requirements horovod pytorch * update * update * update protobuf * update horovod test * add tensorflow * update tensorflow * update * update python exampletet * update python UT * update * update * update * [PPML] Remove XGBoost from PPML guide * Nano : reduce time cost for InferenceOptimizer and update demo (#5740) * update readme and reduce time cost for calculating accuracy * add prune type * filter warnings * add check for input_sample * update based on comment * update readme and add ut * update input_sample * delete redundant lines * add forward_args * Chronos: add conda env to Chronos PRVN Actions (#5744) * [PPML] Fix gbt example spark config error (#5774) * Fix the conflict of `KMP_AFFINITY` and onnxruntime (#5764) * remove `compact` option from KMP_AFFINITY's default value * [PPML] Fix tracker bind issue happens in xgboost examples with SGX enabled (#5766) * Change tracker.py to acquire IP automatically * Remove the dependencies in documents on RABIT_TRACKER_IP * Set xgboost.spark.ignoreSsl to True in xgboost examples * Add deprecate hint for xgboost examples * [PPML]Add log file argument to bigdl-ppml-submit.sh (#5726) * add log file argument * change demo for log argument * Orca OpenVINO estimator add input order param & dataframe input support multi-output (#5471) * update * df support multi-output * temp * update * update test * meet comments * meet comments * del pred * update testcase * [Nano] How-To Guides: Training - PyTorch Lightning (#5643) * Rearrange file structure for PyTorch Inference for docs and add titles for PyTorch-Lightning Training * Add How-to guide: How to accelerate a PyTorch-Lightning application on training workloads through Intel® Extension for PyTorch* * Add how-to guide: How to accelerate a PyTorch-Lightning application on training workloads through multiple instances * Revise: remove '-' in 'PyTorch-Lightning' and some other changes * Add How-to guides: How to use the channels last memory format in your PyTorch Lightning application for training * Add how-to guide: Use BFloat16 Mixed Precision for PyTorch Lightning Training * Add How-to guide: How to accelerate a computer vision data processing pipeline * Small Revision: change comments in several code cells * Disable other nano tests temporarily * Add github action tests for Nano Training PyTorch Lightning tests * Enable other nano tests again * Small revisions: typos and explanation texts changes * Revise: update based on comments * Chronos: fix `quantize` and `to_torch_data_loader` API doc (#5755) * Add region proxy for create-job-badge action (#5761) * add create-job-status-badge action * add selfHostedRunner input * add create badge job * change isSelfHostedRunner to is-self-hosted-runner * update create-job-status-badge * adjust indent * update python-style-check and scala-style-check * update * Update scala-style-check.yml * Update python-style-check.yml * add nightly-test job badge * Update nightly_test.yml * Add region proxy * Chronos: fix install cmd in Chronos User Guide document (#5776) * PPML: Initiate doc for verificationCLI (#5770) * Initiate doc for verificationCLI and refine * Update README.md Co-authored-by: xiangyuT * PPML: mod TPCH document (#5583) * doc: modify markdown grammar typo * doc: md typo * typo * fix * fix * fix * hint * correct argument * add path of password to container * fix syntax errors * fix * fix * [Nano] Update Trainer -> InferenceOptimizer for Related How-to Guides (#5778) * Update Trainer to InferenOptimizer for related how-to guides * Disable other nano tests temporarily * Enable other nano tests again * tdx/scripts (#5636) * Fix runnerHostedOn input (#5784) * add create-job-status-badge action * add selfHostedRunner input * add create badge job * change isSelfHostedRunner to is-self-hosted-runner * update create-job-status-badge * adjust indent * update python-style-check and scala-style-check * update * Update scala-style-check.yml * Update python-style-check.yml * add nightly-test job badge * Update nightly_test.yml * Add region proxy * fix runnerHostedOn input * tdx/docker/bigdl-tdx (#5637) * tdx/docker/bigdl-tdx * address comment * Lightgbmpython (#5696) Port scala lightgbm to python * [Orca] Support save model weights in tf2 ray estimator (#5780) * add save load weights func * hot fix pyspark bug * add ut * code style fix * Nano : add API for InferenceOptimizer and deprecated some Trainer APIs (#5768) * add api and deprecated * add license * add deprecated * update comment for api display * modify comment * update based on comment * [Nano] Update Trainer -> InferenceOptimizer for Related Examples (#5781) * Update Trainer to InferenceOptimizer for related examples * Diasble unrelated nano tests * Enable unrelated nano tests again * Doc: add ray port FAQ to ray overview document. (#5752) * feat: add faq to ray overview doc. * fix: fix doc style. * fix: fix typo. * feat: add ray redis and dashboard port conflict to faq. * fix: fix redis port param name. * fix: fix redis port param name. * fix: fix typo * fix: fix typo * fix: fix wording. * tdx/docker/client-image (#5639) * Add Friesian multi task example code and readme (#5460) * add multi task example codes and readme * fix for Friesian code style * part 1: fix comments in pr * update for comments part 2 * update for pr comments part 3 * update for code style * update codes and add test results * Update README.md * Add Nano openvino async api (#5615) * add openvino async_predict api * add unit test for async_predict api * fix codestyle error * optimize input/output format and doc string * change para name, docstring & add type hint * fix error when calling on_forward_end after the output format of __call__ in OpenVINOModel is changed * fix codestyle check error & disable mypy type check * fix codestyle check error * use model_exist_or_error * Fix orca.pandas.read_csv() only return top 10k rows issue (#5775) * bug fix * add ut * add test file (#5788) * update pod template (#5786) * Add nano pytorch training bf16 example (#5600) * rollback requirement-doc * Add Nano PyTorch BF16 Example * Add comments and remove testing loop * Update * Nano: Add tensorflow quantization example. (#5619) * Nano: Add tensorflow quantization example. * simplify quantize arguments * add some extra comments * delete concrete numbers * change dataset to cifar10 * change file name to cifar10 * improve code organization * remove unnecessary comments * Chronos: how to evaluate using forecaster (#5710) * [PPML] Change E2E XGB example to Spark GBT (#5785) * change to xgb to gbt * use overwrite to save * Update multi-task example (#5787) * update multi-task * fix * minor * minor * [PPML] Fix BigDL Graphene Azure scripts (#5767) * update * update submit scripts * Chronos: fix input feature num != output feature num case for autoformer (#5673) * fix input feature num != output feature num case * add updates * Lightgbm (#5789) * error message * update Databricks user guide (#5779) * update databricks doc * update databricks doc * update databricks doc * update databricks doc * update databricks doc * update databricks doc Co-authored-by: Zhou * [PPML] Update end to end GBT example readme (#5792) * update end to end gbt readme * update header location * add bigdl-ppml jar with denpendency jar in tdx docker (#5793) * Fix PanTest issue（partly） -- ObjectInputStream's readObject method may deserialize untrusted data (#5666) * fix readObject deserialize untrusted data * use classTag[T] and use classof[modue] * remove double init inputStream * add accept pattern * accept Array * add EncodeUtils ut * add success case and refine fail case * add headers * Add warning for Distributed XGBoost examples (#5795) * tdx e2e readme (#5640) * update graphene ppml examples to main (#5804) * update graphene ppml examples * update graphene ppml examples Co-authored-by: Jin Hanyu <476099001@qq.com> * Fix a bug of readthedocs about get_public_dataset (#5808) * Chronos: fix ZooTestCase in tensorflow unit tests (#5800) * Chronos: decouple mtnet forecaster with orca (#5815) * update mtnet model to decouple with orca * rename one of the method * update test-suites scripts (#5823) Co-authored-by: Jin Hanyu <476099001@qq.com> * update tpp blas component information (#5825) Co-authored-by: Jin Hanyu <476099001@qq.com> * Chronos : support confidence interval for forecaster (#5682) * first commit, add predict_interval * add return value of calculate * modify parameter name, only calculate data noise once * add ut for lstm, nbeats, s2s, tcn * add paper link, change evaluate to predict * modify data noise * add ut for loader c and change to mean_yhat * support confidence interval for autoformer * update ppml examples scripts (#5827) Co-authored-by: Jin Hanyu <476099001@qq.com> * Chronos/Nano: update nano and chronos action trigger domain (#5809) * update nano and chronos action trigger domain * update * delete orca.learn as a trigger * Support tf2.1 load model for ray backend (#5830) * support tf2.1 * meet review * Chronos: More autoformer bug fix and enhancement (#5741) * midterm push for working in cafe:) * add updates * remove confusing comments * consider the conflict between tsdataset.label_len and label_len * update according to comments, fix pep8 * add test cmt Co-authored-by: Jin Hanyu <48324756+Hanyu-Jin@users.noreply.github.com> Co-authored-by: Jiaming Co-authored-by: Mingzhi Hu <49382651+y199387@users.noreply.github.com> Co-authored-by: Zhentao Chen <90437536+zhentaocc@users.noreply.github.com> Co-authored-by: Cengguang Zhang Co-authored-by: Juntao_Luo <71260173+AndyLuo1029@users.noreply.github.com> Co-authored-by: binbin Deng <108676127+plusbang@users.noreply.github.com> Co-authored-by: dding3 Co-authored-by: SONG Ge <38711238+sgwhat@users.noreply.github.com> Co-authored-by: Shaojun Liu <61072813+liu-shaojun@users.noreply.github.com> Co-authored-by: Nancy <51090416+yexinyinancy@users.noreply.github.com> Co-authored-by: yansu Co-authored-by: Guancheng Fu <110874468+gc-fu@users.noreply.github.com> Co-authored-by: ZehuaCao <47251317+Romanticoseu@users.noreply.github.com> Co-authored-by: Ruonan Wang <105281011+rnwang04@users.noreply.github.com> Co-authored-by: Shaojie Cui Co-authored-by: Wang Jian <61138589+hzjane@users.noreply.github.com> Co-authored-by: Xiangyu Tian <109123695+xiangyuT@users.noreply.github.com> Co-authored-by: Heyang Sun <60865256+Uxito-Ada@users.noreply.github.com> Co-authored-by: Zhao Changmin Co-authored-by: Kai Huang Co-authored-by: pinggao187 <44044110+pinggao187@users.noreply.github.com> Co-authored-by: shaojie <2779090360@qq.com> Co-authored-by: baishaojie Co-authored-by: Shan Yu Co-authored-by: Aixi Deng <96048273+aixideng@users.noreply.github.com> Co-authored-by: liangs6212 <80952198+liangs6212@users.noreply.github.com> Co-authored-by: Qiyuan Gong Co-authored-by: Jason Dai Co-authored-by: Jiao Wang Co-authored-by: Jian Zhou <41574757+PatrickkZ@users.noreply.github.com> Co-authored-by: Zhou Co-authored-by: Yuwen Hu <54161268+Oscilloscope98@users.noreply.github.com> Co-authored-by: Jin Hanyu <476099001@qq.com> Co-authored-by: Junwei Deng <35031544+TheaperDeng@users.noreply.github.com> Co-authored-by: xiangyuT Co-authored-by: Chaselzxy <47959406+Chaselzxy@users.noreply.github.com> Co-authored-by: Xinyi Zhang Co-authored-by: theaperdeng Co-authored-by: Le-Zheng <30695225+Le-Zheng@users.noreply.github.com> Co-authored-by: fuckyouNoob Co-authored-by: Yishuo Wang Co-authored-by: Guoqiong Song Co-authored-by: binbin Co-authored-by: Hu, Zhaojie Co-authored-by: glorysdj Co-authored-by: David smurf <63043716+smurf-1119@users.noreply.github.com> Co-authored-by: root Co-authored-by: Yina Chen <33650826+cyita@users.noreply.github.com> Co-authored-by: WangBin <32730386+devWangBin@users.noreply.github.com> Co-authored-by: mycprotein <51121231+mycprotein@users.noreply.github.com> --- python/orca/dev/example/requirements-ray.txt | 1 - .../orca/dev/example/run-example-test-ray.sh | 46 ++++++++----------- 2 files changed, 20 insertions(+), 27 deletions(-) diff --git a/python/orca/dev/example/requirements-ray.txt b/python/orca/dev/example/requirements-ray.txt index 5e2edffca7c..063f48c0988 100644 --- a/python/orca/dev/example/requirements-ray.txt +++ b/python/orca/dev/example/requirements-ray.txt @@ -11,7 +11,6 @@ pandas ray[default]==1.9.2 dm_tree gym[atari]==0.17.1 -atari-py==0.2.5 tabulate==0.8.7 tensorflow==1.15.0 tensorflow-gan==2.0.0 diff --git a/python/orca/dev/example/run-example-test-ray.sh b/python/orca/dev/example/run-example-test-ray.sh index bfaa8c33905..5a1e9736c6c 100644 --- a/python/orca/dev/example/run-example-test-ray.sh +++ b/python/orca/dev/example/run-example-test-ray.sh @@ -45,31 +45,26 @@ time3=$((now-start)) set -e ray stop -f -echo "#4 Start rl_pong example" -start=$(date "+%s") -python ${BIGDL_ROOT}/python/orca/example/ray_on_spark/rl_pong/rl_pong.py --iterations 10 -now=$(date "+%s") -time4=$((now-start)) -echo "#5 Start multi_agent example" +echo "#4 Start multi_agent example" start=$(date "+%s") python ${BIGDL_ROOT}/python/orca/example/ray_on_spark/rllib/multi_agent_two_trainers.py --iterations 5 now=$(date "+%s") time5=$((now-start)) -echo "#6 Start async_parameter example" +echo "#5 Start async_parameter example" start=$(date "+%s") python ${BIGDL_ROOT}/python/orca/example/ray_on_spark/parameter_server/async_parameter_server.py --iterations 10 now=$(date "+%s") time6=$((now-start)) -echo "#7 Start sync_parameter example" +echo "#6 Start sync_parameter example" start=$(date "+%s") python ${BIGDL_ROOT}/python/orca/example/ray_on_spark/parameter_server/sync_parameter_server.py --iterations 10 now=$(date "+%s") time7=$((now-start)) -echo "#8 Start mxnet lenet example" +echo "#7 Start mxnet lenet example" start=$(date "+%s") # get_mnist_iterator in MXNet requires the data to be placed in the `data` folder of the running directory. @@ -86,7 +81,7 @@ python ${BIGDL_ROOT}/python/orca/example/learn/mxnet/lenet_mnist.py -e 1 -b 256 now=$(date "+%s") time8=$((now-start)) -echo "#9 Start fashion_mnist example with Tensorboard visualization" +echo "#8 Start fashion_mnist example with Tensorboard visualization" start=$(date "+%s") if [ -d ${BIGDL_ROOT}/python/orca/example/learn/pytorch/fashion_mnist/data ] @@ -106,7 +101,7 @@ now=$(date "+%s") time9=$((now-start)) -echo "#10 start example for orca super-resolution" +echo "#9 start example for orca super-resolution" start=$(date "+%s") if [ ! -f BSDS300-images.tgz ]; then @@ -123,7 +118,7 @@ now=$(date "+%s") time10=$((now-start)) -echo "#11 start example for orca cifar10" +echo "#10 start example for orca cifar10" start=$(date "+%s") if [ -d ${BIGDL_ROOT}/python/orca/example/learn/pytorch/cifar10/data ]; then @@ -139,7 +134,7 @@ now=$(date "+%s") time11=$((now-start)) -echo "#12 Start autoxgboost example" +echo "#11 Start autoxgboost example" if [ -f ${BIGDL_ROOT}/data/incd.csv ] then echo "incd.csv already exists" @@ -153,7 +148,7 @@ now=$(date "+%s") time12=$((now-start)) -echo "#13 Start ray dataset xgboost example" +echo "#12 Start ray dataset xgboost example" if [ -f ${BIGDL_ROOT}/data/incd.csv ] then echo "incd.csv already exists" @@ -166,7 +161,7 @@ python ${BIGDL_ROOT}/python/orca/example/data/ray-dataset-xgboost.py -p ${BIGDL_ now=$(date "+%s") time13=$((now-start)) -echo "#14 start example for orca brainMRI" +echo "#13 start example for orca brainMRI" if [ -f ${BIGDL_ROOT}/python/orca/example/learn/pytorch/brainMRI/kaggle_3m ] then echo "kaggle_3m already exists" @@ -187,14 +182,13 @@ echo "Ray example tests finished" echo "#1 auto-estimator-pytorch time used:$time1 seconds" echo "#2 auto-xgboost-classifier time used:$time2 seconds" echo "#3 auto-xgboost-regressor time used:$time3 seconds" -echo "#4 orca rl_pong time used:$time4 seconds" -echo "#5 orca async_parameter_server time used:$time5 seconds" -echo "#6 orca sync_parameter_server time used:$time6 seconds" -echo "#7 orca multi_agent_two_trainers time used:$time7 seconds" -echo "#8 mxnet_lenet time used:$time8 seconds" -echo "#9 fashion-mnist time used:$time9 seconds" -echo "#10 orca super-resolution example time used:$time10 seconds" -echo "#11 orca cifar10 example time used:$time11 seconds" -echo "#12 auto-xgboost-regressor-spark-df example time used:$time12 seconds" -echo "#13 ray-dataset-xgboost example time used:$time13 seconds" -echo "#14 orca brainMRI example time used:$time14 seconds" +echo "#4 orca async_parameter_server time used:$time5 seconds" +echo "#5 orca sync_parameter_server time used:$time6 seconds" +echo "#6 orca multi_agent_two_trainers time used:$time7 seconds" +echo "#7 mxnet_lenet time used:$time8 seconds" +echo "#8 fashion-mnist time used:$time9 seconds" +echo "#9 orca super-resolution example time used:$time10 seconds" +echo "#10 orca cifar10 example time used:$time11 seconds" +echo "#11 auto-xgboost-regressor-spark-df example time used:$time12 seconds" +echo "#12 ray-dataset-xgboost example time used:$time13 seconds" +echo "#13 orca brainMRI example time used:$time14 seconds"