Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update #8

Merged
merged 253 commits into from
Sep 20, 2022
Merged

Update #8

merged 253 commits into from
Sep 20, 2022

Conversation

ForJadeForest
Copy link
Owner

Update

hkvision and others added 30 commits August 22, 2022 18:57
* add assembly

* Enable FL Sever in SGX

* Change the url in tutorial.md

* Add start-fl-server.py to Docker Image

* rm ppml-assebly.xml

* Add start-python-fl-server-sgx.sh to Docker Image

* Put start-fl-server.py into assembly

* Update 3.1.2 Start FL Server in SGX in pytorch-nn-lr-tutorial.md

* Change the location of start-fl-server.py

* Correct the path in start-python-fl-server-sgx.sh

* Delete -r in cp command

Co-authored-by: Jiaming <[email protected]>
* fix tf inc metric format

* re-run actions

* Apply suggestions from code review

Co-authored-by: Zhentao Chen <[email protected]>

* Update

Co-authored-by: Zhentao Chen <[email protected]>
* fix: update known-issues.

* fix: fix wording.
Migrate current jenkins CI/CD jobs to github action, this is chronos PR validation-Spark2.4-ray job
…5483)

When the input data to AutoformerForecaster.predict() is incorrect, the error message is confusing. Thus, a new error message is added.
* notebook with nano

* load with nano

* modify readme

* clear all outputs and enable benchmark_app

* fix typo
* add assembly

* Enable FL Sever in SGX

* Add start-fl-server.py to Docker Image

* rm ppml-assebly.xml

* Add start-python-fl-server-sgx.sh to Docker Image

* Put start-fl-server.py into assembly

Co-authored-by: Jiaming <[email protected]>
* rollback requirement-doc

* Update

* Update

* Update

* Update

* Fix ipex with amp error

* Fix unit test

* Update

* Remove redundant method

* Fix error

* remove redundant import

* Update ut to conver cases
* add assembly

* Enable FL Sever in SGX

* Add start-fl-server.py to Docker Image

* rm ppml-assebly.xml

* Add start-python-fl-server-sgx.sh to Docker Image

* Put start-fl-server.py into assembly

* Fix the target directory

Co-authored-by: Jiaming <[email protected]>
* remove pyarrow dependency in tf2 ray backend

* update ut to compare prediction results
…d and nightly_test (#5511)

* reorg workflows
* remove duplicate yamls
* add dependencies
…5504)

* feat: add save and load model doc to pytorch estimator quickstart.

* fix: fix typo.

* fix: fix typo

* fix: fix typo.

* feat: add doc to ray backend quickstart

* fix: fix typo.

* feat: add doc to ray backend quickstart
* feat: add shutdown API to tf2 pyspark estimator.

* fix: replace raise with log4Error.

* refactor: refactor the stop log server function.

* fix: fix code style issue.
* fix: use default value if no hdfs port specified.

* fix: add default hdfs port when saving pkl

* refactor: replace pyarrow deps in estimator.

* fix: add process wait()

* fix: resolve unnecessary change
* Update

* Fix code style

* re-run action

* Fix code style

* re-run action

* Update

* Fix code style

* support bf16 multi training

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* reduce ut time and re-run action

* track avx512

* Update lite bf16 training

* Update

* Update

* Update bf16 api

* Update

* Fix typo
* feat: add get remote file to local decorator in orca file utils.

* feat: add unit-test for local and s3 FS.

* fix: update func name and comments.

* fix: fix code style.

* fix: fix code style.

* fix: fix typo.

* fix: fix import.
* fix save model in pyspark est

* use a function to replace

* use orca.data to replace dllib
* Add Xshards test to github actions

* add file path

* fix yml

* fix csv path

* fix csv path

* fix yml

* add wget

* fic typo

* fix typo

* change csv file path

* update pwd

* add new line

* delete a line

* add time

Co-authored-by: yansu <[email protected]>
* add assembly

* Enable FL Sever in SGX

* Add start-fl-server.py to Docker Image

* rm ppml-assebly.xml

* Add start-python-fl-server-sgx.sh to Docker Image

* Put start-fl-server.py into assembly

* Change some packages' edition

* Fix the problem of fl/start-fl-server.py

Co-authored-by: Jiaming <[email protected]>
Reason to change: tracker.py in ${BIGDL_HOME}/jars/bigdl-dllib-spark_${SPARK_VERSION}-${BIGDL_VERSION}.jar is not functional.

Instead of using tracker.py to build ${BIGDL_HOME}/jars/bigdl-dllib-spark_${SPARK_VERSION}-${BIGDL_VERSION}.jar,
we now use it to build xgboost4j_2.12-1.1.2.jar, which can be reached when executing xgboost examples.
* Create nightly-build-example-tests-ppml.yaml

* Update nightly-build-example-tests-ppml.yaml

Triggered by schedule

* Update nightly-build-example-tests-ppml.yaml
* set use_hpo=True and add ut
* feat: add k8s related files

* fix: change secure-argv to secure_argv in Gramine

* fix: check if secured_argv exists with -f

* fix: exit when not ready in init.sh

* fix: change path of encalve-key in k8s template

* feat: copy k8s directory to Gramine directory

* mod: version in development

* mod: enclave-key path

* del: remove tmp build action

* feat: Simplify deployment operations for users

* doc: update README.md

* doc: correct a typo

* doc: correct typo

* add spark pi on k8s scripts
dding3 and others added 28 commits September 15, 2022 12:44
* rollback requirement-doc

* Add Nano PyTorch BF16 Example

* Add comments and remove testing loop

* Update
* Nano: Add tensorflow quantization example.

* simplify quantize arguments

* add some extra comments

* delete concrete numbers

* change dataset to cifar10

* change file name to cifar10

* improve code organization

* remove unnecessary comments
* change to xgb to gbt

* use overwrite to save
* update multi-task

* fix

* minor

* minor
* update

* update submit scripts
…mer (#5673)

* fix input feature num != output feature num case

* add updates
* error message
* update databricks doc

* update databricks doc

* update databricks doc

* update databricks doc

* update databricks doc

* update databricks doc

Co-authored-by: Zhou <[email protected]>
* update end to end gbt readme

* update header location
…y deserialize untrusted data (#5666)

* fix readObject deserialize untrusted data

* use classTag[T] and use classof[modue]

* remove double init inputStream

* add accept pattern

* accept Array

* add EncodeUtils ut

* add success case and refine fail case

* add headers
* update graphene ppml examples

* update graphene ppml examples

Co-authored-by: Jin Hanyu <[email protected]>
* update mtnet model to decouple with orca

* rename one of the method
* first commit, add predict_interval

* add return value of calculate

* modify parameter name, only calculate data noise once

* add ut for lstm, nbeats, s2s, tcn

* add paper link, change evaluate to predict

* modify data noise

* add ut for loader c and change to mean_yhat

* support confidence interval for autoformer
* update nano and chronos action trigger domain

* update

* delete orca.learn as a trigger
* midterm push for working in cafe:)

* add updates

* remove confusing comments

* consider the conflict between tsdataset.label_len and label_len

* update according to comments, fix pep8

* add test cmt
@ForJadeForest ForJadeForest merged commit 44f1608 into ForJadeForest:main Sep 20, 2022
ForJadeForest pushed a commit that referenced this pull request Oct 18, 2022
* vfl LR ckks (intel-analytics#10)

* tests

* add census

* add train

* update census

* update

* add time count

* add new layer

* update vfl client

* add decrypt

* ckksTest

* Merge ckks in FLServer LR Aggregator (#7)

* update ckks in FLServer LR Aggregator

* some changes

* support ckks at scala FL Server

* add

* add

* fix and add ut

* fix and add ut

* add support for customized client module and divide example to 2 parties

* add comments

* split

* Ckks & Dllib Example fix (#8)

* update ckks in FLServer LR Aggregator

* some changes

* support ckks at scala FL Server

* add

* add

* fix and add ut

* fix and add ut

* add support for customized client module and divide example to 2 parties

* add comments

* fix

* fix none target sample

* fix model

* fix train

* fix accuracy

* add shuffle

* enable ckks

* fix ckks

* Ckks fix the bug and add server error message (#9)

* update ckks in FLServer LR Aggregator

* some changes

* support ckks at scala FL Server

* add

* add

* fix and add ut

* fix and add ut

* add support for customized client module and divide example to 2 parties

* add comments

* fix

* fix

* some fix

* update

Co-authored-by: Song Jiaming <[email protected]>

* remove useless  dependency

* update client

* update client

* update

* fix style

* update

* update unit test

* code clean up

* fix ut

* add all-dependencies

* update

* update readme

* update readme and pom

* fix ut

* update pom

* update pom

* update

* add documents

* some update

* revert some code

Co-authored-by: Song Jiaming <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.