Migrate orca context #4768
Merged
Migrate orca context #4768
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* load once * fix * fix * remove num_steps * remove iter * meet comments * wrap once * style fix
* add docs * minor
* initial imple * update * meet review * review and style * remove stopped * add doc * minor * move import * fix mxnet * remove
* update unit tests * minor * update * update mxnet * move barrier * fix mxnet * update * bug fix * update * update test * update mxnet example * update mxnet * minor * minor * minor * update examples * move ray import dependencies * readme * minor * bug fix * remove default
* support numa binding in init_spark_standalone * add doc and add to orca context * address comments * address comments * update scripts * hyperthreading * fix
* support ray on k8s * add to init orca context * style * minor * minor * ut
* update doc * update
* update example * Update README.md * update resnet_finetune.py * delete some file * Create README.md * Update README.md * Update README.md * Update README.md * add detect conda env name * some change * update run example scripte * update init orca context
* attempt to fix ray memory * exclude webui
* update * add back * update * update * minor * meet review * fix typo
* add mem type for dataframe dataset * add ZooTrigger support * update mem type * add test * update * update mem type and orca context * update orca context docs * update orca context with comments * fix style
* add and modify * style
* add shard size to dataframe_to_xshards * add ut and change default shard_size as None * add shard size in orca context and fix style * add ut in pytorch estimator and tf estimator * move shard_size to internal use * fix
* add support for non-barrier mode * fix style * meet review * meet review * move barrier mode to zoocontext * bug fix * modify * update
migrate bigdl to bigdl 2.0
Move visulazition from dllib.utils to dllib
Migrate bigdl python code to bigdl2.0
…ture Update python structure
…ture Update python structure
migrate python estimator
migrate python estimator
migrate orca python file
Keras others uts
Keras others uts
Move orca/src/test to orca/test in BigDL2.0
…tics#2593) * orca context * handle error * enrich error msg * meet review * move log output * style * meet review * add zoocontext * fix ut
shanyu-sys
added a commit
that referenced
this pull request
Sep 22, 2021
This reverts commit 8776fa5.
shanyu-sys
added a commit
that referenced
this pull request
Sep 22, 2021
Le-Zheng
added a commit
to Le-Zheng/analytics-zoo
that referenced
this pull request
Sep 22, 2021
* Orca PyTorch Estimator load data once (intel-analytics#2669) * load once * fix * fix * remove num_steps * remove iter * meet comments * wrap once * style fix * Add Orca Overview and Context Doc (intel-analytics#2748) * add docs * minor * Add init_orca_context (intel-analytics#2774) * initial imple * update * meet review * review and style * remove stopped * add doc * minor * move import * fix mxnet * remove * Update UTs and examples with init_orca_context (intel-analytics#2787) * update unit tests * minor * update * update mxnet * move barrier * fix mxnet * update * bug fix * update * update test * update mxnet example * update mxnet * minor * minor * minor * update examples * move ray import dependencies * readme * minor * bug fix * remove default * Add website doc for init_orca_context (intel-analytics#2822) * Support numa binding in init_spark_standalone (intel-analytics#2847) * support numa binding in init_spark_standalone * add doc and add to orca context * address comments * address comments * update scripts * hyperthreading * fix * shutdown hook (intel-analytics#2853) * Support RayOnSpark for k8s and add docs (intel-analytics#2836) * support ray on k8s * add to init orca context * style * minor * minor * ut * Fix stop_orca_context being called twice (intel-analytics#2878) * Update website doc of init orca context (intel-analytics#2879) * update doc * update * update Torch example (intel-analytics#3022) * update example * Update README.md * update resnet_finetune.py * delete some file * Create README.md * Update README.md * Update README.md * Update README.md * add detect conda env name * some change * update run example scripte * update init orca context * attempt to fix ray memory (intel-analytics#3205) * attempt to fix ray memory * exclude webui * Update doc for orca context (intel-analytics#3287) * update * add back * update * update * minor * meet review * fix typo * Add memory type support for Orca tf estimator (intel-analytics#3280) * add mem type for dataframe dataset * add ZooTrigger support * update mem type * add test * update * update mem type and orca context * update orca context docs * update orca context with comments * fix style * Add get_spark_session in OrcaContext (intel-analytics#3520) * add and modify * style * add shard size to dataframe_to_xshards (intel-analytics#3491) * add shard size to dataframe_to_xshards * add ut and change default shard_size as None * add shard size in orca context and fix style * add ut in pytorch estimator and tf estimator * move shard_size to internal use * fix * Add support for non-barrier mode to launch ray (intel-analytics#4014) * add support for non-barrier mode * fix style * meet review * meet review * move barrier mode to zoocontext * bug fix * modify * update * remove driver cores (intel-analytics#4169) * Add OrcaContext and make spark default read file backend (intel-analytics#2593) * orca context * handle error * enrich error msg * meet review * move log output * style * meet review * add zoocontext * fix ut * change import Co-authored-by: Yina Chen <[email protected]> Co-authored-by: Kai Huang <[email protected]> Co-authored-by: Yang Wang <[email protected]> Co-authored-by: Xin Qiu <[email protected]> Co-authored-by: jenniew <[email protected]> Co-authored-by: dding3 <[email protected]> Co-authored-by: Le-Zheng <[email protected]>
dding3
added a commit
to dding3/analytics-zoo
that referenced
this pull request
Oct 4, 2021
* Orca PyTorch Estimator load data once (intel-analytics#2669) * load once * fix * fix * remove num_steps * remove iter * meet comments * wrap once * style fix * Add Orca Overview and Context Doc (intel-analytics#2748) * add docs * minor * Add init_orca_context (intel-analytics#2774) * initial imple * update * meet review * review and style * remove stopped * add doc * minor * move import * fix mxnet * remove * Update UTs and examples with init_orca_context (intel-analytics#2787) * update unit tests * minor * update * update mxnet * move barrier * fix mxnet * update * bug fix * update * update test * update mxnet example * update mxnet * minor * minor * minor * update examples * move ray import dependencies * readme * minor * bug fix * remove default * Add website doc for init_orca_context (intel-analytics#2822) * Support numa binding in init_spark_standalone (intel-analytics#2847) * support numa binding in init_spark_standalone * add doc and add to orca context * address comments * address comments * update scripts * hyperthreading * fix * shutdown hook (intel-analytics#2853) * Support RayOnSpark for k8s and add docs (intel-analytics#2836) * support ray on k8s * add to init orca context * style * minor * minor * ut * Fix stop_orca_context being called twice (intel-analytics#2878) * Update website doc of init orca context (intel-analytics#2879) * update doc * update * update Torch example (intel-analytics#3022) * update example * Update README.md * update resnet_finetune.py * delete some file * Create README.md * Update README.md * Update README.md * Update README.md * add detect conda env name * some change * update run example scripte * update init orca context * attempt to fix ray memory (intel-analytics#3205) * attempt to fix ray memory * exclude webui * Update doc for orca context (intel-analytics#3287) * update * add back * update * update * minor * meet review * fix typo * Add memory type support for Orca tf estimator (intel-analytics#3280) * add mem type for dataframe dataset * add ZooTrigger support * update mem type * add test * update * update mem type and orca context * update orca context docs * update orca context with comments * fix style * Add get_spark_session in OrcaContext (intel-analytics#3520) * add and modify * style * add shard size to dataframe_to_xshards (intel-analytics#3491) * add shard size to dataframe_to_xshards * add ut and change default shard_size as None * add shard size in orca context and fix style * add ut in pytorch estimator and tf estimator * move shard_size to internal use * fix * Add support for non-barrier mode to launch ray (intel-analytics#4014) * add support for non-barrier mode * fix style * meet review * meet review * move barrier mode to zoocontext * bug fix * modify * update * remove driver cores (intel-analytics#4169) * Add OrcaContext and make spark default read file backend (intel-analytics#2593) * orca context * handle error * enrich error msg * meet review * move log output * style * meet review * add zoocontext * fix ut * change import Co-authored-by: Yina Chen <[email protected]> Co-authored-by: Kai Huang <[email protected]> Co-authored-by: Yang Wang <[email protected]> Co-authored-by: Xin Qiu <[email protected]> Co-authored-by: jenniew <[email protected]> Co-authored-by: dding3 <[email protected]> Co-authored-by: Le-Zheng <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.