Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Move orca/src/test to orca/test in BigDL2.0 #4712

Merged
merged 134 commits into from
Sep 10, 2021
Merged

Move orca/src/test to orca/test in BigDL2.0 #4712

merged 134 commits into from
Sep 10, 2021

Conversation

shanyu-sys
Copy link
Contributor

If not and with all __init__.py added to orca/src/test, pytest will automatedly prepend orca/src to sys.path (refer here). However, since orca/src contains bigdl, python will import all bigdl from orca/src, which only includes orca, instead of from PYTHONPATH. In that case, we cannot import bigdl.dllib

For reference, the tree for bigdl2.0 orca is as below.

image

yangw1234 and others added 30 commits February 26, 2019 18:57
@yangw1234
add tfpark model api (intel-analytics#1094)
bd5673b 
* add tfpark model

* fix some bugs

* add tests

* add tests back

* add more tests

* fix style

* estimator init

* remvoe estimator test

* fix bug

* adjust for new dataset and add example

* rename file

* add example

* fix bugs

* fix style

* remove ut
@yangw1234
add tfpark unit tests (intel-analytics#1185)
9509139 
* add tfpark ut

* add tests

* fix style
@yangw1234
support tensorflow optimizer in keras model (intel-analytics#1205)
2cc5c2e 
* support tensorflow optimizer in keras model

* fix style
@hkvision
Add IntentEntity model in tfpark (intel-analytics#1188)
afb4a7a 
* add intent ner model and example

* add save load

* rename

* refactor

* add ut

* update and add nercrf

* remove example

* add more models

* add nlp_architect packaging

* remove models

* minor

* update

* style

* export env

* exclude py27

* ignore collection of tfpark text

* fix

* change to kwargs

* add ner

* style and meet review

* fix map bug

* rename to test text models

* merge with get bigdl python

* update prepare env

* align with optimizer update

* add doc

* update dist-all
@yangw1234
Add TFPark Estimator API (intel-analytics#1207)
a568b64 
* add estimator training

* add evaluate

* add predict and evaluate

* address comments

* change to metrics to string type

* add unit test

* fix style
@hkvision
Add sequence tagger in tfpark (intel-analytics#1213)
a72ff2a 
* add sequence tagger

* add ut

* style
@yangw1234
using latest tensorflow's freeze funtion, fix keras embedding freeze …
3656a15 
…bug (intel-analytics#1245)

* add graph util

* follow up

* update imports

* fix convert

* fix style

* fix bug
@yangw1234
support tfdataset without batch (intel-analytics#1249)
46594cf 
* support tfdataset without batch

* address comments

* fix style

* add test code

* fix style

* fix bug
@yangw1234
support init dataset from ndarrays (intel-analytics#1256)
039ff75
@hkvision
Add BERTClassifier (intel-analytics#1247)
b8f6145 
* add bert classifier

* add example

* minor

* minor

* minor

* package bert

* minor

* modify api

* minor

* minor

* revert

* update

* fix inputs

* update

* add it

* update

* remove unnecesary

* update and add docs

* update

* remove example

* fix style

* refactor to text/keras

* fix ut

* refactor
@yangw1234
Support Creating TFDataset from ImageSet/TextSet/FeatureSet (intel-an…
19c8344 
…alytics#1270)

* support create a ImageSet from a local folder of images

* fix style

* support create TFDataset from ImageSet

* refactor

* test

* support BigDL dataset

* remove code

* use datset

* add unit tests

* change to featureset

* add tests

* fix style

* fix tests

* fix tests

* add some docs

* add more docs

* fix style
@hkvision
Make bert classifier ut cost less memory (intel-analytics#1317)
9215af2
@hkvision
Change collect to count in bert classifier ut (intel-analytics#1319)
4f27cb6
@yangw1234
Support gradient clipping in TFPark (intel-analytics#1407)
0874afb 
* support gradient clipping

* remove code

* add more check

* add method in tfoptimizer

* add model test

* add tests

* fix style
@zhichao-li
Support init_spark_on_yarn and RayContext (intel-analytics#1344)
4510693 
* rayrunner

* add a jvm killer

* disable killer from spark job and rely on jvm killer only

* add env and verify the cv2 installation

* enhance

* minor

* style

* comments

* local and enhancement

* better local strategy

* doc and style

* doc

* style

* revert

* doc

* disable

* comments

* fix test
@hkvision
Support BERTSQuAD and BERTNER (intel-analytics#1401)
07d67a0 
* add bert squad code

* modify grad clip

* revert

* fix style

* init ner

* update examples

* minor update

* remove examples

* revert adamweightdecay

* update

* add ut
@zhichao-li
Enhance Ray on spark (intel-analytics#1449)
38e4728 
* add more doc, spark_conf and extra_options

* fix pip install

* doc

* fix release.sh
@zhichao-li
We should search and find the necessary jars from python env rather t…
21c37f4 
…han upload them again (intel-analytics#1460)

* fix path

* jars
@yangw1234
Fix tfdataset cannot work with list input (intel-analytics#1478)
e1bd16d 
* fix tfdataset cannot work with list input

* fix style
@zhichao-li
Remove ray dependencies from init_spark_on_yarn (intel-analytics#1500)
b5d0f98 
* remove ray dependencies

* delete
@zhichao-li
Add an example notebook for implementing PS with Ray (intel-analytics…
d8c3dfc 
…#1522)

* upup

* add ps notebook for ray
@zhichao-li
Expose more option for driver in RayContext (intel-analytics#1541)
43d4895 
* expose more option for driver

* minor

* more
@respecteverything
modify run-pytests to check version before test ray (intel-analytics#…
eee4561 
…1532)

* Create .keep

* update .keep path

* add rl_pong example

* move to rl_pong direction

* remove original file

* add parameter server example

* add license

* PEP8 checks

* PEP8 checks

* Add into integration test

* Wrap tests into bash function

* Update license

* PEP8 checks

* Correct syntax of rl_pong

* modify run-pytests to check version before test ray

* test pyspark version and spark home

* add check spark_home's pyspark in case pyspark can't be found

* add check version before run ray examples

* change spark home

* change spark home

* install packages which are needed in ray examples

* check error

* fix error

* change execution to spark-submit

* change memory

* change object memory to test

* add atari_py dependency

* remove .keep

* move ray test to new files

* change some ray-pip lines into function

* remove rl_pong and fix parameter_server iterations

* add iteration

* change iterate, print info

* add more info

* add __init__ files

* change ray to rayexample to avoid conflict and change spark-submit to python to submit tasks

* renamed foreach_evaluator to foreach_worker because rllib update and rename file rllib to rllibexample

* add a dedicated file for the ray test

* PEP8 check fix

* PEP8 check fix

* remove test_split

* remove --doctest-modules about ray

* add time.sleep
@yangw1234
support reading tfrecord in tfdataset (intel-analytics#1584)
42a9815 
* support reading tfrecord in tfdataset

* add doc and test

* fix style

* fix dependency

* fix test

* add resource file

* refactor test

* fix style

* address comment

* add test back

* fix style

* fix tests

* fix python 3.5
@yangw1234
Support TensorFlow metric in TFOptimizer (intel-analytics#1593)
78a5884 
* test

* support stateless metric

* support metric in TFOptimizer

* fix tests

* fix style

* add tests
@respecteverything
fix ray and add more test (intel-analytics#1596)
e2cc857 
* fix ray and add more test

Signed-off-by: Jieru Hong <[email protected]>

* modify raycontext and move test file to func

Signed-off-by: Jieru Hong <[email protected]>

* modify process and add sc.stop in the end

Signed-off-by: Jieru Hong <[email protected]>

* delete one repeat and check PEP8

Signed-off-by: Jieru Hong <[email protected]>

* change file name and remove some useless code

Signed-off-by: Jieru Hong <[email protected]>

* rename test yarn reinit file

Signed-off-by: Jieru Hong <[email protected]>

* ignore test reinit raycontext

Signed-off-by: Jieru Hong <[email protected]>
@yangw1234
Using non-freezing graph for TF training (intel-analytics#1488)
cb17dbc 
* support unfreeze training

* support two version

* fix

* fix

* fix test

* fix test

* fix test

* fix tests

* reduce test

* add tests back

* fix style

* split tests

* isolate tfpark tests

* change back tfnet
@hkvision
Minor fix for detecting sc is local (intel-analytics#1735)
41b019d 
* fix sc local

* update
@yangw1234
TFoptimizer use analytics-zoo estimator (intel-analytics#1749)
55aa5d6 
* tfoptimizer use analytics-zoo estimator

* fix style

* fix bug

* fix tests and add documents

* fix style

* fix test

* fix example

* fix test

* fix tests

* fix tests

* always repartition
@qiuxin2012
add callZooFunc and change all callBigDlFunc to callZooFunc (intel-an…
984ecc4 
…alytics#1793)
dding3 and others added 21 commits September 7, 2021 19:33
@dding3
merge with upstream
c47dfe9
@shanyu-sys
Delete automl (intel-analytics#4680)
595b698 
* rm automl

* rm test automl

* rm automl in tests
@shanyu-sys
Add ray daemon to kill ray processes (intel-analytics#4571)
6fe8fe9 
* add ray daemon

* remove in bigdl

* add ray daemon in start_restricted_worker

* change to static method

* remove ProcessMonitor.register_shutdown_hook and clean_fn

* change name

* clean useless code

* add license
@shanyu-sys
zoo.ray -> bigdl.orca.ray; zoo.util->bigdl.dllib.utils
18f783d
@shanyu-sys
zoo -> bigdl.dllib.utils.nncontext
a344eaa
@shanyu-sys
comment out other tests than ray
5dd473e
@shanyu-sys
change path in run-pytest-ray
254998a
@dding3
merge with upstream
9a4ce70
@dding3
Merge pull request intel-analytics#4697 from dding3/pythonorca
c942977 
migrate orca python file
@dding3
Merge branch 'upstream_bigdl-2.0' into pythontfpark
f6adbdd
@Le-Zheng
Merge branch 'bigdl-2.0' into kerasothers
f6b263a
@Le-Zheng
move nn.keras to keras
5540ca2
@Le-Zheng
Merge pull request intel-analytics#4700 from Le-Zheng/kerasothers
7d6439d 
Keras others uts
@dding3
update test script and rename tfpark package
d90069a
@dding3
Merge pull request intel-analytics#4682 from dding3/pythontfpark
2c60071 
migrate tfpark python to orca
@shanyu-sys
Merge pull request intel-analytics#4688 from yushan111/migrate-ray
98a6032 
Migrate ray to BigDL2.0 python
@dding3
fix pythotfpark ut I
66870b2
@dding3
Merge pull request intel-analytics#4707 from dding3/pytfparkut
a0e62e6 
fix python tfpark ut failure I
@Le-Zheng
turn off scala style check (intel-analytics#4695)
f94a733
@dding3
add resources for python tfpark
d85d731
@dding3
Merge pull request intel-analytics#4710 from dding3/fixpythontfparkII
2d6ed83 
add resources for python tfpark
@shanyu-sys shanyu-sys mentioned this pull request Sep 10, 2021
Merged
@shanyu-sys shanyu-sys merged commit dfeaef7 into intel-analytics:bigdl-2.0 Sep 10, 2021
Le-Zheng pushed a commit to Le-Zheng/analytics-zoo that referenced this pull request Sep 13, 2021
@shanyu-sys
Merge pull request intel-analytics#4712 from yushan111/mv-orca-test
43c9312 
Move orca/src/test to orca/test in BigDL2.0
Le-Zheng pushed a commit to Le-Zheng/analytics-zoo that referenced this pull request Sep 14, 2021
@shanyu-sys
Merge pull request intel-analytics#4712 from yushan111/mv-orca-test
3b4a4b9 
Move orca/src/test to orca/test in BigDL2.0
Le-Zheng pushed a commit to Le-Zheng/analytics-zoo that referenced this pull request Sep 15, 2021
@shanyu-sys
Merge pull request intel-analytics#4712 from yushan111/mv-orca-test
f8cb3bc 
Move orca/src/test to orca/test in BigDL2.0
shanyu-sys added a commit to shanyu-sys/analytics-zoo that referenced this pull request Sep 16, 2021
@shanyu-sys
Merge pull request intel-analytics#4712 from yushan111/mv-orca-test
a61e189 
Move orca/src/test to orca/test in BigDL2.0
Le-Zheng pushed a commit to Le-Zheng/analytics-zoo that referenced this pull request Sep 17, 2021
@shanyu-sys
Merge pull request intel-analytics#4712 from yushan111/mv-orca-test
2dec7c3 
Move orca/src/test to orca/test in BigDL2.0
Le-Zheng pushed a commit to Le-Zheng/analytics-zoo that referenced this pull request Sep 22, 2021
@shanyu-sys
Merge pull request intel-analytics#4712 from yushan111/mv-orca-test
9ea9510 
Move orca/src/test to orca/test in BigDL2.0
Le-Zheng pushed a commit to Le-Zheng/analytics-zoo that referenced this pull request Sep 22, 2021
@shanyu-sys
Merge pull request intel-analytics#4712 from yushan111/mv-orca-test
4d60f83 
Move orca/src/test to orca/test in BigDL2.0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.