Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Estimator Python API and Inception Example #1597

Merged
merged 31 commits into from
Sep 10, 2019
Merged
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Fix & add comments
cyita committed Sep 9, 2019

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
commit c112efea9e9e47769ede39fe71e5dca44ac5152b
2 changes: 1 addition & 1 deletion pyzoo/zoo/examples/inception/README.md
Original file line number Diff line number Diff line change
@@ -67,7 +67,7 @@ ${ANALYTICS_ZOO_HOME}/bin/spark-submit-with-zoo.sh \
```

* Spark yarn client mode, example command
```
```bash
export SPARK_HOME=the root directory of Spark
export ANALYTICS_ZOO_HOME=the dist directory under the Analytics Zoo project

10 changes: 9 additions & 1 deletion pyzoo/zoo/feature/common.py
Original file line number Diff line number Diff line change
@@ -246,10 +246,14 @@ def image_frame(cls, image_frame, memory_type="DRAM", bigdl_type="float"):
def image_set(cls, imageset, memory_type="DRAM", bigdl_type="float"):
"""
Create FeatureSet from ImageFrame.
:param imageset: ImageFrame
:param imageset: ImageSet
:param memory_type: string, DRAM or PMEM
If it's DRAM, will cache dataset into dynamic random-access memory
If it's PMEM, will cache dataset into Intel Optane DC Persistent Memory
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

comment here

If it's a Int number n, will cache dataset into disk, and only hold 1/n
of the data into memory during the training. After going through the
1/n, we will release the current cache, and load another 1/n into
memory.
:param bigdl_type: numeric type
:return: A feature set
"""
@@ -265,6 +269,10 @@ def sample_rdd(cls, rdd, memory_type="DRAM", bigdl_type="float"):
:param memory_type: string, DRAM or PMEM
If it's DRAM, will cache dataset into dynamic random-access memory
If it's PMEM, will cache dataset into Intel Optane DC Persistent Memory
Copy link
Contributor

@qiuxin2012 qiuxin2012 Sep 6, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Append following comment like image_frame

                              If it's a Int number n, will cache dataset into disk, and only hold 1/n
                              of the data into memory during the training. After going through the
                              1/n, we will release the current cache, and load another 1/n into
                              memory.

If it's a Int number n, will cache dataset into disk, and only hold 1/n
of the data into memory during the training. After going through the
1/n, we will release the current cache, and load another 1/n into
memory.
:param bigdl_type:numeric type
:return: A feature set
"""
2 changes: 1 addition & 1 deletion pyzoo/zoo/feature/image/imagePreprocessing.py
Original file line number Diff line number Diff line change
@@ -357,7 +357,7 @@ def __init__(self, bigdl_type="float"):

class ImageFeatureToSample(Preprocessing):
"""
a Transformer that get Sample from ImageFeature.
A transformer that get Sample from ImageFeature.
"""
def __init__(self, bigdl_type="float"):
super(ImageFeatureToSample, self).__init__(bigdl_type)
Original file line number Diff line number Diff line change
@@ -19,6 +19,9 @@ import com.intel.analytics.bigdl.transform.vision.image.opencv.OpenCVMat
import com.intel.analytics.bigdl.transform.vision.image.{FeatureTransformer, ImageFeature}
import org.opencv.core.Core

/**
* Flip the image horizontally and vertically
*/
class Mirror() extends FeatureTransformer {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add comment for all public class.

override def transformMat(feature: ImageFeature): Unit = {
Mirror.transform(feature.opencvMat(), feature.opencvMat())