Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Removed hard-coded paths and provided instructions on running on mac #8

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

javadba
Copy link

@javadba javadba commented Jun 16, 2016

This PR is about usability. It was an arduous task to be able to run the KMeansLearner.ssc script. This PR aims to make it (a) possible to do so (the code has hardcoded paths so is unworkable without these changes) and (b) provides more guidance on how to achieve it. These are applicable to all platforms.

Finally there are details specifically related to Mac.

Here is an excerpt from the Update README.md that shows the changes to the script files to make them usable outside of the specific AWS machine and directory structure that the original developer was using:

System properties to control the testing scripts

The .ssc files under scripts/ directory are used to test the installation. Here are environment variables / System properties to configure them properly for your local environment:

  • bidmach.path: path to the BIDMach installation e.g. -Dbidmach.path=/git/BIDMach
  • hdfs.path: path to the file saved to hdfs e.g. -Dhdfs.path=hdfs://sparkbook:8020/bidmach
  • bidmach.merged.hdfs.path: path to the final merged/combined lz4 output e.g. -Dbidmach.merged.hdfs.path=hdfs://sparkbook:8020/bidmach/BIDMach_MNIST/partsmerged.fmat.lz4
  • spark.executors: number of executors to use in processing e.g. -Dspark.executors=1

@javadba
Copy link
Author

javadba commented Jun 21, 2016

Committers: please examine / provide feedback for this PR.

More info from the updated README.md

Start the spark-shell like this:

spark-shell --executor-memory 6g --total-executor-cores 1 --master spark://sparkbook:7077 --jars /git/BIDMach_Spark/BIDMatHDFS.jar,/git/BIDMach/lib/BIDMat.jar,/git/BIDMach/BIDMach.jar --driver-java-options "-Dbidmach.path=/git/BIDMach -Dbidmach.merged.hdfs.path=hdfs://sparkbook:8020/bidmach/BIDMach_MNIST/partsmerged.fmat.lz4 -Dhdfs.path=hdfs://sparkbook:8020/bidmach -Dspark.executors=1"

Inside the spark-shell:

:load /git/BIDMach_Spark/scripts/load_mnist.ssc
:load /git/BIDMach_Spark/scripts/append_mnist_200k.ssc
:load /git/BIDMach_Spark/scripts/KMeansLearner.ssc

@javadba
Copy link
Author

javadba commented Jun 30, 2016

Is this project alive? No feedback after having submitted 14 days ago.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant