Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot submit python script as spark job #3

Open
yujhongmicrosoft opened this issue Nov 5, 2018 · 1 comment
Open

Cannot submit python script as spark job #3

yujhongmicrosoft opened this issue Nov 5, 2018 · 1 comment

Comments

@yujhongmicrosoft
Copy link

Hello, I am following the command docker run -v $(pwd):/job godatadriven/pyspark /job/samples/word_counter.py with my own python script and am getting this error:
Error: No main class set in JAR; please specify one with --class

In the spark documentation they say:
For Python applications, simply pass a .py file in the place of instead of a JAR,

This is what I'm doing - why am I getting this error?

@NielsZeilemaker
Copy link
Contributor

Hi, could you give me a bit more information? I tried to replicate your error, but couldn't.

I'm running this command locally (on windows):
docker run -v C:\Users\niels\git\pyspark-dockerhub:/job godatadriven/pyspark /job/samples/word_counter.py

To see if your volume mapping is working, you could execute something like:
docker run --entrypoint ls -v C:\Users\niels\git\pyspark-dockerhub:/job godatadriven/pyspark /job
or
docker run --entrypoint ls -v $(pwd):/job godatadriven/pyspark /job

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants