Error: Could not find or load main class scalapb.ScalaPbCodeGenerator

[Parikshit-DartCoder]

Build.sbt
name := "consumers"
version := "0.1"
organization in ThisBuild := "com.abc"
scalaVersion in ThisBuild := "2.12.10"

val sparkV = "3.0.1"

lazy val sparkDependencies = Seq(
"org.apache.spark" %% "spark-core" % sparkV,
"org.apache.spark" %% "spark-streaming" % sparkV,
"org.apache.spark" %% "spark-streaming-kafka-0-10" % sparkV,
"org.apache.spark" %% "spark-sql-kafka-0-10" % sparkV,
"org.apache.spark" %% "spark-sql" % sparkV,
"org.apache.spark" %% "spark-mllib" % sparkV,
"com.thesamet.scalapb" %% "sparksql-scalapb" % "0.11.0"
)

Compile / PB.targets := Seq(
scalapb.gen() -> (Compile / sourceManaged).value / "scalapb"
)

lazy val assemblySettings = Seq(
assemblyJarName in assembly := name.value + ".jar",
assemblyMergeStrategy in assembly := {
case PathList("META-INF", xs @ _*) => MergeStrategy.discard
case x => MergeStrategy.first
}
)

lazy val compilerOptions = Seq(
"-unchecked",
"-feature",
"-language:existentials",
"-language:higherKinds",
"-language:implicitConversions",
"-language:postfixOps",
"-deprecation",
"-encoding",
"utf8"
)

Protoc.sbt

addSbtPlugin("com.thesamet" % "sbt-protoc" % "1.0.2")

libraryDependencies += "com.thesamet.scalapb" %% "compilerplugin" % "0.11.0"

[Parikshit-DartCoder]

I get this eror when trying to execute scalapbc/bin/protoc-gen-scala

[thesamet]

Hi @Parikshit-DartCoder , I am confused about this problem report: in the first message you give your sbt configuration, and in the second message you state you are getting an error from a command line tool that is completely unrelated to your sbt configuration. To help further, I have two follow up questions:

1. Your sbt config includes a plugin (sbt-protoc) that performs source code generation. And if protoc.sbt is in the projects directory, this config should work. Can you elaborate why you are turning to the command line tool given that you seem to have an sbt setup? Is there any problem using sbt to generate code?
2. If you need the command-line tool (protoc-gen-scala) to work - please provide instructions to reproduce the problem. Which operation system, jvm, which link you downloaded protoc-gen-scala from and so on.

Thanks!

[Parikshit-DartCoder]

Hi @thesamet Thanks for the reply.

Your sbt config includes a plugin (sbt-protoc) that performs source code generation. And if protoc.sbt is in the projects directory, this config should work. Can you elaborate why you are turning to the command line tool given that you seem to have an sbt setup? Is there any problem using sbt to generate code?

Answer: Protoc.sbt is in projects directory. We use the script below to generate scala code. I ran below build commands:
sbt clean assembly compile but it fails with message scala class not found as its not generated.Alternatively I have a script that tries to run protoc-gen-scala:

###########SCRIPT############################
#!/usr/bin/env bash
parent_path=$( cd "$(dirname "${BASH_SOURCE[0]}")" ; pwd -P )
cd "$parent_path"
WORKDIR="protosources"
GOGODIR="github.com/gogo/protobuf"
if ! [ -d $WORKDIR ]; then
mkdir $WORKDIR
cp go/streams/apis/pb/.proto $WORKDIR
git clone https://github.com/gogo/protobuf.git $WORKDIR/$GOGODIR
fi
if ! [ -d "scalapbc" ]; then
curl -L https://github.com/scalapb/ScalaPB/releases/download/v0.11.0/scalapbc-0.11.0.zip -o $WORKDIR/scalapbc.zip
unzip $WORKDIR/scalapbc.zip
mv scalapbc-0.11.0 scalapbc
curl -L https://github.com/scalapb/ScalaPB/releases/download/v0.11.0/protoc-gen-scala-0.11.0-linux-x86_64.zip -o $WORKDIR/scalapbc_protoc.zip
unzip $WORKDIR/scalapbc_protoc.zip
mv protoc-gen-scala scalapbc/bin
chmod +X scalapbc/bin/protoc-gen-scala
chmod +X scalapbc/bin/scalapbc
fi
protoc --plugin=scalapbc/bin/protoc-gen-scala -Iprotosources --scala_out=flat_package:src $WORKDIR/.proto
protoc --plugin=scalapbc/bin/protoc-gen-scala -Iprotosources --scala_out=flat_package:src $WORKDIR/$GOGODIR/gogoproto/gogo.proto
WHITELIST=$(cat proto-whitelist.txt | tr -s '\n' '|')
find -E $parent_path/src/pb ! -iregex "./streamhelper/src/pb/.($WHITELIST).*" -delete

2.If you need the command-line tool (protoc-gen-scala) to work - please provide instructions to reproduce the problem. Which operation system, jvm, which link you downloaded protoc-gen-scala from and so on.

config details:

os: 4.14.232-176.381.amzn2.x86_64
jvm :openjdk version "1.8.0_282"
OpenJDK Runtime Environment (build 1.8.0_282-b08)
OpenJDK 64-Bit Server VM (build 25.282-b08, mixed mode)
Links in this : https://github.com/scalapb/ScalaPB/releases/tag/v0.11.0

Plugins Enabled:
org.portablescala.sbtplatformdeps.PlatformDepsPlugin
sbt.plugins.CorePlugin
sbt.plugins.Giter8TemplatePlugin
sbt.plugins.IvyPlugin
sbt.plugins.JUnitXmlReportPlugin
sbt.plugins.JvmPlugin
sbt.plugins.MiniDependencyTreePlugin
sbt.plugins.SemanticdbPlugin
sbtassembly.AssemblyPlugin
sbtprotoc.ProtocPlugin

[thesamet]

Thanks for the info. It looks like the issue with the script is a duplicate of #1114. Please update your script to download protoc-gen-scala to 0.11.3 and try again. It looks like the script also unnecessarily downloads and extracts scalapbc, though it only relies protoc-gen-scala. The download of scalapbc can be safely removed. The chmod also seems redundant since zip files contain the executable bit already, but maybe it's needed for some reason in your specific environment.

Finally, for the failure in sbt you online provided this: "fails with message scala class not found as its not generated" can you provide the full output from sbt? In any case, I didn't understand why you need both source code generation in sbt and the script to work - there's an overlap. My suggestion would be to avoid the script and focus on getting the source code generation to work within sbt - this is the recommended approach.

[Parikshit-DartCoder]

@thesamet Sure I will try the steps you said and update in the thread here. Also the error message I got is from running the script not sbt. When i execute sbt compile I get error as the scala classes not found.My current flow is:
1.Run the script to generate the scala files
2.Run sbt clean compile assembly then sbt package to get the Jar

[thesamet]

sbt-protoc needs to know where your proto files are so it can generate code for them. The default is src/main/protobuf. To override the default, you need to add the following to build.sbt (with the correct path of course): Compile / PB.protoSources := file("/path/to/gogoproto")

Also, it seems like you will need to add flatPackage=true to scalapb.gen to match the script setting.

[Parikshit-DartCoder]

@thesamet ohk let me try all the steps you mentioned and I will update you with the results

[Parikshit-DartCoder]

@thesamet I changed my build.sbt to below, and executing sbt compile,still scala clases not generated

[Parikshit-DartCoder]

name := "protobuf_demo"

version := "0.1"

scalaVersion := "2.12.10"

Compile / PB.protoSources := Seq(sourceDirectory.value / "/home/ec2-user/paysihistory/src/main/protobuf/github.com/gogo/protobuf/gogoproto")

PB.targets in Compile := Seq(
scalapb.gen(flatPackage = true) -> (sourceManaged in Compile).value
)
libraryDependencies += "com.thesamet.scalapb" %% "scalapb-runtime" % scalapb.compiler.Version.scalapbVersion % "protobuf"

[Parikshit-DartCoder]

Directory structure:
root_dir----> build.sbt project src
src----> main/protobuf/github.com/. and main/protobuf/myfile.proto

[thesamet]

The Compile / PB.protoSources line should be as follows:

Compile / PB.protoSources += file("/home/ec2-user/paysihistory/src/main/protobuf/github.com/gogo/protobuf/gogoproto")

because we are appending to protoSources - (you want to keep the default src/main/protobuf path), and since you are giving an absolute path to gogoproto the sourceDirectory.value shouldn't be there. In sbt, you can type show protocSources to see the directories it uses to search for protos.

[Parikshit-DartCoder]

@thesamet updated build.sbt

name := "protobuf_demo"
version := "0.1"
scalaVersion := "2.12.10"
Compile / PB.protoSources += file("/home/ec2-user/paysihistory/src/main/protobuf/github.com/gogo/protobuf/gogoproto")
PB.targets in Compile := Seq(
scalapb.gen(flatPackage = true) -> (sourceManaged in Compile).value
)
libraryDependencies += "com.thesamet.scalapb" %% "scalapb-runtime" % scalapb.compiler.Version.scalapbVersion % "protobuf"

ERROR
gogoproto/gogo.proto: File not found.
github.com/gogo/protobuf/jsonpb/jsonpb_test_proto/test_objects.proto:42:1: Import "gogoproto/gogo.proto" was not found or had errors.
[error] java.lang.RuntimeException: protoc returned exit code: 1
[error] at scala.sys.package$.error(package.scala:30)
[error] at sbtprotoc.ProtocPlugin$.compile(ProtocPlugin.scala:448)
[error] at sbtprotoc.ProtocPlugin$.compileProto$1(ProtocPlugin.scala:651)
[error] at sbtprotoc.ProtocPlugin$.$anonfun$sourceGeneratorTask$17(ProtocPlugin.scala:674)
[error] at sbt.util.Difference.apply(Tracked.scala:414)
[error] at sbt.util.Difference.apply(Tracked.scala:394)
[error] at sbtprotoc.ProtocPlugin$.$anonfun$sourceGeneratorTask$16(ProtocPlugin.scala:663)
[error] at sbtprotoc.ProtocPlugin$.$anonfun$sourceGeneratorTask$16$adapted(ProtocPlugin.scala:659)
[error] at sbt.util.Tracked$.$anonfun$inputChangedW$1(Tracked.scala:219)
[error] at sbtprotoc.ProtocPlugin$.$anonfun$sourceGeneratorTask$1(ProtocPlugin.scala:687)
[error] at scala.Function1.$anonfun$compose$1(Function1.scala:49)
[error] at sbt.internal.util.$tilde$greater.$anonfun$$u2219$1(TypeFunctions.scala:62)
[error] at sbt.std.Transform$$anon$4.work(Transform.scala:68)
[error] at sbt.Execute.$anonfun$submit$2(Execute.scala:282)
[error] at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:23)
[error] at sbt.Execute.work(Execute.scala:291)
[error] at sbt.Execute.$anonfun$submit$1(Execute.scala:282)
[error] at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:265)
[error] at sbt.CompletionService$$anon$2.call(CompletionService.scala:64)
[error] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[error] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[error] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[error] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[error] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[error] at java.lang.Thread.run(Thread.java:748)
[error] (Compile / protocGenerate) protoc returned exit code: 1
[error] Total time: 1 s, completed Jun 8, 2021 4:32:51 PM

[Parikshit-DartCoder]

FYI:
sbt protocSources
[info] welcome to sbt 1.5.3 (Red Hat, Inc. Java 1.8.0_282)
[info] loading settings for project paysihistory-build from plugins.sbt ...
[info] loading project definition from /home/ec2-user/paysihistory/project
[info] loading settings for project paysihistory from build.sbt ...
[info] set current project to protobuf_demo (in build file:/home/ec2-user/paysihistory/)
[info] * /home/ec2-user/paysihistory/src/main/protobuf
[info] * /home/ec2-user/paysihistory/target/protobuf_external_src
[info] * /home/ec2-user/paysihistory/src/main/protobuf/github.com/gogo/protobuf

[Parikshit-DartCoder]

Hi @thesamet I generated the Scala class from protobuf, but now when I try to use them to package I get below errors ,let me know if there is anything I missed:

** Unable to find encoder for type pb.PaysiHistory. An implicit Encoder[pb.PaysiHistory] is needed to store pb.PaysiHistory instances in a Dataset. Primitive types (Int, String, etc) and Product types (case classes) are supported by importing spark.implicits._ Support for serializing other types will be added in future releases.**

[thesamet]

@Parikshit-DartCoder , have you read the section marked "IMPORTANT" around which imports you should and shouldn't have in the docs? If so, what have you tried, and can you provide a reproducible example of the problem?

[Parikshit-DartCoder]

Hi @thesamet I have fixed this in my code it was due to the way I was converting the RDD to data frame which is not supported in Spark 3.0.1 for protobuf based serializations.I have the jar created now using sbt package.

And the earlier issue I was facing because of gogo.proto git repo there are many unwanted photos in that which I don't need so I just extracted out the part I want and copy them into my photo sources dir .

[thesamet]

It sounds like the original issue (reported in the first message) has been resolved through a version upgrade and correcting SBT settings. Closing this issue now, and feel free to open new tickets whether there are other issues. For general questions or guidance, please use gitter or stackoverflow.