-
Notifications
You must be signed in to change notification settings - Fork 78
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add support for parsing copybooks given Spark options #672
Labels
enhancement
New feature or request
Comments
yruslan
added a commit
that referenced
this issue
Apr 19, 2024
yruslan
added a commit
that referenced
this issue
Apr 19, 2024
yruslan
added a commit
that referenced
this issue
Apr 22, 2024
yruslan
added a commit
that referenced
this issue
Apr 22, 2024
yruslan
added a commit
that referenced
this issue
Apr 22, 2024
yruslan
added a commit
that referenced
this issue
Apr 22, 2024
Merged
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Background
Sometime we want to use RDDs and Spark schemas separately for processing input files. In this case it is important to generate Spark schema that matches the record schema exactly. But the parser accepts its own set of options, and Spark reader for the 'cobol' format accepts options via '.option()'. It would be useful for the copybook parser to also be able to parse via options get from a Map[String. String], with the same semantics as the Spark cobol format reader.
Feature
Add support for parsing copybooks given Spark options.
Example
Proposed Solution
As per example
The text was updated successfully, but these errors were encountered: