Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for parsing copybooks given Spark options #672

Closed
yruslan opened this issue Apr 19, 2024 · 0 comments · Fixed by #673
Closed

Add support for parsing copybooks given Spark options #672

yruslan opened this issue Apr 19, 2024 · 0 comments · Fixed by #673
Assignees
Labels
enhancement New feature or request

Comments

@yruslan
Copy link
Collaborator

yruslan commented Apr 19, 2024

Background

Sometime we want to use RDDs and Spark schemas separately for processing input files. In this case it is important to generate Spark schema that matches the record schema exactly. But the parser accepts its own set of options, and Spark reader for the 'cobol' format accepts options via '.option()'. It would be useful for the copybook parser to also be able to parse via options get from a Map[String. String], with the same semantics as the Spark cobol format reader.

Feature

Add support for parsing copybooks given Spark options.

Example

val sparkOptions = Map("generate_record_id" -> "true")
val cobolSchema = CobolSchema.fromSparkOptions(sparkOptions)
val sparkSchema = cobolSchema.getSparkSchema

Proposed Solution

As per example

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant