Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

#678 Add the ability to generate Spark schema based on strict integral precision #691

Merged
merged 3 commits into from
Jul 15, 2024

Conversation

yruslan
Copy link
Collaborator

@yruslan yruslan commented Jul 15, 2024

New option:

.option("strict_integral_precision", "true")

An example copybook:

      01  R.
                03 SEG-ID        PIC X(1).
                03 SEG1.
                  05 NUM1        PIC 9(2).
                03 SEG2 REDEFINES SEG1.
                  05 NUM2        PIC S9(9).
                03 SEG3 REDEFINES SEG1.
                  05 NUM3        PIC S9(15).

A schema without the option:

root
 |-- SEG_ID: string (nullable = true)
 |-- SEG1: struct (nullable = true)
 |    |-- NUM1: integer (nullable = true)
 |-- SEG2: struct (nullable = true)
 |    |-- NUM2: integer (nullable = true)
 |-- SEG3: struct (nullable = true)
 |    |-- NUM3: long (nullable = true)

A schema with the option:

root
 |-- SEG_ID: string (nullable = true)
 |-- SEG1: struct (nullable = true)
 |    |-- NUM1: decimal(2,0) (nullable = true)
 |-- SEG2: struct (nullable = true)
 |    |-- NUM2: decimal(9,0) (nullable = true)
 |-- SEG3: struct (nullable = true)
 |    |-- NUM3: decimal(15,0) (nullable = true)

(note integer and long are replaced with decimals with scale=0)

Copy link

JaCoCo code coverage report - 'cobol-parser'

File Coverage [85.72%] 🍏
ParserVisitor.scala 89.47% 🍏
ANTLRParser.scala 85.79% 🍏
CopybookParser.scala 83.75% 🍏
CobolSchema.scala 81.56% 🍏
RecordExtractors.scala 78.7% 🍏
DecoderSelector.scala 68.7% 🍏
Total Project Coverage 86.99% 🍏

Copy link

JaCoCo code coverage report - 'spark-cobol'

File Coverage [86.16%] 🍏
CobolSchema.scala 89.79% 🍏
CobolParametersParser.scala 85.2% 🍏
Total Project Coverage 80.77% 🍏

@yruslan yruslan marked this pull request as ready for review July 15, 2024 06:53
@yruslan yruslan merged commit 4872933 into master Jul 15, 2024
7 checks passed
@yruslan yruslan deleted the feature/678-strict-integral-precision branch July 15, 2024 07:00
@yruslan yruslan mentioned this pull request Jul 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant