Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Parquet] Add exhaustive integration testing with all possible Parquet types #37943

Closed
danepitkin opened this issue Sep 28, 2023 · 2 comments · Fixed by #38249
Closed

[Parquet] Add exhaustive integration testing with all possible Parquet types #37943

danepitkin opened this issue Sep 28, 2023 · 2 comments · Fixed by #38249
Labels
Component: Parquet Type: enhancement
Milestone
15.0.0

Comments

@danepitkin
Copy link
Member

Describe the enhancement requested

Arrow and Parquet does not have exhaustive integration testing for all possible Parquet data types.

For example, it would be useful if there was a single simple sample Parquet file that had only 1 or 2 rows of data, but covered as much of the type feature space as possible. This would also be useful for testing backwards compatibility of versions e.g. to help catch issues like these[1].

The arrow testing data currently lives in a separate repo[2].

We should:

  • Put together a directory/list/repo of parquet file(s) that can hit the cross section of features/types/encodings to be a good test suite
  • Create the infrastructure for actually testing against them e.g. Parquet reader tests

[1]https://lists.apache.org/thread/4sw2vfmdx60kl2psolwvch8h2297zdkb
[2]https://github.com/apache/arrow-testing/tree/47f7b56b25683202c1fd957668e13f2abafc0f12

Component(s)

Parquet
@mapleFU
Copy link
Member

mapleFU commented Sep 29, 2023

Hi dane, I'd like to do with it fuzzing, but still we have lots of types that we cannot support :-(

@jorisvandenbossche
Copy link
Member

An older issue about this: #22325

jduo added a commit to jduo/arrow that referenced this issue Oct 12, 2023
@jduo
apacheGH-37943: [Java] Add parquet file with all supported types
60f0535 
Add a reference file with all supported types and corresponding test case
to validate that the Dataset API generates this consistently.
@github-actions github-actions bot mentioned this issue Oct 12, 2023
Merged
jduo added a commit to jduo/arrow that referenced this issue Oct 12, 2023
@jduo
apacheGH-37943: [Java] Add parquet file with all supported types
ea8965d 
Add a reference file with all supported types and corresponding test case
to validate that the Dataset API generates this consistently.
jduo added a commit to jduo/arrow that referenced this issue Oct 12, 2023
@jduo
apacheGH-37943: [Java] Add parquet file with all supported types
67cf12b 
Add a reference file with all supported types and corresponding test case
to validate that the Dataset API generates this consistently.
jduo added a commit to jduo/arrow that referenced this issue Oct 13, 2023
@jduo
apacheGH-37943: [Java] Add parquet file with all supported types
ada28d0 
Add a reference file with all supported types and corresponding test case
to validate that the Dataset API generates this consistently.
jduo added a commit to jduo/arrow that referenced this issue Oct 19, 2023
@jduo
apacheGH-37943: [Java] Add parquet file with all supported types
7c54842 
Add a reference file with all supported types and corresponding test case
to validate that the Dataset API generates this consistently.
lidavidm pushed a commit that referenced this issue Oct 20, 2023
@jduo
GH-37943: [Java] Add parquet file with all supported types (#38249)
110fb96 
### Rationale for this change
Validate the types the Dataset APIs support when generating Parquet files.

### What changes are included in this PR?
Add a reference file with all supported types and corresponding test case to validate that the Dataset API generates this consistently.

### Are these changes tested?
Yes.

### Are there any user-facing changes?
No.
* Closes: #37943

Authored-by: James Duong <[email protected]>
Signed-off-by: David Li <[email protected]>
@lidavidm lidavidm added this to the 15.0.0 milestone Oct 20, 2023
JerAguilon pushed a commit to JerAguilon/arrow that referenced this issue Oct 23, 2023
@jduo
apacheGH-37943: [Java] Add parquet file with all supported types (apa…
0b7a289 
…che#38249)

### Rationale for this change
Validate the types the Dataset APIs support when generating Parquet files.

### What changes are included in this PR?
Add a reference file with all supported types and corresponding test case to validate that the Dataset API generates this consistently.

### Are these changes tested?
Yes.

### Are there any user-facing changes?
No.
* Closes: apache#37943

Authored-by: James Duong <[email protected]>
Signed-off-by: David Li <[email protected]>
JerAguilon pushed a commit to JerAguilon/arrow that referenced this issue Oct 25, 2023
@jduo @JerAguilon
apacheGH-37943: [Java] Add parquet file with all supported types (apa…
a43d5ba 
…che#38249)

### Rationale for this change
Validate the types the Dataset APIs support when generating Parquet files.

### What changes are included in this PR?
Add a reference file with all supported types and corresponding test case to validate that the Dataset API generates this consistently.

### Are these changes tested?
Yes.

### Are there any user-facing changes?
No.
* Closes: apache#37943

Authored-by: James Duong <[email protected]>
Signed-off-by: David Li <[email protected]>
loicalleyne pushed a commit to loicalleyne/arrow that referenced this issue Nov 13, 2023
@jduo @loicalleyne
apacheGH-37943: [Java] Add parquet file with all supported types (apa…
deda92c 
…che#38249)

### Rationale for this change
Validate the types the Dataset APIs support when generating Parquet files.

### What changes are included in this PR?
Add a reference file with all supported types and corresponding test case to validate that the Dataset API generates this consistently.

### Are these changes tested?
Yes.

### Are there any user-facing changes?
No.
* Closes: apache#37943

Authored-by: James Duong <[email protected]>
Signed-off-by: David Li <[email protected]>
dgreiss pushed a commit to dgreiss/arrow that referenced this issue Feb 19, 2024
@jduo @dgreiss
apacheGH-37943: [Java] Add parquet file with all supported types (apa…
6bbd46b 
…che#38249)

### Rationale for this change
Validate the types the Dataset APIs support when generating Parquet files.

### What changes are included in this PR?
Add a reference file with all supported types and corresponding test case to validate that the Dataset API generates this consistently.

### Are these changes tested?
Yes.

### Are there any user-facing changes?
No.
* Closes: apache#37943

Authored-by: James Duong <[email protected]>
Signed-off-by: David Li <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Component: Parquet Type: enhancement
Projects
None yet
Milestone
15.0.0
Development

Successfully merging a pull request may close this issue.

GH-37943: [Java] Add parquet file with all supported types jduo/arrow
4 participants
@lidavidm @jorisvandenbossche @mapleFU @danepitkin