[Feature] Support interface value converted to parquet #4
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ongkong
approved these changes
Feb 13, 2024
| @@ -10,7 +10,7 @@ import ( | |||
|
|
|||
| // Pass the obj as the go struct object | |||
| func NewParquetWriterFromProto(pFile source.ParquetFile, obj interface{}, np int64) (*ParquetWriter, error) { | |||
| schemaHandler, err := schema.NewSchemaHandlerFromProtoStruct(obj) | |||
| schemaHandler, err := schema.NewSchemaHandlerFromStruct(obj, false) | |||
There was a problem hiding this comment.
did you mean to swap this out from NewSchemaHandlerFromProtoStruct?
There was a problem hiding this comment.
Yes. I want to just keep one code path since original implementation is actually incomplete.
| case reflect.Array, reflect.Slice: | ||
| tags, err = getListTag(typ) | ||
| case reflect.Map: | ||
| tags, err = getMapTag(typ) | ||
| case reflect.Interface: |
@ongkong Thanks for review. I think it's OK to just review it high level. There are lots of details about how parquet file organize data and encoding data. I will make sure the testing cover the use cases we need. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Support the interface type in schema generation. It only replies on the reflect.Type which doesn't have enough information for the interface type. So I added the reflect.Value into the schema generation
Fix the definition level issue which doesn't have correct repetitionType tag set in the schema
Centralize the pointer processing which is scattered in original solution. Instead of processing it everywhere, just append the pointer or interface into stack let the dedicated function to handle it
Int and uint is not distinguished in the original implementation where it treat all value as int. While when there is uint value it will panic.
Log Analyzer Framework
How this code was tested
unit test.
make testI also updated the proto_write.go to include the interface and nested interface array.