Skip to content
This repository has been archived by the owner on Sep 18, 2023. It is now read-only.

[NSE-979] Support reading parquet with case sensitive #980

Merged

Conversation

jackylee-ch
Copy link
Contributor

What changes were proposed in this pull request?

This PR is trying to fix the problem with reading parquet with case sensitive column names. In our scenarios, we have case sensitive column names in parquet, such as Id7, and scan parquet with case insensitive mode, which means id7 whill be pushed down to source, and then we will get 'Id7 not exists'.

How was this patch tested?

unit tests and additional test.

@github-actions
Copy link

Thanks for opening a pull request!

Could you open an issue for this pull request on Github Issues?

https://github.com/oap-project/native-sql-engine/issues

Then could you also rename commit message and pull request title in the following format?

[NSE-${ISSUES_ID}] ${detailed message}

See also:

@jackylee-ch jackylee-ch changed the title support reading parquet with case sensitive [NSE-979] Support reading parquet with case sensitive Jun 20, 2022
@github-actions
Copy link

#979

@PHILO-HE
Copy link
Collaborator

PHILO-HE commented Jun 20, 2022

Thanks for your great work!

I note there are some similar code logic in ArrowPartitionReaderFactory#buildColumnarReader, which looks also lacking case insensitive handling. Right?

@jackylee-ch
Copy link
Contributor Author

I note there are some similar code logic in ArrowPartitionReaderFactory#buildColumnarReader, which looks also lacking case insensitive handling. Right?

Yeap. And there will be some tests will be added for this PR.

@zhouyuan zhouyuan merged commit b28ec12 into oap-project:main Jun 27, 2022
@jackylee-ch jackylee-ch deleted the fix_read_parquet_with_case_sensitive branch June 27, 2022 00:57
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants