Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Percent encode opensearch index name #2564

Merged
merged 4 commits into from
Mar 19, 2024

Conversation

seankao-az
Copy link
Collaborator

@seankao-az seankao-az commented Mar 19, 2024

Description

Handle special characters when generating Flint index name by percent-encoding the invalid OpenSearch index name characters, since in some cases, sql queries (DROP, ALTER) are handled over here rather than forwarded to spark.
Same fix in opensearch-spark repo: opensearch-project/opensearch-spark#215

Issues Resolved

Check List

  • New functionality includes testing.
    • All tests pass, including unit test, integration test and doctest
  • New functionality has been documented.
    • New functionality has javadoc added
    • New functionality has user manual doc added
  • Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@noCharger
Copy link
Collaborator

It might cause parsing issue. Let's sync up offline.

penghuo
penghuo previously approved these changes Mar 19, 2024
@seankao-az
Copy link
Collaborator Author

seankao-az commented Mar 19, 2024

It might cause parsing issue. Let's sync up offline.

sure I get your concern. Will test if spark submit parameters can parse the backtick stuff

Edit:
actually on second thought, this PR will not introduce any parsing issue for the query in spark submit parameter. Any special characters in backticks is already supported in OpenSearch sql and ppl.
This PR is addressing the case where OS sql / ppl already accepts table names with special character, and we are trying to create an index with special characters due to that, which is forbidden by OpenSearch, hence the encoding

Copy link

codecov bot commented Mar 19, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 95.50%. Comparing base (b375a28) to head (e26053a).

Additional details and impacted files
@@            Coverage Diff            @@
##               main    #2564   +/-   ##
=========================================
  Coverage     95.50%   95.50%           
- Complexity     5113     5117    +4     
=========================================
  Files           489      489           
  Lines         14310    14318    +8     
  Branches        961      963    +2     
=========================================
+ Hits          13667    13675    +8     
  Misses          618      618           
  Partials         25       25           
Flag Coverage Δ
sql-engine 95.50% <100.00%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Signed-off-by: Sean Kao <[email protected]>
@seankao-az seankao-az requested review from dai-chen and penghuo March 19, 2024 20:06
Copy link
Collaborator

@dai-chen dai-chen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the changes!

@penghuo penghuo merged commit e17962f into opensearch-project:main Mar 19, 2024
27 of 28 checks passed
opensearch-trigger-bot bot pushed a commit that referenced this pull request Mar 19, 2024
* percent encode opensearch index name

Signed-off-by: Sean Kao <[email protected]>

* spec test vacuum

Signed-off-by: Sean Kao <[email protected]>

* spotlessApply

Signed-off-by: Sean Kao <[email protected]>

---------

Signed-off-by: Sean Kao <[email protected]>
(cherry picked from commit e17962f)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
vmmusings pushed a commit that referenced this pull request Mar 19, 2024
* percent encode opensearch index name



* spec test vacuum



* spotlessApply



---------


(cherry picked from commit e17962f)

Signed-off-by: Sean Kao <[email protected]>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport 2.x enhancement New feature or request v2.13.0 Issues targeting release v2.13.0
Projects
Status: No status
Development

Successfully merging this pull request may close these issues.

4 participants