You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Spark table restricts only alphabet, digit and underscore in table name. However, table doesn't from Spark/Hive catalog can have special characters. In this case, the Flint index name generated will be rejected by OpenSearch restricts as below:
Index names can’t contain spaces, commas, or the following characters: :, ", *, +, /, , |, ?, #, >, or <
Some extra encoding is required here to solve the problem:
Base64 can represent any string into a sequence of characters that don't include spaces or the restricted characters. However, Base64 might still contain characters like / and + which might not be suitable in this case.
Percent(URI)-encoding converts the character to its corresponding byte value in ASCII and then represent that value as a pair of hexadecimal preceded by a percent sign. Ref: https://en.wikipedia.org/wiki/Percent-encoding
Because percent sign is valid in OpenSearch index name, percent-encoding maybe the preferred solution. Also it preserves the readability for user who wants to access the index directly.
Here is quick test with OpenSearch 2.9:
# Example: loggroup/test
# => loggroup%2Ftest : encode slash
# => loggroup%2ftest : lowercase f (FlintOSClient is doing this already)
# => loggroup%252ftest : encode % (automatic conversion by REST client)
PUT loggroup%252ftest
GET _cat/indices
# green open loggroup%2ftest A8F3EVwqSdCseKOLuD6ULQ 5 2 0 0 3kb 1kb
In Flint index metadata, there is source field which stores the source table name. This can be used to double check in case of any naming conflicts.
What alternatives have you considered?
Define some custom encoding rule to solve this specific problem.
The text was updated successfully, but these errors were encountered:
Is your feature request related to a problem?
Spark table restricts only alphabet, digit and underscore in table name. However, table doesn't from Spark/Hive catalog can have special characters. In this case, the Flint index name generated will be rejected by OpenSearch restricts as below:
Ref: https://opensearch.org/docs/latest/api-reference/index-apis/create-index/#index-naming-restrictions
What solution would you like?
Some extra encoding is required here to solve the problem:
Because percent sign is valid in OpenSearch index name, percent-encoding maybe the preferred solution. Also it preserves the readability for user who wants to access the index directly.
Here is quick test with OpenSearch 2.9:
In Flint index metadata, there is
source
field which stores the source table name. This can be used to double check in case of any naming conflicts.What alternatives have you considered?
Define some custom encoding rule to solve this specific problem.
The text was updated successfully, but these errors were encountered: