Storage: Abort adding vector index if encryption enabled

[JaySon-Huang]

What problem does this PR solve?

Issue Number: close #9505, ref #9032

Problem Summary:

Encryption-at-rest is not compatible with building vector-index. We need more tests.

What is changed and how it works?

Storage: Abort adding vector index if encryption enabled

Encryption-at-rest is not compatible with building vector-index. We need more tests. Abort adding vector-index when encryption-at-rest is enabled in tiflash for now.

One of the reason is DMFileV3IncrementWriter::writeAndIncludeMetaFile call WriteBufferFromWritableFileBuilder::buildPtr(..., /* create_new_encryption_info */ true, ...) and it renew the encryption_info of the DMFile. It causes the encryption_info mismatch after generating the new DMFile meta v1.

There is still something wrong after fixing it. Still under investigation.

Other:
Add info level logging message when vector search is performed in tiflash.

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No code

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

None

[breezewish]

Is it possible to still create the vector index file while encyption is enabled? Looks like setting create_new_encryption_info may resolve the issue I guess. I'm worried about aborting the index may affect dedicated tier or serverless.

[JaySon-Huang]

Is it possible to still create the vector index file while encyption is enabled? Looks like setting create_new_encryption_info may resolve the issue I guess. I'm worried about aborting the index may affect dedicated tier or serverless.

@breezewish After fixing the WriteBufferFromWritableFileBuilder::buildPtr in DMFileV3IncrementWriter::writeAndIncludeMetaFile, I still meet errors when trying to create vector index on a larger vector dataset. I'm still investigating the root cause of it.

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: breezewish, JinheLin

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [JinheLin,breezewish]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ti-chi-bot]

[LGTM Timeline notifier]

Timeline:

• 2024-10-10 02:04:58.382891757 +0000 UTC m=+1098053.803104762: ☑️ agreed by breezewish.
• 2024-10-10 02:32:38.638427302 +0000 UTC m=+1099714.058640312: ☑️ agreed by JinheLin.

[JaySon-Huang]

/hold