[Enhancement]: Add segment id hint in multi-stage operations

[congqixia]

Is there an existing issue for this?

• I have searched the existing issues

What would you like to be added?

Add segment id hint in multi-stage operations like: "Requery", "DeleteByExpression", "L0Compaction", etc.

Why is this needed?

There are lots of duplicate execution in previous mentioned operations, especially to location segment id from input PK values. Some of them are proven to be the bottlenecks in some extreme use case.

Anything else?

No response

[congqixia]

Propose

Record pre-calculated segment id result from PK as segment hints and utilize them in following stages

Fallback mechanism

When compaction happens, the segment hints may be out-of-date so the detection & fallback mechanism is needed. The default fallback behavior is to act like the hint never exists, which is to iterate all candidates and re-calculate the segment ids.

Compatibility

The segment hints fields are newly added so that old components will just ignore the hint and act like fallback behavior. So there shall be no compatibility issue here.

[congqixia]

Delete By expression

the delete data route is relevant to multiple components: proxy, mq, datanode, querynode & storage.

Complex deletion

Delete by expression(aka complex delete) is separated into multiple steps:

• s1. Executing the query by expression with stream query API (Proxy<=> Querynode)
• s2. Reorganize the matched pks and transfer the delete into normal delete by pk operation (Proxy=>MQ)
• s3. The following steps act the same as normal delete by pks (Datanode, storage)

The Action items here:

• (s1) Return segment id hint enhance: Return segment id hint in QueryStream response #36487
• (s2) Add segment hints field in wal(MQ) data struct enhance: Add segment id hint in DeleteRequest milvus-proto#324
• (s3) Add segment hints in deltalog storage

Utilizing segment hint processing delete

Datanode

There are two feature could be related to segment hints

• Writing deltalogs (determining whether some entry shall be recorded via segment bloom filters). The part could be deprecated after removing loading BF in datanodes (See also enhance: Skip loading bf in datanode #36367)
• L0 compaction could utilizing segment hint when splitting the L0 deltalog into L1/L2 segments

QueryNode

The delegator need to forward streaming the streaming and l0 delta data into corresponding segments. Both task could utilize segment id hint as long as the delete data contains them.

• Utilize segment id hint when forwarding streaming delta
• Utilize segment id hint when processing l0 delta data

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.