Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Enhancement]: Refine Delegator L0 segment & delete buffer #35303

Open
1 task done
congqixia opened this issue Aug 6, 2024 · 5 comments
Open
1 task done

[Enhancement]: Refine Delegator L0 segment & delete buffer #35303

congqixia opened this issue Aug 6, 2024 · 5 comments
Assignees
@congqixia
Labels
kind/enhancement Issues or changes related to enhancement
Milestone
2.5.1

Comments

@congqixia
Copy link
Contributor

Is there an existing issue for this?

  • I have searched the existing issues

What would you like to be added?

Reduce the memory usage for delegator with large number of l0 segments or delete buffer

Why is this needed?

Before streaming node implementation, delegator cannot control the size of l0 segments and delete buffer. In some cases the memory pressure could be too large to handle.
We would like to change the l0 "loading" and split process in delegator and do some refinement for delete buffer purging

Anything else?

No response
@congqixia congqixia added the kind/enhancement Issues or changes related to enhancement label Aug 6, 2024
@congqixia congqixia self-assigned this Aug 6, 2024
@congqixia
Copy link
Contributor Author

congqixia commented Aug 6, 2024 •
edited
Loading

congqixia added a commit to congqixia/milvus that referenced this issue Aug 6, 2024
@congqixia
enhance: Purge delete buffer asap all L0 segment ready
9725c52 
See also milvus-io#35303

Signed-off-by: Congqi Xia <[email protected]>
@congqixia congqixia mentioned this issue Aug 6, 2024
Closed
congqixia added a commit to congqixia/milvus that referenced this issue Aug 7, 2024
@congqixia
enhance: Add forward delete max batch size
d2de474 
Separate delete data into batches if any delete data size exceeds the
confgured max batch size

Related to milvus-io#35303

Signed-off-by: Congqi Xia <[email protected]>
@congqixia congqixia mentioned this issue Aug 7, 2024
Closed
congqixia added a commit to congqixia/milvus that referenced this issue Aug 7, 2024
@congqixia
enhance: Add forward delete max batch size
e9785d6 
Separate delete data into batches if any delete data size exceeds the
confgured max batch size

Related to milvus-io#35303

Signed-off-by: Congqi Xia <[email protected]>
@congqixia congqixia mentioned this issue Aug 7, 2024
Closed
congqixia added a commit to congqixia/milvus that referenced this issue Sep 3, 2024
@congqixia
enhance: Add delete buffer related quota logic
266711a 
See also milvus-io#35303

Signed-off-by: Congqi Xia <[email protected]>
@congqixia congqixia mentioned this issue Sep 3, 2024
Merged
sre-ci-robot pushed a commit that referenced this issue Sep 5, 2024
@congqixia
enhance: Add delete buffer related quota logic (#35918)
8593c45 
See also #35303

---------

Signed-off-by: Congqi Xia <[email protected]>
congqixia added a commit to congqixia/milvus that referenced this issue Sep 5, 2024
@congqixia
enhance: Add delete buffer related quota logic (milvus-io#35918)
4850f89 
See also milvus-io#35303

---------

Signed-off-by: Congqi Xia <[email protected]>
@congqixia congqixia mentioned this issue Sep 5, 2024
Merged
congqixia added a commit to congqixia/milvus that referenced this issue Sep 5, 2024
@congqixia
enhance: Add delete buffer related quota logic (milvus-io#35918)
978787d 
See also milvus-io#35303

---------

Signed-off-by: Congqi Xia <[email protected]>
sre-ci-robot pushed a commit that referenced this issue Sep 5, 2024
@congqixia @aoiasd
enhance: [2.4] Add delete buffer related quota logic (#35918) (#35997)
da0bc22 
Cherry pick from master
pr: #35128 #35918
See also #35303

---------

Signed-off-by: aoiasd <[email protected]>
Signed-off-by: Congqi Xia <[email protected]>
Co-authored-by: aoiasd <[email protected]>
foxspy pushed a commit to foxspy/milvus that referenced this issue Sep 6, 2024
@congqixia @foxspy
enhance: Add delete buffer related quota logic (milvus-io#35918)
02c1a44 
See also milvus-io#35303

---------

Signed-off-by: Congqi Xia <[email protected]>
@stale Stale
Copy link

stale bot commented Sep 7, 2024

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.

@stale stale bot added the stale indicates no udpates for 30 days label Sep 7, 2024
congqixia added a commit to congqixia/milvus that referenced this issue Sep 11, 2024
@congqixia
enhance: Add L0 forward policy to support remote load
f19004c 
Related to milvus-io#35303

This PR add a param item to support change l0 forward behavior from bf
filtering and forward to remote load.

Signed-off-by: Congqi Xia <[email protected]>
@congqixia congqixia mentioned this issue Sep 11, 2024
Merged
chyezh pushed a commit to chyezh/milvus that referenced this issue Sep 11, 2024
@congqixia @chyezh
enhance: Add delete buffer related quota logic (milvus-io#35918)
59724a2 
See also milvus-io#35303

---------

Signed-off-by: Congqi Xia <[email protected]>
sre-ci-robot pushed a commit that referenced this issue Sep 12, 2024
@congqixia
enhance: Add L0 forward policy to support remote load (#36189)
11dbe1e 
Related to #35303

This PR add a param item to support change l0 forward behavior from bf
filtering and forward to remote load.

---------

Signed-off-by: Congqi Xia <[email protected]>
@stale stale bot removed the stale indicates no udpates for 30 days label Sep 12, 2024
congqixia added a commit to congqixia/milvus that referenced this issue Sep 12, 2024
@congqixia
enhance: Filter out non-hit delete records during load delte
fc1e771 
Related to milvus-io#35303

This PR utilizes pk index in segment to exclude non-hit delete record
during load delete records. This ability is crucial when l0/delete
forward policy only replies on segment itself(without BF filtering).

Signed-off-by: Congqi Xia <[email protected]>
@congqixia congqixia mentioned this issue Sep 12, 2024
Merged
congqixia added a commit to congqixia/milvus that referenced this issue Sep 12, 2024
@congqixia
enhance: Add L0 forward policy to support remote load (milvus-io#36189)
38d8258 
Related to milvus-io#35303

This PR add a param item to support change l0 forward behavior from bf
filtering and forward to remote load.

---------

Signed-off-by: Congqi Xia <[email protected]>
@congqixia congqixia mentioned this issue Sep 12, 2024
Merged
sre-ci-robot pushed a commit that referenced this issue Sep 12, 2024
@congqixia
enhance: [2.4] Add L0 forward policy to support remote load (#36189) (#…
13d443e 
…36208)

Cherry-pick from master
pr: #36189
Related to #35303

This PR add a param item to support change l0 forward behavior from bf
filtering and forward to remote load.

---------

Signed-off-by: Congqi Xia <[email protected]>
sre-ci-robot pushed a commit that referenced this issue Sep 13, 2024
@congqixia
enhance: Filter out non-hit delete records during load delta (#36207)
58d3200 
Related to #35303

This PR utilizes pk index in segment to exclude non-hit delete record
during load delete records. This ability is crucial when l0/delete
forward policy only replies on segment itself(without BF filtering).

---------

Signed-off-by: Congqi Xia <[email protected]>
congqixia added a commit to congqixia/milvus that referenced this issue Nov 1, 2024
@congqixia
enhance: Move forward l0 logic out of delta lock
b029d0a 
Related to milvus-io#35303

`deleteMut` shall be protecting streaming delete buffer, forward l0
could be move out of the rlock section to reduce tsafe impact from
loading segments.

Signed-off-by: Congqi Xia <[email protected]>
sre-ci-robot pushed a commit that referenced this issue Nov 1, 2024
@congqixia
enhance: [2.4] Move forward l0 logic out of delta lock (#37340)
ce7d409 
Cherry pick from master
pr: #37337
Related to #35303

`deleteMut` shall be protecting streaming delete buffer, forward l0
could be move out of the rlock section to reduce tsafe impact from
loading segments.

Signed-off-by: Congqi Xia <[email protected]>
sre-ci-robot pushed a commit that referenced this issue Nov 4, 2024
@congqixia
enhance: Move forward l0 logic out of delta lock (#37337)
f54cf41 
Related to #35303

`deleteMut` shall be protecting streaming delete buffer, forward l0
could be move out of the rlock section to reduce tsafe impact from
loading segments.

Signed-off-by: Congqi Xia <[email protected]>
congqixia added a commit to congqixia/milvus that referenced this issue Nov 4, 2024
@congqixia
enhance: Move forward l0 logic out of delta lock (milvus-io#37337)
7bbdc24 
Related to milvus-io#35303

`deleteMut` shall be protecting streaming delete buffer, forward l0
could be move out of the rlock section to reduce tsafe impact from
loading segments.

Signed-off-by: Congqi Xia <[email protected]>
@congqixia congqixia mentioned this issue Nov 4, 2024
Closed
congqixia added a commit to congqixia/milvus that referenced this issue Nov 14, 2024
@congqixia
enhance: Enable RemoteLoad l0 forward policy by default
523c038 
Related to milvus-io#35303

Signed-off-by: Congqi Xia <[email protected]>
@congqixia congqixia mentioned this issue Nov 14, 2024
Merged
sre-ci-robot pushed a commit that referenced this issue Nov 15, 2024
@congqixia
enhance: Enable RemoteLoad l0 forward policy by default (#37678)
66bf254 
Related to #35303

Signed-off-by: Congqi Xia <[email protected]>
congqixia added a commit to congqixia/milvus that referenced this issue Nov 15, 2024
@congqixia
enhance: Enable RemoteLoad l0 forward policy by default (milvus-io#…
8046699 
…37678)

Related to milvus-io#35303

Signed-off-by: Congqi Xia <[email protected]>
@congqixia congqixia mentioned this issue Nov 15, 2024
Merged
sre-ci-robot pushed a commit that referenced this issue Nov 15, 2024
@congqixia
enhance: [2.4] Enable RemoteLoad l0 forward policy by default (#37678…
cdf703a 
…) (#37713)

Cherry-pick from master
pr: #37678
Related to #35303

Signed-off-by: Congqi Xia <[email protected]>
@stale Stale
Copy link

stale bot commented Nov 17, 2024

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.

@stale stale bot added the stale indicates no udpates for 30 days label Nov 17, 2024
@congqixia
Copy link
Contributor Author

keep

@stale stale bot removed the stale indicates no udpates for 30 days label Nov 19, 2024
@czs007 czs007 mentioned this issue Nov 21, 2024
Merged
congqixia added a commit to congqixia/milvus that referenced this issue Nov 21, 2024
@congqixia
enhance: Use PrimaryKeys to replace interface slice for segment delete
7b6835e 
Related to milvus-io#35303

Reduce temporary memory usage for PK interface for segment delete.

Signed-off-by: Congqi Xia <[email protected]>
@congqixia congqixia mentioned this issue Nov 21, 2024
Merged
sre-ci-robot pushed a commit that referenced this issue Nov 22, 2024
@congqixia
enhance: Use PrimaryKeys to replace interface slice for segment del…
1ed6867 
…ete (#37880)

Related to #35303

Reduce temporary memory usage for PK interface for segment delete.

Signed-off-by: Congqi Xia <[email protected]>
sre-ci-robot pushed a commit that referenced this issue Nov 22, 2024
@czs007
fix: Revert "enhance: [2.4] Enable RemoteLoad l0 forward policy" (#…
9b3de3a 
…37875)

issue #35303
pr: #37867
This reverts commit cdf703a.

Signed-off-by: zhenshan.cao <[email protected]>
JsDove pushed a commit to JsDove/milvus that referenced this issue Nov 26, 2024
@congqixia @JsDove
enhance: Use PrimaryKeys to replace interface slice for segment del…
be7b8ad 
…ete (milvus-io#37880)

Related to milvus-io#35303

Reduce temporary memory usage for PK interface for segment delete.

Signed-off-by: Congqi Xia <[email protected]>
@stale Stale
Copy link

stale bot commented Dec 20, 2024

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.

@stale stale bot added the stale indicates no udpates for 30 days label Dec 20, 2024
congqixia added a commit to congqixia/milvus that referenced this issue Dec 23, 2024
@congqixia
enhance: Use arrow cdata to avoid proto marshaling delete data
5bd5cd0 
Related to milvus-io#35303

Reference https://arrow.apache.org/docs/format/CDataInterface.html

Signed-off-by: Congqi Xia <[email protected]>
congqixia added a commit to congqixia/milvus that referenced this issue Dec 23, 2024
@congqixia
enhance: Use Arrow cdata to avoid proto serde for delete
9a549d5 
Related to milvus-io#35303

Reference https://arrow.apache.org/docs/format/CDataInterface.html

Signed-off-by: Congqi Xia <[email protected]>
@congqixia congqixia mentioned this issue Dec 23, 2024
Open
@yanliang567 yanliang567 modified the milestones: 2.5.0, 2.5.1 Dec 24, 2024
@stale stale bot removed stale indicates no udpates for 30 days labels Dec 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@congqixia congqixia

Labels
kind/enhancement Issues or changes related to enhancement
Projects
None yet
Milestone
2.5.1
Development

No branches or pull requests
2 participants
@yanliang567 @congqixia