enhance: wal adaptor implementation #34122

chyezh · 2024-06-25T02:12:38Z

add adaptor to implement walimpls into wal interface.
implement timetick sorted and filtering scanner.
add test for wal.

sre-ci-robot · 2024-06-25T02:12:46Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: chyezh
To complete the pull request process, please assign wxyucs after the PR has been reviewed.
You can assign the PR to them by writing /assign @wxyucs in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

codecov · 2024-06-27T11:32:41Z

Codecov Report

Attention: Patch coverage is 90.07444% with 40 lines in your changes missing coverage. Please review.

Project coverage is 80.85%. Comparing base (feb13cd) to head (beeeff8).
Report is 11 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #34122      +/-   ##
==========================================
+ Coverage   80.58%   80.85%   +0.27%     
==========================================
  Files        1095     1114      +19     
  Lines      137929   138464     +535     
==========================================
+ Hits       111145   111958     +813     
+ Misses      22573    22254     -319     
- Partials     4211     4252      +41

Files	Coverage Δ
...ternal/streamingnode/server/wal/adaptor/builder.go	`66.66% <100.00%> (ø)`
...nternal/streamingnode/server/wal/adaptor/opener.go	`100.00% <100.00%> (ø)`
...node/server/wal/utility/immutable_message_queue.go	`100.00% <100.00%> (ø)`
...l/streamingnode/server/wal/utility/message_heap.go	`100.00% <100.00%> (ø)`
...streamingnode/server/wal/utility/reorder_buffer.go	`100.00% <100.00%> (ø)`
internal/util/streamingutil/status/checker.go	`100.00% <100.00%> (ø)`
...util/streamingutil/status/client_stream_wrapper.go	`100.00% <100.00%> (ø)`
internal/util/streamingutil/util/id_allocator.go	`100.00% <100.00%> (ø)`
pkg/metrics/metrics.go	`100.00% <ø> (ø)`
pkg/util/typeutil/type.go	`0.00% <ø> (ø)`
... and 7 more

... and 42 files with indirect coverage changes

chyezh · 2024-07-02T07:13:51Z

internal/streamingnode/server/wal/adaptor/scanner_registry.go

+
+// AllocateScannerName a scanner name for a scanner.
+// The scanner name should be persistent on meta for garbage clean up.
+func (m *scannerRegistry) AllocateScannerName() (string, error) {


If some walimpls can not create stateless consumer.
we need to implement a garbage collector to collect the consumers.

mergify · 2024-07-02T07:26:15Z

@chyezh E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

chyezh · 2024-07-02T07:29:42Z

/run-cpu-e2e

chyezh · 2024-07-02T09:49:53Z

rerun ut

mergify · 2024-07-02T11:14:43Z

@chyezh E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

chyezh · 2024-07-02T11:24:54Z

/run-cpu-e2e

mergify · 2024-07-02T12:04:23Z

@chyezh E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

chyezh · 2024-07-02T12:31:12Z

/run-cpu-e2e

mergify · 2024-07-02T13:03:38Z

@chyezh E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

chyezh · 2024-07-02T13:31:16Z

/run-cpu-e2e

mergify · 2024-07-02T15:45:44Z

@chyezh E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

chyezh · 2024-07-03T00:15:09Z

/run-cpu-e2e

mergify · 2024-07-03T03:33:55Z

@chyezh E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

chyezh · 2024-07-03T04:11:47Z

/run-cpu-e2e

chyezh · 2024-07-03T05:09:56Z

/run-cpu-e2e

chyezh · 2024-07-03T06:10:10Z

/run-cpu-e2e

chyezh · 2024-07-03T07:19:27Z

/run-cpu-e2e

chyezh · 2024-07-03T09:44:25Z

/run-cpu-e2e

- add adaptor to implement walimpls into wal interface. - add test for wal. Signed-off-by: chyezh <[email protected]>

Signed-off-by: chyezh <[email protected]>

mergify · 2024-07-03T15:20:35Z

@chyezh E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

chyezh · 2024-07-04T00:37:44Z

/run-cpu-e2e

mergify · 2024-07-04T00:43:55Z

@chyezh E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

chyezh · 2024-07-04T00:47:05Z

/run-cpu-e2e

mergify · 2024-07-04T00:54:04Z

@chyezh E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

chyezh · 2024-07-04T01:28:23Z

/run-cpu-e2e

chyezh · 2024-07-04T06:35:28Z

internal/streamingnode/server/wal/adaptor/scanner_adaptor.go

+		return scanner.Chan(), nil
+	}
+	// TODO: configurable pending count.
+	if s.pendingQueue.Len()+s.reorderBuffer.Len() > 1024 {


here, if our timetick interval is greater than 1024, the scanner get stucked.
Should be modified into another strategy.

chyezh · 2024-07-04T06:36:36Z

internal/streamingnode/server/wal/interceptors/timetick/timetick_interceptor.go

+}
+
+// blockUntilSyncTimeTickReady blocks until the first time tick message is sent.
+func (impl *timeTickAppendInterceptor) blockUntilSyncTimeTickReady(underlyingWALImpls walimpls.WALImpls) error {


move first timetick sent as a single function.

jaime0815 · 2024-07-04T06:56:38Z

internal/streamingnode/server/wal/adaptor/scanner_adaptor.go

+		return scanner.Chan(), nil
+	}
+	// TODO: configurable pending count.
+	if s.pendingQueue.Len()+s.reorderBuffer.Len() > 1024 {


We shouldn't restrict the pending queue length as it might block receiving all messages within the timetick window. Two key points for message packing are:

Use the pending queue of a bounded length to accommodate all messages within a timetick window.

Activate back pressure mechanism with sendingCh if downstream consumption is too slow.

jaime0815 · 2024-07-04T07:22:53Z

All refinements or fixes will be done in the next PR.

jaime0815 · 2024-07-04T07:23:00Z

/lgtm

sre-ci-robot requested review from bigsheeper, congqixia and godchen0212 June 25, 2024 02:12

sre-ci-robot added size/XXL Denotes a PR that changes 1000+ lines. area/dependency Pull requests that update a dependency file labels Jun 25, 2024

mergify bot added dco-passed DCO check passed. kind/enhancement Issues or changes related to enhancement labels Jun 25, 2024

chyezh mentioned this pull request Jun 25, 2024

[Enhancement]: Support Streaming Service in Milvus #33285

Open

46 tasks

chyezh force-pushed the feat_streaming_service_wal_adaptor branch from ff488e3 to b660a25 Compare June 26, 2024 08:29

sre-ci-robot added the area/compilation label Jun 26, 2024

chyezh force-pushed the feat_streaming_service_wal_adaptor branch 2 times, most recently from 7a1a506 to 5932f22 Compare June 27, 2024 08:12

mergify bot added the ci-passed label Jun 27, 2024

chyezh force-pushed the feat_streaming_service_wal_adaptor branch from 5932f22 to f8de92c Compare July 2, 2024 06:57

mergify bot removed the ci-passed label Jul 2, 2024

chyezh force-pushed the feat_streaming_service_wal_adaptor branch from f8de92c to 3b16b36 Compare July 2, 2024 06:59

chyezh commented Jul 2, 2024

View reviewed changes

chyezh force-pushed the feat_streaming_service_wal_adaptor branch from 3b16b36 to 5b7025b Compare July 2, 2024 10:17

chyezh added 3 commits July 3, 2024 18:17

enhance: wal adaptor implementation

8dcafc8

- add adaptor to implement walimpls into wal interface. - add test for wal. Signed-off-by: chyezh <[email protected]>

fix: move sync first time tick into single function

da871e6

Signed-off-by: chyezh <[email protected]>

fix: add unittest for status package

beeeff8

Signed-off-by: chyezh <[email protected]>

chyezh force-pushed the feat_streaming_service_wal_adaptor branch from 8e65ada to beeeff8 Compare July 3, 2024 10:17

mergify bot added the ci-passed label Jul 4, 2024

chyezh commented Jul 4, 2024

View reviewed changes

jaime0815 reviewed Jul 4, 2024

View reviewed changes

sre-ci-robot assigned jaime0815 Jul 4, 2024

sre-ci-robot added the lgtm label Jul 4, 2024

jaime0815 merged commit 7611128 into milvus-io:master Jul 4, 2024
11 of 12 checks passed

chyezh deleted the feat_streaming_service_wal_adaptor branch July 4, 2024 07:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

enhance: wal adaptor implementation #34122

enhance: wal adaptor implementation #34122

chyezh commented Jun 25, 2024 •

edited

Loading

sre-ci-robot commented Jun 25, 2024

codecov bot commented Jun 27, 2024 •

edited

Loading

chyezh Jul 2, 2024

mergify bot commented Jul 2, 2024

chyezh commented Jul 2, 2024

chyezh commented Jul 2, 2024

mergify bot commented Jul 2, 2024

chyezh commented Jul 2, 2024

mergify bot commented Jul 2, 2024

chyezh commented Jul 2, 2024

mergify bot commented Jul 2, 2024

chyezh commented Jul 2, 2024

mergify bot commented Jul 2, 2024

chyezh commented Jul 3, 2024

mergify bot commented Jul 3, 2024

chyezh commented Jul 3, 2024

chyezh commented Jul 3, 2024

chyezh commented Jul 3, 2024

chyezh commented Jul 3, 2024

chyezh commented Jul 3, 2024

mergify bot commented Jul 3, 2024

chyezh commented Jul 4, 2024

mergify bot commented Jul 4, 2024

chyezh commented Jul 4, 2024

mergify bot commented Jul 4, 2024

chyezh commented Jul 4, 2024

chyezh Jul 4, 2024

chyezh Jul 4, 2024

jaime0815 Jul 4, 2024

jaime0815 commented Jul 4, 2024

jaime0815 commented Jul 4, 2024

enhance: wal adaptor implementation #34122

enhance: wal adaptor implementation #34122

Conversation

chyezh commented Jun 25, 2024 • edited Loading

sre-ci-robot commented Jun 25, 2024

codecov bot commented Jun 27, 2024 • edited Loading

Codecov Report

chyezh Jul 2, 2024

Choose a reason for hiding this comment

mergify bot commented Jul 2, 2024

chyezh commented Jul 2, 2024

chyezh commented Jul 2, 2024

mergify bot commented Jul 2, 2024

chyezh commented Jul 2, 2024

mergify bot commented Jul 2, 2024

chyezh commented Jul 2, 2024

mergify bot commented Jul 2, 2024

chyezh commented Jul 2, 2024

mergify bot commented Jul 2, 2024

chyezh commented Jul 3, 2024

mergify bot commented Jul 3, 2024

chyezh commented Jul 3, 2024

chyezh commented Jul 3, 2024

chyezh commented Jul 3, 2024

chyezh commented Jul 3, 2024

chyezh commented Jul 3, 2024

mergify bot commented Jul 3, 2024

chyezh commented Jul 4, 2024

mergify bot commented Jul 4, 2024

chyezh commented Jul 4, 2024

mergify bot commented Jul 4, 2024

chyezh commented Jul 4, 2024

chyezh Jul 4, 2024

Choose a reason for hiding this comment

chyezh Jul 4, 2024

Choose a reason for hiding this comment

jaime0815 Jul 4, 2024

Choose a reason for hiding this comment

jaime0815 commented Jul 4, 2024

jaime0815 commented Jul 4, 2024

chyezh commented Jun 25, 2024 •

edited

Loading

codecov bot commented Jun 27, 2024 •

edited

Loading