[feature] add write blocking properties to control write blocking. #117
Conversation
| } | ||
|
|
||
| def shouldBlock(exception: String): Boolean = { | ||
| blockTriggerKeysArray.exists(exception.contains) |
There was a problem hiding this comment.
Will there be any problems with this judgment, for example, it is not -235, but -235 appears?
There was a problem hiding this comment.
In order to prevent misjudgment, you can set a more identifiable key, such as 'err=-235,error-238'
| @@ -90,6 +90,16 @@ public interface ConfigurationOptions { | |||
|
|
|||
| int DORIS_SINK_BATCH_INTERVAL_MS_DEFAULT = 50; | |||
|
|
|||
| String DORIS_SINK_MAX_BLOCKING_TIMES = "doris.sink.max.blocking.times"; | |||
There was a problem hiding this comment.
Is this parameter similar to sink.max-retries?
There was a problem hiding this comment.
'sink.max-retries' is to specify the maximum number of retries for other exceptions, and 'doris.sink.max.blocking.times' is to specify the maximum number of retries when 'doris.sink.block.trigger.keys' appears, Often in order to ensure the success of data import, 'doris.sink.max.blocking.times' is usually set larger
CodeCooker17
force-pushed
the
block_write
branch
from
d564e79 to
ec402eb
Compare
CodeCooker17
force-pushed
the
block_write
branch
from
ec402eb to
31db70e
Compare
|
After the optimization PR of #140, doris will stream the data when writing in http chuck mode, which to a certain extent avoids the compaction problems caused by large batch data load. |
Proposed changes
Issue Number: close #xxx
Problem Summary:
Describe the overview of changes.
add three config to control write blocking. Prevent write failure due to cpmpaction not being timely to improve write success rate.
--- max blocking times
"doris.sink.max.blocking.times" = "8",
--- max blocking interval times per blocking
"doris.sink.max.blocking.interval.ms" = "300000",
--- need to blocking write keys
"doris.sink.block.trigger.keys" = "-235,-238,-233"
Checklist(Required)
Further comments
If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...