Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: kudu写Doris为什么有时丢数据 有时候不丢数据 #1183

Open
laixueyong opened this issue Nov 13, 2024 · 6 comments
Open

[Bug]: kudu写Doris为什么有时丢数据 有时候不丢数据 #1183

laixueyong opened this issue Nov 13, 2024 · 6 comments
Assignees
@wgzhao
Labels
bug Something isn't working

Comments

@laixueyong
Copy link

What happened?

选取2个不同的kudu表导入数据进Doris 一个显示完成的数据和Doris进去的一样 一个显示的不一样

Version

4.2.2 (Default)

OS Type

No response

Java JDK Version

Oracle JDK 1.8.0

Relevant log output

2024-11-13 16:35:06.600 [        main] INFO  Engine               - 
  ___      _     _            
 / _ \    | |   | |           
/ /_\ \ __| | __| | __ ___  __
|  _  |/ _` |/ _` |/ _` \ \/ /
| | | | (_| | (_| | (_| |>  < 
\_| |_/\__,_|\__,_|\__,_/_/\_\
:: Addax version ::    (v4.2.2)
2024-11-13 16:35:06.843 [        main] INFO  VMInfo               - VMInfo# operatingSystem class => sun.management.OperatingSystemImpl
2024-11-13 16:35:06.858 [        main] INFO  Engine               - 
{
        "setting":{
                "speed":{
                        "byte":-1,
                        "channel":5
                },
                "errorLimit":{
                        "record":0,
                        "percentage":0
                }
        },
        "content":{
                "reader":{
                        "name":"kudureader",
                        "parameter":{
                                "masterAddress":"idc-cdh-119-73:7051,idc-cdh-119-74:7051,idc-cdh-119-75:7051,idc-cdh-119-80:7051,idc-cdh-119-81:7051",
                                "table":"impala::kududb.b2b_uorders_orderdetail",
                                "splitPk":"iorderid",
                                "lowerBound":0,
                                "upperBound":1000000000,
                                "column":[
                                        "aquantity",
                                        "asalepaymoney",
                                        "cactivities",
                                        "cbizname",
                                        "ccreator",
                                        "cerpcode",
                                        "cmasterunitmeasure",
                                        "cmemo",
                                        "cmodifier",
                                        "cmutualactivities",
                                        "corderno",
                                        "corderproducttype",
                                        "coutsyskey",
                                        "cparentsuitekey",
                                        "cproductauxunitname",
                                        "cproductcode",
                                        "cproductname",
                                        "cproductoutsyskey",
                                        "cproductunitname",
                                        "cprojectid",
                                        "cskucode",
                                        "cskuoutsyskey",
                                        "cspecdescription",
                                        "cstockname",
                                        "csuitekey",
                                        "ctaxid",
                                        "dconsigntime",
                                        "dcreated",
                                        "dcreatedtime",
                                        "dmodifytime",
                                        "fauxexchangerate",
                                        "fmainexchangerate",
                                        "fmarketpricie",
                                        "fmastermeasurequantity",
                                        "fordrealmoney",
                                        "foutboundqty",
                                        "fparticularlymoney",
                                        "fpointsmoney",
                                        "fpromotionmoney",
                                        "frebatemoney",
                                        "freturnpresendqty",
                                        "freturnqty",
                                        "fsalecost",
                                        "fsalepaymoney",
                                        "fsaleprice",
                                        "ftaxrate",
                                        "ftransactionprice",
                                        "iauxunitquantity",
                                        "ibizid",
                                        "ibizproductid",
                                        "ibizskuid",
                                        "iconversionrate",
                                        "icorpid",
                                        "id",
                                        "ideleted",
                                        "idkey",
                                        "igroupid",
                                        "igroupindex",
                                        "imasterunitid",
                                        "iorderdetailpriceid",
                                        "iorderid",
                                        "iorgid",
                                        "iproductauxunitid",
                                        "iproductid",
                                        "iproductunitid",
                                        "iquantity",
                                        "irebatereturnproductid",
                                        "isaledepartmentid",
                                        "isaleorgid",
                                        "isalesorgid",
                                        "isendquantity",
                                        "isettlementorgid",
                                        "ishoppingcartid",
                                        "iskuid",
                                        "istockid",
                                        "istockorgid",
                                        "itransactiontypeid",
                                        "pubuts",
                                        "tquantity",
                                        "tsalepaymoney",
                                        "version",
                                        "ytenant_id"
                                ],
                                "readTimeout":30,
                                "scanTimeout":60,
                                "where":[
                                        "dcreated > 2020-04-30 23:59:59",
                                        "dcreated < 2020-06-01 00:00:00"
                                ]
                        }
                },
                "writer":{
                        "name":"doriswriter",
                        "parameter":{
                                "loadUrl":[
                                        "10.96.119.181:8001",
                                        "10.96.119.183:8001",
                                        "10.96.119.184:8001"
                                ],
                                "writeMode":"insert",
                                "username":"bigdata_job_load",
                                "password":"*****",
                                "loadProps":{
                                        "format":"json",
                                        "strip_outer_array":"true",
                                        "read_json_by_line":"true"
                                },
                                "batchSize":100000,
                                "column":[
                                        "aquantity",
                                        "asalepaymoney",
                                        "cactivities",
                                        "cbizname",
                                        "ccreator",
                                        "cerpcode",
                                        "cmasterunitmeasure",
                                        "cmemo",
                                        "cmodifier",
                                        "cmutualactivities",
                                        "corderno",
                                        "corderproducttype",
                                        "coutsyskey",
                                        "cparentsuitekey",
                                        "cproductauxunitname",
                                        "cproductcode",
                                        "cproductname",
                                        "cproductoutsyskey",
                                        "cproductunitname",
                                        "cprojectid",
                                        "cskucode",
                                        "cskuoutsyskey",
                                        "cspecdescription",
                                        "cstockname",
                                        "csuitekey",
                                        "ctaxid",
                                        "dconsigntime",
                                        "dcreated",
                                        "dcreatedtime",
                                        "dmodifytime",
                                        "fauxexchangerate",
                                        "fmainexchangerate",
                                        "fmarketpricie",
                                        "fmastermeasurequantity",
                                        "fordrealmoney",
                                        "foutboundqty",
                                        "fparticularlymoney",
                                        "fpointsmoney",
                                        "fpromotionmoney",
                                        "frebatemoney",
                                        "freturnpresendqty",
                                        "freturnqty",
                                        "fsalecost",
                                        "fsalepaymoney",
                                        "fsaleprice",
                                        "ftaxrate",
                                        "ftransactionprice",
                                        "iauxunitquantity",
                                        "ibizid",
                                        "ibizproductid",
                                        "ibizskuid",
                                        "iconversionrate",
                                        "icorpid",
                                        "id",
                                        "ideleted",
                                        "idkey",
                                        "igroupid",
                                        "igroupindex",
                                        "imasterunitid",
                                        "iorderdetailpriceid",
                                        "iorderid",
                                        "iorgid",
                                        "iproductauxunitid",
                                        "iproductid",
                                        "iproductunitid",
                                        "iquantity",
                                        "irebatereturnproductid",
                                        "isaledepartmentid",
                                        "isaleorgid",
                                        "isalesorgid",
                                        "isendquantity",
                                        "isettlementorgid",
                                        "ishoppingcartid",
                                        "iskuid",
                                        "istockid",
                                        "istockorgid",
                                        "itransactiontypeid",
                                        "pubuts",
                                        "tquantity",
                                        "tsalepaymoney",
                                        "version",
                                        "ytenant_id"
                                ],
                                "session":[
                                        "set session sql_mode='ANSI'"
                                ],
                                "connection":{
                                        "selectedDatabase":"dshisdatadb",
                                        "database":"dshisdatadb",
                                        "jdbcUrl":"jdbc:mysql://bd-doris-test.pec.com.cn:9030/dstestdb?useUnicode=true&useSSL=false",
                                        "table":[
                                                "b2b_uorders_orderdetail_mpp"
                                        ]
                                }
                        }
                }
        }
}

2024-11-13 16:35:06.882 [        main] INFO  JobContainer         - The jobContainer begins to process the job.
2024-11-13 16:35:06.898 [       job-0] INFO  JobContainer         - The Reader.Job [kudureader] perform prepare work .
2024-11-13 16:35:06.923 [       job-0] INFO  JobContainer         - The Writer.Job [doriswriter] perform prepare work .
2024-11-13 16:35:07.301 [       job-0] INFO  JobContainer         - Job set Channel-Number to 5 channel(s).
2024-11-13 16:35:07.303 [       job-0] INFO  JobContainer         - The Reader.Job [kudureader] is divided into [5] task(s).
2024-11-13 16:35:07.303 [       job-0] INFO  JobContainer         - The Writer.Job [doriswriter] is divided into [5] task(s).
2024-11-13 16:35:07.350 [       job-0] INFO  JobContainer         - The Scheduler launches [1] taskGroup(s).
2024-11-13 16:35:07.361 [ taskGroup-0] INFO  TaskGroupContainer   - The taskGroupId=[0] started [5] channels for [5] tasks.
2024-11-13 16:35:07.364 [ taskGroup-0] INFO  Channel              - The Channel set byte_speed_limit to -1, No bps activated.
2024-11-13 16:35:07.365 [ taskGroup-0] INFO  Channel              - The Channel set record_speed_limit to -1, No tps activated.
2024-11-13 16:35:10.390 [Doris-interval-flush] INFO  DorisWriterManager   - Doris interval Sinking triggered: label[addax_doris_writer_2c436d83-1d5c-47c3-abb5-3d4beea77d19].
2024-11-13 16:35:10.399 [    Thread-4] INFO  DorisStreamLoadObserver - Start to join batch data: rows[13408] bytes[20017917] label[addax_doris_writer_2c436d83-1d5c-47c3-abb5-3d4beea77d19].
2024-11-13 16:35:10.420 [    Thread-4] INFO  DorisStreamLoadObserver - Executing stream load to: 'http://10.96.119.181:8001/api/dshisdatadb/b2b_uorders_orderdetail_mpp/_stream_load', size: '20031326'
2024-11-13 16:35:11.444 [    Thread-4] INFO  DorisWriterManager   - Async stream load finished: label[addax_doris_writer_2c436d83-1d5c-47c3-abb5-3d4beea77d19].
2024-11-13 16:35:14.445 [Doris-interval-flush] INFO  DorisWriterManager   - Doris interval Sinking triggered: label[addax_doris_writer_108c346b-3c3f-41dc-8f54-de2def85f753].
2024-11-13 16:35:14.447 [    Thread-4] INFO  DorisStreamLoadObserver - Start to join batch data: rows[76717] bytes[114919696] label[addax_doris_writer_108c346b-3c3f-41dc-8f54-de2def85f753].
2024-11-13 16:35:14.762 [    Thread-4] INFO  DorisStreamLoadObserver - Executing stream load to: 'http://10.96.119.181:8001/api/dshisdatadb/b2b_uorders_orderdetail_mpp/_stream_load', size: '114996414'
2024-11-13 16:35:18.504 [    Thread-4] INFO  DorisWriterManager   - Async stream load finished: label[addax_doris_writer_108c346b-3c3f-41dc-8f54-de2def85f753].
2024-11-13 16:35:19.390 [       job-0] INFO  StandAloneJobContainerCommunicator - Total 143552 records, 53850588 bytes | Speed 4.28MB/s, 11962 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.000s |  All Task WaitReaderTime 6.155s | Percentage 80.00%
2024-11-13 16:35:20.026 [    Thread-4] INFO  DorisStreamLoadObserver - Start to join batch data: rows[100000] bytes[149565843] label[addax_doris_writer_fd53cc88-5954-4d75-934e-8e776b20d6b3].
2024-11-13 16:35:20.169 [    Thread-4] INFO  DorisStreamLoadObserver - Executing stream load to: 'http://10.96.119.183:8001/api/dshisdatadb/b2b_uorders_orderdetail_mpp/_stream_load', size: '149665844'
2024-11-13 16:35:24.048 [    Thread-4] INFO  DorisWriterManager   - Async stream load finished: label[addax_doris_writer_fd53cc88-5954-4d75-934e-8e776b20d6b3].
2024-11-13 16:35:25.499 [    Thread-4] INFO  DorisStreamLoadObserver - Start to join batch data: rows[100000] bytes[149947608] label[addax_doris_writer_4846bfe2-f426-4af6-9916-9fedda5c715a].
2024-11-13 16:35:25.663 [    Thread-4] INFO  DorisStreamLoadObserver - Executing stream load to: 'http://10.96.119.184:8001/api/dshisdatadb/b2b_uorders_orderdetail_mpp/_stream_load', size: '150047609'
2024-11-13 16:35:30.676 [    Thread-4] INFO  DorisWriterManager   - Async stream load finished: label[addax_doris_writer_4846bfe2-f426-4af6-9916-9fedda5c715a].
2024-11-13 16:35:30.872 [    Thread-4] INFO  DorisStreamLoadObserver - Start to join batch data: rows[100000] bytes[151298038] label[addax_doris_writer_55f1ebde-6ecf-4571-b348-6e19742f057a].
2024-11-13 16:35:31.064 [    Thread-4] INFO  DorisStreamLoadObserver - Executing stream load to: 'http://10.96.119.183:8001/api/dshisdatadb/b2b_uorders_orderdetail_mpp/_stream_load', size: '151398039'
2024-11-13 16:35:31.415 [       job-0] INFO  StandAloneJobContainerCommunicator - Total 326208 records, 122372150 bytes | Speed 5.45MB/s, 15221 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.000s |  All Task WaitReaderTime 11.303s | Percentage 80.00%
2024-11-13 16:35:35.097 [    Thread-4] INFO  DorisWriterManager   - Async stream load finished: label[addax_doris_writer_55f1ebde-6ecf-4571-b348-6e19742f057a].
2024-11-13 16:35:36.903 [    Thread-4] INFO  DorisStreamLoadObserver - Start to join batch data: rows[100000] bytes[149862101] label[addax_doris_writer_34ad4c28-ff1e-45fb-99ad-f967b565d4d0].
2024-11-13 16:35:37.093 [    Thread-4] INFO  DorisStreamLoadObserver - Executing stream load to: 'http://10.96.119.183:8001/api/dshisdatadb/b2b_uorders_orderdetail_mpp/_stream_load', size: '149962102'
2024-11-13 16:35:43.420 [       job-0] INFO  StandAloneJobContainerCommunicator - Total 503936 records, 189542814 bytes | Speed 5.34MB/s, 14810 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.000s |  All Task WaitReaderTime 16.035s | Percentage 80.00%
2024-11-13 16:35:44.565 [    Thread-4] INFO  DorisWriterManager   - Async stream load finished: label[addax_doris_writer_34ad4c28-ff1e-45fb-99ad-f967b565d4d0].
2024-11-13 16:35:44.567 [    Thread-4] INFO  DorisStreamLoadObserver - Start to join batch data: rows[100000] bytes[149847661] label[addax_doris_writer_78ccf59e-150e-493d-b852-8baacd6cc276].
2024-11-13 16:35:44.680 [    Thread-4] INFO  DorisStreamLoadObserver - Executing stream load to: 'http://10.96.119.184:8001/api/dshisdatadb/b2b_uorders_orderdetail_mpp/_stream_load', size: '149947662'
2024-11-13 16:35:53.749 [    Thread-4] INFO  DorisWriterManager   - Async stream load finished: label[addax_doris_writer_78ccf59e-150e-493d-b852-8baacd6cc276].
2024-11-13 16:35:53.753 [    Thread-4] INFO  DorisStreamLoadObserver - Start to join batch data: rows[100000] bytes[149707764] label[addax_doris_writer_963e3829-fa4a-49dc-b762-8a47e3a7aad7].
2024-11-13 16:35:54.042 [    Thread-4] INFO  DorisStreamLoadObserver - Executing stream load to: 'http://10.96.119.184:8001/api/dshisdatadb/b2b_uorders_orderdetail_mpp/_stream_load', size: '149807765'
2024-11-13 16:35:55.561 [       job-0] INFO  StandAloneJobContainerCommunicator - Total 698240 records, 262449488 bytes | Speed 5.79MB/s, 16192 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.000s |  All Task WaitReaderTime 20.823s | Percentage 80.00%
2024-11-13 16:35:58.541 [    Thread-4] INFO  DorisWriterManager   - Async stream load finished: label[addax_doris_writer_963e3829-fa4a-49dc-b762-8a47e3a7aad7].
2024-11-13 16:35:58.543 [    Thread-4] INFO  DorisStreamLoadObserver - Start to join batch data: rows[100000] bytes[150120618] label[addax_doris_writer_c1bfc8f8-a8ef-4f5c-8af7-e8aba5b9a63a].
2024-11-13 16:35:58.669 [    Thread-4] INFO  DorisStreamLoadObserver - Executing stream load to: 'http://10.96.119.184:8001/api/dshisdatadb/b2b_uorders_orderdetail_mpp/_stream_load', size: '150220619'
2024-11-13 16:36:04.298 [    Thread-4] INFO  DorisWriterManager   - Async stream load finished: label[addax_doris_writer_c1bfc8f8-a8ef-4f5c-8af7-e8aba5b9a63a].
2024-11-13 16:36:04.300 [    Thread-4] INFO  DorisStreamLoadObserver - Start to join batch data: rows[100000] bytes[151325225] label[addax_doris_writer_e8490f6f-a447-45b2-b3b9-e2e61ffa03bf].
2024-11-13 16:36:04.586 [    Thread-4] INFO  DorisStreamLoadObserver - Executing stream load to: 'http://10.96.119.183:8001/api/dshisdatadb/b2b_uorders_orderdetail_mpp/_stream_load', size: '151425226'
2024-11-13 16:36:07.595 [       job-0] INFO  StandAloneJobContainerCommunicator - Total 828832 records, 311512321 bytes | Speed 3.90MB/s, 10882 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.000s |  All Task WaitReaderTime 25.894s | Percentage 80.00%
2024-11-13 16:36:08.819 [    Thread-4] INFO  DorisWriterManager   - Async stream load finished: label[addax_doris_writer_e8490f6f-a447-45b2-b3b9-e2e61ffa03bf].
2024-11-13 16:36:09.004 [    Thread-4] INFO  DorisStreamLoadObserver - Start to join batch data: rows[100000] bytes[149754646] label[addax_doris_writer_34823b58-d418-422d-8a77-bf7bdb43f5a6].
2024-11-13 16:36:09.134 [    Thread-4] INFO  DorisStreamLoadObserver - Executing stream load to: 'http://10.96.119.184:8001/api/dshisdatadb/b2b_uorders_orderdetail_mpp/_stream_load', size: '149854647'
2024-11-13 16:36:13.471 [    Thread-4] INFO  DorisWriterManager   - Async stream load finished: label[addax_doris_writer_34823b58-d418-422d-8a77-bf7bdb43f5a6].
2024-11-13 16:36:13.473 [    Thread-4] INFO  DorisStreamLoadObserver - Start to join batch data: rows[100000] bytes[149697850] label[addax_doris_writer_90ac66b3-da10-49b3-a67b-44e2523de44b].
2024-11-13 16:36:13.585 [    Thread-4] INFO  DorisStreamLoadObserver - Executing stream load to: 'http://10.96.119.184:8001/api/dshisdatadb/b2b_uorders_orderdetail_mpp/_stream_load', size: '149797851'
2024-11-13 16:36:17.825 [    Thread-4] INFO  DorisWriterManager   - Async stream load finished: label[addax_doris_writer_90ac66b3-da10-49b3-a67b-44e2523de44b].
2024-11-13 16:36:18.054 [    Thread-4] INFO  DorisStreamLoadObserver - Start to join batch data: rows[100000] bytes[149791614] label[addax_doris_writer_440fe496-7d3b-4624-a07a-4ea3f9345c33].
2024-11-13 16:36:18.162 [    Thread-4] INFO  DorisStreamLoadObserver - Executing stream load to: 'http://10.96.119.184:8001/api/dshisdatadb/b2b_uorders_orderdetail_mpp/_stream_load', size: '149891615'
2024-11-13 16:36:19.599 [       job-0] INFO  StandAloneJobContainerCommunicator - Total 1186144 records, 446040212 bytes | Speed 10.69MB/s, 29776 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.000s |  All Task WaitReaderTime 36.335s | Percentage 80.00%
2024-11-13 16:36:22.207 [    Thread-4] INFO  DorisWriterManager   - Async stream load finished: label[addax_doris_writer_440fe496-7d3b-4624-a07a-4ea3f9345c33].
2024-11-13 16:36:22.916 [    Thread-4] INFO  DorisStreamLoadObserver - Start to join batch data: rows[100000] bytes[150056549] label[addax_doris_writer_45dc6d79-71bc-4369-a864-f55e6831af65].
2024-11-13 16:36:23.040 [    Thread-4] INFO  DorisStreamLoadObserver - Executing stream load to: 'http://10.96.119.183:8001/api/dshisdatadb/b2b_uorders_orderdetail_mpp/_stream_load', size: '150156550'
2024-11-13 16:36:27.381 [    Thread-4] INFO  DorisWriterManager   - Async stream load finished: label[addax_doris_writer_45dc6d79-71bc-4369-a864-f55e6831af65].
2024-11-13 16:36:27.846 [    Thread-4] INFO  DorisStreamLoadObserver - Start to join batch data: rows[100000] bytes[151195492] label[addax_doris_writer_0314cb9a-3be7-4afc-858b-1b76ccfc586b].
2024-11-13 16:36:27.955 [    Thread-4] INFO  DorisStreamLoadObserver - Executing stream load to: 'http://10.96.119.184:8001/api/dshisdatadb/b2b_uorders_orderdetail_mpp/_stream_load', size: '151295493'
2024-11-13 16:36:31.602 [       job-0] INFO  StandAloneJobContainerCommunicator - Total 1390944 records, 523396140 bytes | Speed 6.15MB/s, 17066 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.000s |  All Task WaitReaderTime 41.321s | Percentage 80.00%
2024-11-13 16:36:31.946 [    Thread-4] INFO  DorisWriterManager   - Async stream load finished: label[addax_doris_writer_0314cb9a-3be7-4afc-858b-1b76ccfc586b].
2024-11-13 16:36:33.092 [    Thread-4] INFO  DorisStreamLoadObserver - Start to join batch data: rows[100000] bytes[149881645] label[addax_doris_writer_ffbd2000-8985-454c-86e1-cdb3a5d742d3].
2024-11-13 16:36:33.202 [    Thread-4] INFO  DorisStreamLoadObserver - Executing stream load to: 'http://10.96.119.184:8001/api/dshisdatadb/b2b_uorders_orderdetail_mpp/_stream_load', size: '149981646'
2024-11-13 16:36:42.746 [    Thread-4] INFO  DorisWriterManager   - Async stream load finished: label[addax_doris_writer_ffbd2000-8985-454c-86e1-cdb3a5d742d3].
2024-11-13 16:36:42.748 [    Thread-4] INFO  DorisStreamLoadObserver - Start to join batch data: rows[100000] bytes[149658474] label[addax_doris_writer_579c6c23-7e10-4fc4-9e9a-db4cc098ceed].
2024-11-13 16:36:42.874 [    Thread-4] INFO  DorisStreamLoadObserver - Executing stream load to: 'http://10.96.119.183:8001/api/dshisdatadb/b2b_uorders_orderdetail_mpp/_stream_load', size: '149758475'
2024-11-13 16:36:43.606 [       job-0] INFO  StandAloneJobContainerCommunicator - Total 1587200 records, 597030372 bytes | Speed 5.85MB/s, 16354 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.000s |  All Task WaitReaderTime 46.944s | Percentage 80.00%
2024-11-13 16:36:47.699 [    Thread-4] INFO  DorisWriterManager   - Async stream load finished: label[addax_doris_writer_579c6c23-7e10-4fc4-9e9a-db4cc098ceed].
2024-11-13 16:36:47.701 [    Thread-4] INFO  DorisStreamLoadObserver - Start to join batch data: rows[100000] bytes[149872811] label[addax_doris_writer_9c07981f-57ef-4ac2-9e6d-5beb334d0def].
2024-11-13 16:36:47.860 [    Thread-4] INFO  DorisStreamLoadObserver - Executing stream load to: 'http://10.96.119.181:8001/api/dshisdatadb/b2b_uorders_orderdetail_mpp/_stream_load', size: '149972812'
2024-11-13 16:36:51.894 [    Thread-4] INFO  DorisWriterManager   - Async stream load finished: label[addax_doris_writer_9c07981f-57ef-4ac2-9e6d-5beb334d0def].
2024-11-13 16:36:51.895 [    Thread-4] INFO  DorisStreamLoadObserver - Start to join batch data: rows[100000] bytes[149905690] label[addax_doris_writer_c24b3348-fe2a-4c02-be3e-c1126eb3a6d5].
2024-11-13 16:36:52.015 [    Thread-4] INFO  DorisStreamLoadObserver - Executing stream load to: 'http://10.96.119.183:8001/api/dshisdatadb/b2b_uorders_orderdetail_mpp/_stream_load', size: '150005691'
2024-11-13 16:36:55.688 [       job-0] INFO  StandAloneJobContainerCommunicator - Total 1761600 records, 662527905 bytes | Speed 5.21MB/s, 14533 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.000s |  All Task WaitReaderTime 52.507s | Percentage 80.00%
2024-11-13 16:36:56.518 [    Thread-4] INFO  DorisWriterManager   - Async stream load finished: label[addax_doris_writer_c24b3348-fe2a-4c02-be3e-c1126eb3a6d5].
2024-11-13 16:36:56.520 [    Thread-4] INFO  DorisStreamLoadObserver - Start to join batch data: rows[100000] bytes[151372120] label[addax_doris_writer_3a4bae41-8928-492f-8817-c4ef2e111dd8].
2024-11-13 16:36:56.633 [    Thread-4] INFO  DorisStreamLoadObserver - Executing stream load to: 'http://10.96.119.181:8001/api/dshisdatadb/b2b_uorders_orderdetail_mpp/_stream_load', size: '151472121'
2024-11-13 16:36:58.690 [       job-0] INFO  AbstractScheduler    - The scheduler has completed all tasks.
2024-11-13 16:36:58.691 [       job-0] INFO  JobContainer         - The Writer.Job [doriswriter] perform post work.
2024-11-13 16:36:58.691 [       job-0] INFO  JobContainer         - The Reader.Job [kudureader] perform post work.
2024-11-13 16:36:58.692 [       job-0] INFO  StandAloneJobContainerCommunicator - Total 1961429 records, 738050261 bytes | Speed 6.34MB/s, 17670 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.000s |  All Task WaitReaderTime 57.522s | Percentage 100.00%
2024-11-13 16:36:58.693 [       job-0] INFO  JobContainer         - 
Job start  at             : 2024-11-13 16:35:06
Job end    at             : 2024-11-13 16:36:58
Job took secs             :                111s
Average   bps             :            6.34MB/s
Average   rps             :          17670rec/s
Number of rec             :             1961429
Failed record             :                   0
@laixueyong laixueyong added the bug Something isn't working label Nov 13, 2024
@laixueyong
Copy link
Author

Snipaste_2024-11-13_16-41-39

@laixueyong
Copy link
Author

2024-11-13 16:53:01.831 [ main] INFO Engine -

/ _ \ | | | |
/ /\ \ __| | __| | __ ___ __
| _ |/ |/ _ |/ ` \ / /
| | | | (| | (| | (| |> <
_| |/_,|_,|_,//_
:: Addax version :: (v4.2.2)
2024-11-13 16:53:02.082 [ main] INFO VMInfo - VMInfo# operatingSystem class => sun.management.OperatingSystemImpl
2024-11-13 16:53:02.094 [ main] INFO Engine -
{
"setting":{
"speed":{
"byte":-1,
"channel":5
},
"errorLimit":{
"record":0,
"percentage":0
}
},
"content":{
"reader":{
"name":"kudureader",
"parameter":{
"masterAddress":"idc-cdh-119-73:7051,idc-cdh-119-74:7051,idc-cdh-119-75:7051,idc-cdh-119-80:7051,idc-cdh-119-81:7051",
"table":"impala::kududb.b2b_uorders_deliverydetail",
"splitPk":"id",
"lowerBound":0,
"upperBound":1000000000,
"column":[
"benable",
"cbatchno",
"ccreator",
"cerpcode",
"cexchangeratetypeid",
"cmodifier",
"corderdetailidkey",
"corderno",
"corderproducttype",
"coutsyskey",
"cproductauxunitname",
"cproductcode",
"cproductname",
"cproductunitname",
"cremark",
"cspecdescription",
"dcreated",
"dcreatedtime",
"dmodifytime",
"dvaliddate",
"fsalepaymoney",
"fsaleprice",
"ftransactionprice",
"iauxunitquantity",
"ibizid",
"ibizproductid",
"ibizskuid",
"iconversionrate",
"icorpid",
"id",
"ideleted",
"ideliverydetailpriceid",
"ideliveryid",
"ideliverystockid",
"idkey",
"igroupid",
"iorderdetailid",
"iorgid",
"iproductid",
"ireceivedquantity",
"isalesorgid",
"isendquantity",
"iskuid",
"istockid",
"pubuts",
"version",
"ytenant_id"
],
"readTimeout":30,
"scanTimeout":60,
"where":[
"dcreatedtime > 2021-04-14 23:59:59",
"dcreatedtime < 2021-04-16 00:00:00"
]
}
},
"writer":{
"name":"doriswriter",
"parameter":{
"loadUrl":[
"10.96.119.181:8001",
"10.96.119.183:8001",
"10.96.119.184:8001"
],
"writeMode":"insert",
"username":"bigdata_job_load",
"password":"*****",
"loadProps":{
"format":"json",
"strip_outer_array":"true",
"read_json_by_line":"true"
},
"batchSize":100000,
"column":[
"benable",
"cbatchno",
"ccreator",
"cerpcode",
"cexchangeratetypeid",
"cmodifier",
"corderdetailidkey",
"corderno",
"corderproducttype",
"coutsyskey",
"cproductauxunitname",
"cproductcode",
"cproductname",
"cproductunitname",
"cremark",
"cspecdescription",
"dcreated",
"dcreatedtime",
"dmodifytime",
"dvaliddate",
"fsalepaymoney",
"fsaleprice",
"ftransactionprice",
"iauxunitquantity",
"ibizid",
"ibizproductid",
"ibizskuid",
"iconversionrate",
"icorpid",
"id",
"ideleted",
"ideliverydetailpriceid",
"ideliveryid",
"ideliverystockid",
"idkey",
"igroupid",
"iorderdetailid",
"iorgid",
"iproductid",
"ireceivedquantity",
"isalesorgid",
"isendquantity",
"iskuid",
"istockid",
"pubuts",
"version",
"ytenant_id"
],
"session":[
"set session sql_mode='ANSI'"
],
"connection":{
"selectedDatabase":"dshisdatadb",
"database":"dshisdatadb",
"jdbcUrl":"jdbc:mysql://bd-doris-test.pec.com.cn:9030/dstestdb?useUnicode=true&useSSL=false",
"table":[
"b2b_uorders_deliverydetail_mpp"
]
}
}
}
}
}

2024-11-13 16:53:02.119 [ main] INFO JobContainer - The jobContainer begins to process the job.
2024-11-13 16:53:02.135 [ job-0] INFO JobContainer - The Reader.Job [kudureader] perform prepare work .
2024-11-13 16:53:02.152 [ job-0] INFO JobContainer - The Writer.Job [doriswriter] perform prepare work .
2024-11-13 16:53:02.553 [ job-0] INFO JobContainer - Job set Channel-Number to 5 channel(s).
2024-11-13 16:53:02.556 [ job-0] INFO JobContainer - The Reader.Job [kudureader] is divided into [5] task(s).
2024-11-13 16:53:02.556 [ job-0] INFO JobContainer - The Writer.Job [doriswriter] is divided into [5] task(s).
2024-11-13 16:53:02.600 [ job-0] INFO JobContainer - The Scheduler launches [1] taskGroup(s).
2024-11-13 16:53:02.611 [ taskGroup-0] INFO TaskGroupContainer - The taskGroupId=[0] started [5] channels for [5] tasks.
2024-11-13 16:53:02.615 [ taskGroup-0] INFO Channel - The Channel set byte_speed_limit to -1, No bps activated.
2024-11-13 16:53:02.615 [ taskGroup-0] INFO Channel - The Channel set record_speed_limit to -1, No tps activated.
2024-11-13 16:53:05.634 [Doris-interval-flush] INFO DorisWriterManager - Doris interval Sinking triggered: label[addax_doris_writer_a1486db7-2a25-4620-a9bf-b2f08ad76f74].
2024-11-13 16:53:05.634 [Doris-interval-flush] INFO DorisWriterManager - Doris interval Sinking triggered: label[addax_doris_writer_21887251-a5fc-4f56-a826-aea142edfc8b].
2024-11-13 16:53:05.634 [Doris-interval-flush] INFO DorisWriterManager - Doris interval Sinking triggered: label[addax_doris_writer_2c8aac97-d61d-4979-b84c-3242d6876d83].
2024-11-13 16:53:05.636 [Doris-interval-flush] INFO DorisWriterManager - Doris interval Sinking triggered: label[addax_doris_writer_2edd59bb-63b0-4699-aa04-b3e836f4dddb].
2024-11-13 16:53:05.800 [ Thread-1] INFO DorisStreamLoadObserver - Start to join batch data: rows[20576] bytes[21914643] label[addax_doris_writer_2c8aac97-d61d-4979-b84c-3242d6876d83].
2024-11-13 16:53:05.822 [ Thread-1] INFO DorisStreamLoadObserver - Executing stream load to: 'http://10.96.119.183:8001/api/dshisdatadb/b2b_uorders_deliverydetail_mpp/_stream_load', size: '21935220'
2024-11-13 16:53:06.802 [ Thread-1] INFO DorisWriterManager - Async stream load finished: label[addax_doris_writer_2c8aac97-d61d-4979-b84c-3242d6876d83].
2024-11-13 16:53:08.635 [Doris-interval-flush] INFO DorisWriterManager - Doris interval Sinking triggered: label[addax_doris_writer_c01b5343-4f70-4de4-a8b6-df44e8fe9789].
2024-11-13 16:53:09.751 [ Thread-1] INFO DorisStreamLoadObserver - Start to join batch data: rows[100000] bytes[106482959] label[addax_doris_writer_4ae27e9a-4ce8-4b9c-8aef-c72d5dd24309].
2024-11-13 16:53:10.030 [ Thread-1] INFO DorisStreamLoadObserver - Executing stream load to: 'http://10.96.119.184:8001/api/dshisdatadb/b2b_uorders_deliverydetail_mpp/_stream_load', size: '106582960'
2024-11-13 16:53:12.342 [ Thread-1] INFO DorisWriterManager - Async stream load finished: label[addax_doris_writer_4ae27e9a-4ce8-4b9c-8aef-c72d5dd24309].
2024-11-13 16:53:14.408 [ Thread-1] INFO DorisStreamLoadObserver - Start to join batch data: rows[100000] bytes[106499631] label[addax_doris_writer_9c03356a-8c80-46f4-bf77-df6fc96054fe].
2024-11-13 16:53:14.531 [ Thread-1] INFO DorisStreamLoadObserver - Executing stream load to: 'http://10.96.119.184:8001/api/dshisdatadb/b2b_uorders_deliverydetail_mpp/_stream_load', size: '106599632'
2024-11-13 16:53:14.626 [ job-0] INFO StandAloneJobContainerCommunicator - Total 179328 records, 53227941 bytes | Speed 4.23MB/s, 14944 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.000s | All Task WaitReaderTime 6.777s | Percentage 80.00%
2024-11-13 16:53:17.116 [ Thread-1] INFO DorisWriterManager - Async stream load finished: label[addax_doris_writer_9c03356a-8c80-46f4-bf77-df6fc96054fe].
2024-11-13 16:53:18.825 [ Thread-1] INFO DorisStreamLoadObserver - Start to join batch data: rows[100000] bytes[106467774] label[addax_doris_writer_0e9fe2af-806e-4774-8c2f-97d492e2995a].
2024-11-13 16:53:18.949 [ Thread-1] INFO DorisStreamLoadObserver - Executing stream load to: 'http://10.96.119.183:8001/api/dshisdatadb/b2b_uorders_deliverydetail_mpp/_stream_load', size: '106567775'
2024-11-13 16:53:23.541 [ Thread-1] INFO DorisWriterManager - Async stream load finished: label[addax_doris_writer_0e9fe2af-806e-4774-8c2f-97d492e2995a].
2024-11-13 16:53:23.544 [ Thread-1] INFO DorisStreamLoadObserver - Start to join batch data: rows[100000] bytes[106477965] label[addax_doris_writer_648002ea-b533-49b5-9b7f-77c720195fd0].
2024-11-13 16:53:23.669 [ Thread-1] INFO DorisStreamLoadObserver - Executing stream load to: 'http://10.96.119.181:8001/api/dshisdatadb/b2b_uorders_deliverydetail_mpp/_stream_load', size: '106577966'
2024-11-13 16:53:26.416 [ Thread-1] INFO DorisWriterManager - Async stream load finished: label[addax_doris_writer_648002ea-b533-49b5-9b7f-77c720195fd0].
2024-11-13 16:53:26.631 [ job-0] INFO StandAloneJobContainerCommunicator - Total 410464 records, 121819415 bytes | Speed 5.45MB/s, 19261 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.000s | All Task WaitReaderTime 13.192s | Percentage 80.00%
2024-11-13 16:53:27.585 [ Thread-1] INFO DorisStreamLoadObserver - Start to join batch data: rows[100000] bytes[106481369] label[addax_doris_writer_8551bad6-092d-4721-86d0-4cc4ba484217].
2024-11-13 16:53:27.677 [ Thread-1] INFO DorisStreamLoadObserver - Executing stream load to: 'http://10.96.119.181:8001/api/dshisdatadb/b2b_uorders_deliverydetail_mpp/_stream_load', size: '106581370'
2024-11-13 16:53:30.362 [ Thread-1] INFO DorisWriterManager - Async stream load finished: label[addax_doris_writer_8551bad6-092d-4721-86d0-4cc4ba484217].
2024-11-13 16:53:31.707 [ Thread-1] INFO DorisStreamLoadObserver - Start to join batch data: rows[100000] bytes[106479298] label[addax_doris_writer_440a4d06-e5c0-4ffb-8382-96c59c3df719].
2024-11-13 16:53:31.785 [ Thread-1] INFO DorisStreamLoadObserver - Executing stream load to: 'http://10.96.119.183:8001/api/dshisdatadb/b2b_uorders_deliverydetail_mpp/_stream_load', size: '106579299'
2024-11-13 16:53:32.633 [ job-0] INFO AbstractScheduler - The scheduler has completed all tasks.
2024-11-13 16:53:32.634 [ job-0] INFO JobContainer - The Writer.Job [doriswriter] perform post work.
2024-11-13 16:53:32.634 [ job-0] INFO JobContainer - The Reader.Job [kudureader] perform post work.
2024-11-13 16:53:32.636 [ job-0] INFO StandAloneJobContainerCommunicator - Total 622397 records, 184714596 bytes | Speed 5.87MB/s, 20746 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.000s | All Task WaitReaderTime 18.759s | Percentage 100.00%
2024-11-13 16:53:32.637 [ job-0] INFO JobContainer -
Job start at : 2024-11-13 16:53:02
Job end at : 2024-11-13 16:53:32
Job took secs : 30s
Average bps : 5.87MB/s
Average rps : 20746rec/s
Number of rec : 622397
Failed record : 0

@laixueyong
Copy link
Author

这个导入Doris就是622397

@laixueyong
Copy link
Author

但是为什么有时候 第一次导入数据和第二次导入数据 完成的都是一样 但是清表后 查出来就各不相同

@laixueyong
Copy link
Author

addax完成的条数和源头一样 但是写入Doris的结果条数有时候次次不一样 但是有时候有一样

@wgzhao
Copy link
Owner

wgzhao commented Nov 13, 2024

参考 #1178
原因可能是Doris表用来合并写引擎

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@wgzhao wgzhao

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@wgzhao @laixueyong