Skip to content
This repository has been archived by the owner on May 22, 2023. It is now read-only.

CRDB: problem loading seats #10

Closed
timveil opened this issue Feb 9, 2019 · 2 comments
Closed

CRDB: problem loading seats #10

timveil opened this issue Feb 9, 2019 · 2 comments
Labels
crdb CockroachDB load

Comments

@timveil
Copy link
Collaborator

timveil commented Feb 9, 2019 •
edited
Loading

has difficulty executing statement in JDBC/Application code. Executes fine using cockroachdb cli.

could it be cockroachdb/cockroach#31645

4:59:12,874 (SEATSLoader.java:540) DEBUG - Loading RESERVATION batch [total=100]
Exception in thread "main" java.lang.RuntimeException: Failed to execute threads: Failed to load data files for scaling-sized table 'RESERVATION'
	at com.oltpbenchmark.util.ThreadUtil.run(ThreadUtil.java:298)
	at com.oltpbenchmark.util.ThreadUtil.runNewPool(ThreadUtil.java:262)
	at com.oltpbenchmark.api.BenchmarkModule.loadDatabase(BenchmarkModule.java:296)
	at com.oltpbenchmark.api.BenchmarkModule.loadDatabase(BenchmarkModule.java:256)
	at com.oltpbenchmark.DBWorkload.runLoader(DBWorkload.java:794)
	at com.oltpbenchmark.DBWorkload.main(DBWorkload.java:525)
Caused by: java.lang.RuntimeException: Failed to load data files for scaling-sized table 'RESERVATION'
	at com.oltpbenchmark.benchmarks.seats.SEATSLoader.loadScalingTable(SEATSLoader.java:408)
	at com.oltpbenchmark.benchmarks.seats.SEATSLoader$9.load(SEATSLoader.java:279)
	at com.oltpbenchmark.api.Loader$LoaderThread.run(Loader.java:64)
	at com.oltpbenchmark.util.ThreadUtil$LatchRunnable.run(ThreadUtil.java:332)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: Failed to load table RESERVATION
	at com.oltpbenchmark.benchmarks.seats.SEATSLoader.loadTable(SEATSLoader.java:553)
	at com.oltpbenchmark.benchmarks.seats.SEATSLoader.loadScalingTable(SEATSLoader.java:406)
	... 6 more
Caused by: java.sql.BatchUpdateException: Batch entry 0 INSERT INTO RESERVATION VALUES (0, 12666373951980517, 844487207895738, 0, 108.0, 929920185, 548993814, 999896582, 494515244, 503367956, 960603398, 885447949, 541103370, 293605567),(1, 12666373951981011, 844487207895738, 1, 652.0, 375784014, 1006827424, 834080608, 205529855, 148006672, 771793001, 290270997, 880443185, 136389981),(2, 56576470318841943, 422229648229303, 0, 445.0, 816525642, 610118543, 25568357, 138235164, 805444716, 578128922, 152664618, 343874315, 601703967),(3, 56576470318842005, 422229648229303, 1, 776.0, 752791020, 1012437428, 800810679, 278855619, 415893127, 813534258, 478266136, 287438830, 361670355),(4, 66709569480425793, 1196347038057455, 0, 624.0, 196615509, 860331233, 453402859, 832952953, 484781637, 97249057, 501804113, 788856190, 470679915),(5, 66709569480425866, 1196347038057455, 1, 107.0, 766323961, 250357278, 232660473, 763595255, 694305948, 175048703, 439471436, 337156886, 433579372),(6, 26177172834091068, 563213021692939, 0, 548.0, 389358164, 853406524, 1029204623, 628844628, 224470063, 168800575, 343449689, 959713590, 463441980),(7, 26177172834091902, 563213021692939, 1, 999.0, 91380457, 242603041, 645957593, 890330199, 598006720, 880111553, 265749980, 773359527, 332902439),(8, 43347146413441287, 1196305160765758, 0, 534.0, 1021549946, 174793392, 602742726, 360864762, 617443702, 959078287, 223275205, 791146197, 832117484),(9, 43347146413441906, 1196305160765758, 1, 633.0, 319164882, 976319583, 793950117, 794512353, 1071961776, 702259065, 595276836, 683846371, 495073931),(10, 43347146413442652, 1196305160765758, 2, 795.0, 81409301, 4262155, 529618059, 1005972439, 269826412, 285418173, 1048790930, 135542023, 405192592),(11, 43347146413442541, 1196305160765758, 3, 700.0, 928969780, 368649131, 610563009, 803030759, 1011367050, 704351516, 67272974, 287449588, 896455228),(12, 43347146413442356, 1196305160765758, 4, 161.0, 488202778, 583870016, 871633965, 320690139, 360604007, 44488605, 381431399, 917981012, 641305309),(13, 50384020831207865, 1126130764268265, 0, 694.0, 26883755, 660603216, 82446799, 152746193, 291552260, 107119660, 259990985, 1072731091, 582691140),(14, 59109745109238515, 422372456038440, 0, 765.0, 134742472, 244971683, 634806676, 64209105, 790747020, 954784904, 938206147, 316454255, 288000845),(15, 59109745109238051, 422372456038440, 1, 115.0, 544682297, 540554038, 597503796, 967526759, 923136472, 913950381, 659388683, 26063459, 217458770),(16, 40813871623045128, 563109943329531, 0, 956.0, 926933665, 1011923685, 716643931, 103908847, 276081879, 855620197, 919851676, 1021089062, 1000287352),(17, 68961369294112997, 492837837553701, 0, 402.0, 862936063, 680148975, 834074966, 958408709, 16112824, 113276494, 745450012, 559289525, 172633707),(18, 16888498602639574, 1337023320228403, 0, 149.0, 938035604, 71704814, 988105195, 210729713, 659397398, 132308370, 535913944, 824845890, 558035978),(19, 16888498602641367, 1337023320228403, 1, 663.0, 517988495, 1055766927, 1015299914, 68175388, 509611675, 302143947, 812998864, 115684252, 42784685),(20, 16888498602641031, 1337023320228403, 2, 201.0, 778023386, 923192352, 548349592, 959359424, 180196448, 90907998, 940191807, 901626153, 324132425),(21, 16888498602639368, 1337023320228403, 3, 222.0, 36017187, 995883567, 38905635, 159662053, 162688864, 357474876, 952441900, 946203264, 114936856),(22, 59109745109237987, 1196499508953878, 0, 310.0, 896194487, 606513045, 351299175, 1044518423, 529346638, 1011403680, 678091253, 158990616, 403372705),(23, 59109745109238024, 1196499508953878, 1, 198.0, 618937519, 885744141, 619530158, 79989393, 668439180, 63333097, 1065645437, 391568580, 659965275),(24, 79375943432405035, 633478689751854, 0, 600.0, 788041631, 650364173, 51332944, 49642898, 769907090, 980756797, 962704064, 213227433, 766066711),(25, 35465847065543230, 1337259544511467, 0, 372.0, 898604316, 930275145, 438720620, 841397134, 1063238755, 246683123, 346205851, 844101500, 980696087),(26, 35465847065543038, 1337259544511467, 1, 970.0, 553711774, 71385384, 465034093, 164555363, 391792742, 254805279, 177206177, 971699861, 637935848),(27, 41939771529890149, 563203358933997, 0, 581.0, 592722965, 547518937, 810538787, 982705400, 1069573256, 334346584, 137285379, 725791646, 417703487),(28, 41939771529891113, 563203358933997, 1, 313.0, 190205393, 55841050, 245605170, 876029887, 151939996, 333105077, 434488503, 392691857, 230506713),(29, 68679894317401051, 774282749477030, 0, 792.0, 494220604, 369627565, 306334817, 685459501, 23332689, 170780112, 1021065730, 951228765, 989443054),(30, 56857945295557035, 844473251824291, 0, 214.0, 829676020, 549764225, 107313439, 657887252, 497943971, 645087932, 1014679991, 459992176, 735636193),(31, 44191571343573198, 1337146802127600, 0, 841.0, 771077892, 767708099, 755484066, 325919025, 349350432, 800185598, 793959553, 351473230, 738455114),(32, 41939771529888131, 1337232701342558, 0, 563.0, 237374906, 108706553, 600295962, 875254616, 362416442, 299672871, 358144799, 427221812, 121364251),(33, 28428972647777002, 844507609907931, 0, 443.0, 443410678, 359253065, 431801290, 186170775, 95927688, 582673177, 845515922, 144010812, 916695543),(34, 23362423066985028, 985388979372830, 0, 959.0, 253494224, 361637241, 632608829, 490157459, 821847666, 978418971, 665232884, 459032061, 383878219),(35, 23362423066984759, 985388979372830, 1, 294.0, 901171151, 111672033, 468831289, 498534628, 155401463, 556076551, 200785552, 16085078, 609539137),(36, 23362423066984822, 985388979372830, 2, 388.0, 1022980166, 190676725, 912494854, 535109927, 539400158, 841211479, 797429192, 678055519, 666540092),(37, 26177172834091133, 422229646459974, 0, 987.0, 395754578, 960165169, 133655291, 570833614, 334630618, 661274248, 703868193, 106806892, 819194470),(38, 40813871623045309, 915062112141929, 0, 177.0, 948683027, 36110709, 204974767, 84255180, 902545798, 208243233, 873207718, 975670440, 675644970),(39, 40813871623045228, 915062112141929, 1, 679.0, 62282621, 1049867190, 162003470, 896751816, 589955843, 233822201, 169364804, 962438048, 949998283),(40, 64457769666740541, 1337089894990226, 0, 958.0, 72365699, 323314531, 891602518, 638893387, 1024451483, 945577309, 916173624, 1004591904, 321052318),(41, 71213169107796334, 422378899194045, 0, 908.0, 906548203, 471857516, 819033834, 471365254, 902154533, 101212703, 312194194, 1029791027, 606013974),(42, 71213169107796472, 422378899194045, 1, 927.0, 298797011, 68470824, 479857356, 657479871, 837656528, 1067946376, 642404666, 508735450, 228078113),(43, 71213169107796487, 422378899194045, 2, 691.0, 139580334, 1063805438, 222957616, 381094786, 330208335, 857764834, 551812051, 574242702, 1044927558),(44, 71213169107795998, 422378899194045, 3, 600.0, 691372567, 460754230, 97974016, 626316553, 819616498, 542036104, 931015326, 578148961, 989241797),(45, 17169973579350349, 774273082802951, 0, 512.0, 946753148, 625187169, 238602610, 902250669, 12387421, 285748405, 968417078, 64369310, 178797275),(46, 17169973579350242, 774273082802951, 1, 779.0, 160574041, 933593217, 927982095, 1069734928, 112816125, 709070670, 567569477, 20007216, 540498866),(47, 17169973579350041, 774273082802951, 2, 200.0, 646119588, 85579980, 1009643644, 573876948, 221799405, 716836538, 511333593, 816287062, 931859087),(48, 4503599627375243, 492670330077984, 0, 530.0, 700862823, 544259189, 125539526, 147776695, 304974231, 674549672, 858495681, 296511712, 207549115),(49, 56857945295554805, 1266721150370339, 0, 214.0, 509724676, 198376730, 37119321, 35689078, 294001031, 483318533, 780681891, 266428915, 373057199),(50, 56857945295556489, 1266721150370339, 1, 166.0, 776244467, 961532370, 242658229, 517486332, 800155163, 555973921, 636072699, 792178620, 400333026),(51, 41658296553178314, 492697175785772, 0, 439.0, 809719326, 207376027, 364316740, 229030503, 707781011, 292336034, 991613441, 672524476, 351300976),(52, 41658296553178124, 492697175785772, 1, 907.0, 849041183, 750981753, 872327571, 843721592, 1023566941, 527547255, 361077441, 338328300, 541639385),(53, 41658296553177243, 492697175785772, 2, 262.0, 1013561647, 919047003, 994929199, 881280755, 495380056, 777884085, 154828745, 637269600, 887448481),(54, 41658296553179012, 492697175785772, 3, 847.0, 709266517, 914299700, 10334115, 255528554, 599867174, 564483042, 373708220, 982784843, 590330138),(55, 41658296553177876, 492697175785772, 4, 671.0, 627750939, 223714214, 67891415, 25724135, 392055804, 7612869, 938617034, 1045929571, 834084498),(56, 20547673299878229, 422466943075076, 0, 807.0, 951310806, 1010153085, 396125959, 230713916, 455366004, 812851319, 904220024, 913651439, 270589622),(57, 20547673299878006, 422466943075076, 1, 276.0, 705044699, 656755946, 901760206, 1019910800, 672285309, 267028747, 296133906, 726403474, 221422865),(58, 21673573206723875, 563242012459475, 0, 461.0, 196523792, 844585970, 628110507, 293910588, 911416191, 80869237, 578993733, 495304289, 1022241991),(59, 21673573206724036, 563242012459475, 1, 732.0, 370938297, 597933867, 449469543, 30228486, 788730183, 684824477, 425440182, 629581810, 30167540),(60, 21673573206723702, 563242012459475, 2, 565.0, 623524023, 1018890011, 70583161, 998293522, 376191314, 703767353, 885164742, 391344041, 395449044),(61, 38562071809359976, 703909708579311, 0, 414.0, 804975499, 896398860, 712647686, 747743499, 36023228, 679398199, 738450461, 390660281, 37775914),(62, 21392098230011083, 633535594692802, 0, 407.0, 535911691, 45004593, 865244475, 457203012, 111790140, 925022859, 108274847, 521244433, 880631951),(63, 21392098230010572, 633535594692802, 1, 146.0, 1016277465, 896623055, 407045872, 368750053, 1027291421, 45567436, 583186577, 822551784, 620536345) was aborted: An I/O error occurred while sending to the backend.  Call getNextException to see other errors in the batch.
	at org.postgresql.jdbc.BatchResultHandler.handleError(BatchResultHandler.java:148)
	at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:486)
	at org.postgresql.jdbc.PgStatement.executeBatch(PgStatement.java:840)
	at org.postgresql.jdbc.PgPreparedStatement.executeBatch(PgPreparedStatement.java:1538)
	at com.oltpbenchmark.benchmarks.seats.SEATSLoader.loadTable(SEATSLoader.java:541)
	... 7 more
Caused by: org.postgresql.util.PSQLException: An I/O error occurred while sending to the backend.
	at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:487)
	... 10 more
Caused by: java.io.EOFException
	at org.postgresql.core.PGStream.receiveChar(PGStream.java:308)
	at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1952)
	at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:481)
	... 10 more
@timveil timveil added crdb CockroachDB load labels Feb 9, 2019
@timveil
Copy link
Collaborator Author

timveil commented Feb 12, 2019

starting seeing this on the FLIGHT table before RESERVATION. dropped batch size from 100 to 5 and still see error. running the exact sql on the CLI executes fast, without issue.

[DEBUG] 2019-02-12 08:56:25,959 [Thread-8]  com.oltpbenchmark.benchmarks.seats.SEATSLoader loadTable - Loading FLIGHT batch [total=5]
Exception in thread "main" java.lang.RuntimeException: Failed to execute threads: Failed to load data files for scaling-sized table 'FLIGHT'
	at com.oltpbenchmark.util.ThreadUtil.run(ThreadUtil.java:298)
	at com.oltpbenchmark.util.ThreadUtil.runNewPool(ThreadUtil.java:262)
	at com.oltpbenchmark.api.BenchmarkModule.loadDatabase(BenchmarkModule.java:296)
	at com.oltpbenchmark.api.BenchmarkModule.loadDatabase(BenchmarkModule.java:256)
	at com.oltpbenchmark.DBWorkload.runLoader(DBWorkload.java:794)
	at com.oltpbenchmark.DBWorkload.main(DBWorkload.java:525)
Caused by: java.lang.RuntimeException: Failed to load data files for scaling-sized table 'FLIGHT'
	at com.oltpbenchmark.benchmarks.seats.SEATSLoader.loadScalingTable(SEATSLoader.java:408)
	at com.oltpbenchmark.benchmarks.seats.SEATSLoader$8.load(SEATSLoader.java:261)
	at com.oltpbenchmark.api.Loader$LoaderThread.run(Loader.java:64)
	at com.oltpbenchmark.util.ThreadUtil$LatchRunnable.run(ThreadUtil.java:332)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: Failed to load table FLIGHT
	at com.oltpbenchmark.benchmarks.seats.SEATSLoader.loadTable(SEATSLoader.java:553)
	at com.oltpbenchmark.benchmarks.seats.SEATSLoader.loadScalingTable(SEATSLoader.java:406)
	... 6 more
Caused by: java.sql.BatchUpdateException: Batch entry 0 INSERT INTO FLIGHT VALUES (985321333539915, 1099, 77, '2019-02-11 23:11:25.952-05', 148, '2019-03-25 16:11:25.952-04', 0, 414.0, 150, 148, 911142600, 114247904, 839440230, 709656306, 138951867, 1073135033, 163999472, 506677809, 255380444, 670070668, 95490917, 486021555, 308551645, 78989135, 959361679, 621223907, 816490681, 1028743181, 337970379, 579540622, 94916982, 784005859, 1061456626, 364097268, 726055163, 825855564, 677457136, 747418187, 579993647, 806273741),(774262347694337, 257, 202, '2019-02-11 20:26:25.952-05', 192, '2019-03-25 13:26:25.952-04', 0, 432.0, 150, 149, 64683298, 944907780, 707323451, 1018405189, 708105400, 763153558, 561320109, 507900229, 804102562, 270324237, 223042723, 753542951, 601296076, 476240946, 690327255, 183668210, 1038659515, 1058907293, 178654560, 73865315, 504790752, 326320895, 132641587, 686701789, 919411342, 120794075, 145116478, 632137619, 586437374, 645578883),(703828104380711, 295, 148, '2019-02-11 18:56:25.952-05', 131, '2019-05-06 03:56:25.952-04', 0, 966.0, 150, 150, 658900762, 1016441380, 525489028, 612127428, 742205938, 927101998, 143745504, 872450774, 340139955, 835978293, 205128253, 346899344, 413309609, 487651463, 90825616, 314709598, 747707200, 923899474, 744260968, 325649959, 742562352, 585467872, 1003865524, 73409779, 688831633, 981172306, 404892578, 1029924520, 977177328, 945933880),(774279526023979, 811, 108, '2019-02-11 20:41:25.952-05', 208, '2019-02-11 20:41:25.952-05', 0, 482.0, 150, 145, 825916806, 948509503, 337840775, 170938868, 497180127, 806347051, 604675562, 947049971, 25236558, 770027703, 440485892, 530006902, 1066625579, 638543850, 142850934, 230773132, 96434946, 616426444, 981017702, 962073569, 1060827671, 90330724, 871442517, 902791660, 263038581, 1065479381, 155870855, 354940847, 193377651, 64049918) was aborted: An I/O error occurred while sending to the backend.  Call getNextException to see other errors in the batch.
	at org.postgresql.jdbc.BatchResultHandler.handleError(BatchResultHandler.java:148)
	at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:486)
	at org.postgresql.jdbc.PgStatement.executeBatch(PgStatement.java:840)
	at org.postgresql.jdbc.PgPreparedStatement.executeBatch(PgPreparedStatement.java:1538)
	at com.oltpbenchmark.benchmarks.seats.SEATSLoader.loadTable(SEATSLoader.java:541)
	... 7 more
Caused by: org.postgresql.util.PSQLException: An I/O error occurred while sending to the backend.
	at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:487)
	... 10 more
Caused by: java.io.EOFException
	at org.postgresql.core.PGStream.receiveChar(PGStream.java:308)
	at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1952)
	at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:481)
	... 10 more

@timveil
Copy link
Collaborator Author

timveil commented Feb 12, 2019

this problem occurred because of the way OLTPBench was creating and holding on to connections. by the time the thread was ready (post countdown latch) to actually load the table, the connection was stale. pushed connection creation time to the last possible minute to get around this issue. there is a larger connection pool/management issue that needs to be addressed

@timveil timveil closed this as completed Feb 12, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
crdb CockroachDB load
Milestone
No milestone
Development

No branches or pull requests
1 participant
@timveil