Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

github.com/cockroachdb/pebble/internal/metamorphic: TestMeta failed #3594

Closed
cockroach-teamcity opened this issue May 7, 2024 · 6 comments · Fixed by #3601
Closed

github.com/cockroachdb/pebble/internal/metamorphic: TestMeta failed #3594

cockroach-teamcity opened this issue May 7, 2024 · 6 comments · Fixed by #3601

Comments

@cockroach-teamcity
Copy link
Member

github.com/cockroachdb/pebble/internal/metamorphic.TestMeta failed with artifacts on refs/heads/master @ f03e7efeebfd:

        // 04:09:52.895 INFO: [JOB 4936] compacting: sstable created 011900
        // 04:09:52.896 INFO: [JOB 4931] sstable deleted 011502
        // 04:09:52.896 INFO: [JOB 4803] flushing: sstable created 011901
        // 04:09:52.896 INFO: [JOB 4934] compacting: sstable created 011902
        // 04:09:52.896 INFO: [JOB 4803] flushing: sstable created 011903
        // 04:09:52.897 INFO: [JOB 4934] compacted(default) L3 [011825] (766B) Score=15.13 + L4 [] (0B) Score=8.89 -> L4 [011897 011902] (1.5KB), in 0.6s (0.6s total), output rate 2.3KB/s
        // 04:09:53.529 INFO: [JOB 4938] compacting(move) L2 [011542] (932B) Score=18.71 + L3 [] (0B) Score=13.66; OverlappingRatio: Single 0.00, Multi 0.00
        // 04:09:53.529 INFO: [JOB 4803] flushing: sstable created 011904
        // 04:09:53.531 INFO: [JOB 4934] sstable deleted 011531
        // 04:09:53.531 INFO: [JOB 4936] compacted(default) L3 [011521] (765B) Score=14.31 + L4 [] (0B) Score=9.01 -> L4 [011900] (765B), in 0.0s (0.6s total), output rate 611KB/s
        // 04:09:54.182 INFO: [JOB 4803] flushing: sstable created 011905
        // 04:09:54.183 INFO: [JOB 4940] compacting(default) L3 [011869] (766B) Score=13.98 + L4 [] (0B) Score=9.21; OverlappingRatio: Single 0.00, Multi 0.00
        // 04:09:54.183 INFO: [JOB 4934] sstable deleted 011825
        // 04:09:54.184 INFO: [JOB 4938] compacted(move) L2 [011542] (932B) Score=18.71 + L3 [] (0B) Score=13.66 -> L3 [011542] (932B), in 0.0s (0.7s total), output rate 170MB/s
        // 04:09:54.834 INFO: [JOB 4941] compacting(default) L4 [011858] (771B) Score=9.44 + L5 [009755] (771B) Score=0.99; OverlappingRatio: Single 1.00, Multi 0.00
        // 04:09:54.834 INFO: [JOB 4940] compacting: sstable created 011906
        // 04:09:54.834 INFO: [JOB 4803] flushing: sstable created 011907
        // 04:09:54.834 INFO: [JOB 4936] sstable deleted 011521
        // 04:09:54.836 INFO: [JOB 4940] compacted(default) L3 [011869] (766B) Score=13.98 + L4 [] (0B) Score=9.21 -> L4 [011906] (766B), in 0.7s (0.7s total), output rate 1.1KB/s
        // 04:09:55.484 INFO: [JOB 4803] flushing: sstable created 011908
        // 04:09:55.485 INFO: [JOB 4943] compacting(default) L1 [011533] (767B) Score=19.70 + L2 [] (0B) Score=17.65; OverlappingRatio: Single 0.00, Multi 0.00
        // 04:09:55.485 INFO: [JOB 4803] flushing: sstable created 011910
        // 04:09:55.485 INFO: [JOB 4941] compacting: sstable created 011909
        // 04:09:55.485 INFO: [JOB 4803] flushing: sstable created 011911
        // 04:09:55.485 INFO: [JOB 4943] compacting: sstable created 011912
        // 04:09:55.485 INFO: [JOB 4803] flushing: sstable created 011913
        // 04:09:55.486 INFO: [JOB 4941] compacted(default) L4 [011858] (771B) Score=9.44 + L5 [009755] (771B) Score=0.99 -> L5 [011909] (771B), in 0.7s (0.7s total), output rate 1.2KB/s
        // 04:09:56.158 INFO: [JOB 4945] compacting(move) L2 [011548] (930B) Score=18.33 + L3 [] (0B) Score=13.41; OverlappingRatio: Single 0.00, Multi 0.00
        // 04:09:56.158 INFO: [JOB 4803] flushing: sstable created 011914
        // 04:09:56.159 INFO: [JOB 4941] sstable deleted 011858
        // 04:09:56.160 INFO: [JOB 4945] compacted(move) L2 [011548] (930B) Score=18.33 + L3 [] (0B) Score=13.41 -> L3 [011548] (930B), in 0.0s (0.0s total), output rate 177MB/s
        // 04:09:56.797 INFO: [JOB 4946] compacting(default) L3 [007533] (932B) Score=13.77 + L4 [] (0B) Score=9.36; OverlappingRatio: Single 0.00, Multi 0.00
        // 04:09:56.798 INFO: [JOB 4803] flushing: sstable created 011915
        // 04:09:56.798 INFO: [JOB 4946] compacting: sstable created 011916
        // 04:09:56.798 INFO: [JOB 4941] sstable deleted 011869
        // 04:09:56.799 INFO: [JOB 4943] compacted(default) L1 [011533] (767B) Score=19.70 + L2 [] (0B) Score=17.65 -> L2 [011912] (767B), in 0.0s (1.3s total), output rate 2.4MB/s
        // 04:09:57.464 INFO: [JOB 4948] compacting(default) L4 [011831] (759B) Score=9.49 + L5 [009758] (914B) Score=0.99; OverlappingRatio: Single 1.20, Multi 0.00
        // 04:09:57.464 INFO: [JOB 4803] flushing: sstable created 011917
        // 04:09:57.465 INFO: [JOB 4946] compacted(default) L3 [007533] (932B) Score=13.77 + L4 [] (0B) Score=9.36 -> L4 [011916] (932B), in 0.7s (0.7s total), output rate 1.4KB/s
        // 04:09:58.109 INFO: [JOB 4948] compacting: sstable created 011918
        // 04:09:58.109 INFO: [JOB 4949] compacting(default) L3 [009751] (930B) Score=13.39 + L4 [] (0B) Score=9.22; OverlappingRatio: Single 0.00, Multi 0.00
        // 04:09:58.109 INFO: [JOB 4803] flushing: sstable created 011919
        // 04:09:58.109 INFO: [JOB 4949] compacting: sstable created 011920
        // 04:09:58.109 INFO: [JOB 4946] sstable deleted 011533
        // 04:09:58.111 INFO: [JOB 4948] compacted(default) L4 [011831] (759B) Score=9.49 + L5 [009758] (914B) Score=0.99 -> L5 [011918] (914B), in 0.6s (0.6s total), output rate 1.4KB/s
        // 04:09:58.762 INFO: [JOB 4803] flushing: sstable created 011921
        // 04:09:58.764 INFO: [JOB 4949] compacted(default) L3 [009751] (930B) Score=13.39 + L4 [] (0B) Score=9.22 -> L4 [011920] (930B), in 0.7s (0.7s total), output rate 1.4KB/s
        
        To reduce:  go test ./internal/metamorphic -tags invariants -run 'TestMeta$' --run-dir /artifacts/meta/240507-040640.5302452232842/random-028 --try-to-reduce -v
        --- FAIL: TestMeta/execution/random-028 (198.39s)
Help

To reproduce, try:

go test -tags 'invariants' -exec 'stress -p 1' -timeout 0 -test.v -run 'TestMeta$' ./internal/metamorphic -seed 1715054800530688428 -ops "uniform:5000-10000"

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

github.com/cockroachdb/pebble/internal/metamorphic.TestMeta failed with artifacts on refs/heads/master @ f03e7efeebfd:

        // 05:42:27.497 INFO: [JOB 2577] sstable deleted 010609
        // 05:42:27.497 INFO: [JOB 2577] sstable deleted 010610
        // 05:42:27.498 INFO: [JOB 2577] sstable deleted 010611
        // 05:42:27.498 INFO: [JOB 2577] sstable deleted 010612
        // 05:42:27.498 INFO: [JOB 2577] sstable deleted 010613
        // 05:42:27.498 INFO: [JOB 2577] sstable deleted 010614
        // 05:42:27.498 INFO: [JOB 2577] sstable deleted 010615
        // 05:42:27.499 INFO: [JOB 2577] sstable deleted 010616
        // 05:42:27.499 INFO: [JOB 2577] sstable deleted 010617
        // 05:42:27.499 INFO: [JOB 2577] sstable deleted 010618
        // 05:42:27.499 INFO: [JOB 2577] sstable deleted 010619
        // 05:42:27.499 INFO: [JOB 2577] sstable deleted 010620
        // 05:42:27.499 INFO: [JOB 2577] sstable deleted 010621
        // 05:42:27.500 INFO: [JOB 2577] sstable deleted 010622
        // 05:42:27.500 INFO: [JOB 2577] sstable deleted 010623
        // 05:42:27.501 INFO: [JOB 2853] compacting(default) L4 [011780] (1.0KB) Score=3.32 + L5 [006783] (1.2KB) Score=0.55; OverlappingRatio: Single 1.17, Multi 0.00
        // 05:42:27.501 INFO: [JOB 2577] sstable deleted 010624
        // 05:42:27.502 INFO: [JOB 2577] sstable deleted 010625
        // 05:42:27.511 INFO: [JOB 2852] compacted(default) L3 [011651] (1.0KB) Score=4.66 + L4 [] (0B) Score=3.31 -> L4 [011811] (1.0KB), in 0.0s (0.0s total), output rate 128KB/s
        // 05:42:28.702 INFO: [JOB 2855] compacting(default) L3 [011652] (1.0KB) Score=4.62 + L4 [] (0B) Score=3.27; OverlappingRatio: Single 0.00, Multi 0.00
        // 05:42:28.703 INFO: [JOB 2853] compacting: sstable created 011812
        // 05:42:28.703 INFO: [JOB 2577] sstable deleted 010626
        // 05:42:28.703 INFO: [JOB 2577] sstable deleted 010627
        // 05:42:28.703 INFO: [JOB 2855] compacting: sstable created 011813
        // 05:42:28.704 INFO: [JOB 2577] sstable deleted 010628
        // 05:42:28.704 INFO: [JOB 2577] sstable deleted 010629
        // 05:42:28.704 INFO: [JOB 2577] sstable deleted 010630
        // 05:42:28.704 INFO: [JOB 2577] sstable deleted 010633
        // 05:42:28.713 INFO: [JOB 2853] compacted(default) L4 [011780] (1.0KB) Score=3.32 + L5 [006783] (1.2KB) Score=0.55 -> L5 [011812] (1.0KB), in 1.2s (1.2s total), output rate 859B/s
        // 05:42:29.944 INFO: [JOB 2577] sstable deleted 010644
        // 05:42:29.954 INFO: [JOB 2855] compacted(default) L3 [011652] (1.0KB) Score=4.62 + L4 [] (0B) Score=3.27 -> L4 [011813] (1.0KB), in 0.0s (1.3s total), output rate 536KB/s
        // 05:42:31.340 INFO: [JOB 2577] sstable deleted 010739
        // 05:42:31.342 INFO: [JOB 2858] compacting(default) L4 [011811] (1.0KB) Score=3.33 + L5 [007273] (1.2KB) Score=0.55; OverlappingRatio: Single 1.13, Multi 0.00
        // 05:42:31.343 INFO: [JOB 2577] sstable deleted 010805
        // 05:42:31.343 INFO: [JOB 2577] sstable deleted 010806
        // 05:42:31.343 INFO: [JOB 2577] sstable deleted 010807
        // 05:42:31.344 INFO: [JOB 2577] sstable deleted 010808
        // 05:42:31.344 INFO: [JOB 2577] sstable deleted 010809
        // 05:42:31.344 INFO: [JOB 2577] sstable deleted 010810
        // 05:42:31.344 INFO: [JOB 2577] sstable deleted 010811
        // 05:42:31.345 INFO: [JOB 2577] sstable deleted 010812
        // 05:42:31.345 INFO: [JOB 2858] compacting: sstable created 011814
        // 05:42:31.345 INFO: [JOB 2577] sstable deleted 010813
        // 05:42:31.346 INFO: [JOB 2859] compacting(default) L3 [011653] (1012B) Score=4.60 + L4 [] (0B) Score=3.27; OverlappingRatio: Single 0.00, Multi 0.00
        // 05:42:31.347 INFO: [JOB 2577] sstable deleted 010814
        // 05:42:31.347 INFO: [JOB 2577] sstable deleted 010815
        // 05:42:31.358 INFO: [JOB 2858] compacted(default) L4 [011811] (1.0KB) Score=3.33 + L5 [007273] (1.2KB) Score=0.55 -> L5 [011814] (1.0KB), in 0.0s (0.0s total), output rate 255KB/s
        
        To reduce:  go test ./internal/metamorphic -tags invariants -run 'TestMeta$' --run-dir /artifacts/meta/240507-052709.6402565222102/random-006 --try-to-reduce -v
        --- FAIL: TestMeta/execution/random-006 (908.26s)
Help

To reproduce, try:

go test -tags 'invariants' -exec 'stress -p 1' -timeout 0 -test.v -run 'TestMeta$' ./internal/metamorphic -seed 1715059629640823123 -ops "uniform:5000-10000"

This test on roachdash | Improve this report!

@itsbilal
Copy link
Member

itsbilal commented May 7, 2024

[04:09:59] :	 [Step 1/1] === NAME  TestMeta/execution/random-028
[04:09:59] :	 [Step 1/1]     meta.go:241: 
[04:09:59] :	 [Step 1/1]         ===== SEED =====
[04:09:59] :	 [Step 1/1]         1715054800530688428
[04:09:59] :	 [Step 1/1]         ===== ERR =====
[04:09:59] :	 [Step 1/1]         exit status 2
[04:09:59] :	 [Step 1/1]         ===== OUT =====
[04:09:59] :	 [Step 1/1]         panic: operation took longer than 1m0s: db1.IngestExternalFiles(external0, "ehjujdrmzqgccc" /* start */, "ehjujdrmzqgcccq" /* end */, "@184" /* syntheticSuffix */, "ehj" /* syntheticPrefix */, external0, "ugnujdrmzqgccc" /* start */, "ugnujdrmzqgcccx" /* end */, "" /* syntheticSuffix */, "ugn" /* syntheticPrefix */)
[04:09:59] :	 [Step 1/1]         
[04:09:59] :	 [Step 1/1]         goroutine 16904 [running]:
[04:09:59] :	 [Step 1/1]         github.com/cockroachdb/pebble/metamorphic.(*Test).runOp.func1()
[04:09:59] :	 [Step 1/1]         	github.com/cockroachdb/pebble/metamorphic/external/com_github_cockroachdb_pebble/metamorphic/test.go:412 +0xb4
[04:09:59] :	 [Step 1/1]         created by time.goFunc
[04:09:59] :	 [Step 1/1]         	GOROOT/src/time/sleep.go:177 +0x2d
[04:09:59] :	 [Step 1/1]         
[04:09:59] :	 [Step 1/1]         ===== OPTIONS =====
[04:09:59] :	 [Step 1/1]         [Version]
[04:09:59] :	 [Step 1/1]           pebble_version=0.1
[04:09:59] :	 [Step 1/1]         
[04:09:59] :	 [Step 1/1]         [Options]
[04:09:59] :	 [Step 1/1]           bytes_per_sync=2097152
[04:09:59] :	 [Step 1/1]           cache_size=32768
[04:09:59] :	 [Step 1/1]           cleaner=archive
[04:09:59] :	 [Step 1/1]           compaction_debt_concurrency=1073741824
[04:09:59] :	 [Step 1/1]           comparer=pebble.internal.testkeys
[04:09:59] :	 [Step 1/1]           disable_wal=true
[04:09:59] :	 [Step 1/1]           disable_ingest_as_flushable=true
[04:09:59] :	 [Step 1/1]           flush_delay_delete_range=70ms
[04:09:59] :	 [Step 1/1]           flush_delay_range_key=755ms
[04:09:59] :	 [Step 1/1]           flush_split_bytes=16
[04:09:59] :	 [Step 1/1]           format_major_version=17
[04:09:59] :	 [Step 1/1]           l0_compaction_concurrency=1
[04:09:59] :	 [Step 1/1]           l0_compaction_file_threshold=4
[04:09:59] :	 [Step 1/1]           l0_compaction_threshold=12
[04:09:59] :	 [Step 1/1]           l0_stop_writes_threshold=40
[04:09:59] :	 [Step 1/1]           lbase_max_bytes=1
[04:09:59] :	 [Step 1/1]           max_concurrent_compactions=2
[04:09:59] :	 [Step 1/1]           max_concurrent_downloads=1
[04:09:59] :	 [Step 1/1]           max_manifest_file_size=16
[04:09:59] :	 [Step 1/1]           max_open_files=1000
[04:09:59] :	 [Step 1/1]           mem_table_size=16777216
[04:09:59] :	 [Step 1/1]           mem_table_stop_writes_threshold=6
[04:09:59] :	 [Step 1/1]           min_deletion_rate=16777216
[04:09:59] :	 [Step 1/1]           merger=pebble.concatenate
[04:09:59] :	 [Step 1/1]           multilevel_compaction_heuristic=none
[04:09:59] :	 [Step 1/1]           read_compaction_rate=16000
[04:09:59] :	 [Step 1/1]           read_sampling_multiplier=16
[04:09:59] :	 [Step 1/1]           strict_wal_tail=true
[04:09:59] :	 [Step 1/1]           table_cache_shards=8
[04:09:59] :	 [Step 1/1]           validate_on_ingest=true
[04:09:59] :	 [Step 1/1]           wal_dir=
[04:09:59] :	 [Step 1/1]           wal_bytes_per_sync=0
[04:09:59] :	 [Step 1/1]           max_writer_concurrency=0
[04:09:59] :	 [Step 1/1]           force_writer_parallelism=false
[04:09:59] :	 [Step 1/1]           secondary_cache_size_bytes=33554432
[04:09:59] :	 [Step 1/1]           create_on_shared=2
[04:09:59] :	 [Step 1/1]           disable_delete_only_compactions=true
[04:09:59] :	 [Step 1/1]           disable_elision_only_compactions=true
[04:09:59] :	 [Step 1/1]           disable_lazy_combined_iteration=true
[04:09:59] :	 [Step 1/1]         
[04:09:59] :	 [Step 1/1]         [Level "0"]
[04:09:59] :	 [Step 1/1]           block_restart_interval=32
[04:09:59] :	 [Step 1/1]           block_size=524288
[04:09:59] :	 [Step 1/1]           block_size_threshold=69
[04:09:59] :	 [Step 1/1]           compression=Snappy
[04:09:59] :	 [Step 1/1]           filter_policy=rocksdb.BuiltinBloomFilter
[04:09:59] :	 [Step 1/1]           filter_type=table
[04:09:59] :	 [Step 1/1]           index_block_size=4
[04:09:59] :	 [Step 1/1]           target_file_size=1
[04:09:59] :	 [Step 1/1]         
[04:09:59] :	 [Step 1/1]         [TestOptions]
[04:09:59] :	 [Step 1/1]           ingest_using_apply=true
[04:09:59] :	 [Step 1/1]           delete_sized=true
[04:09:59] :	 [Step 1/1]           replace_single_delete=true
[04:09:59] :	 [Step 1/1]           threads=8
[04:09:59] :	 [Step 1/1]           disable_block_property_collector=true
[04:09:59] :	 [Step 1/1]           enable_value_blocks=true
[04:09:59] :	 [Step 1/1]           disable_value_blocks_for_ingest_sstables=true
[04:09:59] :	 [Step 1/1]           async_apply_to_db=true
[04:09:59] :	 [Step 1/1]           shared_storage_enabled=true
[04:09:59] :	 [Step 1/1]           external_storage_enabled=true
[04:09:59] :	 [Step 1/1]           secondary_cache_enabled=true
[04:09:59] :	 [Step 1/1]           seed_efos=2309337034655293500
[04:09:59] :	 [Step 1/1]           ingest_split=true
[04:09:59] :	 [Step 1/1]           io_latency_mean=877.704µs
[04:09:59] :	 [Step 1/1]           io_latency_probability=0.0098303769
[04:09:59] :	 [Step 1/1]           io_latency_seed=1758674309626369732
[04:09:59] :	 [Step 1/1]           use_shared_replicate=true
[04:09:59] :	 [Step 1/1]         
[04:09:59] :	 [Step 1/1]         ===== OPS =====
[04:09:59] :	 [Step 1/1]         Init(0 /* dbs */, 45 /* batches */, 63 /* iters */, 45 /* snapshots */, 1 /* externalObjs */)
[04:09:59] :	 [Step 1/1]         db1.Merge("wfvuccmac@1", "pnryreo")
[04:09:59] :	 [Step 1/1]         db1.Get("wfvuccmac@1")
[04:09:59] :	 [Step 1/1]         db1.Delete("wfvuccmac@1")
[04:09:59] :	 [Step 1/1]         db1.Set("nrghd@1", "cpi")
[04:09:59] :	 [Step 1/1]         db1.DeleteRange("nrghd@1", "wfvuccmac@1")
[04:09:59] :	 [Step 1/1]         db1.Delete("wfvuccmac@1")
[04:09:59] :	 [Step 1/1]         db1.Merge("nrghd@2", "owceqcpv")
[04:09:59] :	 [Step 1/1]         db1.Merge("nrghd@2", "ozomk")
[04:09:59] :	 [Step 1/1]         db1.Get("nrghd@2")
[04:09:59] :	 [Step 1/1]         db1.Delete("wfvuccmac@1")
[04:09:59] :	 [Step 1/1]         db1.Get("nrghd@2")
[04:09:59] :	 [Step 1/1]         db1.Delete("nrghd@1")
[04:09:59] :	 [Step 1/1]         db1.RangeKeySet("nrghd", "wfvuccmac", "@1", "ibxdqzh")
[04:09:59] :	 [Step 1/1]         db1.DeleteRange("nrghd@2", "nrghd@1")
[04:09:59] :	 [Step 1/1]         batch0 = db1.NewIndexedBatch()
[04:09:59] :	 [Step 1/1]         db1.Merge("nrghd@1", "vmqvqjboxehkwacdn")
[04:09:59] :	 [Step 1/1]         db1.Ingest(batch0)
[04:09:59] :	 [Step 1/1]         db1.SingleDelete("nrghd@1", false /* maybeReplaceDelete */)
[04:09:59] :	 [Step 1/1]         db1.SingleDelete("wfvuccmac@1", false /* maybeReplaceDelete */)
[04:09:59] :	 [Step 1/1]         db1.Get("wfvuccmac@1")

@jbowens
Copy link
Collaborator

jbowens commented May 7, 2024

Saw another failure on #3593.

2024-05-07T17:53:41.7149735Z --- FAIL: TestMeta (402.11s)
2024-05-07T17:53:41.7152974Z     --- FAIL: TestMeta/execution (0.00s)
2024-05-07T17:53:41.7154582Z         --- FAIL: TestMeta/execution/random-027 (377.95s)
2024-05-07T17:53:41.7159241Z             meta.go:241: 
2024-05-07T17:53:41.7160165Z                 ===== SEED =====
2024-05-07T17:53:41.7161634Z                 1715104012873781044
2024-05-07T17:53:41.7162286Z                 ===== ERR =====
2024-05-07T17:53:41.7163534Z                 exit status 2
2024-05-07T17:53:41.7164407Z                 ===== OUT =====
2024-05-07T17:53:41.7165941Z                 panic: operation took longer than 1m0s: db1.Compact("kwdmkna@13", "qjggfqpf@10", false /* parallelize */)
2024-05-07T17:53:41.7167497Z                 
2024-05-07T17:53:41.7168444Z                 goroutine 35068 [running]:
2024-05-07T17:53:41.7170024Z                 github.com/cockroachdb/pebble/metamorphic.(*Test).runOp.func1()
2024-05-07T17:53:41.7171812Z                 	/home/runner/work/pebble/pebble/metamorphic/test.go:412 +0xb4
2024-05-07T17:53:41.7173460Z                 created by time.goFunc
2024-05-07T17:53:41.7174905Z                 	/opt/hostedtoolcache/go/1.22.2/x64/src/time/sleep.go:177 +0x2d
2024-05-07T17:53:41.7175856Z                 
2024-05-07T17:53:41.7176737Z                 goroutine 1 [chan receive, 6 minutes]:
2024-05-07T17:53:41.7179254Z                 testing.(*T).Run(0xc00023a4e0, {0xd31eb5?, 0x0?}, 0xd7f708)
2024-05-07T17:53:41.7180975Z                 	/opt/hostedtoolcache/go/1.22.2/x64/src/testing/testing.go:1750 +0x3ab
2024-05-07T17:53:41.7182332Z                 testing.runTests.func1(0xc00023a4e0)
2024-05-07T17:53:41.7184193Z                 	/opt/hostedtoolcache/go/1.22.2/x64/src/testing/testing.go:2161 +0x37
2024-05-07T17:53:41.7185716Z                 testing.tRunner(0xc00023a4e0, 0xc000051c70)
2024-05-07T17:53:41.7187500Z                 	/opt/hostedtoolcache/go/1.22.2/x64/src/testing/testing.go:1689 +0xfb
2024-05-07T17:53:41.7189301Z                 testing.runTests(0xc000218bd0, {0x1353500, 0x2, 0x2}, {0x1?, 0xc000215520?, 0x0?})
2024-05-07T17:53:41.7191239Z                 	/opt/hostedtoolcache/go/1.22.2/x64/src/testing/testing.go:2159 +0x445
2024-05-07T17:53:41.7192468Z                 testing.(*M).Run(0xc000242460)
2024-05-07T17:53:41.7194161Z                 	/opt/hostedtoolcache/go/1.22.2/x64/src/testing/testing.go:2027 +0x68b
2024-05-07T17:53:41.7195245Z                 main.main()
2024-05-07T17:53:41.7196194Z                 	_testmain.go:49 +0x16c
2024-05-07T17:53:41.7196962Z                 
2024-05-07T17:53:41.7197867Z                 goroutine 18 [semacquire, 6 minutes]:
2024-05-07T17:53:41.7199102Z                 sync.runtime_Semacquire(0xc0000e1a40?)
2024-05-07T17:53:41.7200671Z                 	/opt/hostedtoolcache/go/1.22.2/x64/src/runtime/sema.go:62 +0x25
2024-05-07T17:53:41.7201860Z                 sync.(*WaitGroup).Wait(0x0?)
2024-05-07T17:53:41.7203300Z                 	/opt/hostedtoolcache/go/1.22.2/x64/src/sync/waitgroup.go:116 +0x48
2024-05-07T17:53:41.7204910Z                 golang.org/x/sync/errgroup.(*Group).Wait(0xc000380340)
2024-05-07T17:53:41.7207360Z                 	/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:53 +0x25
2024-05-07T17:53:41.7209268Z                 github.com/cockroachdb/pebble/metamorphic.Execute(0xc00024ac60)
2024-05-07T17:53:41.7210998Z                 	/home/runner/work/pebble/pebble/metamorphic/meta.go:643 +0x116
2024-05-07T17:53:41.7213672Z                 github.com/cockroachdb/pebble/metamorphic.RunOnce({0xe52cf8, 0xc00023a680}, {0x7ffcbcf95c47, 0x2b}, 0x0, {0xc00026ef00, 0x33}, {0xc00022c3e0, 0x2, 0x2})
2024-05-07T17:53:41.7215949Z                 	/home/runner/work/pebble/pebble/metamorphic/meta.go:562 +0xff8
2024-05-07T17:53:41.7217941Z                 github.com/cockroachdb/pebble/internal/metamorphic.runTestMeta(0xc00023a680, 0x0)
2024-05-07T17:53:41.7220119Z                 	/home/runner/work/pebble/pebble/internal/metamorphic/meta_test.go:87 +0x179
2024-05-07T17:53:41.7222418Z                 github.com/cockroachdb/pebble/internal/metamorphic.TestMeta(0xc00023a680?)
2024-05-07T17:53:41.7224631Z                 	/home/runner/work/pebble/pebble/internal/metamorphic/meta_test.go:54 +0x15
2024-05-07T17:53:41.7226153Z                 testing.tRunner(0xc00023a680, 0xd7f708)
2024-05-07T17:53:41.7227755Z                 	/opt/hostedtoolcache/go/1.22.2/x64/src/testing/testing.go:1689 +0xfb
2024-05-07T17:53:41.7229176Z                 created by testing.(*T).Run in goroutine 1
2024-05-07T17:53:41.7230750Z                 	/opt/hostedtoolcache/go/1.22.2/x64/src/testing/testing.go:1742 +0x390
2024-05-07T17:53:41.7231769Z                 
2024-05-07T17:53:41.7232597Z                 goroutine 8 [select]:
2024-05-07T17:53:41.7234269Z                 github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFS).startTickerLocked.func1()
2024-05-07T17:53:41.7236092Z                 	/home/runner/work/pebble/pebble/vfs/disk_health.go:670 +0xf8
2024-05-07T17:53:41.7238172Z                 created by github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFS).startTickerLocked in goroutine 18
2024-05-07T17:53:41.7240430Z                 	/home/runner/work/pebble/pebble/vfs/disk_health.go:659 +0x6e
2024-05-07T17:53:41.7241385Z                 
2024-05-07T17:53:41.7242284Z                 goroutine 9 [select]:
2024-05-07T17:53:41.7243925Z                 github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker.func1()
2024-05-07T17:53:41.7245728Z                 	/home/runner/work/pebble/pebble/vfs/disk_health.go:253 +0xcb
2024-05-07T17:53:41.7247765Z                 created by github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker in goroutine 18
2024-05-07T17:53:41.7249729Z                 	/home/runner/work/pebble/pebble/vfs/disk_health.go:248 +0x58
2024-05-07T17:53:41.7250713Z                 
2024-05-07T17:53:41.7251716Z                 goroutine 11 [select, 6 minutes]:
2024-05-07T17:53:41.7253259Z                 github.com/cockroachdb/pebble/wal.(*dirProber).probeLoop(0xc0003d00c8)
2024-05-07T17:53:41.7255099Z                 	/home/runner/work/pebble/pebble/wal/failover_manager.go:99 +0xd2
2024-05-07T17:53:41.7256781Z                 github.com/cockroachdb/pebble/wal.(*stopper).runAsync.func1()
2024-05-07T17:53:41.7258487Z                 	/home/runner/work/pebble/pebble/wal/failover_manager.go:813 +0x22
2024-05-07T17:53:41.7260374Z                 created by github.com/cockroachdb/pebble/wal.(*stopper).runAsync in goroutine 18
2024-05-07T17:53:41.7262265Z                 	/home/runner/work/pebble/pebble/wal/failover_manager.go:812 +0x79
2024-05-07T17:53:41.7263644Z                 
2024-05-07T17:53:41.7264455Z                 goroutine 12 [select]:
2024-05-07T17:53:41.7266372Z                 github.com/cockroachdb/pebble/wal.(*failoverMonitor).monitorLoop(0xc0003d0000, 0xc000200360)
2024-05-07T17:53:41.7268312Z                 	/home/runner/work/pebble/pebble/wal/failover_manager.go:334 +0x105
2024-05-07T17:53:41.7270018Z                 github.com/cockroachdb/pebble/wal.newFailoverMonitor.func1()
2024-05-07T17:53:41.7271847Z                 	/home/runner/work/pebble/pebble/wal/failover_manager.go:258 +0x25
2024-05-07T17:53:41.7273458Z                 github.com/cockroachdb/pebble/wal.(*stopper).runAsync.func1()
2024-05-07T17:53:41.7275198Z                 	/home/runner/work/pebble/pebble/wal/failover_manager.go:813 +0x22
2024-05-07T17:53:41.7277038Z                 created by github.com/cockroachdb/pebble/wal.(*stopper).runAsync in goroutine 18
2024-05-07T17:53:41.7278878Z                 	/home/runner/work/pebble/pebble/wal/failover_manager.go:812 +0x79
2024-05-07T17:53:41.7279972Z                 
2024-05-07T17:53:41.7280797Z                 goroutine 13 [chan receive]:
2024-05-07T17:53:41.7282360Z                 github.com/cockroachdb/pebble.(*cleanupManager).mainLoop(0xc000242500)
2024-05-07T17:53:41.7284194Z                 	/home/runner/work/pebble/pebble/obsolete_files.go:163 +0xe5
2024-05-07T17:53:41.7286343Z                 github.com/cockroachdb/pebble.openCleanupManager.func1.1({0xe559d0?, 0xc000358090?})
2024-05-07T17:53:41.7288242Z                 	/home/runner/work/pebble/pebble/obsolete_files.go:98 +0x17
2024-05-07T17:53:41.7290065Z                 runtime/pprof.Do({0xe55910?, 0x14128a0?}, {{0xc00022c300?, 0x0?, 0x0?}}, 0xc00006cfc0)
2024-05-07T17:53:41.7292089Z                 	/opt/hostedtoolcache/go/1.22.2/x64/src/runtime/pprof/runtime.go:51 +0x9d
2024-05-07T17:53:41.7293743Z                 github.com/cockroachdb/pebble.openCleanupManager.func1()
2024-05-07T17:53:41.7295361Z                 	/home/runner/work/pebble/pebble/obsolete_files.go:97 +0x56
2024-05-07T17:53:41.7297166Z                 created by github.com/cockroachdb/pebble.openCleanupManager in goroutine 18
2024-05-07T17:53:41.7298885Z                 	/home/runner/work/pebble/pebble/obsolete_files.go:96 +0x265

https://productionresultssa15.blob.core.windows.net/actions-results/4f48879c-2c76-4976-9fa1-b17e3e71f06a/workflow-job-run-145a003c-c470-5a00-1723-1ce9b722fc2c/logs/job/job-logs.txt?rsct=text%2Fplain&se=2024-05-07T18%3A16%3A48Z&sig=pLNTCeKgcZs58SLVgC0%2FxnXZERKngP0CGc2lKjGLggg%3D&sp=r&spr=https&sr=b&st=2024-05-07T18%3A06%3A43Z&sv=2021-12-02

@itsbilal itsbilal self-assigned this May 7, 2024
@itsbilal itsbilal mentioned this issue May 7, 2024
@cockroach-teamcity
Copy link
Member Author

github.com/cockroachdb/pebble/internal/metamorphic.TestMeta failed with artifacts on refs/heads/master @ 98cadc9f9bcd:

        // 04:08:45.696 INFO: [JOB 7927] sstable deleted 021223
        // 04:08:45.696 INFO: [JOB 7927] sstable deleted 021224
        // 04:08:45.696 INFO: [JOB 7927] sstable deleted 021225
        // 04:08:45.696 INFO: [JOB 7927] sstable deleted 021226
        // 04:08:45.700 INFO: [JOB 7927] sstable deleted 021227
        // 04:08:45.700 INFO: [JOB 7927] sstable deleted 021228
        // 04:08:45.700 INFO: [JOB 7927] sstable deleted 021229
        // 04:08:45.700 INFO: [JOB 7956] sstable deleted 021450
        // 04:08:45.700 INFO: [JOB 7961] sstable deleted 021451
        // 04:08:45.700 INFO: [JOB 7962] sstable deleted 021452
        // 04:08:45.700 INFO: [JOB 7967] sstable deleted 021453
        // 04:08:45.700 INFO: [JOB 7968] sstable deleted 021454
        // 04:08:45.700 INFO: [JOB 7968] sstable deleted 021455
        // 04:08:45.700 INFO: [JOB 7975] sstable deleted 021456
        // 04:08:45.700 INFO: [JOB 7978] sstable deleted 021457
        // 04:08:45.700 INFO: [JOB 7981] sstable deleted 021458
        // 04:08:45.700 INFO: [JOB 7984] sstable deleted 021459
        // 04:08:45.700 INFO: [JOB 7985] sstable deleted 021460
        // 04:08:45.700 INFO: [JOB 7985] sstable deleted 021461
        // 04:08:45.700 INFO: [JOB 7987] sstable deleted 021462
        // 04:08:45.700 INFO: [JOB 7995] sstable deleted 021463
        // 04:08:45.700 INFO: [JOB 7996] sstable deleted 021464
        // 04:08:45.700 INFO: [JOB 7998] sstable deleted 021465
        // 04:08:45.700 INFO: [JOB 7998] sstable deleted 021466
        // 04:08:45.700 INFO: [JOB 8004] sstable deleted 021467
        // 04:08:45.700 INFO: [JOB 8007] sstable deleted 021468
        // 04:08:45.700 INFO: [JOB 8007] sstable deleted 021469
        // 04:08:45.700 INFO: [JOB 8012] sstable deleted 021470
        // 04:08:45.700 INFO: [JOB 8015] sstable deleted 021471
        // 04:08:45.700 INFO: [JOB 8017] sstable deleted 021472
        // 04:08:45.700 INFO: [JOB 8017] sstable deleted 021473
        // 04:08:45.700 INFO: [JOB 8023] sstable deleted 021474
        // 04:08:45.700 INFO: [JOB 8027] sstable deleted 021475
        // 04:08:45.706 INFO: [JOB 7933] MANIFEST created 021756
        // 04:08:46.341 INFO: [JOB 7933] flushed 1 memtable (244B) to L0 [021502 021503 021504 021505 021506 021507 021508 021509 021510 021511 021512 021513 021514 021515 021516 021517 021518 021519 021520 021521 021522 021523 021524 021525 021526 021527 021528 021529 021530 021531 021532 021533 021534 021535 021536 021537 021538 021539 021540 021541 021542 021543 021544 021545 021546 021547 021548 021549 021550 021551 021552 021553 021554 021555 021556 021557 021558 021559 021560 021561 021562 021563 021564 021565 021566 021567 021568 021569 021570 021571 021572 021573 021574 021575 021576 021577 021578 021579 021580 021581 021582 021583 021584 021585 021586 021587 021588 021589 021590 021591 021592 021593 021594 021595 021596 021597 021598 021599 021600 021601 021602 021603 021604 021605 021606 021607 021608 021609 021610 021611 021612 021613 021614 021615 021616 021617 021618 021619 021620 021621 021622 021623 021624 021625 021626 021627 021628 021629 021630 021631 021632 021633 021634 021635 021636 021637 021638 021639 021640 021641 021642 021643 021644 021645 021646 021647 021648 021649 021650 021651 021652 021653 021654 021655 021656 021657 021658 021659 021660 021661 021662 021663 021664 021665 021666 021667 021668 021670 021672 021674 021675 021677 021678 021680 021681 021683 021685 021686 021688 021689 021690 021692 021693 021695 021696 021697 021699 021700 021702 021704 021705 021706 021707 021708 021711 021712 021713 021714 021715 021716 021717 021719 021721 021722 021723 021724 021726 021728 021729 021732 021733 021734 021736 021737 021739 021740 021742 021743 021744 021745 021747 021748 021750 021752 021753 021754] (191KB), in 42.1s (42.7s total), output rate 4.5KB/s
        // 04:08:46.341 INFO: [JOB 7933] WAL deleted 021043
        // 04:08:46.341 INFO: [JOB 7933] sstable deleted 021476
        // 04:08:46.341 INFO: [JOB 7933] MANIFEST deleted 018632
        // 04:08:46.342 INFO: [JOB 8028] compacted(default) L4 [021477] (1018B) Score=20.42 + L5 [] (0B) Score=0.93 -> L5 [021755] (1018B), in 1.2s (1.9s total), output rate 845B/s
        // 04:08:46.964 INFO: [JOB 8028] sstable deleted 021477
        // 04:08:46.966 INFO: [JOB 8031] compacted(default) L4 [021478] (1.0KB) Score=21.15 + L5 [] (0B) Score=0.95 -> L5 [021757] (1.0KB), in 0.0s (1.3s total), output rate 3.1MB/s
        // 04:08:47.608 INFO: [JOB 8031] sstable deleted 021478
        // 04:08:47.608 INFO: [JOB 8034] compacting(default) L3 [021490] (981B) Score=13.28 + L4 [] (0B) Score=20.12; OverlappingRatio: Single 0.00, Multi 0.00
        // 04:08:47.608 INFO: [JOB 8034] compacting: sstable created 021758
        // 04:08:47.609 INFO: [JOB 8035] compacting(default) L4 [021479] (1.0KB) Score=20.86 + L5 [] (0B) Score=0.92; OverlappingRatio: Single 0.00, Multi 0.00
        // 04:08:47.609 INFO: [JOB 8034] compacting: sstable created 021759
        // 04:08:47.609 INFO: [JOB 8035] compacting: sstable created 021760
        
        To reduce:  go test ./internal/metamorphic -tags invariants -run 'TestMeta$' --run-dir /artifacts/meta/240508-040626.0551582757471/random-029 --try-to-reduce -v
        --- FAIL: TestMeta/execution/random-029 (142.03s)
Help

To reproduce, try:

go test -tags 'invariants' -exec 'stress -p 1' -timeout 0 -test.v -run 'TestMeta$' ./internal/metamorphic -seed 1715141186055399080 -ops "uniform:5000-10000"

This test on roachdash | Improve this report!

@itsbilal
Copy link
Member

itsbilal commented May 8, 2024

Investigating with seed 1715141186055399080 where we time out on an IngestExternalFileOp, it looks like we time out in makeRoomForWrite where we're trying to force a flush cuz we can't ingest external files as a flushable ingest:

        goroutine 73 [sync.Cond.Wait]:
        runtime.gopark(0xc18718ac7b00e410?, 0x14000600000?, 0x88?, 0x70?, 0x104cf4de0?)
                /opt/homebrew/Cellar/go/1.21.1/libexec/src/runtime/proc.go:398 +0xc8 fp=0x1401d8a7040 sp=0x1401d8a7020 pc=0x104cc3578
        runtime.goparkunlock(...)
                /opt/homebrew/Cellar/go/1.21.1/libexec/src/runtime/proc.go:404
        sync.runtime_notifyListWait(0x14000480af0, 0xb3)
                /opt/homebrew/Cellar/go/1.21.1/libexec/src/runtime/sema.go:527 +0x154 fp=0x1401d8a7090 sp=0x1401d8a7040 pc=0x104cf3804
        sync.(*Cond).Wait(0x14000480ae0)
                /opt/homebrew/Cellar/go/1.21.1/libexec/src/sync/cond.go:70 +0xcc fp=0x1401d8a70d0 sp=0x1401d8a7090 pc=0x104d0012c
        github.com/cockroachdb/pebble.(*DB).makeRoomForWrite(0x14000480900, 0x0)
                /Users/bilal/go/src/github.com/cockroachdb/pebble/db.go:2471 +0x1a0 fp=0x1401d8a71c0 sp=0x1401d8a70d0 pc=0x1051fec30
        github.com/cockroachdb/pebble.(*DB).ingest.func1(0x8b7)
                /Users/bilal/go/src/github.com/cockroachdb/pebble/ingest.go:1601 +0x684 fp=0x1401d8a74e0 sp=0x1401d8a71c0 pc=0x105217964
        github.com/cockroachdb/pebble.(*commitPipeline).AllocateSeqNum(0x140002a6000, 0x1, 0x14017f979b0, 0x1401d8a78f0)
                /Users/bilal/go/src/github.com/cockroachdb/pebble/commit.go:415 +0x280 fp=0x1401d8a7550 sp=0x1401d8a74e0 pc=0x1051d9160
        github.com/cockroachdb/pebble.(*DB).ingest(0x14000480900, {0x14017394360, 0x1, 0x1}, {0x0, 0x0, 0x0}, {{0x0, 0x0, 0x0}, ...}, ...)
                /Users/bilal/go/src/github.com/cockroachdb/pebble/ingest.go:1685 +0x674 fp=0x1401d8a7a40 sp=0x1401d8a7550 pc=0x105216524
        github.com/cockroachdb/pebble.(*DB).Ingest(0x0?, {0x14017394360?, 0x14000463ff0?, 0xe?})
                /Users/bilal/go/src/github.com/cockroachdb/pebble/ingest.go:1106 +0xbc fp=0x1401d8a7af0 sp=0x1401d8a7a40 pc=0x105214f4c
        github.com/cockroachdb/pebble/metamorphic.(*ingestExternalFilesOp).run(0x140003f85c0, 0x14000000900, {0x140001738f0?, 0x1402c9c7f20?, 0x0?})
                /Users/bilal/go/src/github.com/cockroachdb/pebble/metamorphic/ops.go:1034 +0x6cc fp=0x1401d8a7e20 sp=0x1401d8a7af0 pc=0x1052ad66c
        github.com/cockroachdb/pebble/metamorphic.(*Test).runOp(0x14000000900?, 0x140241bff00?, {0x140001738f0?, 0x0?, 0x0?})
                /Users/bilal/go/src/github.com/cockroachdb/pebble/metamorphic/test.go:415 +0x118 fp=0x1401d8a7e90 sp=0x1401d8a7e20 pc=0x1052c6ea8
        github.com/cockroachdb/pebble/metamorphic.Execute.func1()
                /Users/bilal/go/src/github.com/cockroachdb/pebble/metamorphic/meta.go:627 +0x114 fp=0x1401d8a7f60 sp=0x1401d8a7e90 pc=0x1052a5df4
        golang.org/x/sync/errgroup.(*Group).Go.func1()

@cockroach-teamcity
Copy link
Member Author

github.com/cockroachdb/pebble/internal/metamorphic.TestMeta failed with artifacts on refs/heads/master @ 4301b8f511d9:

=== RUN   TestMeta/execution/random-014
=== PAUSE TestMeta/execution/random-014
=== CONT  TestMeta/execution/random-014
=== RUN   TestMeta/execution/random-006
=== PAUSE TestMeta/execution/random-006
=== CONT  TestMeta/execution/random-006
=== RUN   TestMeta/execution/random-021
=== PAUSE TestMeta/execution/random-021
=== CONT  TestMeta/execution/random-021
=== RUN   TestMeta/execution/random-022
=== PAUSE TestMeta/execution/random-022
=== CONT  TestMeta/execution/random-022
=== RUN   TestMeta/execution/random-024
=== PAUSE TestMeta/execution/random-024
=== CONT  TestMeta/execution/random-024
=== RUN   TestMeta/execution/random-029
=== PAUSE TestMeta/execution/random-029
=== CONT  TestMeta/execution/random-029
=== RUN   TestMeta/compare/standard-001
=== RUN   TestMeta/compare/standard-004
=== RUN   TestMeta/execution/random-007
=== PAUSE TestMeta/execution/random-007
=== CONT  TestMeta/execution/random-007
=== RUN   TestMeta/execution/random-025
=== PAUSE TestMeta/execution/random-025
=== CONT  TestMeta/execution/random-025
=== RUN   TestMeta/compare/standard-008
=== RUN   TestMeta/execution/random-020
=== PAUSE TestMeta/execution/random-020
=== CONT  TestMeta/execution/random-020
=== RUN   TestMeta/execution/standard-007
=== PAUSE TestMeta/execution/standard-007
=== CONT  TestMeta/execution/standard-007
=== RUN   TestMeta/execution/standard-029
=== PAUSE TestMeta/execution/standard-029
=== CONT  TestMeta/execution/standard-029
=== RUN   TestMeta/compare/standard-003
=== RUN   TestMeta/compare/standard-006
=== RUN   TestMeta/execution/standard-001
=== PAUSE TestMeta/execution/standard-001
=== CONT  TestMeta/execution/standard-001
=== RUN   TestMeta/execution/standard-003
=== PAUSE TestMeta/execution/standard-003
=== CONT  TestMeta/execution/standard-003
=== RUN   TestMeta/execution/random-016
=== PAUSE TestMeta/execution/random-016
=== CONT  TestMeta/execution/random-016
=== RUN   TestMeta/execution/random-028
=== PAUSE TestMeta/execution/random-028
=== CONT  TestMeta/execution/random-028
Help

To reproduce, try:

go test -tags 'invariants' -exec 'stress -p 1' -timeout 0 -test.v -run 'TestMeta$' ./internal/metamorphic -seed 1715227882649100225 -ops "uniform:5000-10000"

This test on roachdash | Improve this report!

@RaduBerinde RaduBerinde assigned RaduBerinde and unassigned itsbilal May 9, 2024
RaduBerinde added a commit to RaduBerinde/pebble that referenced this issue May 9, 2024
Before the recent compaction cleanup, the output table writer was
initialized lazily, which means that until the first key was written
the estimated size was 0. So even with a tiny target size, we will get
at least one point key and any associated spans.

Now the table writer is initialized up front and the estimated size of
an empty table writer is 8. When the target file size is less than
that, we try to split the table as soon as possible (and emit just a
small piece of a span in many cases). These many tiny tables slow down
the tests a lot, to the point of timing out certain operations.

This change sets the minimum target file size in the metamorphic
tests to 12.

Fixes cockroachdb#3594
Fixes cockroachdb#3595
RaduBerinde added a commit that referenced this issue May 9, 2024
Before the recent compaction cleanup, the output table writer was
initialized lazily, which means that until the first key was written
the estimated size was 0. So even with a tiny target size, we will get
at least one point key and any associated spans.

Now the table writer is initialized up front and the estimated size of
an empty table writer is 8. When the target file size is less than
that, we try to split the table as soon as possible (and emit just a
small piece of a span in many cases). These many tiny tables slow down
the tests a lot, to the point of timing out certain operations.

This change sets the minimum target file size in the metamorphic
tests to 12.

Fixes #3594
Fixes #3595
@jbowens jbowens moved this to Done in [Deprecated] Storage Jun 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
No open projects
Archived in project
Development

Successfully merging a pull request may close this issue.

4 participants