Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

meta: TestOptionsRoundtrip is flaky #3421

Closed
sumeerbhola opened this issue Mar 18, 2024 · 0 comments · Fixed by #3458
Closed

meta: TestOptionsRoundtrip is flaky #3421

sumeerbhola opened this issue Mar 18, 2024 · 0 comments · Fixed by #3458

Comments

@sumeerbhola
Copy link
Collaborator

I suspect what is happening below is that the value of opts.ioLatencyProbability is very close to 0, but not zero, so the initial write has these values written:

if opts.ioLatencyProbability > 0.0 {
fmt.Fprintf(&buf, " io_latency_mean=%s\n", opts.ioLatencyMean)
fmt.Fprintf(&buf, " io_latency_probability=%f\n", opts.ioLatencyProbability)
fmt.Fprintf(&buf, " io_latency_seed=%d\n", opts.ioLatencySeed)
}

But since the default precision for printf of floats is causing it to print as zero, the parsing is seeing it as 0, so when the parsed options are printed, we omit these values.

--- FAIL: TestOptionsRoundtrip (0.68s)
--- FAIL: TestOptionsRoundtrip/random-065 (0.00s)
options_test.go:94: Serialized options:
[Version]
pebble_version=0.1

        [Options]
          bytes_per_sync=65536
          cache_size=33554432
          cleaner=archive
          compaction_debt_concurrency=1073741824
          comparer=pebble.internal.testkeys
          disable_wal=false
          disable_ingest_as_flushable=true
          flush_delay_delete_range=905ms
          flush_delay_range_key=670ms
          flush_split_bytes=128
          format_major_version=17
          l0_compaction_concurrency=1
          l0_compaction_file_threshold=1
          l0_compaction_threshold=9
          l0_stop_writes_threshold=84
          lbase_max_bytes=16
          level_multiplier=80
          max_concurrent_compactions=1
          max_manifest_file_size=1
          max_open_files=1000
          mem_table_size=4194304
          mem_table_stop_writes_threshold=4
          min_deletion_rate=16777216
          merger=pebble.concatenate
          multilevel_compaction_heuristic=wamp(0.00, false)
          read_compaction_rate=16000
          read_sampling_multiplier=16
          strict_wal_tail=true
          table_cache_shards=4
          validate_on_ingest=false
          wal_dir=
          wal_bytes_per_sync=0
          max_writer_concurrency=0
          force_writer_parallelism=false
          secondary_cache_size_bytes=0
          create_on_shared=0
        
        [WAL Failover]
          secondary_dir=data/wal_secondary
          primary_dir_probe_interval=1.951817ms
          healthy_probe_latency_threshold=4.428586ms
          healthy_interval=8.303307ms
          unhealthy_sampling_interval=2.092938ms
          unhealthy_operation_latency_threshold=6.05706ms
          elevated_write_stall_threshold_lag=15.792989ms
        
        [Level "0"]
          block_restart_interval=64
          block_size=16384
          block_size_threshold=90
          compression=ZSTD
          filter_policy=testing_bloom_filter/bits_per_key=8
          filter_type=table
          index_block_size=8192
          target_file_size=1048576
        
        [TestOptions]
          strictfs=true
          replace_single_delete=true
          threads=4
          enable_value_blocks=true
          disable_value_blocks_for_ingest_sstables=true
          external_storage_enabled=true
          seed_efos=5909666[56](https://github.com/cockroachdb/pebble/actions/runs/8331540259/job/22798668146?pr=3407#step:4:57)1378333442
          io_latency_mean=3.836181ms
          io_latency_probability=0.000000
          io_latency_seed=4038267658547851335
          use_excise=true
        
    options_test.go:100: 
        	Error Trace:	/home/runner/work/pebble/pebble/metamorphic/options_test.go:100
        	            				/home/runner/work/pebble/pebble/metamorphic/options_test.go:148
        	Error:      	Not equal: 
        	            	expected: "[Version]\n  pebble_version=0.1\n\n[Options]\n  bytes_per_sync=65536\n  cache_size=33554432\n  cleaner=archive\n  compaction_debt_concurrency=1073741824\n  comparer=pebble.internal.testkeys\n  disable_wal=false\n  disable_ingest_as_flushable=true\n  flush_delay_delete_range=905ms\n  flush_delay_range_key=670ms\n  flush_split_bytes=128\n  format_major_version=17\n  l0_compaction_concurrency=1\n  l0_compaction_file_threshold=1\n  l0_compaction_threshold=9\n  l0_stop_writes_threshold=84\n  lbase_max_bytes=16\n  level_multiplier=80\n  max_concurrent_compactions=1\n  max_manifest_file_size=1\n  max_open_files=1000\n  mem_table_size=4194304\n  mem_table_stop_writes_threshold=4\n  min_deletion_rate=16777216\n  merger=pebble.concatenate\n  multilevel_compaction_heuristic=wamp(0.00, false)\n  read_compaction_rate=16000\n  read_sampling_multiplier=16\n  strict_wal_tail=true\n  table_cache_shards=4\n  validate_on_ingest=false\n  wal_dir=\n  wal_bytes_per_sync=0\n  max_writer_concurrency=0\n  force_writer_parallelism=false\n  secondary_cache_size_bytes=0\n  create_on_shared=0\n\n[WAL Failover]\n  secondary_dir=data/wal_secondary\n  primary_dir_probe_interval=1.951817ms\n  healthy_probe_latency_threshold=4.428586ms\n  healthy_interval=8.303307ms\n  unhealthy_sampling_interval=2.092938ms\n  unhealthy_operation_latency_threshold=6.0[57](https://github.com/cockroachdb/pebble/actions/runs/8331540259/job/22798668146?pr=3407#step:4:58)06ms\n  elevated_write_stall_threshold_lag=15.792989ms\n\n[Level \"0\"]\n  block_restart_interval=64\n  block_size=16384\n  block_size_threshold=90\n  compression=ZSTD\n  filter_policy=testing_bloom_filter/bits_per_key=8\n  filter_type=table\n  index_block_size=8192\n  target_file_size=1048576\n\n[TestOptions]\n  strictfs=true\n  replace_single_delete=true\n  threads=4\n  enable_value_blocks=true\n  disable_value_blocks_for_ingest_sstables=true\n  external_storage_enabled=true\n  seed_efos=5909666561378333442\n  io_latency_mean=3.836181ms\n  io_latency_probability=0.000000\n  io_latency_seed=40382676[58](https://github.com/cockroachdb/pebble/actions/runs/8331540259/job/22798668146?pr=3407#step:4:59)547851335\n  use_excise=true\n"
        	            	actual  : "[Version]\n  pebble_version=0.1\n\n[Options]\n  bytes_per_sync=65536\n  cache_size=33554432\n  cleaner=archive\n  compaction_debt_concurrency=1073741824\n  comparer=pebble.internal.testkeys\n  disable_wal=false\n  disable_ingest_as_flushable=true\n  flush_delay_delete_range=905ms\n  flush_delay_range_key=670ms\n  flush_split_bytes=128\n  format_major_version=17\n  l0_compaction_concurrency=1\n  l0_compaction_file_threshold=1\n  l0_compaction_threshold=9\n  l0_stop_writes_threshold=84\n  lbase_max_bytes=16\n  level_multiplier=80\n  max_concurrent_compactions=1\n  max_manifest_file_size=1\n  max_open_files=1000\n  mem_table_size=4194304\n  mem_table_stop_writes_threshold=4\n  min_deletion_rate=16777216\n  merger=pebble.concatenate\n  multilevel_compaction_heuristic=wamp(0.00, false)\n  read_compaction_rate=16000\n  read_sampling_multiplier=16\n  strict_wal_tail=true\n  table_cache_shards=4\n  validate_on_ingest=false\n  wal_dir=\n  wal_bytes_per_sync=0\n  max_writer_concurrency=0\n  force_writer_parallelism=false\n  secondary_cache_size_bytes=0\n  create_on_shared=0\n\n[WAL Failover]\n  secondary_dir=data/wal_secondary\n  primary_dir_probe_interval=1.951817ms\n  healthy_probe_latency_threshold=4.428586ms\n  healthy_interval=8.303307ms\n  unhealthy_sampling_interval=2.092938ms\n  unhealthy_operation_latency_threshold=6.05706ms\n  elevated_write_stall_threshold_lag=15.792989ms\n\n[Level \"0\"]\n  block_restart_interval=64\n  block_size=16384\n  block_size_threshold=90\n  compression=ZSTD\n  filter_policy=testing_bloom_filter/bits_per_key=8\n  filter_type=table\n  index_block_size=8192\n  target_file_size=1048576\n\n[TestOptions]\n  strictfs=true\n  replace_single_delete=true\n  threads=4\n  enable_value_blocks=true\n  disable_value_blocks_for_ingest_sstables=true\n  external_storage_enabled=true\n  seed_efos=[59](https://github.com/cockroachdb/pebble/actions/runs/8331540259/job/22798668146?pr=3407#step:4:60)096665[61](https://github.com/cockroachdb/pebble/actions/runs/8331540259/job/22798668146?pr=3407#step:4:62)378333442\n  use_excise=true\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -68,5 +68,2 @@
        	            	   seed_efos=590966[65](https://github.com/cockroachdb/pebble/actions/runs/8331540259/job/22798668146?pr=3407#step:4:66)61378333442
        	            	-  io_latency_mean=3.836181ms
        	            	-  io_latency_probability=0.000000
        	            	-  io_latency_seed=40382[67](https://github.com/cockroachdb/pebble/actions/runs/8331540259/job/22798668146?pr=3407#step:4:68)65854[78](https://github.com/cockroachdb/pebble/actions/runs/8331540259/job/22798668146?pr=3407#step:4:79)5[133](https://github.com/cockroachdb/pebble/actions/runs/8331540259/job/22798668146?pr=3407#step:4:134)5
        	            	   use_excise=true
        	Test:       	TestOptionsRoundtrip/random-065

FAIL

RaduBerinde added a commit to RaduBerinde/pebble that referenced this issue Mar 26, 2024
Print 10 decimal digits of `ioLatencyProbability` when stringifying
the options and make sure that it's at least that value when
generating a random probability.

Fixes cockroachdb#3421.
RaduBerinde added a commit to RaduBerinde/pebble that referenced this issue Mar 26, 2024
Print 10 decimal digits of `ioLatencyProbability` when stringifying
the options and make sure that it's at least that value when
generating a random probability.

Fixes cockroachdb#3421.
RaduBerinde added a commit that referenced this issue Mar 27, 2024
Print 10 decimal digits of `ioLatencyProbability` when stringifying
the options and make sure that it's at least that value when
generating a random probability.

Fixes #3421.
@jbowens jbowens moved this to Done in [Deprecated] Storage Jun 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Archived in project
Development

Successfully merging a pull request may close this issue.

2 participants