Use tikv-jemallocator instead of jemallocator #6354

estensen · 2024-09-04T12:07:04Z

Proposed Changes

Replace jemallocator with the successor tikv-jemallocator

chong-he · 2024-09-04T13:16:58Z

The link for tikv-jemallocator: https://github.com/tikv/jemallocator

estensen · 2024-09-04T13:28:04Z

The link for tikv-jemallocator: https://github.com/tikv/jemallocator

whops, sorry, added now

michaelsproul · 2024-09-05T04:45:46Z

common/malloc_utils/Cargo.toml


 # Jemalloc's background_threads feature requires Linux (pthreads).
 [target.'cfg(target_os = "linux")'.dependencies]
-jemallocator = { version = "0.5.0", optional = true, features = ["stats", "background_threads"] }
+tikv-jemallocator = { version = "0.5.0", optional = true, features = [


Let's bump to 0.6 while we're at it.

It seems jemallocator and tikv-jemallocator were identical until 0.5.4 anyway, and then the 0.6 version was published only to the tikv-jemallocator crate:

The project is also published as jemallocator for historical reasons. The two crates are the same except names. For new projects, it's recommended to use tikv-xxx versions instead.

https://crates.io/crates/tikv-jemallocator/versions

https://crates.io/crates/jemallocator/versions

Bumped to 0.6.0. Let me know if you wanted to not pin patch version.

Now I'm getting stack overflow when testing locally. Increasing the stack to 8MB it doesn't happen.

❯ cargo test Compiling beacon_node v5.3.0 (/Users/anda/Developer/lighthouse/beacon_node) Compiling network v0.2.0 (/Users/anda/Developer/lighthouse/beacon_node/network) Compiling types v0.2.1 (/Users/anda/Developer/lighthouse/consensus/types) Compiling beacon_chain v0.2.0 (/Users/anda/Developer/lighthouse/beacon_node/beacon_chain) Compiling state_processing v0.2.0 (/Users/anda/Developer/lighthouse/consensus/state_processing) Compiling lcli v5.3.0 (/Users/anda/Developer/lighthouse/lcli) Compiling validator_client v0.3.5 (/Users/anda/Developer/lighthouse/validator_client) Compiling state_transition_vectors v0.1.0 (/Users/anda/Developer/lighthouse/testing/state_transition_vectors) Compiling node_test_rig v0.2.0 (/Users/anda/Developer/lighthouse/testing/node_test_rig) Compiling watch v0.1.0 (/Users/anda/Developer/lighthouse/watch) Compiling database_manager v0.1.0 (/Users/anda/Developer/lighthouse/database_manager) Compiling boot_node v5.3.0 (/Users/anda/Developer/lighthouse/boot_node) Compiling lighthouse v5.3.0 (/Users/anda/Developer/lighthouse/lighthouse) Compiling simulator v0.2.0 (/Users/anda/Developer/lighthouse/testing/simulator) Compiling store v0.2.0 (/Users/anda/Developer/lighthouse/beacon_node/store) Compiling timer v0.2.0 (/Users/anda/Developer/lighthouse/beacon_node/timer) Finished `test` profile [unoptimized + debuginfo] target(s) in 6m 19s Running unittests src/lib.rs (target/debug/deps/account_manager-dba67dee002bdd47) running 0 tests test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s Running unittests src/lib.rs (target/debug/deps/account_utils-d8946cbf523eb6cc) running 13 tests test test::test_password_exactly_min_length ... ok test test::test_password_over_min_length ... ok test test::unicode_characters ... ok test test::test_zeroize_strip_off ... ok test test::test_strip_off ... ok test test::test_password_too_short - should panic ... ok test validator_definitions::tests::voting_keystore_filename_lighthouse ... ok test validator_definitions::tests::graffiti_checks ... ok test validator_definitions::tests::gas_limit_checks ... ok test validator_definitions::tests::builder_proposals_checks ... ok test validator_definitions::tests::suggested_fee_recipient_checks ... ok test validator_definitions::tests::voting_keystore_filename_launchpad ... ok test validator_definitions::tests::voting_keystore_filename_prysm ... ok test result: ok. 13 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.02s Running unittests src/lib.rs (target/debug/deps/beacon_chain-9ba3fc78ed49044f) running 79 tests test block_times_cache::test::observed_time_uses_minimum ... ok test data_availability_checker::overflow_lru_cache::pending_components_tests::invalid_blobs_block_valid_blobs ... ok test data_availability_checker::overflow_lru_cache::pending_components_tests::invalid_blobs_valid_blobs_block ... ok test data_availability_checker::overflow_lru_cache::pending_components_tests::valid_blobs_invalid_blobs_block ... ok test data_availability_checker::overflow_lru_cache::pending_components_tests::block_valid_blobs_invalid_blobs ... ok test data_availability_checker::overflow_lru_cache::pending_components_tests::valid_blobs_block_invalid_blobs ... ok test data_availability_checker::overflow_lru_cache::pending_components_tests::valid_block_invalid_blobs_valid_blobs ... ok test eth1_chain::test::collect_valid_votes::distinct_votes_in_state ... ok test eth1_chain::test::collect_valid_votes::no_votes_in_state ... ok test eth1_chain::test::collect_valid_votes::duplicate_votes_in_state ... ok test eth1_chain::test::eth1_chain_json_backend::deposits_empty_cache ... ok test eth1_chain::test::eth1_chain_json_backend::eth1_data_empty_cache ... ok test eth1_chain::test::eth1_chain_json_backend::default_vote ... ok test eth1_chain::test::slot_start_time ... ok test eth1_chain::test::winning_vote::all_tying_votes ... ok test eth1_chain::test::winning_vote::equal_votes ... ok test eth1_chain::test::winning_vote::no_votes ... ok test eth1_chain::test::winning_vote::some_votes ... ok test eth1_chain::test::winning_vote::tying_votes ... ok test eth1_finalization_cache::tests::fork_at_epoch_boundary ... ok test eth1_finalization_cache::tests::fully_imported_deposits ... ok test eth1_chain::test::eth1_data_sets::empty_cache ... ok test eth1_finalization_cache::tests::partially_imported_deposits ... ok test eth1_finalization_cache::tests::massive_deposit_queue ... ok test eth1_chain::test::eth1_data_sets::ideal_scenario ... ok test eth1_chain::test::eth1_chain_json_backend::deposits_with_cache ... ok test head_tracker::test::bad_length ... ok test head_tracker::test::block_add ... ok test head_tracker::test::empty_round_trip ... ok test head_tracker::test::non_empty_round_trip ... ok test graffiti_calculator::tests::check_graffiti_with_validator_specified_value ... ok test naive_aggregation_pool::tests::attestation_key_tree_hash_tests ... ok test graffiti_calculator::tests::check_graffiti_with_el_version_support ... ok test graffiti_calculator::tests::check_graffiti_without_el_version_support ... ok test naive_aggregation_pool::tests::attestation_tests_base::auto_pruning_item ... ok test naive_aggregation_pool::tests::attestation_tests_base::single_item ... ok test naive_aggregation_pool::tests::attestation_tests_base::multiple_items ... ok test naive_aggregation_pool::tests::attestation_tests_electra::auto_pruning_item ... ok test naive_aggregation_pool::tests::attestation_tests_electra::multiple_items ... ok test naive_aggregation_pool::tests::attestation_tests_electra::single_item ... ok test naive_aggregation_pool::tests::sync_contribution_tests::auto_pruning_item ... ok test naive_aggregation_pool::tests::sync_contribution_tests::max_items ... ok test naive_aggregation_pool::tests::sync_contribution_tests::multiple_items ... ok test naive_aggregation_pool::tests::sync_contribution_tests::single_item ... ok test observed_attesters::tests::observed_aggregators::mulitple_contiguous_periods ... ok test observed_attesters::tests::observed_aggregators::mulitple_non_contiguous_periods ... ok test observed_attesters::tests::observed_aggregators::single_period ... ok test observed_attesters::tests::observed_attesters::mulitple_contiguous_periods ... ok test observed_attesters::tests::observed_attesters::mulitple_non_contiguous_periods ... ok test observed_attesters::tests::observed_attesters::single_period ... ok test observed_attesters::tests::observed_sync_aggregators::mulitple_contiguous_periods_different_subcommittee ... ok test observed_attesters::tests::observed_sync_aggregators::mulitple_contiguous_periods_same_subcommittee ... ok test observed_attesters::tests::observed_sync_aggregators::mulitple_non_contiguous_periods_different_subcommitte ... ok test observed_attesters::tests::observed_sync_aggregators::mulitple_non_contiguous_periods_same_subcommitte ... ok test observed_attesters::tests::observed_sync_aggregators::single_period ... ok test observed_attesters::tests::observed_sync_aggregators::single_period_multiple_subcommittees ... ok test observed_attesters::tests::observed_sync_contributors::mulitple_contiguous_periods_different_subcommittee ... ok test observed_attesters::tests::observed_sync_contributors::mulitple_contiguous_periods_same_subcommittee ... ok test observed_attesters::tests::observed_sync_contributors::mulitple_non_contiguous_periods_different_subcommitte ... ok test observed_attesters::tests::observed_sync_contributors::mulitple_non_contiguous_periods_same_subcommitte ... ok test observed_attesters::tests::observed_sync_contributors::single_period ... ok test observed_attesters::tests::observed_sync_contributors::single_period_multiple_subcommittees ... ok test observed_attesters::tests::value_storage ... ok test observed_block_producers::tests::pruning ... ok test observed_block_producers::tests::simple_observations ... ok test observed_data_sidecars::tests::pruning ... ok test observed_data_sidecars::tests::simple_observations ... ok test observed_slashable::tests::pruning ... ok test observed_slashable::tests::simple_observations ... ok test state_advance_timer::tests::lock ... ok test validator_pubkey_cache::test::basic_operation ... ok test validator_pubkey_cache::test::persistence ... ok test naive_aggregation_pool::tests::attestation_tests_base::max_items ... ok test naive_aggregation_pool::tests::attestation_tests_electra::max_items ... ok test data_availability_checker::overflow_lru_cache::test::overflow_cache_test_insert_components ... ok thread 'tokio-runtime-worker' has overflowed its stack fatal runtime error: stack overflow error: test failed, to rerun pass `-p beacon_chain --lib` Caused by: process didn't exit successfully: `/Users/anda/Developer/lighthouse/target/debug/deps/beacon_chain-9ba3fc78ed49044f` (signal: 6, SIGABRT: process abort signal) ~/Developer/lighthouse tikv-jemallocator *2 !2 ❯ 6m 25s 04:39:03 pm

I think that's a separate issue:

stack overflow when running overflow_cache_test_state_cache in debug #6120

Generally we avoid running the tests in debug mode, because the Rust compiler is very conservative and ends up wasting a lot of stack space. Still, it's something we'll work on.

If you want to run the tests locally, use make nextest-release.

michaelsproul · 2024-09-05T23:59:03Z

The clippy failure is due to the new stable compiler release. I'll fix that in another PR now.

michaelsproul · 2024-09-06T02:14:35Z

@mergify queue

mergify · 2024-09-06T02:14:38Z

queue

🛑 The pull request has been removed from the queue `default`

The merge conditions cannot be satisfied due to failing checks.

You can take a look at Queue: Embarked in merge queue check runs for more details.

In case of a failure due to a flaky test, you should first retrigger the CI.
Then, re-embark the pull request into the merge queue by posting the comment
@mergifyio refresh on the pull request.

michaelsproul · 2024-09-06T02:49:16Z

@mergify requeue

mergify · 2024-09-06T02:49:24Z

requeue

✅ This pull request will be re-embarked automatically

The followup queue command will be automatically executed to re-embark the pull request

mergify · 2024-09-06T02:49:24Z

queue

✅ The pull request has been merged automatically

The pull request has been merged automatically at 0c5e25b

Add retropgf funding (sigp#6324) Migrate from `ethereum-types` to `alloy-primitives` (sigp#6078) * Remove use of ethers_core::RlpStream * Merge branch 'unstable' of https://github.com/sigp/lighthouse into remove_use_of_ethers_core * Remove old code * Simplify keccak call * Remove unused package * Merge branch 'unstable' of https://github.com/ethDreamer/lighthouse into remove_use_of_ethers_core * Merge branch 'unstable' into remove_use_of_ethers_core * Run clippy * Merge branch 'remove_use_of_ethers_core' of https://github.com/dospore/lighthouse into remove_use_of_ethers_core * Check all cargo fmt * migrate to alloy primitives init * fix deps * integrate alloy-primitives * resolve dep issues * more changes based on dep changes * add TODOs * Merge branch 'unstable' of https://github.com/sigp/lighthouse into remove_use_of_ethers_core * Revert lock * Add BeaconBlocksByRange v3 * continue migration * Revert "Add BeaconBlocksByRange v3" This reverts commit e3ce7fc. * impl hash256 extended trait * revert some uneeded diffs * merge conflict resolved * fix subnet id rshift calc * rename to FixedBytesExtended * debugging * Merge branch 'unstable' of https://github.com/sigp/lighthouse into migrate-to-alloy-primitives * fix failed test * fixing more tests * Merge branch 'unstable' of https://github.com/sigp/lighthouse into remove_use_of_ethers_core * introduce a shim to convert between the two u256 types * move alloy to wrokspace * align alloy versions * update * update web3signer test certs * refactor * resolve failing tests * linting * fix graffiti string test * fmt * fix ef test * resolve merge conflicts * remove udep and revert cert * cargo patch * cyclic dep * fix build error * Merge branch 'unstable' of https://github.com/sigp/lighthouse into migrate-to-alloy-primitives * resolve conflicts, update deps * merge unstable * fmt * fix deps * Merge branch 'unstable' of https://github.com/sigp/lighthouse into migrate-to-alloy-primitives * resolve merge conflicts * resolve conflicts, make necessary changes * Remove patch * fmt * remove file * merge conflicts * sneaking in a smol change * bump versions * Merge remote-tracking branch 'origin/unstable' into migrate-to-alloy-primitives * Updates for peerDAS * Update ethereum_hashing to prevent dupe * updated alloy-consensus, removed TODOs * cargo update * endianess fix * Merge branch 'unstable' of https://github.com/sigp/lighthouse into migrate-to-alloy-primitives * fmt * fix merge * fix test * fixed_bytes crate * minor fixes * convert u256 to i64 * panic free mixin to_low_u64_le * from_str_radix * computbe_subnet api and ensuring we use big-endian * Merge branch 'unstable' of https://github.com/sigp/lighthouse into migrate-to-alloy-primitives * fix test * Simplify subnet_id test * Simplify some more tests * Add tests to fixed_bytes crate * Merge branch 'unstable' into migrate-to-alloy-primitives update libp2p to version 0.54 (sigp#6249) * update libp2p to version 0.54.0 * address review * Merge branch 'unstable' of github.com:sigp/lighthouse into update-libp2p * Merge branch 'update-libp2p' of github.com:sigp/lighthouse into update-libp2p Drop block data from BlockError and BlobError (sigp#5735) * Drop block data from BlockError and BlobError * Debug release tests * Fix release tests * Revert unnecessary changes to lighthouse_metrics Update manual checkpoint sync content in Lighthouse book (sigp#6338) * Update manual checkpiont sync * Update faq * Minor revision * Minor revision Light Client Bug Fix (sigp#6299) * Light Client Bug Fix Ignore Rust 1.82 warnings about void patterns (sigp#6357) * Ignore Rust 1.82 warnings about void patterns Enable `large_stack_frames` lint (sigp#6343) * Enable `large_stack_frames` lint Add blob count label to `DATA_COLUMN_SIDECAR_COMPUTATION` metric (sigp#6340) * Add blob count label to `DATA_COLUMN_SIDECAR_COMPUTATION` metric, and move metrics into the compute function, recording only successful computation. * Move `discard_timer_on_break` usage to caller site. * Merge branch 'unstable' into compute-data-column-metric * Merge branch 'unstable' into compute-data-column-metric Metadata request ordering (sigp#6336) * Send metadata request ordering * Merge branch 'unstable' into metadata-order Clarify validator monitor block log (sigp#6342) * Clarify validator monitor block log * Merge branch 'unstable' into clarify-block-log Check known parent on rpc blob process (sigp#5893) * Check known parent on rpc blob process * fix test * Merge branch 'unstable' of https://github.com/sigp/lighthouse into blob-unknown-parent Remove beta tag from gossipsub 1.2 (sigp#6344) * Remove the beta tag from gossipsub v1.2 * fix clippy * Merge branch 'unstable' into remove-beta-tag Fix lints for Rust 1.81 (sigp#6363) * Fix lints for Rust 1.81 Use tikv-jemallocator instead of jemallocator (sigp#6354) * Use tikv-jemallocator instead of jemallocator * Merge branch 'unstable' into tikv-jemallocator * Bump tikv-jemallocator and tikv-jemalloc-ctl Improve `get_custody_columns` validation, caching and error handling (sigp#6308) * Improve `get_custody_columns` validation, caching and error handling. * Merge branch 'unstable' into get-custody-columns-error-handing * Fix failing test and add more test. * Fix failing test and add more test. * Merge branch 'unstable' into get-custody-columns-error-handing * Add unit test to make sure the default specs won't panic on the `compute_custody_requirement_subnets` function. * Add condition when calling `compute_custody_subnets_from_metadata` and update logs. * Validate `csc` when returning from enr. Remove `csc` computation on connection since we get them on metadata anyway. * Add `peers_per_custody_subnet_count` to track peer csc and supernodes. * Disconnect peers with invalid metadata and find other peers instead. * Fix sampling tests. * Merge branch 'unstable' into get-custody-columns-error-handing * Merge branch 'unstable' into get-custody-columns-error-handing Delete legacy payload reconstruction (sigp#6213) * Delete legacy payload reconstruction * Delete unneeded failing test * Merge remote-tracking branch 'origin/unstable' into remove-more-ethers * Merge remote-tracking branch 'origin/unstable' into remove-more-ethers * Cleanups simplify rpc codec logic (sigp#6304) * simplify rpc codec logic * Merge branch 'unstable' of github.com:sigp/lighthouse into simplify-rpc-codec * Merge branch 'unstable' of github.com:sigp/lighthouse into simplify-rpc-codec * Merge branch 'unstable' of github.com:sigp/lighthouse into simply-rpc-codec * Merge branch 'unstable' into simplify-rpc-codec * Merge branch 'unstable' into simplify-rpc-codec

Use tikv-jemallocator instead of jemallocator

cc1415e

dapplion requested a review from michaelsproul September 4, 2024 13:01

Merge branch 'unstable' into tikv-jemallocator

57ca6ec

michaelsproul added ready-for-review The code is ready for review dependencies Pull requests that update a dependency file labels Sep 5, 2024

michaelsproul reviewed Sep 5, 2024

View reviewed changes

michaelsproul added the waiting-on-author The reviewer has suggested changes and awaits thier implementation. label Sep 5, 2024

Bump tikv-jemallocator and tikv-jemalloc-ctl

4ca3108

michaelsproul approved these changes Sep 6, 2024

View reviewed changes

michaelsproul added ready-for-merge This PR is ready to merge. and removed ready-for-review The code is ready for review waiting-on-author The reviewer has suggested changes and awaits thier implementation. labels Sep 6, 2024

mergify bot added a commit that referenced this pull request Sep 6, 2024

Merge of #6354

48607c8

mergify bot mentioned this pull request Sep 6, 2024

merge queue: embarking unstable (c824142) and #6354 together #6364

Closed

5 tasks

mergify bot added a commit that referenced this pull request Sep 6, 2024

Merge of #6354

39c9827

mergify bot mentioned this pull request Sep 6, 2024

merge queue: embarking unstable (c824142) and #6354 together #6365

Closed

5 tasks

mergify bot merged commit 0c5e25b into sigp:unstable Sep 6, 2024
27 of 28 checks passed

estensen deleted the tikv-jemallocator branch September 6, 2024 06:20

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use tikv-jemallocator instead of jemallocator #6354

Use tikv-jemallocator instead of jemallocator #6354

estensen commented Sep 4, 2024 •

edited

Loading

chong-he commented Sep 4, 2024

estensen commented Sep 4, 2024

michaelsproul Sep 5, 2024

estensen Sep 5, 2024

estensen Sep 5, 2024

michaelsproul Sep 5, 2024 •

edited

Loading

michaelsproul commented Sep 5, 2024

michaelsproul commented Sep 6, 2024

mergify bot commented Sep 6, 2024 •

edited

Loading

michaelsproul commented Sep 6, 2024

mergify bot commented Sep 6, 2024

mergify bot commented Sep 6, 2024 •

edited

Loading

Use tikv-jemallocator instead of jemallocator #6354

Use tikv-jemallocator instead of jemallocator #6354

Conversation

estensen commented Sep 4, 2024 • edited Loading

Proposed Changes

chong-he commented Sep 4, 2024

estensen commented Sep 4, 2024

michaelsproul Sep 5, 2024

Choose a reason for hiding this comment

estensen Sep 5, 2024

Choose a reason for hiding this comment

estensen Sep 5, 2024

Choose a reason for hiding this comment

michaelsproul Sep 5, 2024 • edited Loading

Choose a reason for hiding this comment

michaelsproul commented Sep 5, 2024

michaelsproul commented Sep 6, 2024

mergify bot commented Sep 6, 2024 • edited Loading

🛑 The pull request has been removed from the queue default

michaelsproul commented Sep 6, 2024

mergify bot commented Sep 6, 2024

✅ This pull request will be re-embarked automatically

mergify bot commented Sep 6, 2024 • edited Loading

✅ The pull request has been merged automatically

estensen commented Sep 4, 2024 •

edited

Loading

michaelsproul Sep 5, 2024 •

edited

Loading

mergify bot commented Sep 6, 2024 •

edited

Loading

🛑 The pull request has been removed from the queue `default`

mergify bot commented Sep 6, 2024 •

edited

Loading