Seed Driven Cluster Bootstrap #2
Draft
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
andrwng
force-pushed
the
node_id_assignment
branch
2 times, most recently
from
82a3849
to
960e599
Compare
dlex
force-pushed
the
333_symseed-cluster-formation
branch
2 times, most recently
from
c6d5a01
to
7ee41a0
Compare
dlex
changed the title
|
andrwng
reviewed
Oct 6, 2022
| // The controller log has already been written to | ||
| ss::future<> | ||
| controller::cluster_creation_hook(const bool local_node_is_seed_server) { | ||
| if (config::node().empty_seed_starts_cluster()) { |
There was a problem hiding this comment.
I'm surprised that this has the behavior of not writing a cluster UUID. IMO supporting "legacy" doesn't mean maintaining the exact previous behavior -- we just need to be able to allow for configurations with an empty seed server.
I think ideally, the only thing empty_seed_starts_cluster changes is the set of seed servers returned from initial_seed_brokers()
There was a problem hiding this comment.
Hmmm right we've been discussing that last week, and this is what is left from my original perspective. I think you're right about it but I just will think twice and will do the command driven bootstrap in both cases. In particular that would mean the original way of creating the initial user will go away, and even with empty_seed_starts_cluster, there will be bootstrap_cluster_cmd instead of create_user_cmd. That should work, though.
This will be used when we first start a node that has no node UUID. I exposed this as storage because the UUID uniquely identifies the contents of the kv-store / data directory.
This encapsulates some pieces of cluster discovery into its own class. I added some descriptions on how each method is expected to be implemented, but I added implementations that match the existing behavior.
This adds the skeleton of an RPC service that will be used for cluster bootstrap. It includes an initial `cluster_bootstrap_info` RPC type that is currently empty. Introducing as a standalone commit to parallelize some workstreams.
This plumbs the cluster discovery module into `redpanda::application` as one of the first subsystems initialized during startup. Cluster discovery will have an initial dependency on the storage subsystem, to get node UUID, cluster UUID, etc., and on the RPC subsystem, to serve the RPCs required for the initial bootstrapping sequence. The rest of the subsystems will have a dependency on cluster discovery, which, once automatic node ID assignment is supported, will be what determines the node ID. To that end, this commit orders the startup of the storage and RPC subsystems up front, followed by cluster discovery, followed by the rest of the subsystems. This commit only plumbs cluster discovery infrastructure. The cluster discovery sequence as a whole is unchanged.
It can be convenient to construct a named_type using constructors of the underlying type. A simple solution today is to construct the underlying type separately and then construct the named_type with it, but this spills the underlying type into call sites. Another approach would be to construct with the named_type::type constructor, but this can be verbose. This commit adds a templatized static constructor that callers can pass constructor args into.
Upon first starting up the storage layer, Redpanda will now check to see if a node UUID exists in the controller keyspace of the key-value store. If not, it will generate one and persist it to disk, and make it available on all shards. This will be used as a unique identifier when registering a node (particularly important in the future when we auto-assign node IDs).
This adds a new controller command to register a node UUID. It is activated by the existing join RPC endpoint and has the following semantics: - Register: all new versions of Redpanda will send their UUID along with an optional node ID in their initial join request. If the UUID is not in the leader controller's `members_table`, the UUID is registered by replicating a `register_node_uuid_cmd`. - Join: all new version of Redpanda will, after registering, send another join request that will actually add the node to the Raft group. Old versions of Redpanda will send a join request with no UUID.
This commit makes the node_id config property optional. It doesn't make any checks that the field is set. A follow-up will change the Redpanda's behavior when seeing an empty node ID.
This commit adds the ability for Redpanda to assign node ID based on the node UUID when a node ID isn't set.
When the controller starts up, we check to ensure the configured node ID is identical to that in the kvstore if one exists, persisting one if not. We should use this node ID at startup if weren't configured with one, e.g. if we're upgrading an existing cluster and begin omitting node ID from the node config.
RPC `client_probe` metrics currently leverage a labeling scheme defined by Seastar in which, for a given metric, a set of metric labels can only be registered once per metric. Currently, the label used is solely based on the server address associated with a given `rpc::transport`. As such, we currently cannot start multiple `rpc::transport`s pointed at the same server. This functionality could be useful though: consider when a node is restarted empty with a new node ID. Redpanda currently has a check that nodes being added to the controller Raft group don't overlap with existing group members' addresses. But if we were to remove this check, when the new node _is_ added to the Raft group, each node will try to create a new `rpc::transport` pointing at the new node, and register metrics with identical labels to those registered by the old node, and be met with a Seastar `double_registration` exception. To enable the above scenario, this commit adds the `node_id` as a label for client metrics, and aggregates them by this label.
andrwng
force-pushed
the
node_id_assignment
branch
from
960e599
to
5972afd
Compare
Also a corresponding parameter is added to redpanda test fixture
cluster_discovery works with node config directly assert message in replicate_and_wait adjusted comments
initial_seed_brokers() now support both ESCB and SDCB modes determine_node_id() now supports index-based node_ids
plus some factoring
cluster_uuid type New bootstrap controller log command carrying cluster_uuid New backend to handle cluster bootstrap, w/separate batch type Persistence of cluster_uuid in kvstore
Supply clusterDiscovery's initial_seed_brokers list to raft0 as the set of nodes to start the raft0 group with
If there is no cluster_uuid in SDCB mode, cluster is initialized by posting the new bootstrap command to the controller log.
cluster_test_fixure now supports the empty_seed_starts_cluster parameter
bootstrap_backend operates on credentail_storage for that. In security_frontend, maybe_create_bootstrap_user() is split into two: get_bootstrap_user_creds_from_env(), and the actual user creation.
Client: parallel querying of all seed_servers w/o timeout, until results are obtained from all peer seed servers. Verify that both versions and configurations match Server: supply data initial_seed_brokers() now returns a future
dlex
force-pushed
the
333_symseed-cluster-formation
branch
from
7ee41a0
to
87604ad
Compare
andrwng
force-pushed
the
node_id_assignment
branch
from
5972afd
to
e80e9eb
Compare
andrwng
reviewed
Oct 7, 2022
There was a problem hiding this comment.
Some thoughts so far.
Also sorry but I rebased the node ID assignment branch again -- rebased this branch here https://github.com/andrwng/redpanda/tree/cluster_bootstrap
| @@ -114,9 +118,12 @@ class cluster_test_fixture { | |||
| int proxy_port_base = 8082, | |||
| int schema_reg_port_base = 8081, | |||
| int coproc_supervisor_port_base = 43189, | |||
| configure_node_id use_node_id = configure_node_id::yes) { | |||
| configure_node_id use_node_id = configure_node_id::yes, | |||
| empty_seed_starts_cluster empty_seed_starts_cluster_val | |||
There was a problem hiding this comment.
nit: we should probably distinguish between setting the empty_seed_starts_cluster config and configuring with empty seeds.
andrwng
pushed a commit
that referenced
this pull request
Feb 3, 2023
Without this change there is a data race that can result in a
heap-use-after-free.
```
==3190349==ERROR: AddressSanitizer: heap-use-after-free on address 0x60b000257420 at pc 0x14995faf26e4 bp 0x7ffd3e04b710 sp 0x7ffd3e04b708
READ of size 8 at 0x60b000257420 thread T0
#0 0x14995faf26e3 in profile_flush_file_data /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/krb5-prefix/src/krb5/src/util/profile/prof_file.c:468:24
#1 0x14995faf0cef in profile_close_file /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/krb5-prefix/src/krb5/src/util/profile/prof_file.c:559:14
#2 0x14995fb0c086 in profile_release /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/krb5-prefix/src/krb5/src/util/profile/prof_init.c:514:13
#3 0x14995fa924c6 in k5_os_free_context /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/krb5-prefix/src/krb5/src/lib/krb5/os/init_os_ctx.c:506:9
redpanda-data#4 0x14995f938b87 in krb5_free_context /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/krb5-prefix/src/krb5/src/lib/krb5/krb/init_ctx.c:294:5
redpanda-data#5 0x14995ffc41c3 in krb5_gss_delete_sec_context /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/krb5-prefix/src/krb5/src/lib/gssapi/krb5/delete_sec_context.c:87:9
redpanda-data#6 0x14995ff399a6 in gssint_delete_internal_sec_context /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/krb5-prefix/src/krb5/src/lib/gssapi/mechglue/g_glue.c:606:15
redpanda-data#7 0x14995ff25450 in gss_delete_sec_context /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/krb5-prefix/src/krb5/src/lib/gssapi/mechglue/g_delete_sec_context.c:91:11
redpanda-data#8 0x55e258b9c641 in security::gss::ctx_id::reset() /home/ben/development/src/github.com/BenPope/redpanda-clang-13/src/v/security/gssapi.h:170:13
redpanda-data#9 0x55e258b9c408 in security::gss::ctx_id::~ctx_id() /home/ben/development/src/github.com/BenPope/redpanda-clang-13/src/v/security/gssapi.h:165:17
redpanda-data#10 0x55e258b9c37c in security::gssapi_authenticator::impl::~impl() /home/ben/development/src/github.com/BenPope/redpanda-clang-13/src/v/security/gssapi_authenticator.cc:116:29
redpanda-data#11 0x55e258b9c23c in std::__1::default_delete<security::gssapi_authenticator::impl>::operator()(security::gssapi_authenticator::impl*) const /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/llvm/install/bin/../include/c++/v1/__memory/unique_ptr.h:54:5
redpanda-data#12 0x55e258b65324 in std::__1::unique_ptr<security::gssapi_authenticator::impl, std::__1::default_delete<security::gssapi_authenticator::impl> >::reset(security::gssapi_authenticator::impl*) /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/llvm/install/bin/../include/c++/v1/__memory/unique_ptr.h:315:7
redpanda-data#13 0x55e258b5f302 in security::gssapi_authenticator::authenticate(seastar::basic_sstring<unsigned char, unsigned int, 31u, false>) (.resume) /home/ben/development/src/github.com/BenPope/redpanda-clang-13/src/v/security/gssapi_authenticator.cc:202:15
redpanda-data#14 0x55e258b11830 in std::__1::coroutine_handle<seastar::internal::coroutine_traits_base<boost::outcome_v2::basic_result<seastar::basic_sstring<unsigned char, unsigned int, 31u, false>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::basic_sstring<unsigned char, unsigned int, 31u, false>, std::__1::error_code, void> > >::promise_type>::resume() const /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/llvm/install/bin/../include/c++/v1/__coroutine/coroutine_handle.h:168:9
redpanda-data#15 0x55e258b11315 in seastar::internal::coroutine_traits_base<boost::outcome_v2::basic_result<seastar::basic_sstring<unsigned char, unsigned int, 31u, false>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::basic_sstring<unsigned char, unsigned int, 31u, false>, std::__1::error_code, void> > >::promise_type::run_and_dispose() /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/rp_deps_install/include/seastar/core/coroutine.hh:78:20
redpanda-data#16 0x55e2594a1ead in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2330:14
redpanda-data#17 0x55e2594a7d0e in seastar::reactor::run_some_tasks() /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2737:9
redpanda-data#18 0x55e2594ac86f in seastar::reactor::do_run() /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2906:9
redpanda-data#19 0x55e2594aa3f8 in seastar::reactor::run() /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2789:16
redpanda-data#20 0x55e2591c0a7c in seastar::app_template::run_deprecated(int, char**, std::__1::function<void ()>&&) /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/app-template.cc:265:31
redpanda-data#21 0x55e2591be135 in seastar::app_template::run(int, char**, std::__1::function<seastar::future<int> ()>&&) /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/app-template.cc:156:12
redpanda-data#22 0x55e2470c46e4 in application::run(int, char**) /home/ben/development/src/github.com/BenPope/redpanda-clang-13/src/v/redpanda/application.cc:323:16
redpanda-data#23 0x55e247081d16 in main /home/ben/development/src/github.com/BenPope/redpanda-clang-13/src/v/redpanda/main.cc:22:16
redpanda-data#24 0x14995cc29d8f in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16
redpanda-data#25 0x14995cc29e3f in __libc_start_main csu/../csu/libc-start.c:392:3
redpanda-data#26 0x55e246fc1844 in _start (/home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/bin/redpanda+0x3a678844) (BuildId: 0a81be2927716d279ef7fc96d20cc5d5dfbd1cb2)
0x60b000257420 is located 0 bytes inside of 104-byte region [0x60b000257420,0x60b000257488)
freed by thread T0 here:
#0 0x55e2470443e2 in free /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/llvm/src/compiler-rt/lib/asan/asan_malloc_linux.cpp:52:3
#1 0x14995faf3ed4 in profile_free_file_data /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/krb5-prefix/src/krb5/src/util/profile/prof_file.c:552:5
#2 0x14995faf3860 in profile_dereference_data_locked /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/krb5-prefix/src/krb5/src/util/profile/prof_file.c:509:9
#3 0x14995faf0b74 in profile_dereference_data /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/krb5-prefix/src/krb5/src/util/profile/prof_file.c:502:5
redpanda-data#4 0x14995faf3fc1 in profile_free_file /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/krb5-prefix/src/krb5/src/util/profile/prof_file.c:523:5
redpanda-data#5 0x14995faf0d38 in profile_close_file /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/krb5-prefix/src/krb5/src/util/profile/prof_file.c:562:5
redpanda-data#6 0x14995fb0c086 in profile_release /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/krb5-prefix/src/krb5/src/util/profile/prof_init.c:514:13
redpanda-data#7 0x14995fa924c6 in k5_os_free_context /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/krb5-prefix/src/krb5/src/lib/krb5/os/init_os_ctx.c:506:9
redpanda-data#8 0x14995f938b87 in krb5_free_context /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/krb5-prefix/src/krb5/src/lib/krb5/krb/init_ctx.c:294:5
redpanda-data#9 0x14995ffc41c3 in krb5_gss_delete_sec_context /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/krb5-prefix/src/krb5/src/lib/gssapi/krb5/delete_sec_context.c:87:9
redpanda-data#10 0x14995ff399a6 in gssint_delete_internal_sec_context /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/krb5-prefix/src/krb5/src/lib/gssapi/mechglue/g_glue.c:606:15
redpanda-data#11 0x14995ff25450 in gss_delete_sec_context /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/krb5-prefix/src/krb5/src/lib/gssapi/mechglue/g_delete_sec_context.c:91:11
redpanda-data#12 0x55e258b9c641 in security::gss::ctx_id::reset() /home/ben/development/src/github.com/BenPope/redpanda-clang-13/src/v/security/gssapi.h:170:13
redpanda-data#13 0x55e258b9c408 in security::gss::ctx_id::~ctx_id() /home/ben/development/src/github.com/BenPope/redpanda-clang-13/src/v/security/gssapi.h:165:17
redpanda-data#14 0x55e258b9c37c in security::gssapi_authenticator::impl::~impl() /home/ben/development/src/github.com/BenPope/redpanda-clang-13/src/v/security/gssapi_authenticator.cc:116:29
redpanda-data#15 0x55e258b9c23c in std::__1::default_delete<security::gssapi_authenticator::impl>::operator()(security::gssapi_authenticator::impl*) const /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/llvm/install/bin/../include/c++/v1/__memory/unique_ptr.h:54:5
redpanda-data#16 0x55e258b65324 in std::__1::unique_ptr<security::gssapi_authenticator::impl, std::__1::default_delete<security::gssapi_authenticator::impl> >::reset(security::gssapi_authenticator::impl*) /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/llvm/install/bin/../include/c++/v1/__memory/unique_ptr.h:315:7
redpanda-data#17 0x55e258b5f302 in security::gssapi_authenticator::authenticate(seastar::basic_sstring<unsigned char, unsigned int, 31u, false>) (.resume) /home/ben/development/src/github.com/BenPope/redpanda-clang-13/src/v/security/gssapi_authenticator.cc:202:15
redpanda-data#18 0x55e258b11830 in std::__1::coroutine_handle<seastar::internal::coroutine_traits_base<boost::outcome_v2::basic_result<seastar::basic_sstring<unsigned char, unsigned int, 31u, false>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::basic_sstring<unsigned char, unsigned int, 31u, false>, std::__1::error_code, void> > >::promise_type>::resume() const /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/llvm/install/bin/../include/c++/v1/__coroutine/coroutine_handle.h:168:9
redpanda-data#19 0x55e258b11315 in seastar::internal::coroutine_traits_base<boost::outcome_v2::basic_result<seastar::basic_sstring<unsigned char, unsigned int, 31u, false>, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<seastar::basic_sstring<unsigned char, unsigned int, 31u, false>, std::__1::error_code, void> > >::promise_type::run_and_dispose() /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/rp_deps_install/include/seastar/core/coroutine.hh:78:20
redpanda-data#20 0x55e2594a1ead in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2330:14
redpanda-data#21 0x55e2594a7d0e in seastar::reactor::run_some_tasks() /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2737:9
redpanda-data#22 0x55e2594ac86f in seastar::reactor::do_run() /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2906:9
redpanda-data#23 0x55e2594aa3f8 in seastar::reactor::run() /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2789:16
redpanda-data#24 0x55e2591c0a7c in seastar::app_template::run_deprecated(int, char**, std::__1::function<void ()>&&) /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/app-template.cc:265:31
redpanda-data#25 0x55e2591be135 in seastar::app_template::run(int, char**, std::__1::function<seastar::future<int> ()>&&) /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/app-template.cc:156:12
redpanda-data#26 0x55e2470c46e4 in application::run(int, char**) /home/ben/development/src/github.com/BenPope/redpanda-clang-13/src/v/redpanda/application.cc:323:16
redpanda-data#27 0x55e247081d16 in main /home/ben/development/src/github.com/BenPope/redpanda-clang-13/src/v/redpanda/main.cc:22:16
redpanda-data#28 0x14995cc29d8f in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16
previously allocated by thread T4 here:
#0 0x55e24704468e in malloc /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/llvm/src/compiler-rt/lib/asan/asan_malloc_linux.cpp:69:3
#1 0x14995faee48d in profile_make_prf_data /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/krb5-prefix/src/krb5/src/util/profile/prof_file.c:150:9
#2 0x14995faf0360 in profile_open_file /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/krb5-prefix/src/krb5/src/util/profile/prof_file.c:233:12
#3 0x14995fb0ab1e in profile_init_flags /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/krb5-prefix/src/krb5/src/util/profile/prof_init.c:190:22
redpanda-data#4 0x14995fa90bc0 in os_init_paths /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/krb5-prefix/src/krb5/src/lib/krb5/os/init_os_ctx.c:387:18
redpanda-data#5 0x14995fa9090e in k5_os_init_context /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/krb5-prefix/src/krb5/src/lib/krb5/os/init_os_ctx.c:438:18
redpanda-data#6 0x14995f936ece in krb5_init_context_profile /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/krb5-prefix/src/krb5/src/lib/krb5/krb/init_ctx.c:209:14
redpanda-data#7 0x14995f936871 in krb5_init_context /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/krb5-prefix/src/krb5/src/lib/krb5/krb/init_ctx.c:139:12
redpanda-data#8 0x14995fffc70b in krb5_gss_init_context /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/krb5-prefix/src/krb5/src/lib/gssapi/krb5/init_sec_context.c:1047:12
redpanda-data#9 0x14995ffa0719 in kg_accept_krb5 /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/krb5-prefix/src/krb5/src/lib/gssapi/krb5/accept_sec_context.c:694:12
redpanda-data#10 0x14995ff9e686 in krb5_gss_accept_sec_context_ext /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/krb5-prefix/src/krb5/src/lib/gssapi/krb5/accept_sec_context.c:1311:12
redpanda-data#11 0x14995ffaaf9c in krb5_gss_accept_sec_context /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/krb5-prefix/src/krb5/src/lib/gssapi/krb5/accept_sec_context.c:1340:12
redpanda-data#12 0x14995ff05dbd in gss_accept_sec_context /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/krb5-prefix/src/krb5/src/lib/gssapi/mechglue/g_accept_sec_context.c:266:15
redpanda-data#13 0x55e258b2b157 in security::gssapi_authenticator::impl::more(std::__1::basic_string_view<unsigned char, std::__1::char_traits<unsigned char> >) /home/ben/development/src/github.com/BenPope/redpanda-clang-13/src/v/security/gssapi_authenticator.cc:292:25
redpanda-data#14 0x55e258b2946c in security::gssapi_authenticator::impl::authenticate(seastar::basic_sstring<unsigned char, unsigned int, 31u, false>) /home/ben/development/src/github.com/BenPope/redpanda-clang-13/src/v/security/gssapi_authenticator.cc:218:16
redpanda-data#15 0x55e258b3b562 in security::gssapi_authenticator::authenticate(seastar::basic_sstring<unsigned char, unsigned int, 31u, false>)::$_0::operator()() /home/ben/development/src/github.com/BenPope/redpanda-clang-13/src/v/security/gssapi_authenticator.cc:195:25
redpanda-data#16 0x55e258b3a96a in ssx::impl::worker_task<security::gssapi_authenticator::authenticate(seastar::basic_sstring<unsigned char, unsigned int, 31u, false>)::$_0>::process(seastar::alien::instance&, unsigned int) /home/ben/development/src/github.com/BenPope/redpanda-clang-13/src/v/ssx/thread_worker.h:67:37
redpanda-data#17 0x55e24764d1bc in ssx::impl::thread_worker::run() /home/ben/development/src/github.com/BenPope/redpanda-clang-13/src/v/ssx/thread_worker.h:159:20
redpanda-data#18 0x55e24764ca9f in ssx::impl::thread_worker::start()::'lambda'()::operator()() const /home/ben/development/src/github.com/BenPope/redpanda-clang-13/src/v/ssx/thread_worker.h:104:49
redpanda-data#19 0x55e24764c910 in decltype(static_cast<ssx::impl::thread_worker::start()::'lambda'()>(fp)()) std::__1::__invoke<ssx::impl::thread_worker::start()::'lambda'()>(ssx::impl::thread_worker::start()::'lambda'()&&) /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/llvm/install/bin/../include/c++/v1/type_traits:3640:23
redpanda-data#20 0x55e24764c808 in void std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, ssx::impl::thread_worker::start()::'lambda'()>(std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, ssx::impl::thread_worker::start()::'lambda'()>&, std::__1::__tuple_indices<>) /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/llvm/install/bin/../include/c++/v1/thread:282:5
redpanda-data#21 0x55e24764ad39 in void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, ssx::impl::thread_worker::start()::'lambda'()> >(void*) /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/llvm/install/bin/../include/c++/v1/thread:293:5
redpanda-data#22 0x14995cc94b42 in start_thread nptl/./nptl/pthread_create.c:442:8
Thread T4 created by T0 here:
#0 0x55e24702db0c in pthread_create /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/llvm/src/compiler-rt/lib/asan/asan_interceptors.cpp:208:3
#1 0x55e24764ab0c in std::__1::__libcpp_thread_create(unsigned long*, void* (*)(void*), void*) /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/llvm/install/bin/../include/c++/v1/__threading_support:375:10
#2 0x55e24764a106 in std::__1::thread::thread<ssx::impl::thread_worker::start()::'lambda'(), void>(ssx::impl::thread_worker::start()::'lambda'()&&) /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/llvm/install/bin/../include/c++/v1/thread:309:16
#3 0x55e247606d40 in ssx::impl::thread_worker::start() /home/ben/development/src/github.com/BenPope/redpanda-clang-13/src/v/ssx/thread_worker.h:104:19
redpanda-data#4 0x55e247305329 in ssx::thread_worker::start() (.resume) /home/ben/development/src/github.com/BenPope/redpanda-clang-13/src/v/ssx/thread_worker.h:199:15
redpanda-data#5 0x55e24756b680 in std::__1::coroutine_handle<seastar::internal::coroutine_traits_base<void>::promise_type>::resume() const /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/llvm/install/bin/../include/c++/v1/__coroutine/coroutine_handle.h:168:9
redpanda-data#6 0x55e24756b1a5 in seastar::internal::coroutine_traits_base<void>::promise_type::run_and_dispose() /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/rp_deps_install/include/seastar/core/coroutine.hh:120:20
redpanda-data#7 0x55e2594a1ead in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2330:14
redpanda-data#8 0x55e2594a7d0e in seastar::reactor::run_some_tasks() /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2737:9
redpanda-data#9 0x55e2594ac86f in seastar::reactor::do_run() /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2906:9
redpanda-data#10 0x55e2594aa3f8 in seastar::reactor::run() /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2789:16
redpanda-data#11 0x55e2591c0a7c in seastar::app_template::run_deprecated(int, char**, std::__1::function<void ()>&&) /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/app-template.cc:265:31
redpanda-data#12 0x55e2591be135 in seastar::app_template::run(int, char**, std::__1::function<seastar::future<int> ()>&&) /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/app-template.cc:156:12
redpanda-data#13 0x55e2470c46e4 in application::run(int, char**) /home/ben/development/src/github.com/BenPope/redpanda-clang-13/src/v/redpanda/application.cc:323:16
redpanda-data#14 0x55e247081d16 in main /home/ben/development/src/github.com/BenPope/redpanda-clang-13/src/v/redpanda/main.cc:22:16
redpanda-data#15 0x14995cc29d8f in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16
SUMMARY: AddressSanitizer: heap-use-after-free /home/ben/development/src/github.com/BenPope/redpanda-clang-13/vbuild/debug/clang/v_deps_build/krb5-prefix/src/krb5/src/util/profile/prof_file.c:468:24 in profile_flush_file_data
Shadow bytes around the buggy address:
0x0c1680042e30: fd fd fd fd fd fd fd fd fd fd fa fa fa fa fa fa
0x0c1680042e40: fa fa fd fd fd fd fd fd fd fd fd fd fd fd fd fa
0x0c1680042e50: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
0x0c1680042e60: fd fd fd fd fd fd fa fa fa fa fa fa fa fa fd fd
0x0c1680042e70: fd fd fd fd fd fd fd fd fd fd fd fd fa fa fa fa
=>0x0c1680042e80: fa fa fa fa[fd]fd fd fd fd fd fd fd fd fd fd fd
0x0c1680042e90: fd fa fa fa fa fa fa fa fa fa fd fd fd fd fd fd
0x0c1680042ea0: fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa
0x0c1680042eb0: fd fd fd fd fd fd fd fd fd fd fd fd fd fa fa fa
0x0c1680042ec0: fa fa fa fa fa fa fd fd fd fd fd fd fd fd fd fd
0x0c1680042ed0: fd fd fd fd fa fa fa fa fa fa fa fa fd fd fd fd
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
==3190349==ABORTING
```
Signed-off-by: Ben Pope <[email protected]>
andrwng
pushed a commit
that referenced
this pull request
Apr 12, 2023
The tee_wrapper helper was being used to set the ostream pointer for global logger to be an ostream instance that lived only on the stack, and not resetting it to something global like std::cerr after the test finished. This resulted in the following segfault in other tests that ran after the change but didn't make their own change to ostream pointer: ``` #0 0x7f875b0bbc1f in std::__1::basic_ostream<char, std::__1::char_traits<char> >::sentry::sentry(std::__1::basic_ostream<char, std::__1::char_traits<char> >&) (/vectorized/lib/libc++.so.1+0x65c1f) #1 0x5597c3979afc in std::__1::basic_ostream<char, std::__1::char_traits<char> >& std::__1::__put_character_sequence<char, std::__1::char_traits<char> >(std::__1::basic_ostream<char, std::__1::char_traits<char> >&, char const*, unsigned long) /vectorized/llvm/bin/../include/c++/v1/ostream:722:57 #2 0x5597c4079e44 in std::__1::basic_ostream<char, std::__1::char_traits<char> >& std::__1::operator<<<char, std::__1::char_traits<char> >(std::__1::basic_ostream<char, std::__1::char_traits<char> >&, std::__1::basic_string_view<char, std::__1::char_traits<char> >) /vectorized/llvm/bin/../include/c++/v1/ostream:1057:12 #3 0x5597c602c9f3 in seastar::logger::do_log(seastar::log_level, seastar::logger::log_writer&) /v/build/v_deps_build/seastar-prefix/src/seastar/src/util/log.cc:342:15 redpanda-data#4 0x5597c3c4ad20 in void seastar::logger::log<char const*, int, seastar::basic_sstring<char, unsigned int, 15u, true>&>(seastar::log_level, seastar::logger::format_info, char const*&&, int&&, seastar::basic_sstring<char, unsigned int, 15u, true>&) /vectorized/include/seastar/util/log.hh:231:17 redpanda-data#5 0x5597c3f23c51 in void seastar::logger::trace<char const*, int, seastar::basic_sstring<char, unsigned int, 15u, true>&>(seastar::logger::format_info, char const*&&, int&&, seastar::basic_sstring<char, unsigned int, 15u, true>&) /vectorized/include/seastar/util/log.hh:385:9 redpanda-data#6 0x5597c3f14b95 in cloud_roles::signature_v4::sign_header(boost::beast::http::header<true, boost::beast::http::basic_fields<std::__1::allocator<char> > >&, std::__1::basic_string_view<char, std::__1::char_traits<char> >) const /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-00e717c5e253e2891-1/redpanda/vtools/src/v/cloud_roles/signature.cc:382:5 redpanda-data#7 0x5597c3bf1dfb in cloud_roles::apply_aws_credentials::add_auth(boost::beast::http::header<true, boost::beast::http::basic_fields<std::__1::allocator<char> > >&) const /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-00e717c5e253e2891-1/redpanda/vtools/src/v/cloud_roles/apply_aws_credentials.cc:83:23 redpanda-data#8 0x5597c3b8f88d in cloud_roles::apply_credentials::add_auth(boost::beast::http::header<true, boost::beast::http::basic_fields<std::__1::allocator<char> > >&) const /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-00e717c5e253e2891-1/redpanda/vtools/src/v/cloud_roles/apply_credentials.h:47:23 redpanda-data#9 0x5597c3b8c578 in test_aws_headers::test_method() /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-00e717c5e253e2891-1/redpanda/vtools/src/v/cloud_roles/tests/credential_tests.cc:50:17 ``` This change resets the ostream pointer in the wrapper destructor. Signed-off-by: Noah Watkins <[email protected]>
andrwng
pushed a commit
that referenced
this pull request
Sep 27, 2024
=================================================================
==524==ERROR: AddressSanitizer: container-overflow on address 0x52100028a100 at pc 0x7f29c0bf51fd bp 0x7ffda75397d0 sp 0x7ffda75397c8
READ of size 4 at 0x52100028a100 thread T0
#0 0x7f29c0bf51fc in util::mem_tracker::pretty_print_ascii() const /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-024f8676eb7b139e9-1/redpanda/vtools/src/v/utils/tracking_allocator.cc:75:31
#1 0x55be1ac7eb7b in mem_tracker_pretty_printing::test_method() /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-024f8676eb7b139e9-1/redpanda/vtools/src/v/utils/tests/tracking_allocator_tests.cc:82:5
#2 0x55be1ac7d651 in mem_tracker_pretty_printing_invoker() /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-024f8676eb7b139e9-1/redpanda/vtools/src/v/utils/tests/tracking_allocator_tests.cc:50:1
Signed-off-by: Noah Watkins <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.