riak kv 3.2.0 OTP25 turns off when stanchion starts

[Konstantin74R]

When i first start riak, then riak-cs and stanchion, riak kv self turns off. (riak ping -- Node [email protected] is not responding to pings).
Version riak 3.2.0 OTP25
riak-cs-3.0.1.OTP22
stanchion-3.0.0.OTP22

How can I fixed? After reboot OS it normal works, but before restarting riak applications.

In logs
2023-07-25T12:01:17.107424+00:00 [warning] <0.786.0>@riak_kv_vnode:log_key_amnesia/7:4493 Inbound clock entry for <<35,9,254,249,50,191,134,141>> in <<"moss.users">>/<<"SE4FRXVTWHVXCBEFAQW2">> greater than local.Epochs: {In:210001 Local:0}. Counters: {In:1 Local:0}. 2023-07-25T12:01:17.109281+00:00 [warning] <0.786.0>@riak_kv_vnode:log_key_amnesia/7:4493 Inbound clock entry for <<35,9,254,249,50,191,134,141>> in <<"moss.users">>/<<"SE4FRXVTWHVXCBEFAQW2">> greater than local.Epochs: {In:210001 Local:0}. Counters: {In:1 Local:0}. 2023-07-25T12:01:17.110775+00:00 [warning] <0.786.0>@riak_kv_vnode:log_key_amnesia/7:4493 Inbound clock entry for <<35,9,254,249,50,191,134,141>> in <<"moss.users">>/<<"SE4FRXVTWHVXCBEFAQW2">> greater than local.Epochs: {In:210001 Local:0}. Counters: {In:1 Local:0}. 2023-07-25T12:01:17.112909+00:00 [info] <0.671.0>@riak_repl_app:stop/1:139 Stopped application riak_repl 2023-07-25T12:01:17.114183+00:00 [info] <0.404.0>@riak_kv_app:prep_stop/1:273 Stopping application riak_kv - marked service down. 2023-07-25T12:01:17.114289+00:00 [info] <0.404.0>@riak_kv_app:prep_stop/1:277 Unregistered pb services 2023-07-25T12:01:17.115158+00:00 [info] <0.404.0>@riak_kv_app:prep_stop/1:282 unregistered webmachine routes 2023-07-25T12:01:17.115246+00:00 [info] <0.404.0>@riak_kv_app:prep_stop/1:284 all active put FSMs completed 2023-07-25T12:01:17.115463+00:00 [info] <0.404.0>@riak_kv_app:stop/1:295 Stopped application riak_kv. 2023-07-25T12:01:17.143085+00:00 [info] <0.840.0>@riak_kv_index_hashtree:close_trees/2:1169 Deliberately marking KV hashtree {205523667749658222872393179600727299639115513856,3} for full rebuild on next restart 2023-07-25T12:01:17.143167+00:00 [info] <0.844.0>@riak_kv_index_hashtree:close_trees/2:1169 Deliberately marking KV hashtree {182687704666362864775460604089535377456991567872,3} for full rebuild on next restart 2023-07-25T12:01:17.143220+00:00 [info] <0.1463.0>@riak_kv_index_hashtree:close_trees/2:1169 Deliberately marking KV hashtree {1050454301831586472458898473514828420377701515264,3} for full rebuild on next restart 2023-07-25T12:01:17.143265+00:00 [info] <0.841.0>@riak_kv_index_hashtree:close_trees/2:1169 Deliberately marking KV hashtree {228359630832953580969325755111919221821239459840,3} for full rebuild on next restart 2023-07-25T12:01:17.143322+00:00 [info] <0.1465.0>@riak_kv_index_hashtree:close_trees/2:1169 Deliberately marking KV hashtree {1073290264914881830555831049026020342559825461248,3} for full rebuild on next restart 2023-07-25T12:01:17.145574+00:00 [info] <0.1451.0>@riak_kv_index_hashtree:close_trees/2:1169 Deliberately marking KV hashtree {1027618338748291114361965898003636498195577569280,3} for full rebuild on next restart 2023-07-25T12:01:17.151856+00:00 [info] <0.840.0>@riak_kv_index_hashtree:close_trees/2:1169 Deliberately marking KV hashtree {228359630832953580969325755111919221821239459840,3} for full rebuild on next restart 2023-07-25T12:01:17.151913+00:00 [info] <0.844.0>@riak_kv_index_hashtree:close_trees/2:1169 Deliberately marking KV hashtree {205523667749658222872393179600727299639115513856,3} for full rebuild on next restart 2023-07-25T12:01:17.151959+00:00 [info] <0.1463.0>@riak_kv_index_hashtree:close_trees/2:1169 Deliberately marking KV hashtree {1073290264914881830555831049026020342559825461248,3} for full rebuild on next restart 2023-07-25T12:01:17.152623+00:00 [info] <0.841.0>@riak_kv_index_hashtree:close_trees/2:1169 Deliberately marking KV hashtree {251195593916248939066258330623111144003363405824,3} for full rebuild on next restart 2023-07-25T12:01:17.152679+00:00 [info] <0.1465.0>@riak_kv_index_hashtree:close_trees/2:1169 Deliberately marking KV hashtree {1096126227998177188652763624537212264741949407232,3} for full rebuild on next restart 2023-07-25T12:01:17.167194+00:00 [info] <0.1451.0>@riak_kv_index_hashtree:close_trees/2:1169 Deliberately marking KV hashtree {1050454301831586472458898473514828420377701515264,3} for full rebuild on next restart 2023-07-25T12:01:17.171056+00:00 [info] <0.840.0>@riak_kv_index_hashtree:close_trees/2:1169 Deliberately marking KV hashtree {251195593916248939066258330623111144003363405824,3} for full rebuild on next restart 2023-07-25T12:01:17.171115+00:00 [info] <0.844.0>@riak_kv_index_hashtree:close_trees/2:1169 Deliberately marking KV hashtree {228359630832953580969325755111919221821239459840,3} for full rebuild on next restart 2023-07-25T12:01:17.171164+00:00 [info] <0.1463.0>@riak_kv_index_hashtree:close_trees/2:1169 Deliberately marking KV hashtree {1096126227998177188652763624537212264741949407232,3} for full rebuild on next restart 2023-07-25T12:01:17.171219+00:00 [info] <0.841.0>@riak_kv_index_hashtree:close_trees/2:1169 Deliberately marking KV hashtree {274031556999544297163190906134303066185487351808,3} for full rebuild on next restart 2023-07-25T12:01:17.171802+00:00 [info] <0.1465.0>@riak_kv_index_hashtree:close_trees/2:1169 Deliberately marking KV hashtree {1118962191081472546749696200048404186924073353216,3} for full rebuild on next restart 2023-07-25T12:01:17.181151+00:00 [info] <0.1451.0>@riak_kv_index_hashtree:close_trees/2:1169 Deliberately marking KV hashtree {1073290264914881830555831049026020342559825461248,3} for full rebuild on next restart 2023-07-25T12:01:17.240497+00:00 [info] <0.215.0>@riak_core_app:stop/1:118 Stopped application riak_core.

In log stanchion
2023-07-25 11:53:47.276 [warning] <0.838.0>@stanchion_app:check_admin_creds:77 admin.secret is ignored. 2023-07-25 11:59:37.510 [warning] <0.838.0>@stanchion_app:check_admin_creds:77 admin.secret is ignored. 2023-07-25 12:01:47.939 [error] <0.838.0>@stanchion_app:start:59 Couldn't connect to Riak: {riak_connect_failed,{{tcp,econnrefused},"10.32.101.175",8087}} 2023-07-25 12:01:47.939 [error] <0.839.0> CRASH REPORT Process <0.839.0> with 0 neighbours exited with reason: {tcp,econnrefused} in gen_server:init_it/6 line 358 2023-07-25 12:01:47.939 [error] <0.837.0> CRASH REPORT Process <0.837.0> with 0 neighbours exited with reason: {{riak_connect_failed,{{tcp,econnrefused},"10.32.101.175",8087}},{stanchion_app,start,[normal,[]]}} in application_master:init/4 line 138 2023-07-25 12:01:47.939 [info] <0.681.0> Application stanchion exited with reason: {{riak_connect_failed,{{tcp,econnrefused},"10.32.101.175",8087}},{stanchion_app,start,[normal,[]]}}

[Bob-The-Marauder]

When running Stanchion and KV on the same node, I've sometimes noticed permissions issues with the /tmp/erl_pipes folder - both KV and Stanchion like to claim it as their own which means that the other cannot write to it. I'm not sure if this will fix your problem but maybe try moving changing permissions on that folder so that anyone can write to it?

Another option is to look at Riak CS 3.1.0 or 3.2.1 where Stanchion is integrated into all Riak CS nodes, so if one instance fails, another one takes over.

[Konstantin74R]

Thank you, I will try.