[WIP] Fix/4849 add cluster stress spec #4899

Aaronontheweb · 2021-04-03T22:36:00Z

No description provided.

Aaronontheweb · 2021-04-06T02:33:40Z

Looks like I found a small bug in the MNTR here:

private IEnumerable<RoleName> RoleNames(Type specType)
        {
            var ctorWithConfig = FindConfigConstructor(specType);
            var configType = ctorWithConfig.GetParameters().First().ParameterType;
            var args = ConfigConstructorParamValues(configType);
            var configInstance = Activator.CreateInstance(configType, args);
            var roleType = typeof(RoleName);
            var configProps = configType.GetProperties(BindingFlags.Instance | BindingFlags.Public);
            var roleProps = configProps.Where(p => p.PropertyType == roleType && p.Name != "Myself").Select(p => (RoleName)p.GetValue(configInstance));
            var configFields = configType.GetFields(BindingFlags.Instance | BindingFlags.Public);
            var roleFields = configFields.Where(f => f.FieldType == roleType && f.Name != "Myself").Select(f => (RoleName)f.GetValue(configInstance));
            var roles = roleProps.Concat(roleFields).Distinct();
            return roles;
        }

This can only find RoleNames that have been declared as fields on an MNTR config - any code that dynamically returns a number of Roles, such as the StressSpec, will not run. I'll have to patch this also.

Aaronontheweb · 2021-04-06T15:01:07Z

Not sure why the build failed on Linux here - didn't get much of an error message back.

Aaronontheweb · 2021-04-07T01:42:54Z

StressSpec is still unstable at the moment - flips on my machine. Suspect that there is a problem with Reachability or PhiAccrualFailureDetector that is responsible for some issues with cluster stability. Going to keep working that problem until I can get the flip rate down.

…operties (akkadotnet#4902) * close akkadotnet#4901 - replace reflection magic in MNTR with reading of MultiNodeConfig properties * fixed outdated DiscoverySpec

Changed this to make it consistent with the JVM

This format error would cause the StandardOutLogger to throw a `FormatException` internally

…ronment variable

This format error would cause the StandardOutLogger to throw a `FormatException` internally

…ronment variable

…egistry

…ronontheweb/akka.net into fix/4849-addClusterStressSpec

Aaronontheweb · 2021-04-16T02:19:35Z

build.fsx

@@ -348,6 +348,7 @@ Target "MultiNodeTests" (fun _ ->

 Target "MultiNodeTestsNetCore" (fun _ ->
    if not skipBuild.Value then
+        setEnvironVar "akka.cluster.assert" "on" // needed to enable assert invariants for Akka.Cluster


Asserts gossip invariants when the MNTR is run

Aaronontheweb

Left some comments on these changes thus far - but I have some cleanup to do on the PR.

Aaronontheweb · 2021-04-16T02:19:52Z

src/core/Akka.Cluster.TestKit/MultiNodeClusterSpec.cs

@@ -175,7 +175,7 @@ protected override void AfterTermination()

        //TODO: ExpectedTestDuration?

-        void MuteLog(ActorSystem sys = null)
+        public virtual void MuteLog(ActorSystem sys = null)


Needed to add this in order to subclass this behavior in other specs

Aaronontheweb · 2021-04-16T02:20:39Z

src/core/Akka.Cluster/Cluster.cs

@@ -62,13 +62,29 @@ public static Cluster Get(ActorSystem system)
            return system.WithExtension<Cluster, ClusterExtension>();
        }

+        static Cluster()
+        {
+            bool GetAssertInvariants()


Enabled assert invariants inside the Cluster when the "akka.cluster.assert" environment variable is set

Aaronontheweb · 2021-04-16T02:21:12Z

src/core/Akka.Cluster/ClusterHeartbeat.cs

@@ -649,11 +655,12 @@ public ImmutableHashSet<UniqueAddress> Receivers(UniqueAddress sender)
                {
                    if (iter.MoveNext() == false || n == 0)
                    {
+                        iter.Dispose(); // dispose enumerator


We weren't properly disposing the enumerator from the HeartbeatNodeRing before.

Aaronontheweb · 2021-04-16T02:21:41Z

src/core/Akka.Cluster/ClusterHeartbeat.cs

-                    ? slice1 
-                    : take(remaining, NodeRing().Until(sender).Where(c => !c.Equals(sender)).GetEnumerator(), slice1).Item2;
+                    ? slice1 // or, wrap0around
+                    : take(remaining, NodeRing().TakeWhile(x => x != sender).GetEnumerator(), slice1).Item2;


cleaned up the LINQ syntax here

Aaronontheweb · 2021-04-16T02:23:27Z

src/core/Akka.Cluster/ClusterHeartbeat.cs

-                    else if (x.Uid == y.Uid) return 0;
-                    else return 1;
-                return result;
+                var ha = x.Uid;


Shifts the ring to primarily factor in the UID in the node ring comparisons, rather than the address. Cheaper and faster while remaining stable so long as the membership doesn't change.

Aaronontheweb · 2021-04-16T02:24:42Z

src/core/Akka.Cluster/Reachability.cs

+    ///     Aggregated status of a subject node is defined as (in this order):
+    ///     - Terminated if any observer node considers it as Terminated
+    ///     - Unreachable if any observer node considers it as Unreachable
+    ///     - Reachable otherwise, i.e. no observer node considers it as Unreachable
    /// </summary>
    internal class Reachability //TODO: ISerializable?


Didn't actually change anything in this file - just ReSharper'd it.

Aaronontheweb · 2021-04-16T02:25:57Z

src/core/Akka.Remote.TestKit/MultiNodeSpec.cs

@@ -508,7 +508,6 @@ public int InitialParticipants
        /// </summary>
        public void RunOn(Action thunk, params RoleName[] nodes)
        {
-            if (nodes.Length == 0) throw new ArgumentException("No node given to run on.");


Needed to allow the MNTR to call RunOn with no nodes

Aaronontheweb · 2021-04-16T02:26:08Z

src/core/Akka.Remote/DefaultFailureDetectorRegistry.cs

@@ -30,11 +31,11 @@ public DefaultFailureDetectorRegistry(Func<FailureDetector> factory)

        private readonly Func<FailureDetector> _factory;

-        private AtomicReference<Dictionary<T, FailureDetector>> _resourceToFailureDetector = new AtomicReference<Dictionary<T, FailureDetector>>(new Dictionary<T, FailureDetector>());
+        private AtomicReference<ImmutableDictionary<T, FailureDetector>> _resourceToFailureDetector = new AtomicReference<ImmutableDictionary<T, FailureDetector>>(ImmutableDictionary<T, FailureDetector>.Empty);


Cleaned up some potential mutability issues here.

Aaronontheweb

Left some additional comments - this PR is ready for review now.

Aaronontheweb · 2021-04-16T13:13:08Z

src/core/Akka.Cluster/ClusterDaemon.cs

@@ -1964,8 +1965,11 @@ public ReceiveGossipType ReceiveGossip(GossipEnvelope envelope)
            // for all new joining nodes we remove them from the failure detector
            foreach (var node in _latestGossip.Members)
            {
-                if (node.Status == MemberStatus.Joining && !localGossip.Members.Contains(node))
+                if (!localGossip.Members.Contains(node))


This is the major bugfix for #4849

Aaronontheweb · 2021-04-16T13:14:26Z

src/core/Akka.Cluster/Reachability.cs

    /// </summary>
-    internal class Reachability //TODO: ISerializable?


Made no real changes to this class - just ReSharper'd it.

Aaronontheweb · 2021-04-16T13:14:50Z

src/core/Akka.Remote/PhiAccrualFailureDetector.cs

@@ -154,7 +154,7 @@ public State(HeartbeatHistory history, long? timeStamp)

        private AtomicReference<State> _state;

-        private State state
+        internal State state


Made internal for debugging purposes

Aaronontheweb · 2021-04-16T13:37:47Z

Replaced via #4940

Aaronontheweb mentioned this pull request Apr 6, 2021

MNTR: needs to discover roles from Config, not reflection #4901

Closed

Aaronontheweb force-pushed the fix/4849-addClusterStressSpec branch 2 times, most recently from 93b3a47 to 67b703f Compare April 6, 2021 21:52

Aaronontheweb mentioned this pull request Apr 7, 2021

Akka.Cluster: quarantining / reachability changes appear to be extremely sensitive #4849

Closed

Aaronontheweb added 23 commits April 7, 2021 13:07

partial implementation of Akka.Cluster.Tests.MultiNode.StressSpec

a1a6088

working

33d63c9

stashing changes

00bcb8a

stash

5a8e995

finished porting all StressSpec actor and message types

97a5d81

stashing work

70d2912

finished first cut of MNTR

c0f4d00

fixed some bugs with StressSpec definition

eb40a29

made MuteLog overrideable in Akka.Cluster.Testkit

0fffc1a

replace reflection magic in MNTR with reading of MultiNodeConfig pr…

7ff0f83

…operties (akkadotnet#4902) * close akkadotnet#4901 - replace reflection magic in MNTR with reading of MultiNodeConfig properties * fixed outdated DiscoverySpec

if Roles is empty, then don't run the thunk on any nodes

4e7744d

Changed this to make it consistent with the JVM

enable Infolog

d84660a

cleaned up memory reporting for Infolog

ace3263

added more stages to StressSpec

b9610c1

fixed InternalStats logging

4cdbf3c

fixed SBR logging error that blew up StandardOutLogger

9f20beb

This format error would cause the StandardOutLogger to throw a `FormatException` internally

fixed GossipStats reporting

ef10d3a

made it possible to actually enable Cluster.AssertInvariants via envi…

bbe1140

…ronment variable

added assert invariants to build script

c5079f7

cleaned up gossip class to assert more invariants

c35ed29

have StressSpec passing

92fc09a

completed port of StressSpec

a442820

completed StressSpec port

c257e3b

Aaronontheweb added 21 commits April 10, 2021 10:22

cleaned up memory reporting for Infolog

8795bc0

added more stages to StressSpec

4d0e8e6

fixed InternalStats logging

743ea34

fixed SBR logging error that blew up StandardOutLogger

13ec0f6

This format error would cause the StandardOutLogger to throw a `FormatException` internally

fixed GossipStats reporting

8c55f6e

made it possible to actually enable Cluster.AssertInvariants via envi…

29ab4c7

…ronment variable

added assert invariants to build script

9e2f528

cleaned up gossip class to assert more invariants

bc06c32

have StressSpec passing

bbda529

completed port of StressSpec

8651d4f

completed StressSpec port

feca0d6

removed unused code

faa9d17

ReSharper'd Reachability.cs

032455a

cleaning up cluster failure detector issues

a2dea58

added some debug logging

88726f6

cleaned up immutability and CAS issues inside DefaultFailureDetectorR…

3171e82

…egistry

Merge branch 'dev' into fix/4849-addClusterStressSpec

776ec91

Merge branch 'fix/4849-addClusterStressSpec' of https://github.com/Aa…

7ed9634

…ronontheweb/akka.net into fix/4849-addClusterStressSpec

cleaned up environment variable declaration

464089e

added more debug logging

4c26804

reverted changes to UniqueMember comparators and HashCode

5a14ed3

Aaronontheweb commented Apr 16, 2021

View reviewed changes

Aaronontheweb added 2 commits April 16, 2021 08:08

cleanup debug logging and added bugfix from akka/akka#23595

9109e80

Merge branch 'dev' into fix/4849-addClusterStressSpec

1bea32a

Aaronontheweb commented Apr 16, 2021

View reviewed changes

cleaned up more debug logging

2103a84

Aaronontheweb mentioned this pull request Apr 16, 2021

ClusterStressSpec and Cluster Failure Detector Cleanup #4940

Merged

Aaronontheweb closed this Apr 16, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Fix/4849 add cluster stress spec #4899

[WIP] Fix/4849 add cluster stress spec #4899

Aaronontheweb commented Apr 3, 2021

Aaronontheweb commented Apr 6, 2021

Aaronontheweb commented Apr 6, 2021

Aaronontheweb commented Apr 7, 2021

Aaronontheweb Apr 16, 2021

Aaronontheweb left a comment

Aaronontheweb Apr 16, 2021

Aaronontheweb Apr 16, 2021

Aaronontheweb Apr 16, 2021

Aaronontheweb Apr 16, 2021

Aaronontheweb Apr 16, 2021

Aaronontheweb Apr 16, 2021

Aaronontheweb Apr 16, 2021

Aaronontheweb Apr 16, 2021

Aaronontheweb left a comment

Aaronontheweb Apr 16, 2021

Aaronontheweb Apr 16, 2021

Aaronontheweb Apr 16, 2021

Aaronontheweb commented Apr 16, 2021

		/// </summary>
		internal class Reachability //TODO: ISerializable?

[WIP] Fix/4849 add cluster stress spec #4899

[WIP] Fix/4849 add cluster stress spec #4899

Conversation

Aaronontheweb commented Apr 3, 2021

Aaronontheweb commented Apr 6, 2021

Aaronontheweb commented Apr 6, 2021

Aaronontheweb commented Apr 7, 2021

Choose a reason for hiding this comment

Aaronontheweb left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Aaronontheweb left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Aaronontheweb commented Apr 16, 2021