Reported validator balances inconsistent

[torfbolt]

For my validator client (V0.3.9) the epoch summaries show a very inconsistent picture of the balance. The reported percent change is always negative (see #5247), but the absolute values are sometimes rising and sometimes falling. Also, the new balance of the previous epoch is not equal to the old balance of the current epoch.

Also the inclusion slot looks very strange to me, being larger than the current slot number.

Apr 04 09:16:48 ethnode-4ae769e8 validator[20051]: time="2020-04-04 09:16:48" level=info msg="Previous epoch voting summary" correctlyVotedHead=false correctlyVotedSource=false correctlyVotedTarget=false epoch=19211 inclusionDistance=18446744073709551615 inclusionSlot=18446744073709551615 newBalance=3.133435656 oldBalance=3.133479487 percentChange="-0.00140%" prefix=validator pubKey=0x8a8cd1076aaec9a4
...
Apr 04 09:23:12 ethnode-4ae769e8 validator[20051]: time="2020-04-04 09:23:12" level=info msg="Previous epoch voting summary" correctlyVotedHead=false correctlyVotedSource=false correctlyVotedTarget=false epoch=19212 inclusionDistance=18446744073709551615 inclusionSlot=18446744073709551615 newBalance=3.133510021 oldBalance=3.133550157 percentChange="-0.00128%" prefix=validator pubKey=0x8a8cd1076aaec9a4
...
Apr 04 09:29:36 ethnode-4ae769e8 validator[20051]: time="2020-04-04 09:29:36" level=info msg="Previous epoch voting summary" correctlyVotedHead=false correctlyVotedSource=false correctlyVotedTarget=false epoch=19213 inclusionDistance=18446744073709551615 inclusionSlot=18446744073709551615 newBalance=3.133345222 oldBalance=3.133390901 percentChange="-0.00146%" prefix=validator pubKey=0x8a8cd1076aaec9a4

[0xKiwi]

Hello @torfbolt ! Thanks for reporting this, although I believe mild fluctuations in rewards/penalties are normal, those inclusion slots definitely look off.
Maybe @terencechain can take a closer look at whats going on?

[terencechain]

@torfbolt please provide more evidence such as:
absolute values are sometimes rising and sometimes falling
new balance of the previous epoch is not equal to the old balance of the current epoch

[torfbolt]

If you look at the reported balances for consecutive epochs in the log I posted, you see that the new balance of the previous epoch is not equal to the old balance of the current epoch. However, I did not see this issue again for some time. I will check if it occurs again on Topaz.

[nisdas]

Hey @torfbolt lets track this issue in #5377 . This is pretty much the same issue, and we have been looking at potential solutions for it. Closing this issue and moving conversation there.