All keys are marked as changed on update() in case of a dual gateway change

[gbregman]

The update() function checks to see if the OMAP file version is greater than the local version. In case it is the code gets the state of the OMAP file and the local state. It goes over the keys which are identical in both states and compares their value. In case the value is different, the resource is marked as needed to be deleted from the OMAP file and added again. Only, it seems that all these keys are marked as different, even ones which are the same. The reason is that the OMAP value and the local value are kept in different Python types. The OMAP value is stored in a bytes[] object while the local value is stored in a structure. Adding debug messages to the code and creating a new bdev on the second gateway I could see this on the first gateway:

OMAP:
b'{\n "subsystem_nqn": "nqn.2016-06.io.spdk:cnode1",\n "serial_number": "SPDK87782274058122"\n}'

Local:
{ "subsystem_nqn": "nqn.2016-06.io.spdk:cnode1", "serial_number": "SPDK87782274058122" }

So, even though the actual NQN and serial number here are identical the values are still regarded to be different.

I think we need to parse the bytes[] object from OMAP and compare the fields properly to the local values.

[gbregman]

The exact scenario I was using:

• docker-compose up --scale nvmeof=2 ceph nvmeof
• make demo
• open a shell on the gateway which didn't run the CLI commands (usually gw-1)
• in this shell create a bdev using CLI

Now looking at the log we can see that all the existing resources, which were created by "make demo", get deleted and created again