Refactoring Rx-placement and Rx-mode

[milanlenco]

The original implementation:

• didn't allow to set rx-placement and rx-mode for multiple queues at once
• didn't work correctly with memifs - placement and mode cannot be set until memif has established connection, because only then the queues are created are ready to accept the configuration (therefore interface link state has to be watched and scheduler needs to be notified about changes)

As always I have also included small cleanup and bug fixing from inside kvscheduler...

[ondrej-fabry]

Was just thinking if this is actually not gonna make GetDescriptorForKey more expensive performance-wise. Can you run the perf tests with 20k to compare difference?

[milanlenco]

It varies quite a bit on my system, but there doesn't seem to be measurable drop in the performance:

BEFORE:

level=info msg="Client #0 done, 20000 tunnels took 1m26.308s" loc="grpc-perf/main.go(302)" logger=grpc-stress-test-client
level=info msg="All clients done, took: 1m16.7757s" loc="grpc-perf/main.go(219)" logger=grpc-stress-test-client
level=info msg="All clients done, took: 1m15.096s" loc="grpc-perf/main.go(219)" logger=grpc-stress-test-client

AFTER:

level=info msg="All clients done, took: 1m5.0734s" loc="grpc-perf/main.go(219)" logger=grpc-stress-test-client
level=info msg="All clients done, took: 1m18.1164s" loc="grpc-perf/main.go(219)" logger=grpc-stress-test-client
level=info msg="All clients done, took: 1m12.1412s" loc="grpc-perf/main.go(219)" logger=grpc-stress-test-client

This lookup is cached (for the last 500 different keys), so effectively each distinct key goes through the cycle only once and the number of descriptors isn't that significant.

[ondrej-fabry]

Awesome, merging then.

[ondrej-fabry]

LGTM.