Grpc Async Client Call #241

zzxgzgz · 2021-04-19T21:52:08Z

This PR tries to address the following:

Change the GRPC client call from ACA from synchronous to asynchronous, by using different threads to call NCM and process GoalState(s) returned from NCM, in order to make ACA's on-demand functionality un-blocking.
Modified GRPC client test code from sync to async, to test && prove the feasibility of async calls.

Current results:

ACA is able to call NCM asynchronously, and so far it looks like that the on-demand workflow goes as usual.

The above screenshot shows port_1, which is on host_one and has an IP of 10.0.0.2, was able to ping 10.0.0.3, which is another port inside of another host, after ACA on host_one requested the needed GoalStates from the NCM.

Next:
Need to test scenarios in which multiple ports are involved, to see if the change really works.

zzxgzgz · 2021-04-20T23:55:33Z

Update where on-demand-engine sleeps, so it can wait until goalstates are updated before processing the packet.

Updated ping test speed, notice that the first ping packet went through much faster than the first picture:

…onnectivity_test

zzxgzgz · 2021-04-26T23:28:08Z

Unit tests passed:

root@dcf1e7a6a94b:/mnt/host/code# ./build/tests/aca_tests
Trying to init a new sub to connect to the NCM
[==========] Running 51 tests from 10 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from dhcp_config_test_cases
[ RUN      ] dhcp_config_test_cases.add_dhcp_entry_valid
****** a lot of outputs that were deleted as length of content is limited****** 
[       OK ] zeta_programming_test_cases.delete_zeta_config_valid (8 ms)
[----------] 2 tests from zeta_programming_test_cases (21 ms total)

[----------] 5 tests from arp_config_test_cases
[ RUN      ] arp_config_test_cases.add_arp_entry_valid
Arp Entry with ip: 10.0.1.1 and vlan id 1201 added
[       OK ] arp_config_test_cases.add_arp_entry_valid (0 ms)
[ RUN      ] arp_config_test_cases.add_arp_entry_invalid
Arp Entry with ip: 10.0.0.1 and vlan id 1201 added
Entry already existed! (ip = 10.0.0.1 and vlan id = 1201)
[       OK ] arp_config_test_cases.add_arp_entry_invalid (0 ms)
[ RUN      ] arp_config_test_cases.delete_arp_entry_valid
Entry already existed! (ip = 10.0.0.1 and vlan id = 1201)
[       OK ] arp_config_test_cases.delete_arp_entry_valid (0 ms)
[ RUN      ] arp_config_test_cases.update_arp_entry_valid
Arp Entry with ip: 10.0.0.1 and vlan id 1201 added
[       OK ] arp_config_test_cases.update_arp_entry_valid (0 ms)
[ RUN      ] arp_config_test_cases.update_arp_entry_invalid
Entry not exist! (ip = 10.0.0.1 and vlan id = 1201)
Arp Entry with ip: 10.0.0.1 and vlan id 1201 added
[       OK ] arp_config_test_cases.update_arp_entry_invalid (0 ms)
[----------] 5 tests from arp_config_test_cases (0 ms total)

[----------] 1 test from arp_request_test_cases
[ RUN      ] arp_request_test_cases.arps_recv_valid
[       OK ] arp_request_test_cases.arps_recv_valid (0 ms)
[----------] 1 test from arp_request_test_cases (0 ms total)

[----------] Global test environment tear-down
[==========] 51 tests from 10 test suites ran. (4190 ms total)
[  PASSED  ] 51 tests.

  YOU HAVE 21 DISABLED TESTS

==========INIT TIMES:==========
g_initialize_execute_system_time = 192704 microseconds or 192 milliseconds
g_initialize_execute_ovsdb_time = 136118 microseconds or 136 milliseconds
g_initialize_execute_openflow_time = 49656 microseconds or 49 milliseconds
==========EXECUTION TIMES:==========
g_total_execute_system_time = 827784 microseconds or 827 milliseconds
g_total_execute_ovsdb_time = 366614 microseconds or 366 milliseconds
g_total_execute_openflow_time = 263328 microseconds or 263 milliseconds
==========UPDATE GS TIMES:==========
g_total_update_GS_time = 680836 microseconds or 680 milliseconds
Program exiting, cleaning up...
ACA_Zeta_Programming::clear_all_data ---> Entering
ACA_Zeta_Programming::clear_all_data <--- Exiting
ACA_OVS_L2_Programmer::execute_openflow_command ---> Entering
Executing command: ovs-ofctl del-flows br-tun "arp,arp_op=1"
Command succeeded!
 Elapsed time for system command took: 3202 microseconds or 3 milliseconds.
Elapsed time for openflow client call took: 3295 microseconds or 3 milliseconds. rc: 0
ACA_OVS_L2_Programmer::execute_openflow_command <--- Exiting, rc = 0
ACA_OVS_L2_Programmer::execute_openflow_command ---> Entering
Executing command: ovs-ofctl del-flows br-int "udp,udp_src=68,udp_dst=67"
Command succeeded!
 Elapsed time for system command took: 3203 microseconds or 3 milliseconds.
Elapsed time for openflow client call took: 3299 microseconds or 3 milliseconds. rc: 0
ACA_OVS_L2_Programmer::execute_openflow_command <--- Exiting, rc = 0

DISABLED_grpc_client_connectivity_test result:

/zzxgzgz/alcor-control-agent# ./build/tests/aca_tests --gtest_also_run_disabled_tests --gtest_filter=*DISABLED_grpc_client_connectivity_test -a 0.0.0.0 -p 54321
Trying to init a new sub to connect to the NCM
Note: Google Test filter = *DISABLED_grpc_client_connectivity_test
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from aca_on_demand_testcases
[ RUN      ] aca_on_demand_testcases.DISABLED_grpc_client_connectivity_test
After initing a new sub to connect to the NCM
Streaming capable GRPC server listening on 0.0.0.0:
Start of DISABLED_grpc_client_connectivity_test
Made sure that g_grpc_server is not null
Channel state: 0
Request one's request ID: 12345
Request two's request ID: 54321
Beginning of process_async_grpc_replies
Inside the while loop, counter: 1
This call is OK
Static cast done successfully
Got an GRPC reply that is OK, need to process it.
This reply has request ID: 54321
Return from NCM - Reply Status: 1
Inside the while loop, counter: 2
This call is OK
Static cast done successfully
Got an GRPC reply that is OK, need to process it.
This reply has request ID: 12345
Return from NCM - Reply Status: 0
[       OK ] aca_on_demand_testcases.DISABLED_grpc_client_connectivity_test (10002 ms)
[----------] 1 test from aca_on_demand_testcases (10002 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (10002 ms total)
[  PASSED  ] 1 test.
==========INIT TIMES:==========
g_initialize_execute_system_time = 0 microseconds or 0 milliseconds
g_initialize_execute_ovsdb_time = 0 microseconds or 0 milliseconds
g_initialize_execute_openflow_time = 0 microseconds or 0 milliseconds
==========EXECUTION TIMES:==========
g_total_execute_system_time = 0 microseconds or 0 milliseconds
g_total_execute_ovsdb_time = 0 microseconds or 0 milliseconds
g_total_execute_openflow_time = 0 microseconds or 0 milliseconds
==========UPDATE GS TIMES:==========
g_total_update_GS_time = 0 microseconds or 0 milliseconds
Program exiting, cleaning up...

zzxgzgz · 2021-04-26T23:53:55Z

Future improvement:

Right now, the on-demand engine, after receiving the operation status from NCM, sleeps for a period of time, with the hope of it waking up with the needed goalstate(s) received and applied.

This leads to, in many cases, a long time for the first packet to go out(around 1000ms). In some cases, when only sending 1 packet to the destination, the packet got timed out eventually; in other lucky case, all first packets were sent out in less than 2 ms (futurewei-cloud/alcor#560 (comment)).

I believe this highly depends on the connection between ACAs and the NCM, and how fast NCM sends back data. If it is fast enough, then on-demand engine wakes up with needed goalstates, it sends out the packet really quickly; otherwise, it is slow.

Possible solutions:

Change the reply message from the NCM, to include the goalstates needed, but it will break out intent to let only one thread to take care of the goalstates.
Find out a way to let the on-deand engine know that the goalstate has been received and applied, and it can go ahead and let the packet out.

…grammed

zzxgzgz · 2021-04-29T19:49:41Z

Implemented a "Smart" sleep here, which checks if the target arp entry exists,
if exists, stop sleeping and go forward;
otherwise, sleep for another 1000 us (0.001) seconds and then check again.
If arp still not found after one second, it lets the packet drop.

Posting results of 10 ping tests:

************************* 1 **************************
Start of executing command [docker exec test1 ping -I 10.0.0.2 -c1 10.0.0.3] on host: 192.168.20.92
Connected
Start of executing command [docker exec test3 ping -I 10.0.0.4 -c1 10.0.0.5] on host: 192.168.20.92
PING 10.0.0.3 (10.0.0.3) from 10.0.0.2: 56 data bytes
64 bytes from 10.0.0.3: seq=0 ttl=64 time=19.656 ms

--- 10.0.0.3 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 19.656/19.656/19.656 ms
exit-status: 0
End of executing command [docker exec test1 ping -I 10.0.0.2 -c1 10.0.0.3] on host: 192.168.20.92
DONE
PING 10.0.0.5 (10.0.0.5) from 10.0.0.4: 56 data bytes
64 bytes from 10.0.0.5: seq=0 ttl=64 time=16.067 ms

--- 10.0.0.5 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 16.067/16.067/16.067 ms
exit-status: 0
End of executing command [docker exec test3 ping -I 10.0.0.4 -c1 10.0.0.5] on host: 192.168.20.92
DONE
************************* 2 **************************
Start of executing command [docker exec test3 ping -I 10.0.0.4 -c1 10.0.0.5] on host: 192.168.20.92
Connected
Start of executing command [docker exec test1 ping -I 10.0.0.2 -c1 10.0.0.3] on host: 192.168.20.92
PING 10.0.0.5 (10.0.0.5) from 10.0.0.4: 56 data bytes
64 bytes from 10.0.0.5: seq=0 ttl=64 time=17.286 ms

--- 10.0.0.5 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 17.286/17.286/17.286 ms
exit-status: 0
End of executing command [docker exec test3 ping -I 10.0.0.4 -c1 10.0.0.5] on host: 192.168.20.92
DONE
PING 10.0.0.3 (10.0.0.3) from 10.0.0.2: 56 data bytes
64 bytes from 10.0.0.3: seq=0 ttl=64 time=20.379 ms

--- 10.0.0.3 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 20.379/20.379/20.379 ms
exit-status: 0
End of executing command [docker exec test1 ping -I 10.0.0.2 -c1 10.0.0.3] on host: 192.168.20.92
************************* 3 **************************
Start of executing command [docker exec test1 ping -I 10.0.0.2 -c1 10.0.0.3] on host: 192.168.20.92
Connected
Start of executing command [docker exec test3 ping -I 10.0.0.4 -c1 10.0.0.5] on host: 192.168.20.92
PING 10.0.0.3 (10.0.0.3) from 10.0.0.2: 56 data bytes
64 bytes from 10.0.0.3: seq=0 ttl=64 time=19.064 ms

--- 10.0.0.3 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 19.064/19.064/19.064 ms
exit-status: 0
End of executing command [docker exec test1 ping -I 10.0.0.2 -c1 10.0.0.3] on host: 192.168.20.92
DONE
PING 10.0.0.5 (10.0.0.5) from 10.0.0.4: 56 data bytes
64 bytes from 10.0.0.5: seq=0 ttl=64 time=14.508 ms

--- 10.0.0.5 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 14.508/14.508/14.508 ms
exit-status: 0
End of executing command [docker exec test3 ping -I 10.0.0.4 -c1 10.0.0.5] on host: 192.168.20.92
DONE
************************* 4 **************************
Start of executing command [docker exec test3 ping -I 10.0.0.4 -c1 10.0.0.5] on host: 192.168.20.92
Connected
Start of executing command [docker exec test1 ping -I 10.0.0.2 -c1 10.0.0.3] on host: 192.168.20.92
PING 10.0.0.5 (10.0.0.5) from 10.0.0.4: 56 data bytes
64 bytes from 10.0.0.5: seq=0 ttl=64 time=18.134 ms

--- 10.0.0.5 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 18.134/18.134/18.134 ms
exit-status: 0
End of executing command [docker exec test3 ping -I 10.0.0.4 -c1 10.0.0.5] on host: 192.168.20.92
DONE
PING 10.0.0.3 (10.0.0.3) from 10.0.0.2: 56 data bytes
64 bytes from 10.0.0.3: seq=0 ttl=64 time=24.816 ms

--- 10.0.0.3 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 24.816/24.816/24.816 ms
exit-status: 0
End of executing command [docker exec test1 ping -I 10.0.0.2 -c1 10.0.0.3] on host: 192.168.20.92
DONE
************************* 5 **************************
Need to execute this command concurrently: [docker exec test3 ping -I 10.0.0.4 -c1 10.0.0.5]
Need to execute this command concurrently: [docker exec test1 ping -I 10.0.0.2 -c1 10.0.0.3]
Connected
Start of executing command [docker exec test1 ping -I 10.0.0.2 -c1 10.0.0.3] on host: 192.168.20.92
Connected
Start of executing command [docker exec test3 ping -I 10.0.0.4 -c1 10.0.0.5] on host: 192.168.20.92
PING 10.0.0.5 (10.0.0.5) from 10.0.0.4: 56 data bytes
64 bytes from 10.0.0.5: seq=0 ttl=64 time=14.699 ms

--- 10.0.0.5 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 14.699/14.699/14.699 ms
exit-status: 0
End of executing command [docker exec test3 ping -I 10.0.0.4 -c1 10.0.0.5] on host: 192.168.20.92
DONE
PING 10.0.0.3 (10.0.0.3) from 10.0.0.2: 56 data bytes
64 bytes from 10.0.0.3: seq=0 ttl=64 time=20.137 ms

--- 10.0.0.3 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 20.137/20.137/20.137 ms
exit-status: 0
End of executing command [docker exec test1 ping -I 10.0.0.2 -c1 10.0.0.3] on host: 192.168.20.92
DONE
************************* 6 **************************
Start of executing command [docker exec test1 ping -I 10.0.0.2 -c1 10.0.0.3] on host: 192.168.20.92
Connected
Start of executing command [docker exec test3 ping -I 10.0.0.4 -c1 10.0.0.5] on host: 192.168.20.92
PING 10.0.0.5 (10.0.0.5) from 10.0.0.4: 56 data bytes
64 bytes from 10.0.0.5: seq=0 ttl=64 time=20.966 ms

--- 10.0.0.5 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 20.966/20.966/20.966 ms
exit-status: 0
End of executing command [docker exec test3 ping -I 10.0.0.4 -c1 10.0.0.5] on host: 192.168.20.92
DONE
PING 10.0.0.3 (10.0.0.3) from 10.0.0.2: 56 data bytes
64 bytes from 10.0.0.3: seq=0 ttl=64 time=18.274 ms

--- 10.0.0.3 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 18.274/18.274/18.274 ms
exit-status: 0
End of executing command [docker exec test1 ping -I 10.0.0.2 -c1 10.0.0.3] on host: 192.168.20.92
************************* 7 **************************
Start of executing command [docker exec test1 ping -I 10.0.0.2 -c1 10.0.0.3] on host: 192.168.20.92
Connected
Start of executing command [docker exec test3 ping -I 10.0.0.4 -c1 10.0.0.5] on host: 192.168.20.92
PING 10.0.0.3 (10.0.0.3) from 10.0.0.2: 56 data bytes
64 bytes from 10.0.0.3: seq=0 ttl=64 time=15.625 ms

--- 10.0.0.3 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 15.625/15.625/15.625 ms
exit-status: 0
End of executing command [docker exec test1 ping -I 10.0.0.2 -c1 10.0.0.3] on host: 192.168.20.92
DONE
PING 10.0.0.5 (10.0.0.5) from 10.0.0.4: 56 data bytes
64 bytes from 10.0.0.5: seq=0 ttl=64 time=18.547 ms

--- 10.0.0.5 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 18.547/18.547/18.547 ms
exit-status: 0
End of executing command [docker exec test3 ping -I 10.0.0.4 -c1 10.0.0.5] on host: 192.168.20.92
DONE
************************* 8 **************************
Start of executing command [docker exec test1 ping -I 10.0.0.2 -c1 10.0.0.3] on host: 192.168.20.92
Connected
Start of executing command [docker exec test3 ping -I 10.0.0.4 -c1 10.0.0.5] on host: 192.168.20.92
PING 10.0.0.5 (10.0.0.5) from 10.0.0.4: 56 data bytes
64 bytes from 10.0.0.5: seq=0 ttl=64 time=19.819 ms

--- 10.0.0.5 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 19.819/19.819/19.819 ms
exit-status: 0
End of executing command [docker exec test3 ping -I 10.0.0.4 -c1 10.0.0.5] on host: 192.168.20.92
DONE
PING 10.0.0.3 (10.0.0.3) from 10.0.0.2: 56 data bytes
64 bytes from 10.0.0.3: seq=0 ttl=64 time=22.099 ms

--- 10.0.0.3 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 22.099/22.099/22.099 ms
exit-status: 0
End of executing command [docker exec test1 ping -I 10.0.0.2 -c1 10.0.0.3] on host: 192.168.20.92
DONE
************************* 9 **************************
Start of executing command [docker exec test1 ping -I 10.0.0.2 -c1 10.0.0.3] on host: 192.168.20.92
Connected
Start of executing command [docker exec test3 ping -I 10.0.0.4 -c1 10.0.0.5] on host: 192.168.20.92
PING 10.0.0.3 (10.0.0.3) from 10.0.0.2: 56 data bytes
64 bytes from 10.0.0.3: seq=0 ttl=64 time=16.524 ms

--- 10.0.0.3 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 16.524/16.524/16.524 ms
exit-status: 0
End of executing command [docker exec test1 ping -I 10.0.0.2 -c1 10.0.0.3] on host: 192.168.20.92
DONE
PING 10.0.0.5 (10.0.0.5) from 10.0.0.4: 56 data bytes
64 bytes from 10.0.0.5: seq=0 ttl=64 time=21.236 ms

--- 10.0.0.5 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 21.236/21.236/21.236 ms
exit-status: 0
End of executing command [docker exec test3 ping -I 10.0.0.4 -c1 10.0.0.5] on host: 192.168.20.92
DONE
************************* 10 *************************
Start of executing command [docker exec test1 ping -I 10.0.0.2 -c1 10.0.0.3] on host: 192.168.20.92
Connected
Start of executing command [docker exec test3 ping -I 10.0.0.4 -c1 10.0.0.5] on host: 192.168.20.92
PING 10.0.0.5 (10.0.0.5) from 10.0.0.4: 56 data bytes
64 bytes from 10.0.0.5: seq=0 ttl=64 time=18.801 ms

--- 10.0.0.5 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 18.801/18.801/18.801 ms
exit-status: 0
End of executing command [docker exec test3 ping -I 10.0.0.4 -c1 10.0.0.5] on host: 192.168.20.92
DONE
PING 10.0.0.3 (10.0.0.3) from 10.0.0.2: 56 data bytes
64 bytes from 10.0.0.3: seq=0 ttl=64 time=23.033 ms

--- 10.0.0.3 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 23.033/23.033/23.033 ms
exit-status: 0
End of executing command [docker exec test1 ping -I 10.0.0.2 -c1 10.0.0.3] on host: 192.168.20.92
DONE

include/aca_on_demand_engine.h

src/on_demand/aca_on_demand_engine.cpp

cj-chung · 2021-05-03T17:56:25Z

src/on_demand/aca_on_demand_engine.cpp

+
+      do {
+        found_arp_entry =
+                aca_arp_responder::ACA_ARP_Responder::get_instance().does_arp_entry_exist(stData);


@zzxgzgz do we know how long it will take for this call? if it takes too much time, we better increase wait_time. If it takes very short, we can reduce wait_time.

For now, I am keeping the max wait time to be one second.

If the call takes more than one second to reply, the reply is useless, even if the reply is SUCCESS.

@zzxgzgz Sorry, I mean the check_frequency_us. That will affect how often we need to make this call.

I see. No worries, it is configurable in include/aca_config.h.

For now it is set to 10 seconds, we can adjust it how ever we prefer.

cj-chung · 2021-05-03T18:16:42Z

src/on_demand/aca_on_demand_engine.cpp

+    data_for_on_demand_call *data = new data_for_on_demand_call;
+    // auto size_of_packet_void_pointer = sizeof(packet);
+    void *packet_copy = malloc(packet_size);
+    memcpy(packet_copy, packet, packet_size);
+    data->in_port = in_port;
+    data->packet = packet_copy;
+    data->packet_size = packet_size;
+    data->protocol = _protocol;
+    request_uuid_on_demand_data_map[uuid_str] = data;


@zzxgzgz So you put every unknown packet into a map including it's packet_payload. I am afraid of you will run out of memory if the unknown request never got processed by NCM and there are many unknown packets coming in.
We need to figure out how to handle this exception. For example, set up a threshold for each unknown packet. If threshold time-up, just drop packet and remove it from the map.

I think we can clear the map periodically, like every one minute.

A few scenarios to think about:

Packet comes in, only a few entries in the map -> no problem.

Packet comes in, a lot of entries in the map -> means that a lot of entries haven't been taken care of, probably NCM is down, then we clear the map if the time exceeds the threshold.

Packet doesn't come in, we don't touch the map.

Added another thread to clean the request_uuid_on_demand_data_map periodically.

…INE to handle cleaning expired entries

include/aca_config.h

include/aca_on_demand_engine.h

…dded time measurement for map writes

zzxgzgz · 2021-05-04T18:33:08Z

Added time measurement for request_uuid_on_demand_payload_map, below are some example outputs:

Cleaned up [0] entries in the map, which took [9]us, which is [0]ms

Inserting one entry into request_uuid_on_demand_payload_map took [3]us, which is [0]ms

Erasing one entry into request_uuid_on_demand_payload_map took [3]us, which is [0]ms

Also, a new variable called REQUEST_UUID_ON_DEMAND_PAYLOAD_MAP_MAX_SIZE can be found in include/aca_config.h. This variable defines the maximum size of request_uuid_on_demand_payload_map. Before an entry is inserted into request_uuid_on_demand_payload_map, the on-demand engine checks its current size, and waits until the size is less than the max size.

zzxgzgz · 2021-05-07T20:42:42Z

Did a comparison with the ACA on zzxgzgz:master branch, which is only one commit behind the official master with the License changes, and zzxgzgz:grpc_client_async branch, with includes the change for the grpc async client calls.

100 ports were generated on both hosts, before 100 ping commands are issued from host_one to host_two, which shall generate 100 on-demand calls. Below are the outputs from the Test_Controller:

Sync call output:

PING 10.0.0.10 (10.0.0.10) from 10.0.0.11: 56 data bytes

--- 10.0.0.10 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
PING 10.0.0.4 (10.0.0.4) from 10.0.0.41: 56 data bytes

--- 10.0.0.4 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
PING 10.0.0.22 (10.0.0.22) from 10.0.0.25: 56 data bytes

--- 10.0.0.22 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
PING 10.0.0.6 (10.0.0.6) from 10.0.0.61: 56 data bytes

--- 10.0.0.6 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
PING 10.0.0.38 (10.0.0.38) from 10.0.0.39: 56 data bytes

--- 10.0.0.38 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
PING 10.0.0.36 (10.0.0.36) from 10.0.0.37: 56 data bytes

--- 10.0.0.36 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
PING 10.0.0.30 (10.0.0.30) from 10.0.0.31: 56 data bytes

--- 10.0.0.30 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
PING 10.0.0.24 (10.0.0.24) from 10.0.0.27: 56 data bytes

--- 10.0.0.24 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
PING 10.0.0.26 (10.0.0.26) from 10.0.0.29: 56 data bytes

--- 10.0.0.26 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
PING 10.0.0.58 (10.0.0.58) from 10.0.0.59: 56 data bytes
PING 10.0.0.40 (10.0.0.40) from 10.0.0.43: 56 data bytes

--- 10.0.0.40 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss

--- 10.0.0.58 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
PING 10.0.0.2 (10.0.0.2) from 10.0.0.21: 56 data bytes

--- 10.0.0.2 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
PING 10.0.0.20 (10.0.0.20) from 10.0.0.23: 56 data bytes

--- 10.0.0.20 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
PING 10.0.0.62 (10.0.0.62) from 10.0.0.65: 56 data bytes

--- 10.0.0.62 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
PING 10.0.0.28 (10.0.0.28) from 10.0.0.3: 56 data bytes

--- 10.0.0.28 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
PING 10.0.0.32 (10.0.0.32) from 10.0.0.33: 56 data bytes

--- 10.0.0.32 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
PING 10.0.0.34 (10.0.0.34) from 10.0.0.35: 56 data bytes

--- 10.0.0.34 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
PING 10.0.0.18 (10.0.0.18) from 10.0.0.19: 56 data bytes

--- 10.0.0.18 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
PING 10.0.0.64 (10.0.0.64) from 10.0.0.67: 56 data bytes

--- 10.0.0.64 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
PING 10.0.0.54 (10.0.0.54) from 10.0.0.55: 56 data bytes

--- 10.0.0.54 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
PING 10.0.0.14 (10.0.0.14) from 10.0.0.15: 56 data bytes

--- 10.0.0.14 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
PING 10.0.0.16 (10.0.0.16) from 10.0.0.17: 56 data bytes

--- 10.0.0.16 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
PING 10.0.0.12 (10.0.0.12) from 10.0.0.13: 56 data bytes

--- 10.0.0.12 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
PING 10.0.0.66 (10.0.0.66) from 10.0.0.69: 56 data bytes

--- 10.0.0.66 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
PING 10.0.0.68 (10.0.0.68) from 10.0.0.7: 56 data bytes

--- 10.0.0.68 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
PING 10.0.0.46 (10.0.0.46) from 10.0.0.49: 56 data bytes

--- 10.0.0.46 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
PING 10.0.0.60 (10.0.0.60) from 10.0.0.63: 56 data bytes

--- 10.0.0.60 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
PING 10.0.0.50 (10.0.0.50) from 10.0.0.51: 56 data bytes

--- 10.0.0.50 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
PING 10.0.0.56 (10.0.0.56) from 10.0.0.57: 56 data bytes

--- 10.0.0.56 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
PING 10.0.0.70 (10.0.0.70) from 10.0.0.71: 56 data bytes

--- 10.0.0.70 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
PING 10.0.0.52 (10.0.0.52) from 10.0.0.53: 56 data bytes

--- 10.0.0.52 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
PING 10.0.0.78 (10.0.0.78) from 10.0.0.79: 56 data bytes

--- 10.0.0.78 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
PING 10.0.0.44 (10.0.0.44) from 10.0.0.47: 56 data bytes

--- 10.0.0.44 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
PING 10.0.0.48 (10.0.0.48) from 10.0.0.5: 56 data bytes

--- 10.0.0.48 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
PING 10.0.0.42 (10.0.0.42) from 10.0.0.45: 56 data bytes

--- 10.0.0.42 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
PING 10.0.0.74 (10.0.0.74) from 10.0.0.75: 56 data bytes

--- 10.0.0.74 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
PING 10.0.0.72 (10.0.0.72) from 10.0.0.73: 56 data bytes

--- 10.0.0.72 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
PING 10.0.0.76 (10.0.0.76) from 10.0.0.77: 56 data bytes

--- 10.0.0.76 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
PING 10.0.0.82 (10.0.0.82) from 10.0.0.85: 56 data bytes

--- 10.0.0.82 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
PING 10.0.0.80 (10.0.0.80) from 10.0.0.83: 56 data bytes

--- 10.0.0.80 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
PING 10.0.0.8 (10.0.0.8) from 10.0.0.81: 56 data bytes

--- 10.0.0.8 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
PING 10.0.0.96 (10.0.0.96) from 10.0.0.97: 56 data bytes

--- 10.0.0.96 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
PING 10.0.0.98 (10.0.0.98) from 10.0.0.99: 56 data bytes

--- 10.0.0.98 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
PING 10.0.1.12 (10.0.1.12) from 10.0.1.11: 56 data bytes

--- 10.0.1.12 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
PING 10.0.0.90 (10.0.0.90) from 10.0.0.91: 56 data bytes

--- 10.0.0.90 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
PING 10.0.0.84 (10.0.0.84) from 10.0.0.87: 56 data bytes

--- 10.0.0.84 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
PING 10.0.0.88 (10.0.0.88) from 10.0.0.9: 56 data bytes

--- 10.0.0.88 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
PING 10.0.0.94 (10.0.0.94) from 10.0.0.95: 56 data bytes

--- 10.0.0.94 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
PING 10.0.0.86 (10.0.0.86) from 10.0.0.89: 56 data bytes

--- 10.0.0.86 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
PING 10.0.1.10 (10.0.1.10) from 10.0.1.1: 56 data bytes

--- 10.0.1.10 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
PING 10.0.1.18 (10.0.1.18) from 10.0.1.17: 56 data bytes

--- 10.0.1.18 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
PING 10.0.1.14 (10.0.1.14) from 10.0.1.13: 56 data bytes

--- 10.0.1.14 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
PING 10.0.1.20 (10.0.1.20) from 10.0.1.21: 56 data bytes

--- 10.0.1.20 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
PING 10.0.1.2 (10.0.1.2) from 10.0.1.19: 56 data bytes

--- 10.0.1.2 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
PING 10.0.0.92 (10.0.0.92) from 10.0.0.93: 56 data bytes

--- 10.0.0.92 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
PING 10.0.1.28 (10.0.1.28) from 10.0.1.29: 56 data bytes

--- 10.0.1.28 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
PING 10.0.1.24 (10.0.1.24) from 10.0.1.25: 56 data bytes

--- 10.0.1.24 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
PING 10.0.1.32 (10.0.1.32) from 10.0.1.31: 56 data bytes

--- 10.0.1.32 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
PING 10.0.1.36 (10.0.1.36) from 10.0.1.35: 56 data bytes

--- 10.0.1.36 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
PING 10.0.1.16 (10.0.1.16) from 10.0.1.15: 56 data bytes

--- 10.0.1.16 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
PING 10.0.1.34 (10.0.1.34) from 10.0.1.33: 56 data bytes

--- 10.0.1.34 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
PING 10.0.1.26 (10.0.1.26) from 10.0.1.27: 56 data bytes

--- 10.0.1.26 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
PING 10.0.1.30 (10.0.1.30) from 10.0.1.3: 56 data bytes

--- 10.0.1.30 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
PING 10.0.1.40 (10.0.1.40) from 10.0.1.41: 56 data bytes

--- 10.0.1.40 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
PING 10.0.1.22 (10.0.1.22) from 10.0.1.23: 56 data bytes

--- 10.0.1.22 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
PING 10.0.1.4 (10.0.1.4) from 10.0.1.39: 56 data bytes

--- 10.0.1.4 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
PING 10.0.1.46 (10.0.1.46) from 10.0.1.47: 56 data bytes

--- 10.0.1.46 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
PING 10.0.1.38 (10.0.1.38) from 10.0.1.37: 56 data bytes

--- 10.0.1.38 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
PING 10.0.1.54 (10.0.1.54) from 10.0.1.53: 56 data bytes

--- 10.0.1.54 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss

Async call output:

Executing BASH command:
   docker exec test197 ping -I 10.0.1.99 -c1 10.0.2.2
PING 10.0.0.24 (10.0.0.24) from 10.0.0.27: 56 data bytes
64 bytes from 10.0.0.24: seq=0 ttl=64 time=643.420 ms

--- 10.0.0.24 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 643.420/643.420/643.420 ms
PING 10.0.0.44 (10.0.0.44) from 10.0.0.47: 56 data bytes
64 bytes from 10.0.0.44: seq=0 ttl=64 time=655.562 ms

--- 10.0.0.44 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 655.562/655.562/655.562 ms
PING 10.0.0.72 (10.0.0.72) from 10.0.0.73: 56 data bytes
64 bytes from 10.0.0.72: seq=0 ttl=64 time=717.470 ms

--- 10.0.0.72 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 717.470/717.470/717.470 ms
PING 10.0.0.26 (10.0.0.26) from 10.0.0.29: 56 data bytes
64 bytes from 10.0.0.26: seq=0 ttl=64 time=725.599 ms

--- 10.0.0.26 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 725.599/725.599/725.599 ms
PING 10.0.0.60 (10.0.0.60) from 10.0.0.63: 56 data bytes
64 bytes from 10.0.0.60: seq=0 ttl=64 time=753.748 ms

--- 10.0.0.60 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 753.748/753.748/753.748 ms
PING 10.0.0.36 (10.0.0.36) from 10.0.0.37: 56 data bytes
64 bytes from 10.0.0.36: seq=0 ttl=64 time=780.435 ms

--- 10.0.0.36 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 780.435/780.435/780.435 ms
PING 10.0.0.28 (10.0.0.28) from 10.0.0.3: 56 data bytes
64 bytes from 10.0.0.28: seq=0 ttl=64 time=711.346 ms

--- 10.0.0.28 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 711.346/711.346/711.346 ms
PING 10.0.0.70 (10.0.0.70) from 10.0.0.71: 56 data bytes
64 bytes from 10.0.0.70: seq=0 ttl=64 time=838.695 ms

--- 10.0.0.70 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 838.695/838.695/838.695 ms
PING 10.0.0.20 (10.0.0.20) from 10.0.0.23: 56 data bytes
64 bytes from 10.0.0.20: seq=0 ttl=64 time=883.040 ms

--- 10.0.0.20 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 883.040/883.040/883.040 ms
PING 10.0.0.80 (10.0.0.80) from 10.0.0.83: 56 data bytes
64 bytes from 10.0.0.80: seq=0 ttl=64 time=821.786 ms

--- 10.0.0.80 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 821.786/821.786/821.786 ms
PING 10.0.0.68 (10.0.0.68) from 10.0.0.7: 56 data bytes
64 bytes from 10.0.0.68: seq=0 ttl=64 time=858.270 ms

--- 10.0.0.68 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 858.270/858.270/858.270 ms
PING 10.0.0.22 (10.0.0.22) from 10.0.0.25: 56 data bytes
64 bytes from 10.0.0.22: seq=0 ttl=64 time=906.880 ms

--- 10.0.0.22 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 906.880/906.880/906.880 ms
PING 10.0.0.4 (10.0.0.4) from 10.0.0.41: 56 data bytes
64 bytes from 10.0.0.4: seq=0 ttl=64 time=817.721 ms

--- 10.0.0.4 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 817.721/817.721/817.721 ms
PING 10.0.0.78 (10.0.0.78) from 10.0.0.79: 56 data bytes
64 bytes from 10.0.0.78: seq=0 ttl=64 time=857.674 ms

--- 10.0.0.78 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 857.674/857.674/857.674 ms
PING 10.0.0.6 (10.0.0.6) from 10.0.0.61: 56 data bytes
64 bytes from 10.0.0.6: seq=0 ttl=64 time=883.553 ms

--- 10.0.0.6 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 883.553/883.553/883.553 ms
PING 10.0.0.42 (10.0.0.42) from 10.0.0.45: 56 data bytes
64 bytes from 10.0.0.42: seq=0 ttl=64 time=877.786 ms

--- 10.0.0.42 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 877.786/877.786/877.786 ms
PING 10.0.0.34 (10.0.0.34) from 10.0.0.35: 56 data bytes
64 bytes from 10.0.0.34: seq=0 ttl=64 time=876.790 ms

--- 10.0.0.34 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 876.790/876.790/876.790 ms
PING 10.0.0.56 (10.0.0.56) from 10.0.0.57: 56 data bytes
64 bytes from 10.0.0.56: seq=0 ttl=64 time=858.521 ms

--- 10.0.0.56 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 858.521/858.521/858.521 ms
PING 10.0.0.50 (10.0.0.50) from 10.0.0.51: 56 data bytes
64 bytes from 10.0.0.50: seq=0 ttl=64 time=873.138 ms

--- 10.0.0.50 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 873.138/873.138/873.138 ms
PING 10.0.0.32 (10.0.0.32) from 10.0.0.33: 56 data bytes
64 bytes from 10.0.0.32: seq=0 ttl=64 time=886.771 ms

--- 10.0.0.32 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 886.771/886.771/886.771 ms
PING 10.0.0.66 (10.0.0.66) from 10.0.0.69: 56 data bytes
64 bytes from 10.0.0.66: seq=0 ttl=64 time=906.233 ms

--- 10.0.0.66 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 906.233/906.233/906.233 ms
PING 10.0.0.62 (10.0.0.62) from 10.0.0.65: 56 data bytes
64 bytes from 10.0.0.62: seq=0 ttl=64 time=905.499 ms

--- 10.0.0.62 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 905.499/905.499/905.499 ms
PING 10.0.0.8 (10.0.0.8) from 10.0.0.81: 56 data bytes
64 bytes from 10.0.0.8: seq=0 ttl=64 time=911.913 ms

--- 10.0.0.8 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 911.913/911.913/911.913 ms
PING 10.0.0.76 (10.0.0.76) from 10.0.0.77: 56 data bytes
64 bytes from 10.0.0.76: seq=0 ttl=64 time=924.993 ms

--- 10.0.0.76 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 924.993/924.993/924.993 ms
PING 10.0.0.48 (10.0.0.48) from 10.0.0.5: 56 data bytes
64 bytes from 10.0.0.48: seq=0 ttl=64 time=917.159 ms

--- 10.0.0.48 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 917.159/917.159/917.159 ms
PING 10.0.0.10 (10.0.0.10) from 10.0.0.11: 56 data bytes
64 bytes from 10.0.0.10: seq=0 ttl=64 time=928.928 ms

--- 10.0.0.10 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 928.928/928.928/928.928 ms
PING 10.0.0.54 (10.0.0.54) from 10.0.0.55: 56 data bytes
64 bytes from 10.0.0.54: seq=0 ttl=64 time=887.624 ms

--- 10.0.0.54 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 887.624/887.624/887.624 ms
PING 10.0.0.86 (10.0.0.86) from 10.0.0.89: 56 data bytes
64 bytes from 10.0.0.86: seq=0 ttl=64 time=871.164 ms

--- 10.0.0.86 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 871.164/871.164/871.164 ms
PING 10.0.0.84 (10.0.0.84) from 10.0.0.87: 56 data bytes
64 bytes from 10.0.0.84: seq=0 ttl=64 time=919.057 ms

--- 10.0.0.84 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 919.057/919.057/919.057 ms
PING 10.0.0.58 (10.0.0.58) from 10.0.0.59: 56 data bytes
64 bytes from 10.0.0.58: seq=0 ttl=64 time=895.059 ms

--- 10.0.0.58 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 895.059/895.059/895.059 ms
PING 10.0.0.2 (10.0.0.2) from 10.0.0.21: 56 data bytes
64 bytes from 10.0.0.2: seq=0 ttl=64 time=890.422 ms

--- 10.0.0.2 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 890.422/890.422/890.422 ms
PING 10.0.0.46 (10.0.0.46) from 10.0.0.49: 56 data bytes
64 bytes from 10.0.0.46: seq=0 ttl=64 time=897.734 ms

--- 10.0.0.46 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 897.734/897.734/897.734 ms
PING 10.0.0.52 (10.0.0.52) from 10.0.0.53: 56 data bytes
64 bytes from 10.0.0.52: seq=0 ttl=64 time=919.265 ms

--- 10.0.0.52 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 919.265/919.265/919.265 ms
PING 10.0.0.74 (10.0.0.74) from 10.0.0.75: 56 data bytes
64 bytes from 10.0.0.74: seq=0 ttl=64 time=904.590 ms

--- 10.0.0.74 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 904.590/904.590/904.590 ms
PING 10.0.1.16 (10.0.1.16) from 10.0.1.15: 56 data bytes
64 bytes from 10.0.1.16: seq=0 ttl=64 time=820.856 ms

--- 10.0.1.16 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 820.856/820.856/820.856 ms
PING 10.0.0.96 (10.0.0.96) from 10.0.0.97: 56 data bytes
64 bytes from 10.0.0.96: seq=0 ttl=64 time=853.203 ms

--- 10.0.0.96 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 853.203/853.203/853.203 ms
PING 10.0.0.82 (10.0.0.82) from 10.0.0.85: 56 data bytes
64 bytes from 10.0.0.82: seq=0 ttl=64 time=872.606 ms

--- 10.0.0.82 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 872.606/872.606/872.606 ms
PING 10.0.1.12 (10.0.1.12) from 10.0.1.11: 56 data bytes
64 bytes from 10.0.1.12: seq=0 ttl=64 time=827.207 ms

--- 10.0.1.12 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 827.207/827.207/827.207 ms
PING 10.0.0.90 (10.0.0.90) from 10.0.0.91: 56 data bytes
64 bytes from 10.0.0.90: seq=0 ttl=64 time=786.290 ms

--- 10.0.0.90 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 786.290/786.290/786.290 ms
PING 10.0.0.92 (10.0.0.92) from 10.0.0.93: 56 data bytes
64 bytes from 10.0.0.92: seq=0 ttl=64 time=812.966 ms

--- 10.0.0.92 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 812.966/812.966/812.966 ms
PING 10.0.0.12 (10.0.0.12) from 10.0.0.13: 56 data bytes
64 bytes from 10.0.0.12: seq=0 ttl=64 time=872.772 ms

--- 10.0.0.12 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 872.772/872.772/872.772 ms
PING 10.0.0.94 (10.0.0.94) from 10.0.0.95: 56 data bytes
64 bytes from 10.0.0.94: seq=0 ttl=64 time=826.409 ms

--- 10.0.0.94 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 826.409/826.409/826.409 ms
PING 10.0.0.88 (10.0.0.88) from 10.0.0.9: 56 data bytes
64 bytes from 10.0.0.88: seq=0 ttl=64 time=933.231 ms

--- 10.0.0.88 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 933.231/933.231/933.231 ms
PING 10.0.1.14 (10.0.1.14) from 10.0.1.13: 56 data bytes
64 bytes from 10.0.1.14: seq=0 ttl=64 time=777.029 ms

--- 10.0.1.14 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 777.029/777.029/777.029 ms
PING 10.0.0.18 (10.0.0.18) from 10.0.0.19: 56 data bytes
64 bytes from 10.0.0.18: seq=0 ttl=64 time=844.203 ms

--- 10.0.0.18 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 844.203/844.203/844.203 ms
PING 10.0.1.2 (10.0.1.2) from 10.0.1.19: 56 data bytes
64 bytes from 10.0.1.2: seq=0 ttl=64 time=776.686 ms

--- 10.0.1.2 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 776.686/776.686/776.686 ms
PING 10.0.1.10 (10.0.1.10) from 10.0.1.1: 56 data bytes
64 bytes from 10.0.1.10: seq=0 ttl=64 time=762.564 ms

--- 10.0.1.10 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 762.564/762.564/762.564 ms
PING 10.0.0.98 (10.0.0.98) from 10.0.0.99: 56 data bytes
64 bytes from 10.0.0.98: seq=0 ttl=64 time=757.895 ms

--- 10.0.0.98 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 757.895/757.895/757.895 ms
PING 10.0.1.18 (10.0.1.18) from 10.0.1.17: 56 data bytes
64 bytes from 10.0.1.18: seq=0 ttl=64 time=759.883 ms

--- 10.0.1.18 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 759.883/759.883/759.883 ms
PING 10.0.1.26 (10.0.1.26) from 10.0.1.27: 56 data bytes
64 bytes from 10.0.1.26: seq=0 ttl=64 time=749.425 ms

--- 10.0.1.26 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 749.425/749.425/749.425 ms
PING 10.0.1.20 (10.0.1.20) from 10.0.1.21: 56 data bytes
64 bytes from 10.0.1.20: seq=0 ttl=64 time=740.204 ms

--- 10.0.1.20 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 740.204/740.204/740.204 ms
PING 10.0.1.24 (10.0.1.24) from 10.0.1.25: 56 data bytes
64 bytes from 10.0.1.24: seq=0 ttl=64 time=738.996 ms

--- 10.0.1.24 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 738.996/738.996/738.996 ms
PING 10.0.1.22 (10.0.1.22) from 10.0.1.23: 56 data bytes
64 bytes from 10.0.1.22: seq=0 ttl=64 time=733.761 ms

--- 10.0.1.22 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 733.761/733.761/733.761 ms
PING 10.0.1.38 (10.0.1.38) from 10.0.1.37: 56 data bytes
64 bytes from 10.0.1.38: seq=0 ttl=64 time=716.634 ms

--- 10.0.1.38 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 716.634/716.634/716.634 ms
PING 10.0.1.28 (10.0.1.28) from 10.0.1.29: 56 data bytes
64 bytes from 10.0.1.28: seq=0 ttl=64 time=715.287 ms

--- 10.0.1.28 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 715.287/715.287/715.287 ms
PING 10.0.1.36 (10.0.1.36) from 10.0.1.35: 56 data bytes
64 bytes from 10.0.1.36: seq=0 ttl=64 time=699.051 ms

--- 10.0.1.36 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 699.051/699.051/699.051 ms
PING 10.0.1.32 (10.0.1.32) from 10.0.1.31: 56 data bytes
64 bytes from 10.0.1.32: seq=0 ttl=64 time=671.325 ms

--- 10.0.1.32 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 671.325/671.325/671.325 ms
PING 10.0.1.30 (10.0.1.30) from 10.0.1.3: 56 data bytes
64 bytes from 10.0.1.30: seq=0 ttl=64 time=661.857 ms

--- 10.0.1.30 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 661.857/661.857/661.857 ms
PING 10.0.1.48 (10.0.1.48) from 10.0.1.49: 56 data bytes
64 bytes from 10.0.1.48: seq=0 ttl=64 time=646.442 ms

--- 10.0.1.48 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 646.442/646.442/646.442 ms
PING 10.0.1.40 (10.0.1.40) from 10.0.1.41: 56 data bytes
64 bytes from 10.0.1.40: seq=0 ttl=64 time=662.053 ms

--- 10.0.1.40 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 662.053/662.053/662.053 ms
PING 10.0.1.34 (10.0.1.34) from 10.0.1.33: 56 data bytes
64 bytes from 10.0.1.34: seq=0 ttl=64 time=660.812 ms

--- 10.0.1.34 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 660.812/660.812/660.812 ms
PING 10.0.1.4 (10.0.1.4) from 10.0.1.39: 56 data bytes
64 bytes from 10.0.1.4: seq=0 ttl=64 time=653.870 ms

--- 10.0.1.4 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 653.870/653.870/653.870 ms
PING 10.0.1.44 (10.0.1.44) from 10.0.1.45: 56 data bytes
64 bytes from 10.0.1.44: seq=0 ttl=64 time=667.869 ms

--- 10.0.1.44 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 667.869/667.869/667.869 ms
PING 10.0.1.42 (10.0.1.42) from 10.0.1.43: 56 data bytes
64 bytes from 10.0.1.42: seq=0 ttl=64 time=655.059 ms

--- 10.0.1.42 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 655.059/655.059/655.059 ms
PING 10.0.1.56 (10.0.1.56) from 10.0.1.55: 56 data bytes
64 bytes from 10.0.1.56: seq=0 ttl=64 time=615.847 ms

--- 10.0.1.56 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 615.847/615.847/615.847 ms
PING 10.0.1.50 (10.0.1.50) from 10.0.1.5: 56 data bytes
64 bytes from 10.0.1.50: seq=0 ttl=64 time=626.481 ms

--- 10.0.1.50 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 626.481/626.481/626.481 ms
PING 10.0.1.46 (10.0.1.46) from 10.0.1.47: 56 data bytes
64 bytes from 10.0.1.46: seq=0 ttl=64 time=575.784 ms

--- 10.0.1.46 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 575.784/575.784/575.784 ms
PING 10.0.1.6 (10.0.1.6) from 10.0.1.59: 56 data bytes
64 bytes from 10.0.1.6: seq=0 ttl=64 time=542.029 ms

--- 10.0.1.6 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 542.029/542.029/542.029 ms
PING 10.0.1.58 (10.0.1.58) from 10.0.1.57: 56 data bytes
64 bytes from 10.0.1.58: seq=0 ttl=64 time=539.340 ms

--- 10.0.1.58 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 539.340/539.340/539.340 ms
PING 10.0.1.52 (10.0.1.52) from 10.0.1.51: 56 data bytes
64 bytes from 10.0.1.52: seq=0 ttl=64 time=541.272 ms

--- 10.0.1.52 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 541.272/541.272/541.272 ms
PING 10.0.1.60 (10.0.1.60) from 10.0.1.61: 56 data bytes
64 bytes from 10.0.1.60: seq=0 ttl=64 time=505.150 ms

--- 10.0.1.60 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 505.150/505.150/505.150 ms
PING 10.0.1.64 (10.0.1.64) from 10.0.1.65: 56 data bytes
64 bytes from 10.0.1.64: seq=0 ttl=64 time=496.296 ms

--- 10.0.1.64 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 496.296/496.296/496.296 ms
PING 10.0.1.70 (10.0.1.70) from 10.0.1.7: 56 data bytes
64 bytes from 10.0.1.70: seq=0 ttl=64 time=468.713 ms

--- 10.0.1.70 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 468.713/468.713/468.713 ms
PING 10.0.1.68 (10.0.1.68) from 10.0.1.69: 56 data bytes
64 bytes from 10.0.1.68: seq=0 ttl=64 time=453.509 ms

--- 10.0.1.68 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 453.509/453.509/453.509 ms
PING 10.0.1.62 (10.0.1.62) from 10.0.1.63: 56 data bytes
64 bytes from 10.0.1.62: seq=0 ttl=64 time=490.202 ms

--- 10.0.1.62 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
PING 10.0.1.72 (10.0.1.72) from 10.0.1.71: 56 data bytes
64 bytes from 10.0.1.72: seq=0 ttl=64 time=430.125 ms

--- 10.0.1.72 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 430.125/430.125/430.125 ms
round-trip min/avg/max = 490.202/490.202/490.202 ms
PING 10.0.1.66 (10.0.1.66) from 10.0.1.67: 56 data bytes
64 bytes from 10.0.1.66: seq=0 ttl=64 time=430.596 ms

--- 10.0.1.66 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 430.596/430.596/430.596 ms
PING 10.0.1.54 (10.0.1.54) from 10.0.1.53: 56 data bytes
64 bytes from 10.0.1.54: seq=0 ttl=64 time=443.033 ms

--- 10.0.1.54 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 443.033/443.033/443.033 ms
PING 10.0.1.88 (10.0.1.88) from 10.0.1.89: 56 data bytes
64 bytes from 10.0.1.88: seq=0 ttl=64 time=353.291 ms

--- 10.0.1.88 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 353.291/353.291/353.291 ms
PING 10.0.1.98 (10.0.1.98) from 10.0.1.97: 56 data bytes
64 bytes from 10.0.1.98: seq=0 ttl=64 time=296.510 ms

--- 10.0.1.98 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 296.510/296.510/296.510 ms
PING 10.0.1.80 (10.0.1.80) from 10.0.1.81: 56 data bytes
64 bytes from 10.0.1.80: seq=0 ttl=64 time=430.256 ms

--- 10.0.1.80 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 430.256/430.256/430.256 ms
PING 10.0.1.86 (10.0.1.86) from 10.0.1.87: 56 data bytes
64 bytes from 10.0.1.86: seq=0 ttl=64 time=327.286 ms

--- 10.0.1.86 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 327.286/327.286/327.286 ms
PING 10.0.1.76 (10.0.1.76) from 10.0.1.75: 56 data bytes
64 bytes from 10.0.1.76: seq=0 ttl=64 time=367.373 ms

--- 10.0.1.76 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 367.373/367.373/367.373 ms
PING 10.0.2.2 (10.0.2.2) from 10.0.1.99: 56 data bytes
64 bytes from 10.0.2.2: seq=0 ttl=64 time=290.242 ms

--- 10.0.2.2 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 290.242/290.242/290.242 ms
PING 10.0.1.94 (10.0.1.94) from 10.0.1.93: 56 data bytes
64 bytes from 10.0.1.94: seq=0 ttl=64 time=338.170 ms

--- 10.0.1.94 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 338.170/338.170/338.170 ms
PING 10.0.1.8 (10.0.1.8) from 10.0.1.79: 56 data bytes
64 bytes from 10.0.1.8: seq=0 ttl=64 time=393.017 ms

--- 10.0.1.8 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 393.017/393.017/393.017 ms
PING 10.0.1.74 (10.0.1.74) from 10.0.1.73: 56 data bytes
64 bytes from 10.0.1.74: seq=0 ttl=64 time=355.027 ms

--- 10.0.1.74 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 355.027/355.027/355.027 ms
PING 10.0.1.78 (10.0.1.78) from 10.0.1.77: 56 data bytes
64 bytes from 10.0.1.78: seq=0 ttl=64 time=367.685 ms

--- 10.0.1.78 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 367.685/367.685/367.685 ms
PING 10.0.1.84 (10.0.1.84) from 10.0.1.85: 56 data bytes
64 bytes from 10.0.1.84: seq=0 ttl=64 time=334.472 ms

--- 10.0.1.84 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 334.472/334.472/334.472 ms
PING 10.0.1.82 (10.0.1.82) from 10.0.1.83: 56 data bytes
64 bytes from 10.0.1.82: seq=0 ttl=64 time=315.054 ms

--- 10.0.1.82 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 315.054/315.054/315.054 ms
PING 10.0.1.96 (10.0.1.96) from 10.0.1.95: 56 data bytes
64 bytes from 10.0.1.96: seq=0 ttl=64 time=294.397 ms

PING 10.0.1.90 (10.0.1.90) from 10.0.1.9: 56 data bytes
PING 10.0.1.92 (10.0.1.92) from 10.0.1.91: 56 data bytes
64 bytes from 10.0.1.92: seq=0 ttl=64 time=350.711 ms

--- 10.0.1.92 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 350.711/350.711/350.711 ms
64 bytes from 10.0.1.90: seq=0 ttl=64 time=327.605 ms

--- 10.0.1.90 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 327.605/327.605/327.605 ms
--- 10.0.1.96 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 294.397/294.397/294.397 ms

One thing to notice is that, when testing with the master branch, the above output only shows 69 ping results, which is less than the total ping commands executed(99). The reason of this issue needs to be investigated.

… gs sent->received&&programmed operation

cj-chung

LGTM

xieus · 2021-05-10T18:16:57Z

Future improvement:

Right now, the on-demand engine, after receiving the operation status from NCM, sleeps for a period of time, with the hope of it waking up with the needed goalstate(s) received and applied.

This leads to, in many cases, a long time for the first packet to go out(around 1000ms). In some cases, when only sending 1 packet to the destination, the packet got timed out eventually; in other lucky case, all first packets were sent out in less than 2 ms (futurewei-cloud/alcor#560 (comment)).

I believe this highly depends on the connection between ACAs and the NCM, and how fast NCM sends back data. If it is fast enough, then on-demand engine wakes up with needed goalstates, it sends out the packet really quickly; otherwise, it is slow.

Possible solutions:

Change the reply message from the NCM, to include the goalstates needed, but it will break out intent to let only one thread to take care of the goalstates.

Find out a way to let the on-deand engine know that the goalstate has been received and applied, and it can go ahead and let the packet out.

@zzxgzgz This is very good thinking. You've made quite a bit of optimization in this PR, this is very positive way of approaching an optimized solution.

Also, for any planed items or ideas you plan to do, you could create issue(s) in this repo so that you could continue tracking it.

Thanks!

zzxgzgz added 10 commits April 1, 2021 17:11

initial comment for the grpc client async call change

8b04635

Added initial implementation for the async grpc client call

8f6756c

initial comment for the grpc client async call change

2cb3615

Added initial implementation for the async grpc client call

143561d

trying to keep up with upstream master's code

be99522

Modifed unit test code so that it is async.

8722296

Fixed weird request ID problem in grpc async client call test

09b9249

Added logs, and modified code in the effort of fixing the core dumped

07ac23a

Try to fix pointer problem

5f24657

Try to fix pointer problem

317f91e

zzxgzgz self-assigned this Apr 20, 2021

zzxgzgz added the enhancement New feature or request label Apr 20, 2021

zzxgzgz added 9 commits April 20, 2021 20:24

Added more code for a more proper exit of ACA

c2ac8ba

try to fix core dumped caused by double free

e950320

try to improve speed by using normal unordered_map

63800eb

try to improve speed by diabling usleep for on-demand code

4ea3ee4

Added time to measure which call takes longest time

8e5ee58

Added time to measure which call takes longest time

6890d34

Added more time to sleep

6c7ec20

Adjusted where on-demand engine sleeps

24cfc0c

Reverted back how much time on-damand engine sleeps

b0eb021

Removed unused/commented code; added assert in DISABLED_grpc_client_c…

6478223

…onnectivity_test

zzxgzgz changed the title ~~[DO NOT MERGE] Grpc Async Client Call~~ Grpc Async Client Call Apr 26, 2021

zzxgzgz requested review from cj-chung and xieus April 26, 2021 23:37

zzxgzgz added 2 commits April 27, 2021 18:29

Try to let on-demand engine wait until goalstate is delivered and pro…

6b6c82e

…grammed

Try to modify the hashmap used, and the way to wait for the goalstate

790b5ea

zzxgzgz added 3 commits April 29, 2021 18:16

Changed the size of the copied void pointer to the size of packet

6ce9cbd

Try to implement a smarter sleep

7e2a039

Removed unneeded code

9eaa0be

zzxgzgz added 3 commits April 30, 2021 16:21

Added comment for the packet void pointer

e49179f

Revert aca_arp_responder's map to CTSL:HashMap

b41b804

Added back the missing !

b9cd70a

cj-chung suggested changes May 3, 2021

View reviewed changes

include/aca_on_demand_engine.h Outdated Show resolved Hide resolved

include/aca_on_demand_engine.h Outdated Show resolved Hide resolved

include/aca_on_demand_engine.h Outdated Show resolved Hide resolved

cj-chung reviewed May 3, 2021

View reviewed changes

src/on_demand/aca_on_demand_engine.cpp Outdated Show resolved Hide resolved

src/on_demand/aca_on_demand_engine.cpp Outdated Show resolved Hide resolved

cj-chung reviewed May 3, 2021

View reviewed changes

zzxgzgz added 6 commits May 3, 2021 21:34

Address comments in the PR; added the 3rd thread in ACA_ON_DEMAND_ENG…

00793c0

…INE to handle cleaning expired entries

Trying to find out the reason of crashing

73325ea

Trying to find out the reason of crashing

4c1639c

Trying to find out the reason of crashing

597dba0

Try to move back to unordered map

e97bc4d

Added comment && removed unused code

a598170

zzxgzgz requested a review from cj-chung May 3, 2021 23:52

cj-chung reviewed May 4, 2021

View reviewed changes

include/aca_config.h Outdated Show resolved Hide resolved

cj-chung reviewed May 4, 2021

View reviewed changes

include/aca_on_demand_engine.h Outdated Show resolved Hide resolved

zzxgzgz added 2 commits May 4, 2021 18:15

Adjected lock when cleaning up the map; added max size for the map; a…

6b1824a

…dded time measurement for map writes

Adjusted variables' names

3b968a4

zzxgzgz requested a review from cj-chung May 4, 2021 18:33

Adjusted variables' names

3780af2

zzxgzgz added 2 commits May 7, 2021 20:58

Refactored on_demand function, in order to measure time for the whole…

d510d94

… gs sent->received&&programmed operation

Added comment for use of unordered_map in on-demand engine

26d93f5

cj-chung approved these changes May 10, 2021

View reviewed changes

xieus merged commit 3e13ccd into futurewei-cloud:master May 10, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Grpc Async Client Call #241

Grpc Async Client Call #241

zzxgzgz commented Apr 19, 2021

zzxgzgz commented Apr 20, 2021

zzxgzgz commented Apr 26, 2021

zzxgzgz commented Apr 26, 2021

zzxgzgz commented Apr 29, 2021

cj-chung May 3, 2021

zzxgzgz May 3, 2021

cj-chung May 4, 2021

zzxgzgz May 4, 2021

cj-chung May 3, 2021

zzxgzgz May 3, 2021

zzxgzgz May 3, 2021

zzxgzgz commented May 4, 2021

zzxgzgz commented May 7, 2021

cj-chung left a comment

xieus commented May 10, 2021

Grpc Async Client Call #241

Grpc Async Client Call #241

Conversation

zzxgzgz commented Apr 19, 2021

zzxgzgz commented Apr 20, 2021

zzxgzgz commented Apr 26, 2021

zzxgzgz commented Apr 26, 2021

zzxgzgz commented Apr 29, 2021

cj-chung May 3, 2021

Choose a reason for hiding this comment

zzxgzgz May 3, 2021

Choose a reason for hiding this comment

cj-chung May 4, 2021

Choose a reason for hiding this comment

zzxgzgz May 4, 2021

Choose a reason for hiding this comment

cj-chung May 3, 2021

Choose a reason for hiding this comment

zzxgzgz May 3, 2021

Choose a reason for hiding this comment

zzxgzgz May 3, 2021

Choose a reason for hiding this comment

zzxgzgz commented May 4, 2021

zzxgzgz commented May 7, 2021

cj-chung left a comment

Choose a reason for hiding this comment

xieus commented May 10, 2021