Different types of ovs-dpctl source available when execute test separately with one-shot and binary-search mode

[pradiptapks]

Hi @mrbojangles3 @atheurer ,

I could see there are two types of ovs-dpctl source types when I execute the test separately with one-shot and binary-search mode. Is it a bug? If not then could you please share the reason behind it?

[root@crucible tool-ovs]# git remote -v
origin  https://github.com/perftool-incubator/tool-ovs (fetch)
origin  https://github.com/perftool-incubator/tool-ovs (push)
[root@crucible tool-ovs]# git branch
* master

One-shot mode results for ovs-dpctl source

run-id: 1cdc01e6-0819-4514-94e5-53d84fe0d030
  tags: CQE_COMPRESSION=BALANCED DUT=perf196.perf.lab.eng.bos.redhat.com DUTKernel=5.14.0-283.el9.x86_64 GuestDPDK=22.11.2_1Q GuestOS=RHEL9.2-kernel-5.14.0-284.11.1.el9_2.x86_64 MaxlossPct=0.002 MlxFw=20.34.1002_MT_0000000224 OpenvSwitch=3.1.0-23 RHOSP=RHOS-17.1-RHEL-9-20230511.n.1 Tgen=perf122.perf.lab.eng.bos.redhat.com Tgen=bench-trafficgen-8x25G-XXV710 Topo=HWOL-PVP-2x100G-Mlx-Cx6 Tuned=cpu-partitioning VM=192.168.0.102 
  benchmark: trafficgen
  common params: active-devices=0000:06:00.0,0000:06:00.1,0000:83:00.0,0000:83:00.1,0000:08:00.0,0000:08:00.1,0000:85:00.0,0000:85:00.1 compress-files=ON devices=0000:05:00.0,0000:06:00.0 devices=0000:06:00.0,0000:06:00.1,0000:83:00.0,0000:83:00.1,0000:08:00.0,0000:08:00.1,0000:85:00.0,0000:85:00.1 frame-size=64 max-loss-pct=0.002 measure-latency=1 num-flows=1024 one-shot=1 process-all-profiler-data=ON rate-tolerance-failure=fail rate-unit=% rate=100 result-output=none search-runtime=60 send-teaching-measurement=ON send-teaching-warmup=ON switch-type=testpmd teaching-measurement-packet-type=generic teaching-warmup-packet-type=generic testpmd-burst=64 testpmd-descriptors=2048 testpmd-enable-rss-udp=on testpmd-forward-mode=io testpmd-mtu=9000 testpmd-queues-per-pmd=1 testpmd-queues=1 testpmd-smt=on traffic-direction=bidirectional traffic-generator=trex-txrx use-dst-ip-flows=1 use-dst-mac-flows=0 use-dst-port-flows=0 use-protocol-flows=0 use-src-ip-flows=1 use-src-mac-flows=0 use-src-port-flows=0 validation-runtime=120 warmup-trial=ON 
  metrics:
    source: procstat
      types: interrupts-sec 
    source: mpstat
      types: Busy-CPU NonBusy-CPU 
    source: trafficgen
      types: l1-rx-bps l1-tx-bps l2-rx-bps l2-tx-bps lost-rx-pps max-roundtrip-usec mean-roundtrip-usec rx-pps tx-pps trial-result 
    source: sar-net
      types: L2-Gbps packets-sec errors-sec 
    source: ovs-dpctl
      types: ufid-Gbps ufid-packets-sec ct-stats-show lookups-sec masks-sec ufid-expired-flows-sec ufid-new-flows-sec 
    source: ovs-ofctl
      types: Gbps packets-sec 
    source: sar-scheduler
      types: Load-Average-01m Load-Average-05m Load-Average-15m Process-List-Size Run-Queue-Length IO-Blocked-Tasks 
    source: sar-mem
      types: Page-faults-sec KB-Paged-in-sec KB-Paged-out-sec Pages-freed-sec 
    source: ovs-appctl
      types: mem-show 
    source: sar-tasks
      types: Context-switches-sec Processes-created-sec 
    iteration-id: 7E1936C2-10BA-11EE-8641-12E9FCD178AA
      unique params: 
      primary-period name: measurement
      samples:
        sample-id: 7EA33D40-10BA-11EE-B4B6-12E9FCD178AA
          primary period-id: 7ED77614-10BA-11EE-B4E2-12E9FCD178AA
          period range: begin: 1687410132756 end: 1687410258842
            result: (trafficgen::rx-pps) samples: 38923053.880826 mean: 38923053.880826 min: 38923053.880826 max: 38923053.880826 stddev: NaN stddevpct: NaN

binary-search mode results for ovs-dpctl source

run-id: 8291a5c3-740d-4ec8-a437-81a9c02265be
  tags: CQE_COMPRESSION=AGGRESSIVE DUT=perf196.perf.lab.eng.bos.redhat.com DUTKernel=5.14.0-283.el9.x86_64 GuestDPDK=22.11.2_1Q GuestOS=RHEL9.2-kernel-5.14.0-284.11.1.el9_2.x86_64 MaxlossPct=0.002 MlxFw=20.34.1002_MT_0000000224 OpenvSwitch=3.1.0-23 RHOSP=RHOS-17.1-RHEL-9-20230511.n.1 Tgen=perf122.perf.lab.eng.bos.redhat.com Tgen=bench-trafficgen-8x25G-XXV710 Topo=HWOL-PVP-2x100G-Mlx-Cx6 Tuned=cpu-partitioning VM=192.168.0.102 
  benchmark: trafficgen
  common params: active-devices=0000:06:00.0,0000:06:00.1,0000:83:00.0,0000:83:00.1,0000:08:00.0,0000:08:00.1,0000:85:00.0,0000:85:00.1 compress-files=ON devices=0000:05:00.0,0000:06:00.0 devices=0000:06:00.0,0000:06:00.1,0000:83:00.0,0000:83:00.1,0000:08:00.0,0000:08:00.1,0000:85:00.0,0000:85:00.1 max-loss-pct=0.002 measure-latency=1 num-flows=1024 one-shot=0 process-all-profiler-data=ON rate-tolerance-failure=fail rate-unit=% rate=100 result-output=none search-runtime=60 send-teaching-measurement=ON send-teaching-warmup=ON switch-type=testpmd teaching-measurement-packet-type=generic teaching-warmup-packet-type=generic testpmd-burst=64 testpmd-descriptors=2048 testpmd-enable-rss-udp=on testpmd-forward-mode=io testpmd-mtu=9000 testpmd-queues-per-pmd=1 testpmd-queues=1 testpmd-smt=on traffic-direction=bidirectional traffic-generator=trex-txrx use-dst-ip-flows=1 use-dst-mac-flows=0 use-dst-port-flows=0 use-protocol-flows=0 use-src-ip-flows=1 use-src-mac-flows=0 use-src-port-flows=0 validation-runtime=300 warmup-trial=ON 
  metrics:
    source: trafficgen
      types: l1-rx-bps l1-tx-bps l2-rx-bps l2-tx-bps max-roundtrip-usec mean-roundtrip-usec rx-pps tx-pps lost-rx-pps trial-result 
    source: procstat
      types: interrupts-sec 
    source: mpstat
      types: Busy-CPU NonBusy-CPU 
    source: sar-net
      types: L2-Gbps packets-sec errors-sec 
    source: ovs-ofctl
      types: Gbps packets-sec 
    source: sar-scheduler
      types: IO-Blocked-Tasks Load-Average-01m Load-Average-05m Load-Average-15m Process-List-Size Run-Queue-Length 
    source: sar-mem
      types: Page-faults-sec KB-Paged-in-sec KB-Paged-out-sec Pages-freed-sec 
    source: ovs-dpctl
      types: ct-stats-show masks-sec flows-count lookups-sec 
    source: ovs-appctl
      types: mem-show 
    source: sar-tasks
      types: Context-switches-sec Processes-created-sec 
    iteration-id: 657E806E-109B-11EE-98DA-1B4FAA1B5805
      unique params: frame-size=64 
      primary-period name: measurement
      samples:
        sample-id: 7FB37110-109B-11EE-93B2-1B4FAA1B5805
          primary period-id: 84328456-109B-11EE-93B2-1B4FAA1B5805
          period range: begin: 1687394358050 end: 1687394658104
        sample-id: 67D41C34-109B-11EE-9A16-1B4FAA1B5805
          primary period-id: 7565FA02-109B-11EE-93B2-1B4FAA1B5805
          period range: begin: 1687379761597 end: 1687380061650
        sample-id: 758C6E80-109B-11EE-93B2-1B4FAA1B5805
          primary period-id: 7F8D55C0-109B-11EE-93B2-1B4FAA1B5805
          period range: begin: 1687391607400 end: 1687391907449
            result: (trafficgen::rx-pps) samples: 33980551.419308 33961214.546762 34241731.485681 mean: 34061165.817250 min: 33961214.546762 max: 34241731.485681 stddev: 156673.064443 stddevpct: 0.459976

[pradiptapks]

None of flows-count has reflect as the data collected properly during the test.

# crucible get metric --run 8291a5c3-740d-4ec8-a437-81a9c02265be --period 84328456-109B-11EE-93B2-1B4FAA1B5805 --source ovs-dpctl --type flows-count
Checking for httpd...appears to be running
Checking for elasticsearch...appears to be running
{
  "values": {
    "": [
      {
        "begin": 1687394358050,
        "end": 1687394658104,
        "value": 0
      }
    ]
  },
  "usedBreakouts": [],
  "valueSeriesLabelDecoder": "",
  "remainingBreakouts": [
    "counter",
    "csid",
    "cstype",
    "interface"
  ]
}

[pradiptapks]

logs:

1. http://storage.scalelab.redhat.com/psahoo/PerfTaskLog/OSP17.1/nfv_hwol/trafficgen--2023-06-21_11%3A41%3A28_UTC--8291a5c3-740d-4ec8-a437-81a9c02265be.tar.xz
2. http://storage.scalelab.redhat.com/psahoo/PerfTaskLog/OSP17.1/nfv_hwol/trafficgen--2023-06-22_04%3A56%3A44_UTC--1cdc01e6-0819-4514-94e5-53d84fe0d030.tar.xz

[atheurer]

We will need to see the output from post-processing ovs tools. The file is in /run/tool-data/<client|server|profiler>//ovs/post-process-output.txt