[FEATURE]: extract line number in metrics declaration

[shahzebsiddiqui]

Please describe your feature

Buildtest should have ability to extract line from stdout/stderr that can be assigned to a metric which can be used later for status check. For instance let's take the following results from OSU bandwidth test result and we want to extract a particular line

# OSU MPI Bandwidth Test v7.3                                                                                                                                                                                                                                              
# Size      Bandwidth (MB/s)                                                                                                                                                                                                                                               
# Datatype: MPI_CHAR.                                                                                                                                                                                                                                                      
1                       5.65                                                                                                                                                                                                                                               
2                       3.57                                                                                                                                                                                                                                               
4                       9.26                                                                                                                                                                                                                                               
8                      33.72                                                                                                                                                                                                                                               
16                     20.76                                                                                                                                                                                                                                               
32                    161.53                                                                                                                                                                                                                                               
64                    133.98                                                                                                                                                                                                                                               
128                   705.10                                                                                                                                                                                                                                               
256                   534.20                                                                                                                                                                                                                                               
512                   708.01                                                                                                                                                                                                                                               
1024                  390.97                                                                                                                                                                                                                                               
2048                  899.16                                                                                                                                                                                                                                               
4096                 5240.26                                                                                                                                                                                                                                               
8192                 3275.13                                                                                                                                                                                                                                               
16384                4527.77                                                                                                                                                                                                                                               
32768                6407.88                                                                                                                                                                                                                                               
65536                9635.86                                                                                                                                                                                                                                               
131072              15361.67                                                                                                                                                                                                                                               
262144              10873.81                                                                                                                                                                                                                                               
524288              18213.85                                                                                                                                                                                                                                               
1048576             10275.43                                                                                                                                                                                                                                               
2097152             11507.50                                                                                                                                                                                                                                               
4194304              4837.92     

At the moment we have support for regex in metrics that can be used as comparison which works fine.

buildspecs:
  osu_bandwidth_perf:
    type: script
    executor: generic.local.bash
    description: Run OSU Bandwidth Performance Test
    run:  mpirun -np 2 osu_bw
    metrics:
      osu_bw:
        type: float
        regex:
          exp: '^16384\s+([\d.]+)$'
          stream: stdout
          item: 1
    status:
      assert_ge:
        comparisons:
          - name: osu_bw
            ref: 10000

This regex is applied on entire content of file and not on a particular line.

For this feature, we would like to add a key something like linenum that will be used to extract a particular line where regex is applied. To extend this example we can simply do the following

buildspecs:
  osu_bandwidth_perf:
    type: script
    executor: generic.local.bash
    description: Run OSU Bandwidth Performance Test
    run:  mpirun -np 2 osu_bw
    metrics:
      osu_bw:
        type: float
        regex:
          exp: '^16384\s+([\d.]+)$'
          stream: stdout
          linenum: 10
          item: 1
    status:
      assert_ge:
        comparisons:
          - name: osu_bw
            ref: 10000

The linenum must be an integer and it can be any value including negative values. The idea is that we simply extract the line from file and apply regex on the specific line. linenum: -1 would mean last line of file and linenum: 0 would be first line of file.

Suggest potential solution

No response

Additional Information

No response

Post question in Slack

• I agree that I posted my question in slack before creating this issue

Is there an existing issue

• I confirm there is no existing issue for this issue

[Xiangs18]

Is there a particular reason you repeat this line?:

buildtest/buildtest/builders/base.py

Line 947 in 44e9da7

self.metadata["metrics"][key] = ""

[Xiangs18]

@shahzebsiddiqui
Could you please remind me what's the command to test osu_bandwidth_test.yml file.
I doubt exp: '^16384\s+([\d.]+)$' will extract anything ...

[shahzebsiddiqui]

@Xiangs18 that line should be removed now that I see it is redundant. The metrics are initialized as quotes as their initial value. I think that was the case because we didn't want None as the value due to the way they are written into the report file.

[shahzebsiddiqui]

@Xiangs18 https://github.com/buildtesters/buildtest/blob/devel/aws_tutorial/osu_bandwidth_test.yml is the test we have but it works in aws image so u won't be able to run it locally the osu microbenchmark can be installed locally but also requires MPI.

This is typically provided by mvapich package.

For this PR u can basically simulate this by simply echo this output into file and try to assign a metric value.

[shahzebsiddiqui]

You might find it helpful seeing this example using re property https://buildtest.readthedocs.io/en/devel/writing_buildspecs/status_check.html#using-re-property-to-specify-regular-expression-type

By default we use re.search which is gonna let u search anywhere in string. The re.match will only search pattern at beginning of string whereas re.fullmatch must match entire string. See the example and it will make more sense