[BUG] mixed cluster bwc tests are failing for elasticsearch version v7.10.2

[adnapibar]

Describe the bug

The backwards compatibility tests for Elasticsearch version v7.10.2 are failing for the mixed-cluster tests. These tests verify that the current version is wire compatible with a BWC version.

Failing Tests:

MixedClusterClientYamlTestSuiteIT.test {p0=cat.health/10_basic/Empty cluster}
MixedClusterClientYamlTestSuiteIT.test {p0=cat.health/10_basic/Help}
MixedClusterClientYamlTestSuiteIT.test {p0=cat.health/10_basic/With ts parameter}

To Reproduce

./gradlew :qa:mixed-cluster:v7.10.2#mixedClusterTest

Expected behavior
The tests should pass.

Host/Environment (please complete the following information):

• OS: Amazon Linux 2
• Version 2

Additional context
Add any other context about the problem here.

[tlfeng]

I can reproduce the failure, though different 2 test failures out of 3 during my 2 test runs.

Some logs in my test runs:

test {p0=cat.health/10_basic/Empty cluster}

java.lang.AssertionError: Failure at [cat.health/10_basic:34]: field [$body] was expected to match the provided regex but didn't
Expected: ^
  ( \d+            \s+ # epoch
    \d\d:\d\d:\d\d \s+ # timestamp
    \S+            \s+ # cluster
    \w+            \s+ # status
    \d+            \s+ # node.total
    \d+            \s+ # node.data
    \w+            \s+ # discovered_master
    \d+            \s+ # shards
    \d+            \s+ # pri
    \d+            \s+ # relo
    \d+            \s+ # init
    \d+            \s+ # unassign
    \d+            \s+ # pending_tasks
    (-|\d+(?:[.]\d+)?m?s) \s+ # max task waiting time
    \d+\.\d+%             # active shards percent
    \n
  )+
$
     but: was "1625182044 23:27:24 v7.10.2 green 4 4 0 0 0 0 0 0 - 100.0%\n"
	at __randomizedtesting.SeedInfo.seed([64755F9C74EE4E2E:EC216046DA1223D6]:0)
	at org.opensearch.test.rest.yaml.OpenSearchClientYamlSuiteTestCase.executeSection(OpenSearchClientYamlSuiteTestCase.java:427)
...

test {p0=cat.health/10_basic/With ts parameter}

java.lang.AssertionError: Failure at [cat.health/10_basic:64]: field [$body] was expected to match the provided regex but didn't
Expected: ^
  ( \S+            \s+ # cluster
    \w+            \s+ # status
    \d+            \s+ # node.total
    \d+            \s+ # node.data
    \w+            \s+ # discovered_master
    \d+            \s+ # shards
    \d+            \s+ # pri
    \d+            \s+ # relo
    \d+            \s+ # init
    \d+            \s+ # unassign
    \d+            \s+ # pending_tasks
    (-|\d+(?:[.]\d+)?m?s) \s+ # max task waiting time
    \d+\.\d+%             # active shards percent
    \n
  )+
$
     but: was "v7.10.2 green 4 4 0 0 0 0 0 0 - 100.0%\n"
	at __randomizedtesting.SeedInfo.seed([64755F9C74EE4E2E:EC216046DA1223D6]:0)
	at org.opensearch.test.rest.yaml.OpenSearchClientYamlSuiteTestCase.executeSection(OpenSearchClientYamlSuiteTestCase.java:427)
...

[nknize]

This bug was introduced here in #656. The bwc cat.health test was unilaterally changed without bwc support. So a response is coming from a 7.10.2 node that doesn't include the discovered_master parameter and the regex fails. We need to be careful that we change the rest api spec tests so it correctly tests breaking changes or the bwc tests will sporadically fail. /cc @mgodwan