[CI] failure WatcherRestIT yaml=painless/20_minimal_body/Test execute watch api with minimal body

[mayya-sharipova]

Test failed on master:

./gradlew :x-pack:qa:smoke-test-watcher:integTestRunner --tests "org.elasticsearch.smoketest.WatcherRestIT.test {yaml=painless/20_minimal_body/Test execute watch api with minimal body}" \
  -Dtests.seed=BA6FF797971F1689 \
  -Dtests.security.manager=true \
  -Dtests.locale=naq \
  -Dtests.timezone=Africa/Ndjamena \
  -Dcompiler.java=12 \
  -Druntime.java=11

[2019-07-04T16:56:45,842][INFO ][o.e.s.WatcherRestIT      ] [test] [yaml=painless/20_minimal_body/Test execute watch api with minimal body] before test
[2019-07-04T16:57:44,770][INFO ][o.e.s.WatcherRestIT      ] [test] Stash dump on test failure [{
  "stash" : {
    "body" : {
      "_id" : "my_logging_watch_18583bc1-b462-4db0-909a-49a98420fd91-2019-07-04T15:57:16.316845Z",
      "watch_record" : {
        "watch_id" : "my_logging_watch",
        "node" : "KX66h-VtS5KCzbPplYWwDA",
        "state" : "not_executed_already_queued",
        "trigger_event" : {
          "type" : "manual",
          "triggered_time" : "2019-07-04T15:57:16.316Z",
          "manual" : {
            "schedule" : {
              "scheduled_time" : "2019-07-04T15:57:16.316Z"
            }
          }
        },
        "messages" : [
          "Watch is already queued in thread pool"
        ]
      }
    }
  }
}]

Log: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob+fast+part2/52/console
Build Scans: https://gradle.com/s/kftq7vpb5p36s

Doesn't reproduce locally for me. Could be simply slow test due the network.

[elasticmachine]

Pinging @elastic/es-core-features

[davidkyle]

Another failure

java.lang.AssertionError: Failure at [painless/20_minimal_body:41]: watch_record.state didn't match expected value:
           watch_record.state: expected String [executed] but was String [not_executed_already_queued]

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob+fast+part2/488/console

I grabbed the log file
integTest.log

0   1>   "stash" : {
13:44:40   1>     "body" : {
13:44:40   1>       "_id" : "my_logging_watch_25d32db9-0de5-44c8-b92c-3f258f62c9e7-2019-07-30T12:44:08.951835Z",
13:44:40   1>       "watch_record" : {
13:44:40   1>         "watch_id" : "my_logging_watch",
13:44:40   1>         "node" : "ZdW-FukyT-2dXpGjoAnWWw",
13:44:40   1>         "state" : "not_executed_already_queued",
13:44:40   1>         "trigger_event" : {
13:44:40   1>           "type" : "manual",
13:44:40   1>           "triggered_time" : "2019-07-30T12:44:08.951Z",
13:44:40   1>           "manual" : {
13:44:40   1>             "schedule" : {
13:44:40   1>               "scheduled_time" : "2019-07-30T12:44:08.951Z"
13:44:40   1>             }
13:44:40   1>           }
13:44:40   1>         },
13:44:40   1>         "messages" : [
13:44:40   1>           "Watch is already queued in thread pool"
13:44:40   1>         ]
13:44:40   1>       }
13:44:40   1>     }
13:44:40   1>   }

[davidkyle]

Muted on master 8ab1c9b and 7.x d0cbf0c