[ci] 'FTR Configs' check title => Functional tests

[mattkime]

As part of some recent ci improvements, the github check title for functional tests has become FTR Configs. This is rather confusing. It would more accurate to label it 'Functional tests' to avoid jargon and better match where the tests are located in the repo.

I thiiiink this is a side effect of #130983

[elasticmachine]

Pinging @elastic/kibana-operations (Team:Operations)

[spalger]

This is a side effect of my PR and the decision was made intentionally because these jobs are just executing FTR Configs, which might be running functional tests, api integration tests, or other config types which are just collecting metrics from the Kibana build and not even testing.

[mattkime]

@spalger I'm disappointed by this. While it is important to run tests in an efficient manner, I would hope that it wouldn't come at the expense of clear and concise communication. Previously I had some insight into what was running or completed, now it takes too much effort to bother with.

It would be nice if separation between functional tests, api integration tests, and other similar divisions could be restored.

---

Its likely possible to separate the github checks reporting from the individual process execution.

[spalger]

Yeah, efficiency is getting us closer to each config being it's own job on CI. This will give us individualized logs and statuses for each config but we need to be a bit more efficient before it makes sense to go that route.

Previously I had some insight into what was running or completed, now it takes too much effort to bother with.

I'm sorry to hear that, but combining the configs into a single pool saves a ton of time and money and having a clear understanding of what's running right now and what is already completed doesn't feel like it even starts to compare to the value we're getting out of this change. Can you give me an example of a situation where this information was/would have been helpful? We can look at features which enable this if we can tell that story that shows it's value.

Right now our focus is getting results delivered in well under an hour, hoping for something like 45 minutes soon-ish.

[mattkime]

Yeah, efficiency is getting us closer to each config being it's own job on CI. This will give us individualized logs and statuses for each config but we need to be a bit more efficient before it makes sense to go that route.

Is that a goal thats actively being worked toward?

understanding of what's running right now and what is already completed doesn't feel like it even starts to compare to the value we're getting out of this change.

I don't think those should be viewed as mutually exclusive choices.

Can you give me an example of a situation where this information was/would have been helpful?

I know my test in suite x failed last time, so if that suite passes this time I know that my PR is likely to fully pass and I can change my focus to something else before the full test suite passes.

[spalger]

Is that a goal thats actively being worked toward?

We see the value of it but we don't know if we'll be able to do it without wasting a lot of resources. We want to and making CI as efficient as possible gives us a little more slack to do things like this just because they're nice to have.

I know my test in suite x failed last time, so if that suite passes this time I know that my PR is likely to fully pass and I can change my focus to something else before the full test suite passes.

Okay, yeah, that makes sense. Let's discuss addressing that here: #131879

Something else to consider if you're specifically interested in the results of a specifically tricky config or you're iterating with CI: Try commenting out the majority of the config paths in .buildkite/ftr_configs.yml under enabled: and only running the configs you care about the most. Then, once those pass you can uncomment things and do a full run.

[jloleysens]

Thanks for explaining @spalger ! The primary point of confusion for me was seeing the FTR Config #x is failing message. From the CI user perspective I guessed that had nothing to do with me but realised over time that it was a functional test that was failing I needed to address. Something like "Functional test batch #x" may be slightly clearer?

I get this is is not the end-state for CI so perhaps nothing to really action for now, just wanted to share that experience here too.

Something else to consider if you're specifically interested in the results of a specifically tricky...

Really useful tip!