[data views] Allow data views created on hidden and system indices

[mattkime]

Summary

Previously, the 'Allow hidden and system indices' advanced option when creating a data view was only a UI convenience. It allowed you to see which hidden and system indices you were matching but they would be would be selected just the same once the data view was loaded. At some point something changed and now there are system and hidden indices that require expandWildcards: hidden to be passed to field caps in order to see anything. allowHidden: boolean is added to the DataView and DataViewSpec and passed through when field caps requests are made.

This is primarily a tool for troubleshooting. For instance, instead of hitting a full data stream across a number of data tiers you can select a specific index to compare its performance.

Closes: #164652

[elasticmachine]

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)

[davismcphee]

Code changes LGTM and it worked well when testing, although I encountered something strange when running the functional test. For some reason the test didn't seem to check the "Allow hidden" toggle, but it was still able to find the index and save the data view. I couldn't reproduce while testing though, hidden indices were only showing when the toggle was enabled:

func_test_2.mp4

The test also still succeeded for me when I changed the allowHidden param of PageObjects.settings.createIndexPattern to false, which didn't seem like it should work either.

I also noticed while I was testing that when editing, the advanced settings menu was collapsed, the "Allow hidden" toggle was off, and the index wasn't found in the sources, but I think maybe that's a known issue and unrelated to this PR?

Approving now since the approach looks good and it worked locally, but might be worth following up on the functional test thing to see if you can reproduce.

[Ikuni17]

limits.yml

[kibana-ci]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: 2598670

Failed CI Steps

• Osquery Cypress Tests #5
• Serverless Osquery Cypress Tests #3

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
data	2549	2554	+5
dataViews	257	262	+5
total			+10

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
dataViewEditor	47.7KB	47.8KB	+26.0B

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
dataViews	46.0KB	46.2KB	+197.0B

Unknown metric groups

API count

id	before	after	diff
data	3284	3290	+6
dataViews	1041	1048	+7
total			+13

History

• 💔 Build #167186 failed f03ebed
• 💔 Build #167179 failed 9369027
• 💛 Build #166808 was flaky 332a235
• 💛 Build #166497 was flaky d544631
• 💛 Build #166487 was flaky 3439c5d

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @mattkime

[sorenlouv]

@mattkime Would it be possible to backport this to 8.11.0?

[mattkime]

@sqren Unfortunately due to changes in data view persistence and field caps calls, I think it would be too risky