[FLINK-36460] Refactoring the CI matrix

[SamBarker]

What is the purpose of the change

As observed in FLINK-36332 the CI workflow has become difficult to mange. This PR is attempting to split tests along sensible lines to make the matrix more manageable.

To do so this PR splits the edge to edge workflow in two. A common workflow definition: e2e.yaml which is then reused from multiple different test matrices.

I think this split allows better control over the variables without having to vary each dimension every time. By making the simplifying assumption that variations in the Java version are adequately tested by the maven builds and the smoke tests it effectively reduces the number of test jobs from ~225 to about 100 (TBC)

Brief change log

Simplify CI workflows.

Verifying this change

This change is a trivial rework / code cleanup without any test coverage.

Does this pull request potentially affect one of the following parts:

• Dependencies (does it add or upgrade a dependency): no
• The public API, i.e., is any changes to the CustomResourceDescriptors: no
• Core observer or reconciler logic that is regularly executed: no

Documentation

• Does this pull request introduce a new feature? no
• If yes, how is the feature documented? not applicable

[SamBarker]

@gyfora alternative CI matrix as requested. Even if the matrix proposed in this PR doesn't feel "right" it sets things up so that the matrix is considerably easier to change.

[SamBarker]

This shows it in action on my fork:

[gyfora]

Looks great @SamBarker !

[SamBarker]

I think I've finally got the excludes to align with main. sigh That was a lot more fuss than I wanted and doesn't end up as clean as I was hoping. However I think this represents a much cleaner way to setup test matrices.

The tests are still very slow to run as each test has to build its own copies of the containers. I think thats resolvable without publishing temporary artefacts to a container registry. Some combination of docker advice and GitHub sharing data docs along with this discussion.

That feels like a separate PR.

@gyfora could you trigger another run of the PR please (its gone green on my fork)

[mxm]

Cool! This looks a lot cleaner.