Default MatchFunction configuration for Backend

[TBBle]

Is your feature request related to a problem? Please describe.

For backfill/join-in-progress support, as well as out-of-cluster development and testing, our Arena Servers also function as Directors.

However, this means we need to distribute the match-function connection details (we only have MatchFunction) to these Arena Servers, and the Arena Servers may not have been deployed by the Helm chart that deployed the MatchFunction, so we don't get auto-populated configuration the way we do for other components like the Backend or Frontend.

Describe the solution you'd like

I'd like to be able to specify a default FunctionConfig in the Backend's configuration, if the FetchMatches call does not provide one.

I imagine this would look just like the evaluator configuration, which already appears in the config.

Describe alternatives you've considered

We're considering having a service in our cluster that just returns the MatchFunction Service address for Arena Servers to query. It seems a super-wasteful thing to do.

Additional context

This also makes the fact that FunctionConfig is optional in the API rational. Right now, leaving that out will just generate a failure, because the code is treating it as required.

That said, a quick check suggests that right now the generated OpenAPI spec is failing to mark any object fields as required, when they are required in the code. I'm not sure if that's a problem in the source API declarations, or the GRPC->OpenAPI conversion.

[Laremere]

We don't recommend using match function invocations per server for backfill (I know it was the suggestion at one point, but we've since changed it).
We have a separate issue for supporting backfill: #1240 It's biggest request from the community, and we're currently requirement gathering, so now is a great time to contribute your requirements. I have a todo item to explain why per server match function invocation is not what we're recommending.

Can you explain more about your out-of-cluster development and testing scenarios?

I'm not against this idea - it's fairly easy to implement - just want to make sure we're doing it for the right reasons.

[TBBle]

The Out-of-Cluster Arena workflow in our case is this:

We have a Director (or Directors, for multiple clusters) which wraps an Agones instance.

However, developers often need to run a copy of their UE4 Arena Server on their desktop -- an Out of Cluster (OoC) Arena Server, and have a match run on it instead of on an Agones-allocated Server in the cluster.

Previously to moving to Open Match, all such OoC Arena Servers, and the Agones-wrapping service, would register with a single service (Match Distributor) which was responsible for allocating matches, preferencing OoC Arena Servers when present.

We used this same mechanism for supporting UE4's FindSession/JoinSession flow to implement Game Rooms or Private on-Arena Lobbies, by turning JoinSession into a matchmaking request with a specific Server Id (which would be allocated by the running Arena Server and available to clients calling FindSession) and specific "Accept All" match-type, and then Match Distributor would pass it directly to the relevant Arena Server.

We hadn't implemented Backfill for general match types (as far as I know) but the intent was to have that work similarly, by having the Arena Server register as a Match Distributor, but in this case, not using the magic pass-through match-type, but using the regular match-type (e.g. 1v1, 5v5, 1v5, 8v0 PvE) from which its current match was formed, along with informing our Match Maker of the current match population, so it could backfill appropriately. So we're treating the Game Rooms system as a kind of degenerate-Backfill, which lets use keep the data-flow and control-flow complexity down.

We moved to Open Match in part because we wanted to remove that singleton service, the Match Distributor. Open Match inverts the control flow compared to our earlier approach, so the only way to get a match to a running Arena is to have that Arena act as a Director, since the fetchMatches call to find JoinSession callers has a unique SearchFilter, so we cannot usefully aggregate this call.

Since we needed this to work anyway, we saw this as an appropriate mechanism for backfilling matches, by leaning on a custom Evaluator implementation to deal with preferring backfills over new matches when that occurred.

There's not really any alternatives in the current Open Match structure we could see, apart from reintroducing a Match Distributor, which would be a singleton Director which is aggregating all the match making needs from Agones, OoC Arena Servers waiting to be assigned a match, Game Room Arena Servers, and Arena Servers running Backfill for their match-type. We are considering doing exactly that, perhaps using a MetaDirector, that exposes the om-backend API so that the various things in the previous sentence still only need to speak one protocol whether the MetaDirector is present or not. The MetaDirector collates those requests and produces the appropriate set of fetchMatches requests (in almost all cases this would be 1:1, so no difference is seen on the Open Match side) but is also able to act on the list of known Directors for other system logic actions.

This also gives us an opportunity to make the FindSession/JoinSession flow use a different MMF, rather than our current MMF with a special match-type of "Accept All". This means that, unlike when I opened this ticket, we no longer have a single MMF in our system. The second MMF is a specific use-case, so a default MMF (this feature request) for the majority use-case is still desirable, and reduces configuration load on our Agones-wrapping service, as it's only backfilling arena servers that need to use the special MMF.

We're still working on how we can preference OoC Arena Servers when present in this flow, or even balance preferences between multiple Agones instances. That's the main driver behind the idea of a MetaDirector, because Open Match's world-view seems to best-fit a single-Director approach. (There's even parts of the docs that assume One True Director, such as the Evaluator/Synchroniser documentation).

Looking at #1240 (comment), which was posted while I was writing this up, I think our system would be much easier to implement if instead of FetchMatches, we had a Backend Tickets concept similar to how player are added. Then Open Match would be able to do (via the MMF and Evaluator) more-interesting match distribution decisions amongst Directors, the way it does for Players. I like the idea of "Open Match then matches demand with supply" as a general approach.

[TBBle]

As we've worked through the workflows locally, I want to add/amend to this feature request: It'd be really handy if the om-backend API could identify MatchFunctions by alias, so that when using multiple MatchFunctions, the Directors don't need to know the full k8s Service DNS name, but an alias, and the om-backend can map those aliases to DNS entries.

Specifically, this means in multiple instances of a project, the Directors only need to find their om-backend, and can then refer to the MMFs by a project-specific name, they don't also need to know the deployment-specific parts of the naming convention.

The simple alias idea would require that the om-backend knows about the om-functions by either registration or configuration. If this is undesirable, a way to configure a mapping from om-function alias to om-function Service DNS hostname would work too, as that configuration can be done by deployment systems, e.g., Helm, and then we just need to deploy our MMFs with Service DNS names that match the pattern.

[joeholley]

I know this one has been open for a long time without response, but we do have plans to do another pass on how MMFs are reference dand if 'registering' them with OM core would work well (which requires us to think through a few different use cases). We'll probably try to address this in a future version.