Do not send clusterReceiver metrics through gateway

[dmitryax]

Reasoning for this change:

• Currently platform metrics are not being sent through the gateway even if the gateway is enabled. K8s event collection is broken if the gateway is enabled (k8s events collection is broken if gateway is enabled #490). It has to be consistent: send everything from cluster receiver through gateway or send everything directly to the backends.
• Sending telemetry though the gateway from one pod doesn't look like providing any real value except for additional overhead:
  • If gateway is assigned on some special nodes that have internet connection, the same way cluster receiver can be assigned to the same nodes.
  • Cluster receiver has to talk a lot to k8s API in k8scluster receiver, that part cannot be moved to the gateway as it's done for for agents. Gateway also talks a lot to k8s API, so it makes sense to collocate them together.

[rmfitzpatrick]

Currently platform metrics are not being sent through the gateway even if the gateway is enabled. K8s event collection is broken if the gateway is enabled (#490).

What changes are required to resolve these issues while still aiming to report to the gateway?

[dmitryax]

What changes are required to resolve these issues while still aiming to report to the gateway?

Not much, I'm just proposing not to do that to avoid additional complexity and computational overhead. Please let me if you think there are reasons to send it through gateway that I miss?

[rmfitzpatrick]

If gateway is assigned on some special nodes that have internet connection, the same way cluster receiver can be assigned to the same nodes.

Mainly to avoid this new requirement for isolated use cases where potentially not necessary.

[dmitryax]

The main reason to run gateway is to reduce the load on K8s API from each agent. This is not applicable if we send data from cluster receiver to the gateway.

Another reason for the gateway, that I believe is not very popular, is to move it to specific nodes that have internet access. This may justify sending the data from the cluster receiver through the gateway. But I don't think it can justify additional complexity and computational overhead. Big clusters can require pretty big cluster receiver pod, and all the resources allocated for the pod are likely need to be added to the gateway as well to process the data additionally, especially for doubled signalfx<->otlp translation (we cannot use otlp for metrics between the cluster receiver and gateway). So I think adding couple lines of nodeSelector config to clusterReceiver section is a better option.

[dmitryax]

IDK, it feels like waste of resources and redundant complexity. But I'm ok with sending it all through the gateway if other folks think that's the right thing to do.

cc @signalfx/gdi-data-collection