-
Notifications
You must be signed in to change notification settings - Fork 834
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Duplicate ports defined in seldon-container-engine container #1799
Comments
I believe this is the culprit. |
This was because this port serves not only |
That was my guess as well. About side-effects, haven't noticed any. Istio picks 2 application ports instead of one but no real effect. But this is not clean and confusing. |
My understanding was that this section is mostly informative + used by other systems configured to do so. This was more when I was looking at models nodes before I decided to add
but initially I was thinking about having
Maybe for
to
in order to make it clear at glance what purpose these ports serve. I need to check in Kubernetes documentation for a potential downsides and/or side-effects, @cliveseldon any thoughts on this? |
The name of the I'm also not sure of any requirements on this for k8s so would need to research. |
I think #1798 and this issue are somewhat related, in a contradictory manner. If we use separate ports for metrics (as is the case now) then the second port declaration can be leveraged (with a more suited name) for efficient prometheus scraping. I don't see another reason to declare the port twice. But if we want to keep it like this, then naming every port is definitely an improvement. The greater issue (that motivated all this from my part) is that there are multiple ports declared in the pod (for metrics and regular traffic). Seldon adds prometheus annotations for scraping ( At this point, I can think of 2 options:
cc: @axsaucedo |
That's exactly the reason for naming the port. There is this issue prometheus/prometheus#3756 in Prometheus that will hopefully lift the limitation. Pod-wise we will for sure have multiple |
@cliveseldon actually the name ( |
It seems that |
We should probably think of naming it. Though at present it could be This will be made simpler when we multiplex |
in the second option I outline, a call to pod:8000 could subsequently call localhost/6000 (, localhost/6001 etc) and and send all sets of metrics to prometheus. Here's an example implementation: exporter-merger |
@cliveseldon @axsaucedo now that one option is possible, I'm willing to give the alternative a try. I believe exposing a single metrics endpoint is the better option soI can create a PR for evaluation if you guys thing it's worth the while. |
Can you explain further what you mean by a single metrics endpoint? At present both the executor and each individual model in the inference graph exposes metrics on a port names |
There are indeed multiple ports exposed, for regular traffic (9000 and 5001) metrics/traffic (8000) and exclusively for metrics (6000, 6001 etc). Using pod annotations, we can only configure a single port per pod for metrics scraping; so with the current setup we cannot use this mechanism. The only viable solution is therefore to use the discovery feature of prometheus (i.e. probe every defined port). This ends up as noise on the other ports, as Prometheus tries to scrape all service ports. We can restrict this using regex on port names (e.g. metrics); the downside is that it requires specific configuration for Seldon Deployments on the Prometheus side. Another way would be to expose a single central endpoint for metrics collection (port 8000, /metrics). We can then define the pod annotation for that port and Prometheus will leave the other ones alone. We could do this either for each individual pod or for the entire SeldonDeployment. For this we need to build the necessary login into the executor; it would be configured to collect metrics from each model (or other nodes in the infrerence graph) and export them to Prometheus. This way we eliminate unwanted noise to the other ports and make seldon-specific configuration on Prometheus server unnecessary. In short, increased complexity in the seldon controller in favor of a simpler experience for the user, that's my proposal :) |
I'd prefer not to increase coupling between the executor and the graph components if at all possible. It sounds like you are proposing the executor works like prometheus and scrapes the graph components and then exposing it in 1 place. The only extra config for prometheus is the following: seldon-core/helm-charts/seldon-core-analytics/files/prometheus/prometheus-config.yaml Lines 97 to 99 in dc6e7ea
|
Indeed, this makes assumptions specific to the seldon setup and naming scheme. If we want to discover other pods that don't comply with this, we need another job. Additionally, using the annotation mechanism we could simply use the default prometheus instance deployed by istio (which the one of seldon-core-analytics is intended to replace if I'm not mistaken).
Yes, this is the shorter version of my proposal :) If it doesn't fit the design philosophy we can drop it. |
Closing via #1806 |
Hello,
more info: kubernetes/kubernetes#118261 |
port 8000 is defined twice in the seldon-container-engine spec, as in the following example:
using seldon-core v1.1.0
The text was updated successfully, but these errors were encountered: