Fix 1340 prometheus counters

[deepak-muley]

Due to error in merging, PR #1365 received few changes from master in the history. hence to keep the branch clean, have created this PR. Sorry for the inconvenience.
Ref #1340
@Jeffwan @andreyvelich @johnugeorge

Testing done:

• Created 2 jobs of tensorflow and deleted one job quickly i saw only following even after long time

HELP training_operator_jobs_created_total Counts number of jobs created
TYPE training_operator_jobs_created_total counter
training_operator_jobs_created_total{framework="tensorflow",job_namespace="test-tf-operator"} 2

HELP training_operator_jobs_successful_total Counts number of jobs successful
TYPE training_operator_jobs_successful_total counter
training_operator_jobs_successful_total{framework="tensorflow",job_namespace="test-tf-operator"} 1

Note: somehow was expecting deletedjob count to go up but it did not. Need more debugging @Jeffwan @andreyvelich

[deepak-muley]

Also to open up the discussion of how should the counters be named (metrics name vs label) here are current thoughts

Currently we have following:
training_operator_mxjobs_created_total {job_namespace: "ns"}
training_operator_pytorchjobs_created_total {job_namespace: "ns"}
training_operator_tfjobs_created_total {job_namespace: "ns"}
training_operator_xgboostjobs_created_total {job_namespace: "ns"}

training_operator_mxjobs_deleted_total {job_namespace: "ns"}
training_operator_pytorchjobs_deleted_total {job_namespace: "ns"}
training_operator_tfjobs_deleted_total {job_namespace: "ns"}
training_operator_xgboostjobs_deleted_total {job_namespace: "ns"}

training_operator_mxjobs_successful_total {job_namespace: "ns"}
training_operator_pytorchjobs_successful_total {job_namespace: "ns"}
training_operator_tfjobs_successful_total {job_namespace: "ns"}
training_operator_xgboostjobs_successful_total {job_namespace: "ns"}

training_operator_mxjobs_failed_total {job_namespace: "ns"}
training_operator_pytorchjobs_failed_total {job_namespace: "ns"}
training_operator_tfjobs_failed_total {job_namespace: "ns"}
training_operator_xgboostjobs_failed_total {job_namespace: "ns"}

training_operator_mxjobs_restarted_total {job_namespace: "ns"}
training_operator_pytorchjobs_restarted_total {job_namespace: "ns"}
training_operator_tfjobs_restarted_total {job_namespace: "ns"}
training_operator_xgboostjobs_restarted_total {job_namespace: "ns"}
—

Current suggestion is as follows:
training_operator_jobs_created_total {job_namespace: "ns", framework: "mxnet”}
training_operator_jobs_created_total {job_namespace: "ns", framework: “pytorch”}
training_operator_jobs_created_total {job_namespace: "ns", framework: “tensor flow”}
training_operator_jobs_created_total {job_namespace: "ns", framework: “xgboost”}

training_operator_jobs_deleted_total {job_namespace: "ns", framework: "mxnet”}
training_operator_jobs_deleted_total {job_namespace: "ns", framework: “pytorch”}
training_operator_jobs_deleted_total {job_namespace: "ns", framework: “tensor flow”}
training_operator_jobs_deleted_total {job_namespace: "ns", framework: “xgboost”}

training_operator_jobs_successful_total {job_namespace: "ns", framework: "mxnet”}
training_operator_jobs_successful_total {job_namespace: "ns", framework: “pytorch”}
training_operator_jobs_successful_total {job_namespace: "ns", framework: “tensor flow”}
training_operator_jobs_successful_total {job_namespace: "ns", framework: “xgboost”}

training_operator_jobs_failed_total {job_namespace: "ns", framework: "mxnet”}
training_operator_jobs_failed_total {job_namespace: "ns", framework: “pytorch”}
training_operator_jobs_failed_total {job_namespace: "ns", framework: “tensor flow”}
training_operator_jobs_failed_total {job_namespace: "ns", framework: “xgboost”}

training_operator_jobs_restarted_total {job_namespace: "ns", framework: "mxnet”}
training_operator_jobs_restarted_total {job_namespace: "ns", framework: “pytorch”}
training_operator_jobs_restarted_total {job_namespace: "ns", framework: “tensor flow”}
training_operator_jobs_restarted_total {job_namespace: "ns", framework: “xgboost”}

————
Why not to go then following route where we add type as well in label.
training_operator_jobs_total {job_namespace: "ns", framework: "mxnet”, type: “created”}
training_operator_jobs_total {job_namespace: "ns", framework: "mxnet”, type: “deleted”}
training_operator_jobs_total {job_namespace: "ns", framework: "mxnet”, type: “successful”}
training_operator_jobs_total {job_namespace: "ns", framework: "mxnet”, type: “failed”}
training_operator_jobs_total {job_namespace: "ns", framework: "mxnet”, type: “restarted”}

training_operator_jobs_total {job_namespace: "ns", framework: “tensorflow”, type: “created”}
training_operator_jobs_total {job_namespace: "ns", framework: "tensorflow”, type: “deleted”}
training_operator_jobs_total {job_namespace: "ns", framework: "tensorflow”, type: “successful”}
training_operator_jobs_total {job_namespace: "ns", framework: "tensorflow”, type: “failed”}
training_operator_jobs_total {job_namespace: "ns", framework: "tensorflow”, type: “restarted”}

training_operator_jobs_total {job_namespace: "ns", framework: “pytorch”, type: “created”}
training_operator_jobs_total {job_namespace: "ns", framework: "pytorch”, type: “deleted”}
training_operator_jobs_total {job_namespace: "ns", framework: "pytorch”, type: “successful”}
training_operator_jobs_total {job_namespace: "ns", framework: "pytorch”, type: “failed”}
training_operator_jobs_total {job_namespace: "ns", framework: "pytorch”, type: “restarted”}

training_operator_jobs_total {job_namespace: "ns", framework: “xgboost”, type: “created”}
training_operator_jobs_total {job_namespace: "ns", framework: "xgboost”, type: “deleted”}
training_operator_jobs_total {job_namespace: "ns", framework: "xgboost”, type: “successful”}
training_operator_jobs_total {job_namespace: "ns", framework: "xgboost”, type: “failed”}
training_operator_jobs_total {job_namespace: "ns", framework: "xgboost”, type: “restarted”}

any concerns on 3rd naming convention?

[aws-kf-ci-bot]

Hi @deepak-muley. Thanks for your PR.

I'm waiting for a kubeflow member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[Jeffwan]

/ok-to-test

[Jeffwan]

To me, I think option 2 is better?

Option 1. "training_operator_mxjobs_created_total " It might be hard to create dashboard.
Option 3. I think this is not recommended?

When you have a successful request count and a failed request count, the best way to expose this is as one metric for total requests and another metric for failed requests. This makes it easy to calculate the failure ratio. Do not use one metric with a failed or success label. Similarly, with hit or miss for caches, it’s better to have one metric for total and another for hits.

ref: https://prometheus.io/docs/instrumenting/writing_exporters/

[deepak-muley]

/test kubeflow-tf-operator-presubmit

[deepak-muley]

To me, I think option 2 is better?

Option 1. "training_operator_mxjobs_created_total " It might be hard to create dashboard.
Option 3. I think this is not recommended?

When you have a successful request count and a failed request count, the best way to expose this is as one metric for total requests and another metric for failed requests. This makes it easy to calculate the failure ratio. Do not use one metric with a failed or success label. Similarly, with hit or miss for caches, it’s better to have one metric for total and another for hits.

ref: https://prometheus.io/docs/instrumenting/writing_exporters/

Per https://prometheus.io/docs/prometheus/latest/querying/basics/ does not really highlight in a clean way of any issue with more labels

[johnugeorge]

Option 2 looks more intuitive to me. How about starting with Option 2 for now?

[deepak-muley]

Yes currently I have pushed the changes with option2

[deepak-muley]

/retest

[google-oss-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Jeffwan

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [Jeffwan]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[Jeffwan]

LGTM. leave it to @johnugeorge to double check and approve

[johnugeorge]

Can you delete previous tfjob prometheus counters? Any reason for keeping it?

tfJobsSuccessCount, tfJobsFailureCount, tfJobsCreatedCount, tfJobsRestartCount

[deepak-muley]

They will go away when we remove those files. They are not registered

[johnugeorge]

Related : #1367

[johnugeorge]

Thanks @deepak-muley
/lgtm