how to remove stopped task from metrics

[mxlxm]

Question

Before asking a question, make sure you have:

• Searched existing Stack Overflow questions.
• Googled your question.
• Searched open and closed GitHub issues
• Read the documentation:
  • Detailed documentation
  • 简体中文文档

how to remove stopped task(some test tasks) from dm-worker metrics?

[csuzhangxc]

@mxlxm DM's metrics are stored in Prometheus, you can try https://prometheus.io/docs/prometheus/latest/querying/api/#delete-series.

[mxlxm]

@mxlxm DM's metrics are stored in Prometheus, you can try https://prometheus.io/docs/prometheus/latest/querying/api/#delete-series.

no, I mean some stopped task's metric info still present on dm-worker's metric page

[csuzhangxc]

stopped task's metrics already stored in Prometheus, then Grafana still presents them.
so, you should delete them in Prometheus.

[mxlxm]

no, there's nothing about prometheus.
I created some test task, and then stopped them, query-status shows empty, but on dm-worker's x.x.x.x:8262/metrics page, those test task's metric still present. is there any possible way to remove them?

[csuzhangxc]

@mxlxm I understand your question now.
This is because when stopping the task, metrics with task-name label were not removed. Maybe we should call some functions like DeleteLabelValues for metrics.

Now, I think you should restart the DM-worker to remove these metrics, and we can remove them automatically when stopping the task later. Or, you can open a PR to do that if you want. 😁

[mxlxm]

😅 haven't read the code, I'll try to do this this weekend.

[csuzhangxc]

fixed in #575.