tsdb: Query and other benchmarks

[gouthamve]

We don't have proper benchmarks yet and that means we cannot see the potential impact of the changes we are introducing.

[AlekSi]

"Help wanted"? How can I help?

[krasi-georgiev]

we haven't discussed anything yet so it is a greenfield project, but here some random thoughts.
we already have a benchmarking CI for prometheus and can probably reuse a lot from the logic and pieces.
We can do a very simplified version for the tsdb as well.

1. Implement some tsdb bench tool.
2. When a test is triggered it will build and run the versions: one from the PR and another one from master or a release.
3. Than just use grafana to display the exposed metrics and compare the results.

@sipian and I have worked on the Prombench project so can help along the way as well.

@gouthamve should have some better ideas where to start.

[krasi-georgiev]

not sure if you have looked at the tsdb cli tool, but it already has some benchmarking already so maybe this is a good starting point. Can keep it simple to start with.
https://github.com/prometheus/tsdb/blob/9be883670eaf846e8f5bb65ea577dcd7d28d469e/cmd/tsdb/main.go#L43

[krasi-georgiev]

@AlekSi just found PR: prometheus-junkyard/tsdb#240 that adds some more benchmarking.
I guess our first goal is to somehow automate these and display the results.

The way I see it:

1. comment /benchmark
2. Prow starts a job
3. The job runs go bench against master and another one against the PR and just displays the logs in the prow job dashboard.

[krasi-georgiev]

just came across https://github.com/timescale/tsbs which might help.

[Sangeeta04Sharma]

Hello contributors, I would like to work on it as my GSOC project. I have used and worked Prometheus, Kubernetes, and Golang. Please guide me and help to start work on it.

[krasi-georgiev]

@Sangeeta04Sharma can you ping me on IRC and will try to give some directions.

[rajibmitra]

@krasi-georgiev I would like to work on this issue as part of community bridge program if no one is actively working on it.

[krasi-georgiev]

We did this one as part of the last GSOC program. There are still some things to finish, but another contributor has taken the initiative and don't think there is much left to do here to make it an antyre project. I think it landed in the communiyt bridge program by mistake.

[rajibmitra]

okay , thanks for the update @krasi-georgiev

[brian-brazil]

We have plenty of benchmarks, where do you think there are gaps?

[roidelapluie]

@brian-brazil I think that #6512 highlights that we probably don't stress enough the tsdb reads.

[imrajdas]

Hi @krasi-georgiev @gouthamve . I would like to take this as my GSOC project. Can you help me with it?

[brian-brazil]

We've looked at this as part of the bug scrub, and we currently have benchmarks in the tsdb for the querier so this is being closed. If there's specific cases not covered, an issue/PR should be created for them.