[Speculative decoding] Add periodic log with time spent in proposal/scoring/verification #6963
Conversation
cadedaniel
commented
Jul 30, 2024
cadedaniel
commented
- Log values whenever rejection sampler produces metrics
- Measures average time spent proposing each token, time spent scoring, and time spent verifiying.
|
👋 Hi! Thank you for contributing to the vLLM project. Once the PR is approved and ready to go, please make sure to run full CI as it is required to merge (or just use auto-merge). To run full CI, you can do one of these:
🚀 |
| class Timer: | ||
|
|
||
| def __enter__(self): | ||
| self.start_time = time.time() | ||
| return self | ||
|
|
||
| def __exit__(self, exc_type, exc_value, traceback): | ||
| self.end_time = time.time() | ||
| self.elapsed_time_s = self.end_time - self.start_time | ||
| self.elapsed_time_ms = self.elapsed_time_s * 1000 |
There was a problem hiding this comment.
Put it in a more common space like utils?
| (average_time_per_proposal_tok_ms, scoring_time_ms, | ||
| verification_time_ms) = stage_times | ||
| logger.info( | ||
| "SpecDecodeWorker stage times: " | ||
| "average_time_per_proposal_tok_ms=%.02f " | ||
| "scoring_time_ms=%.02f verification_time_ms=%.02f", | ||
| average_time_per_proposal_tok_ms, scoring_time_ms, | ||
| verification_time_ms) |
There was a problem hiding this comment.
It looks like we will print this log every step? Did I miss something?
There was a problem hiding this comment.
only when rejection sampler emits metrics, which is every 5s
There was a problem hiding this comment.
I'll add comment
What is the motivation ? |
oh because normally many users don't want to emit this kind of logs (and I thought that's why we have the disable_log_stats as an argument) |
|
Hmm good point. Let me see if I can pipe down that argument. |
github-actions
bot
added
the
ready
…coring/verification (vllm-project#6963)
…coring/verification (vllm-project#6963)
…coring/verification (vllm-project#6963) Signed-off-by: Alvant <[email protected]>
