Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[test] Ensure that the first token generation is not included into TPOT #1414

Merged
merged 3 commits into from
Dec 23, 2024
Merged

[test] Ensure that the first token generation is not included into TPOT #1414

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
e7347a7
imporove perf_metrics tests
pavel-esir Dec 19, 2024
dad4c6d
add left cpp file
pavel-esir Dec 19, 2024
a1e0d35
relax a bit prefill time comparison
pavel-esir Dec 20, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion src/cpp/src/perf_metrics.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -101,7 +101,7 @@ void PerfMetrics::evaluate_statistics(std::optional<TimePoint> start_time) {

auto ttft = tok_times[0] - start_time_val;
raw_metrics.m_times_to_first_token = std::vector<MicroSeconds>();
raw_metrics.m_times_to_first_token.emplace_back(ttft / batch_sizes[0]);
raw_metrics.m_times_to_first_token.emplace_back(ttft);
Copy link
Contributor Author

@pavel-esir pavel-esir Dec 19, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we have batch 10 and it takes 1 sec until the first token is generated then time to the first token is still 1 sec not 100 ms! Therefore i removed / batch_sizes[0]

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree, but should we notify llm_bench about the change?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I asked @eaidova what she thinks. Let's wait her answer

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@peterchen-intel @wgzintel do you have objections? What the reason to divide first token latency on batch?

Copy link
Contributor Author

@pavel-esir pavel-esir Dec 20, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As i see from blame, this division was added by @ialbrecht. Do you have any objection to remove it?

num_generated_tokens = batch_sizes[0];

// The very first infer request (prefill stage) is slower than subsequent ones since we process a sequence of tokens.
Expand Down
10 changes: 9 additions & 1 deletion tests/python_tests/test_generate_api.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -798,6 +798,12 @@ def test_perf_metrics(model_descr, generation_config, prompt):
assert (mean_ttft, std_ttft) == (perf_metrics.get_ttft().mean, perf_metrics.get_ttft().std)
assert mean_ttft > 0 and mean_ttft < 1000.0

raw_metrics = perf_metrics.raw_metrics
durations = np.array(raw_metrics.m_durations) / 1000
# Check that prefill is not included in durations for TPOT calculation.
# For the very long prompt prefill is slow and TTFT is much larger than any other token genration duration.
assert np.all(mean_ttft > durations * 10)

mean_tpot, std_tpot = perf_metrics.get_tpot()
assert (mean_tpot, std_tpot) == (perf_metrics.get_tpot().mean, perf_metrics.get_tpot().std)
assert mean_tpot > 0 and mean_ttft < 1000.0
Expand All @@ -822,7 +828,9 @@ def test_perf_metrics(model_descr, generation_config, prompt):
assert std_detok_duration == 0

# assert that calculating statistics manually from the raw counters we get the same restults as from PerfMetrics
raw_metrics = perf_metrics.raw_metrics
assert np.allclose(mean_tpot, np.mean(durations))
assert np.allclose(std_tpot, np.std(durations))

raw_dur = np.array(raw_metrics.generate_durations) / 1000
assert np.allclose(mean_gen_duration, np.mean(raw_dur))
assert np.allclose(std_gen_duration, np.std(raw_dur))
Expand Down
Loading