Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(sdk, api): fix types, enhance chat, audio and completion tests #1038

Merged
merged 107 commits into from
Sep 20, 2024

Conversation

justinthelaw
Copy link
Contributor

@justinthelaw justinthelaw commented Sep 17, 2024

Description

Creates and enables vLLM E2E tests that touch each SDK gRPC endpoint at least once with a "happy path" test. Standardizes the new Completions test from vLLM's E2E tests to the LLaMA-CPP-Python E2E tests. Fixes the FinishReason enum typing and object field issues across Completions and ChatCompletions. Adds Audio unit tests and refactors Repeater to enable running those tests.

BREAKING CHANGES

  • fixes FinishReason to be an enum in both Completions and ChatCompletions protobufs
    • modifies API gRPC handler, typing utils, and helper utils to use a Enum class to define and transform the stub responses
  • fixes object and created field for Completions type as defined in the OpenAI API specification
    • uses Literal text_completion for object
    • uses ChatCompletions type's created for Completions created

CHANGES

  • adds and enables simple ("happy path") E2E vLLM testing for local environments only (GPU runner is broken, see commit history)
  • adds Completions E2E test to llama-cpp-python and vLLM to catch more potential issues, like FinishReason being the wrong type
  • adds Audio and Completions unit testing
  • condenses tests into a single test file with an ENV parser (default, warning and helper text included) for model name
  • adds more comprehensive Make target for test and typing artifact clean-up
  • improves testing documentation and Makefile targets

Related Issue

Fixes #1037

Relates to #854

Checklist before merging

@justinthelaw justinthelaw added tech-debt Not a feature, but still necessary blocked 🛑 Something needs to happen before this issues is worked labels Sep 17, 2024
@justinthelaw justinthelaw self-assigned this Sep 17, 2024
@justinthelaw justinthelaw requested a review from a team as a code owner September 17, 2024 01:07
@justinthelaw justinthelaw linked an issue Sep 17, 2024 that may be closed by this pull request
Copy link

netlify bot commented Sep 17, 2024

Deploy Preview for leapfrogai-docs canceled.

Name Link
🔨 Latest commit 366cc79
🔍 Latest deploy log https://app.netlify.com/sites/leapfrogai-docs/deploys/66edbef41176bd00083aad64

@justinthelaw
Copy link
Contributor Author

There seems to be an issue with our GPU runner configurations that is blocking the vLLM E2E tests from running: https://github.com/defenseunicorns/leapfrogai/actions/runs/10894219888/job/30230952202

The upstream UDS Common action is unable to install UDS CLI into the bin directory. It seems like it needs sudo or higher permissions, even though that is not usually required in our CPU (large or regular) runners. As a side note, the vLLM E2E tests run locally.

@justinthelaw justinthelaw marked this pull request as draft September 17, 2024 18:00
@justinthelaw justinthelaw changed the title fix(sdk, test): fix finish reason, chat and completion e2e tests fix(sdk, test): fix finish reason, chat, audsio and completion tests Sep 19, 2024
@justinthelaw justinthelaw changed the title fix(sdk, test): fix finish reason, chat, audsio and completion tests fix(sdk, test): fix finish reason, chat, audio and completion tests Sep 19, 2024
@justinthelaw justinthelaw changed the title fix(sdk, test): fix finish reason, chat, audio and completion tests test(sdk, api): fix finish reason, chat, audio and completion tests Sep 19, 2024
@justinthelaw justinthelaw marked this pull request as draft September 19, 2024 15:03
@justinthelaw justinthelaw force-pushed the 1037-testvllm-implement-e2e-testing-for-vllm branch from 70f58ad to a1a03c1 Compare September 19, 2024 20:07
@justinthelaw justinthelaw marked this pull request as ready for review September 19, 2024 22:11
@justinthelaw justinthelaw removed the api label Sep 20, 2024
@justinthelaw justinthelaw changed the title test(sdk, api): fix finish reason, chat, audio and completion tests chore(sdk, api): fix finish reason, chat, audio and completion tests Sep 20, 2024
@justinthelaw justinthelaw force-pushed the 1037-testvllm-implement-e2e-testing-for-vllm branch from cac0fdb to 3387974 Compare September 20, 2024 13:47
@justinthelaw justinthelaw marked this pull request as draft September 20, 2024 13:49
@justinthelaw
Copy link
Contributor Author

Blocked until #1085 is merged into main.

@justinthelaw justinthelaw marked this pull request as ready for review September 20, 2024 16:22
docs/DEVELOPMENT.md Outdated Show resolved Hide resolved
@justinthelaw justinthelaw changed the title chore(sdk, api): fix finish reason, chat, audio and completion tests chore(sdk, api): fix types, enhance chat, audio and completion tests Sep 20, 2024
@justinthelaw justinthelaw requested review from gphorvath and a team September 20, 2024 19:27
@justinthelaw justinthelaw merged commit 014329c into main Sep 20, 2024
32 checks passed
@justinthelaw justinthelaw deleted the 1037-testvllm-implement-e2e-testing-for-vllm branch September 20, 2024 19:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
chore tech-debt Not a feature, but still necessary
Projects
None yet
Development

Successfully merging this pull request may close these issues.

chore(vllm): implement e2e testing for vllm
5 participants