feat: model preheat #1

frostming · 2024-08-20T02:02:06Z

Signed-off-by: Frost Ming [email protected]

What does this PR address?

Fixes #(issue)

Before submitting:

Does the Pull Request follow Conventional Commits specification naming? Here are GitHub's
guide on how to create a pull request.
Does the code follow BentoML's code style, pre-commit run -a script has passed (instructions)?
Did you read through contribution guidelines and follow development guidelines?
Did your changes require updates to the documentation? Have you updated
those accordingly? Here are documentation guidelines and tips on writting docs.
Did you write tests to cover your changes?

* feat: export component context at top namespace Signed-off-by: Frost Ming <[email protected]>

Signed-off-by: Frost Ming <[email protected]>

* feat: mount asgi app and fastapi support Signed-off-by: Frost Ming <[email protected]> * add test cases Signed-off-by: Frost Ming <[email protected]> * fix: update dependencies Signed-off-by: Frost Ming <[email protected]> * fix: only collect metrics when enabled Signed-off-by: Frost Ming <[email protected]> * fix types Signed-off-by: Frost Ming <[email protected]> --------- Signed-off-by: Frost Ming <[email protected]>

* fix: add envs to image * ci: auto fixes from pre-commit.ci For more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* fix: crash when pulling bento from organization with proxy mode Signed-off-by: Frost Ming <[email protected]>

* fix: add str in enum for deployment schema Signed-off-by: FogDong <[email protected]> * ci: auto fixes from pre-commit.ci For more information, see https://pre-commit.ci * fix test Signed-off-by: FogDong <[email protected]> --------- Signed-off-by: FogDong <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

Add envs field in bentofile Signed-off-by: Sherlock113 <[email protected]>

Add GitHub repo links for use case docs Signed-off-by: Sherlock113 <[email protected]>

Update the vllm use case doc Signed-off-by: Sherlock113 <[email protected]>

* Add how to get deployment info doc Signed-off-by: Sherlock113 <[email protected]> * Add other methods and attributes Signed-off-by: Sherlock113 <[email protected]> * Update the docs Signed-off-by: Sherlock113 <[email protected]> --------- Signed-off-by: Sherlock113 <[email protected]>

Signed-off-by: Frost Ming <[email protected]>

* fix: wait until server ready when initializing client Signed-off-by: Frost Ming <[email protected]> * fix format Signed-off-by: Frost Ming <[email protected]> * fix: read token from env * fix: don't wait for proxy client Signed-off-by: Frost Ming <[email protected]> * fix: fetch schema url Signed-off-by: Frost Ming <[email protected]> * fix tests Signed-off-by: Frost Ming <[email protected]> * try fixing Signed-off-by: Frost Ming <[email protected]> * why readyz isn't working Signed-off-by: Frost Ming <[email protected]> * revert change Signed-off-by: Frost Ming <[email protected]>

…gic (#4561) Signed-off-by: Frost Ming <[email protected]>

Add doc for server_ready_timeout Signed-off-by: Sherlock113 <[email protected]>

* fix: depracate all enum in schema Signed-off-by: FogDong <[email protected]> * ci: auto fixes from pre-commit.ci For more information, see https://pre-commit.ci * fix: fix test Signed-off-by: FogDong <[email protected]> * ci: auto fixes from pre-commit.ci For more information, see https://pre-commit.ci --------- Signed-off-by: FogDong <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* feat: threaded serving Signed-off-by: Frost Ming <[email protected]> * fix: add context manager Signed-off-by: Frost Ming <[email protected]> * remove test code from quickstart Signed-off-by: Frost Ming <[email protected]> * fix: change the server url Signed-off-by: Frost Ming <[email protected]> * fix: add server on deployment Signed-off-by: Frost Ming <[email protected]>

Adjust existing Bento deployment wording Signed-off-by: Sherlock113 <[email protected]>

Signed-off-by: FogDong <[email protected]>

* Add doc for using pydantic model at the top level Signed-off-by: Sherlock113 <[email protected]> * Import bentoml package Signed-off-by: Sherlock113 <[email protected]> * Add explanations for accessing parameters Signed-off-by: Sherlock113 <[email protected]> --------- Signed-off-by: Sherlock113 <[email protected]>

Signed-off-by: FogDong <[email protected]>

* set upload timeouts to 10 hours * ci: auto fixes from pre-commit.ci For more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* fix: respect the content type config for file output Signed-off-by: Frost Ming <[email protected]> * fix issubclass check Signed-off-by: Frost Ming <[email protected]> --------- Signed-off-by: Frost Ming <[email protected]>

* Update the adaptive batching doc Signed-off-by: Sherlock113 <[email protected]> * Add @bentoml.api decorator Signed-off-by: Sherlock113 <[email protected]> * Add a section to explain how to handle multiple parameters Signed-off-by: Sherlock113 <[email protected]> --------- Signed-off-by: Sherlock113 <[email protected]>

* feat: Service.to_asgi() method Signed-off-by: Frost Ming <[email protected]>

* Add asgi doc Signed-off-by: Sherlock113 <[email protected]> * ci: auto fixes from pre-commit.ci For more information, see https://pre-commit.ci --------- Signed-off-by: Sherlock113 <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* fix: allow legacy runnable methods when calling API Signed-off-by: Frost Ming <[email protected]> * fix: output content type Signed-off-by: Frost Ming <[email protected]> * fix compatibility with python 3.9 Signed-off-by: Frost Ming <[email protected]>

fix(stub): update env params construction from __init__ Signed-off-by: Aaron <[email protected]>

* Add testing doc Signed-off-by: Sherlock113 <[email protected]> * ci: auto fixes from pre-commit.ci For more information, see https://pre-commit.ci --------- Signed-off-by: Sherlock113 <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

chore(stub): use TypedDict Signed-off-by: Aaron <[email protected]>

…nexpected keyword argument 'task'" (#4880) Fixes #4879 Signed-off-by: Frost Ming <[email protected]>

* fix: silence git output Signed-off-by: Frost Ming <[email protected]>

* fix: surface config error on bento build Signed-off-by: Frost Ming <[email protected]>

* Refactor tasks doc Signed-off-by: Sherlock113 <[email protected]> * ci: auto fixes from pre-commit.ci For more information, see https://pre-commit.ci --------- Signed-off-by: Sherlock113 <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

…page (#4891) * Sync the BYOC doc with blog Signed-off-by: Sherlock113 <[email protected]> * ci: auto fixes from pre-commit.ci For more information, see https://pre-commit.ci --------- Signed-off-by: Sherlock113 <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

…4893) * fix: move user setup to base template to make custom base image work Signed-off-by: Frost Ming <[email protected]>

…rvice(metrics=...)` (#4895) Fixes #4859 Signed-off-by: Frost Ming <[email protected]>

* ci: pre-commit autoupdate [skip ci] updates: - [github.com/astral-sh/ruff-pre-commit: v0.5.0 → v0.5.6](astral-sh/ruff-pre-commit@v0.5.0...v0.5.6) - [github.com/pdm-project/pdm: 2.16.1 → 2.17.3](pdm-project/pdm@2.16.1...2.17.3) - [github.com/bufbuild/buf: v1.34.0 → v1.35.1](bufbuild/buf@v1.34.0...v1.35.1) * chore: update check to use isinstance Signed-off-by: Aaron Pham <[email protected]> --------- Signed-off-by: Aaron Pham <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Aaron Pham <[email protected]>

Update doc images Signed-off-by: Sherlock113 <[email protected]>

* fix: fix resource type Signed-off-by: FogDong <[email protected]> * ci: auto fixes from pre-commit.ci For more information, see https://pre-commit.ci --------- Signed-off-by: FogDong <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* fix: direct return response if tracer id is null Signed-off-by: FogDong <[email protected]> * ci: auto fixes from pre-commit.ci For more information, see https://pre-commit.ci --------- Signed-off-by: FogDong <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

…4903)

…ng it around (#4907)

Signed-off-by: Frost Ming <[email protected]>

frostming and others added 30 commits March 5, 2024 11:37

feat: export component context at top namespace (#4551)

70bd0de

* feat: export component context at top namespace Signed-off-by: Frost Ming <[email protected]>

feat: use starlette request directly in service context (#4552)

6da7d91

Signed-off-by: Frost Ming <[email protected]>

fix(sdk): build envs into dockerfile (#4546)

934773d

* fix: add envs to image * ci: auto fixes from pre-commit.ci For more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

fix: rename server context attributes (#4553)

70fa6a2

fix: crash when pulling bento from organization with proxy mode (#4554)

5b3e752

* fix: crash when pulling bento from organization with proxy mode Signed-off-by: Frost Ming <[email protected]>

docs: Add envs field in bentofile (#4555)

f15210a

Add envs field in bentofile Signed-off-by: Sherlock113 <[email protected]>

docs: Add GitHub repo links for use case docs (#4556)

e095010

Add GitHub repo links for use case docs Signed-off-by: Sherlock113 <[email protected]>

docs: Update the vllm use case doc (#4558)

41456ff

Update the vllm use case doc Signed-off-by: Sherlock113 <[email protected]>

fix: rename the _bentoml_sdk.api module (#4557)

ca021df

Signed-off-by: Frost Ming <[email protected]>

feat: Add server.on_deployment hook to execute deployment-specific lo…

064e087

…gic (#4561) Signed-off-by: Frost Ming <[email protected]>

docs: Add doc for server_ready_timeout (#4563)

cb01f10

Add doc for server_ready_timeout Signed-off-by: Sherlock113 <[email protected]>

docs: Adjust existing Bento deployment wording (#4567)

80ad614

Adjust existing Bento deployment wording Signed-off-by: Sherlock113 <[email protected]>

fix: fix list instance type (#4566)

49022b0

Signed-off-by: FogDong <[email protected]>

fix: delete access in admin console (#4569)

b8c30cf

Signed-off-by: FogDong <[email protected]>

fix(client): set upload timeouts to 10 hours (#4564)

408078e

* set upload timeouts to 10 hours * ci: auto fixes from pre-commit.ci For more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

fix: respect the content type config for file output (#4570)

4a42052

* fix: respect the content type config for file output Signed-off-by: Frost Ming <[email protected]> * fix issubclass check Signed-off-by: Frost Ming <[email protected]> --------- Signed-off-by: Frost Ming <[email protected]>

feat: Service.to_asgi() method (#4572)

264958c

* feat: Service.to_asgi() method Signed-off-by: Frost Ming <[email protected]>

fix(stub): add envs to __init__ construction (#4576)

349b4ae

fix(stub): update env params construction from __init__ Signed-off-by: Aaron <[email protected]>

chore(type): use TypedDict for inferred stubs (#4577)

100c1cc

chore(stub): use TypedDict Signed-off-by: Aaron <[email protected]>

frostming and others added 29 commits July 24, 2024 20:42

fix(framework): transformers vs 1.3.0 "TypeError: __init__() got an u…

a148bb3

…nexpected keyword argument 'task'" (#4880) Fixes #4879 Signed-off-by: Frost Ming <[email protected]>

fix: silence git output (#4883)

bd145ca

* fix: silence git output Signed-off-by: Frost Ming <[email protected]>

fix: surface config error on bento build (#4888)

d34749f

* fix: surface config error on bento build Signed-off-by: Frost Ming <[email protected]>

fix: move user setup to base template to make custom base image work (#…

1764c69

…4893) * fix: move user setup to base template to make custom base image work Signed-off-by: Frost Ming <[email protected]>

fix: Cannot define custom duration histogram buckets via `@bentoml.se…

51468b6

…rvice(metrics=...)` (#4895) Fixes #4859 Signed-off-by: Frost Ming <[email protected]>

fix: press text (#4898)

7e1f10c

docs: Update doc images (#4901)

be4abeb

Update doc images Signed-off-by: Sherlock113 <[email protected]>

docs: update azure byoc instructions (#4905)

03a7f85

fix: metrics duration defined with min,max,factor not taking effect (#…

29214bf

…4903)

refactor: store the cloud context into the container instead of passi…

da6c30b

…ng it around (#4907)

feat: support model loading preheat

87b7ccb

feat: lazy model reference

bdaea53

Signed-off-by: Frost Ming <[email protected]>

feat: add huggingface endpoint

4b62006

Signed-off-by: Frost Ming <[email protected]>

fix: export members at top module

da70440

Signed-off-by: Frost Ming <[email protected]>

feat: save huggingface models to BENTOML_HF_CACHE_DIR

5c9b977

Signed-off-by: Frost Ming <[email protected]>

fix: freeze model version when serving a bento

fc03f05

Signed-off-by: Frost Ming <[email protected]>

fix: memoize the resolved value

ab5768f

Signed-off-by: Frost Ming <[email protected]>

feat: create placeholders for non-bentoml models

87123e4

Signed-off-by: Frost Ming <[email protected]>

fix: hf models root

ce395e1

Signed-off-by: Frost Ming <[email protected]>

fix: models push

3f641fd

Signed-off-by: Frost Ming <[email protected]>

fix: pulling model

be7e326

Signed-off-by: Frost Ming <[email protected]>

fix: add endpoint and url to metadata

82c2e0d

Signed-off-by: Frost Ming <[email protected]>

feat: calc hf model size

7d844dd

Signed-off-by: Frost Ming <[email protected]>

fix: use revision

ea68ef3

Signed-off-by: Frost Ming <[email protected]>

frostming closed this Aug 20, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: model preheat #1

feat: model preheat #1

frostming commented Aug 20, 2024

feat: model preheat #1

feat: model preheat #1

Conversation

frostming commented Aug 20, 2024

What does this PR address?

Before submitting: