Skip to content

frostming/BentoML

Folders and files

NameName
Last commit message
Last commit date
Jul 20, 2023
Feb 1, 2024
Oct 26, 2022
May 27, 2024
May 27, 2024
Jan 22, 2024
Feb 1, 2024
May 31, 2024
May 27, 2024
Jul 20, 2023
Feb 7, 2024
Sep 27, 2022
Jun 2, 2023
Jul 20, 2023
Feb 1, 2024
May 14, 2024
Feb 1, 2024
Jul 20, 2023
Jul 13, 2022
Nov 16, 2022
Jul 6, 2023
Jun 12, 2020
Jun 2, 2023
Feb 1, 2024
Jun 2, 2023
Jan 20, 2020
Feb 23, 2024
May 21, 2024
Jun 2, 2023
Oct 26, 2022
Oct 20, 2022
Feb 1, 2024
May 8, 2024
May 8, 2024

Repository files navigation

BentoML: Unified Model Serving Framework

Unified Model Serving Framework

🍱 Build model inference APIs and multi-model serving systems with any open-source or custom AI models. πŸ‘‰ Join our Slack community!

License: Apache-2.0 Releases CI Twitter Community

What is BentoML?

BentoML is an open-source model serving framework, simplifying how AI/ML models gets into production:

  • 🍱 Easily build APIs for Any AI/ML Model. Turn any model inference script into a REST API server with just a few lines of code and standard Python type hints.
  • 🐳 Docker Containers made simple. No more dependency hell! Manage your environments, dependencies and models with a simple config file. BentoML automatically generates Docker images, ensures reproducibility, and simplifies how you run inference across different environments.
  • 🧭 Maximize CPU/GPU utilization. Improve your API throughput and latency performance leveraging built-in serving optimization features like dynamic batching, model parallelism, multi-stage pipeline and multi-model inference-graph orchestration.
  • πŸ‘©β€πŸ’» Build Custom AI Applications. BentoML is highly flexible for advanced customizations. Easily implement your own API specifications, asynchronous inference tasks; customize pre/post-processing, model inference logic; and define model composition; all using Python code. Supports any ML framework, modality, and inference runtime.
  • πŸš€ Build for Production. Develop, run and debug locally. Seamlessly deploy to production with Docker containers or BentoCloud.

Getting started

Install BentoML:

# Requires Pythonβ‰₯3.8
pip install bentoml torch transformers

Define APIs in a service.py file.

import bentoml
from transformers import pipeline
from typing import List

@bentoml.service
class Summarization:
    def __init__(self):
        self.pipeline = pipeline('summarization')

    @bentoml.api(batchable=True)
    def summarize(self, texts: List[str]) -> List[str]:
        results = self.pipeline(texts)
        return list(map(lambda res: res['summary_text'], results))

Run the service code locally (serving at http://localhost:3000 by default):

bentoml serve service.py:Summarization

Now you can run inference from your browser at http://localhost:3000 or with a Python script:

import bentoml

with bentoml.SyncHTTPClient('http://localhost:3000') as client:
    text_to_summarize: str = input("Enter text to summarize: ")
    summarized_text: str = client.summarize([text_to_summarize])[0]
    print(f"Summarized text: {summarized_text}")

Deploying your first Bento

To deploy your BentoML Service code, first create a bentofile.yaml file to define its dependencies and environments. Find the full list of bentofile options here.

service: "service:Summarization" # Entry service import path
include:
  - "*.py" # Include all .py files in current directory
python:
  packages: # Python dependencies to include
  - torch
  - transformers

Then, choose one of the following ways for deployment:

🐳 Docker Container

Run bentoml build to package necessary code, models, dependency configs into a Bento - the standardized deployable artifact in BentoML:

bentoml build

Ensure Docker is running. Generate a Docker container image for deployment:

bentoml containerize summarization:latest

Run the generated image:

docker run --rm -p 3000:3000 summarization:latest
☁️ BentoCloud

BentoCloud is the AI inference platform for fast moving AI teams. It lets you easily deploy your BentoML code in a fast-scaling infrastructure. Sign up for BentoCloud for personal access; for enterprise use cases, contact our team.

# After signup, follow login instructions upon API token creation:
bentoml cloud login --api-token <your-api-token>

# Deploy from current directory:
bentoml deploy .

bentocloud-ui

For detailed explanations, read Quickstart.

Use cases

Check out the examples folder for more sample code and usage.

Advanced topics

See Documentation for more tutorials and guides.

Community

Get involved and join our Community Slack πŸ’¬, where thousands of AI/ML engineers help each other, contribute to the project, and talk about building AI products.

To report a bug or suggest a feature request, use GitHub Issues.

Contributing

There are many ways to contribute to the project:

Thanks to all of our amazing contributors!

Usage tracking and feedback

The BentoML framework collects anonymous usage data that helps our community improve the product. Only BentoML's internal API calls are being reported. This excludes any sensitive information, such as user code, model data, model names, or stack traces. Here's the code used for usage tracking. You can opt-out of usage tracking by the --do-not-track CLI option:

bentoml [command] --do-not-track

Or by setting the environment variable:

export BENTOML_DO_NOT_TRACK=True

License

Apache License 2.0

About

Unified Model Serving Framework 🍱

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Python 94.4%
  • Shell 2.7%
  • Starlark 0.9%
  • Jinja 0.6%
  • Dockerfile 0.4%
  • Swift 0.2%
  • Other 0.8%