Skip to content

πŸŽ‰ An awesome & curated list of best LLMOps tools.

Notifications You must be signed in to change notification settings

InftyAI/Awesome-LLMOps

Folders and files

NameName
Last commit message
Last commit date

Latest commit

Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 

Repository files navigation

Awesome-LLMOps Awesome

πŸŽ‰ An awesome & curated list of best LLMOps tools. But more about LLMOps.

Table of Contents

LLMOps

Name Stars Release Contributors About Tag
BentoML Stars Release Contributors Build Production-Grade AI Applications
Dify Stars Release Contributors One API for plugins and datasets, one interface for prompt engineering and visual operation, all for creating powerful AI applications
FastChat Stars Release Contributors An open platform for training, serving, and evaluating large language models. Release repo for Vicuna and Chatbot Arena.
Flowise Stars Release Contributors Drag & drop UI to build your customized LLM flow
Haystack Stars Release Contributors πŸ” LLM orchestration framework to build customizable, production-ready LLM applications. Connect components (models, vector DBs, file converters) to pipelines or agents that can interact with your data. With advanced retrieval methods, it's best suited for building RAG, question answering, semantic search or conversational agent chatbots.
LangChain Stars Release Contributors ⚑ Building applications with LLMs through composability ⚑
LiteLLM Stars Release Contributors lightweight package to simplify LLM API calls - Azure, OpenAI, Cohere, Anthropic, Replicate. Manages input/output translation
LLaMa-Factory Stars Release Contributors Easy-to-use LLM fine-tuning framework (LLaMA, BLOOM, Mistral, Baichuan, Qwen, ChatGLM)
LlamaIndex Stars Release Contributors LlamaIndex is a data framework for your LLM applications
Mem0 Stars Release Contributors The memory layer for Personalized AI
Open WebUI Stars Release Contributors User-friendly WebUI for LLMs (Formerly Ollama WebUI)
PrivateGPUT Stars Release Contributors Interact with your documents using the power of GPT, 100% privately, no data leaks
Swift GitHub Repo stars GitHub Release GitHub contributors SWIFT supports training(PreTraining/Fine-tuning/RLHF), inference, evaluation and deployment of 350+ LLMs and 90+ MLLMs (multimodal large models).

MLOps

Name Stars Release Contributors About Tag
Flyte Stars Release Contributors Scalable and flexible workflow orchestration platform that seamlessly unifies data, ML and analytics stacks.
Kubeflow Stars Release Contributors Machine Learning Toolkit for Kubernetes cloud
Metaflow Stars Release Contributors πŸš€ Build and manage real-life data science projects with ease!
MLflow Stars Release Contributors Open source platform for the machine learning lifecycle
Seldon-Core Stars Release Contributors An MLOps framework to package, deploy, monitor and manage thousands of production machine learning models. cloud
ZenML Stars Release Contributors ZenML πŸ™: Build portable, production-ready MLOps pipelines. https://zenml.io.

Inference

Name Stars Release Contributors About Tag
DeepSpeed-MII Stars Release Contributors MII makes low-latency and high-throughput inference possible, powered by DeepSpeed.
Inference Stars Release Contributors A fast, easy-to-use, production-ready inference server for computer vision supporting deployment of many popular model architectures and fine-tuned models. vision
ipex-llm Stars Release Contributors Accelerate local LLM inference and finetuning (LLaMA, Mistral, ChatGLM, Qwen, Baichuan, Mixtral, Gemma, Phi, MiniCPM, etc.) on Intel CPU and GPU (e.g., local PC with iGPU, discrete GPU such as Arc, Flex and Max); seamlessly integrate with llama.cpp, Ollama, HuggingFace, LangChain, LlamaIndex, GraphRAG, DeepSpeed, vLLM, FastChat, Axolotl, etc. device
llmaz Stars Release Contributors ☸️ Effortlessly serve state-of-the-art LLMs on Kubernetes.
LMDeploy Stars Release Contributors LMDeploy is a toolkit for compressing, deploying, and serving LLMs.
MaxText Stars Release Contributors A simple, performant and scalable Jax LLM! Jax
llama.cpp Stars Release Contributors LLM inference in C/C++ device
MInference Stars Release Contributors To speed up Long-context LLMs' inference, approximate and dynamic sparse calculate the attention, which reduces inference latency by up to 10x for pre-filling on an A100 while maintaining accuracy.
MLC LLM Stars Release Contributors Universal LLM Deployment Engine with ML Compilation
MLServer Stars Release Contributors MLServer aims to provide an easy way to start serving your machine learning models through a REST and gRPC interface, fully compliant with KFServing's V2 Dataplane spec.
Nanoflow Stars Release Contributors A throughput-oriented high-performance serving framework for LLMs
Ollama Stars Release Contributors Get up and running with Llama 3, Mistral, Gemma 2, and other large language models. device
OpenLLM Stars Release Contributors Operating LLMs in production
OpenVINO Stars Release Contributors OpenVINOβ„’ is an open-source toolkit for optimizing and deploying AI inference
Ratchet Stars Release Contributors A cross-platform browser ML framework. browser
RayServe Stars Release Contributors Ray is a unified framework for scaling AI and Python applications. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.
RouteLLM Stars Release Contributors A framework for serving and evaluating LLM routers - save LLM costs without compromising quality. cost
SGLang Stars Release Contributors SGLang is a structured generation language designed for large language models (LLMs). It makes your interaction with models faster and more controllable.
transformers.js Stars Release Contributors State-of-the-art Machine Learning for the web. Run πŸ€— Transformers directly in your browser, with no need for a server! browser
Triton Inference Server Stars Release Contributors The Triton Inference Server provides an optimized cloud and edge inferencing solution.
TensorRT-LLM GitHub Repo stars GitHub Release GitHub contributors TensorRT-LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and build TensorRT engines that contain state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs.
Text Generation Inference Stars Release Contributors Large Language Model Text Generation Inference
vLLM Stars Release Contributors A high-throughput and memory-efficient inference and serving engine for LLMs
web-llm Stars Release Contributors A high-throughput and memory-efficient inference and serving engine for LLMs browser
zml Stars Release Contributors High performance AI inference stack. Built for production.

Training

Name Stars Release Contributors About Tag
ColossalAI Stars Release Contributors Making large AI models cheaper, faster and more accessible
Ludwig Stars Release Contributors Low-code framework for building custom LLMs, neural networks, and other AI models
MLX Stars Release Contributors MLX: An array framework for Apple silicon

FineTune

Name Stars Release Contributors About Tag
Axolotl Stars Release Contributors Go ahead and axolotl questions
torchtune Stars Release Contributors A Native-PyTorch Library for LLM Fine-tuning
unsloth Stars Release Contributors Finetune Llama 3, Mistral, Phi & Gemma LLMs 2-5x faster with 80% less memory

Agent

Name Stars Release Contributors About Tag
AutoGPT Stars Release Contributors An experimental open-source attempt to make GPT-4 fully autonomous.
MetaGPT Stars Release Contributors 🌟 The Multi-Agent Framework: Given one line Requirement, return PRD, Design, Tasks, Repo
Swarm Stars Release Contributors Framework for building, orchestrating and deploying multi-agent systems. Managed by OpenAI Solutions team. Experimental framework.
XAgent Stars Release Contributors An Autonomous LLM Agent for Complex Task Solving

Evaluation

Name Stars Release Contributors About Tag
AgentBench Stars Release Contributors A Comprehensive Benchmark to Evaluate LLMs as Agents
lm-evaluation-harness Stars Release Contributors A framework for few-shot evaluation of language models.
LongBench Stars Release Contributors LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding long-context

DB Store

Name Stars Release Contributors About Tag
chroma Stars Release Contributors the AI-native open-source embedding database vector
deeplake Stars Release Contributors Database for AI. Store Vectors, Images, Texts, Videos, etc. Use with LLMs/LangChain. Store, query, version, & visualize any AI data. Stream data in real-time to PyTorch/TensorFlow. https://activeloop.ai
Faiss Stars Release Contributors A library for efficient similarity search and clustering of dense vectors. vector
milvus Stars Release Contributors A cloud-native vector database, storage for next generation AI applications cloud,vector
weaviate Stars Release Contributors Weaviate is an open-source vector database that stores both objects and vectors, allowing for the combination of vector search with structured filtering with the fault tolerance and scalability of a cloud-native database​. cloud,vector

Observation

Name Stars Release Contributors About Tag
OpenLLMetry Stars Release Contributors Open-source observability for your LLM application, based on OpenTelemetry
Helicone AI Stars Release Contributors 🧊 The open-source LangSmith alternative for logging, monitoring, and debugging AI applications.
phoenix Stars Release Contributors ML Observability in a Notebook - Uncover Insights, Surface Problems, Monitor, and Fine Tune your Generative LLM, CV and Tabular Models
wandb Stars Release Contributors πŸ”₯ A tool for visualizing and tracking your machine learning experiments. This repo contains the CLI and Python API.

Alignment

Name Stars Release Contributors About Tag
OpenRLHF Stars Release Contributors An Easy-to-use, Scalable and High-performance RLHF Framework (70B+ PPO Full Tuning & Iterative DPO & LoRA & Mixtral)
Self-RLHF Stars Release Contributors Safe RLHF: Constrained Value Alignment via Safe Reinforcement Learning from Human Feedback

Outputs

Name Stars Release Contributors About Tag
Instructor Stars Release Contributors structured outputs for llms
Outlines Stars Release Contributors Structured Text Generation

About

πŸŽ‰ An awesome & curated list of best LLMOps tools.

Topics

Resources

Stars

Watchers

Forks