Skip to content

Commit

Permalink
📦 v0.0.1 (Private Preview) (#115)
Browse files Browse the repository at this point in the history
## Changelog

### Features

- ✨ #81: Allow to chat message based for specific models (@mkrueger12)

### Improvements

- 🔧 #78: Normalize response latency by response token count (@roma-glushko)
- 📝 #112 added the CLI banner info (@roma-glushko)

### Miscellaneous 

- 📝 #114 Make links actual across the project (@roma-glushko)
  • Loading branch information
roma-glushko authored Jan 31, 2024
1 parent 2c8ca01 commit b02c949
Show file tree
Hide file tree
Showing 33 changed files with 560 additions and 81 deletions.
2 changes: 1 addition & 1 deletion .goreleaser.yml
Original file line number Diff line number Diff line change
Expand Up @@ -144,7 +144,7 @@ brews:
#
# Default depends on the client.
# Templates: allowed
url_template: "https://github.mycompany.com/foo/bar/releases/download/{{ .Tag }}/{{ .ArtifactName }}"
url_template: "https://github.com/einstack/glide/releases/download/{{ .Tag }}/{{ .ArtifactName }}"

# Allows you to set a custom download strategy. Note that you'll need
# to implement the strategy and add it to your tap repository.
Expand Down
15 changes: 15 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,21 @@ The changelog consists of three categories:
- **Improvements** - bugfixes, performance and other types of improvements to existing functionality
- **Miscellaneous** - all other updates like build, release, CLI, etc.

## 0.0.1 (Jan 31st, 2024)

### Features

-#81: Allow to chat message based for specific models (@mkrueger12)

### Improvements

- 🔧 #78: Normalize response latency by response token count (@roma-glushko)
- 📝 #112 added the CLI banner info (@roma-glushko)

### Miscellaneous

- 📝 #114 Make links actual across the project (@roma-glushko)

## 0.0.1-rc.2 (Jan 22nd, 2024)

### Improvements
Expand Down
2 changes: 1 addition & 1 deletion CODE_OF_CONDUCT.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ a project may be further defined and clarified by project maintainers.
## Enforcement

Instances of abusive, harassing, or otherwise unacceptable behavior may be
reported by contacting the project team at [email protected]. All
reported by contacting the project team at [[email protected]](mailto:[email protected]). All
complaints will be reviewed and investigated and will result in a response that
is deemed necessary and appropriate to the circumstances. The project team is
obligated to maintain confidentiality with regard to the reporter of an incident.
Expand Down
121 changes: 93 additions & 28 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,16 @@
# Glide: Cloud-Native LLM Gateway for Seamless LLMOps
<div align="center">
<img src="docs/images/glide.png" width="400px" alt="Glide GH Header" />
<img src="docs/logo/glide_no_bgd.png" width="300px" alt="Glide GH Header" />
</div>

[![LICENSE](https://img.shields.io/github/license/modelgateway/glide.svg?style=flat-square&color=%233f90c8)](https://github.com/modelgateway/glide/blob/main/LICENSE)

[![codecov](https://codecov.io/github/EinStack/glide/graph/badge.svg?token=F7JT39RHX9)](https://codecov.io/github/EinStack/glide)
[![Discord](https://img.shields.io/discord/1181281407813828710)](https://discord.gg/pt53Ej7rrc)
[![Documentation](https://img.shields.io/badge/build-view-violet%20?style=flat&logo=books&label=docs&link=https%3A%2F%2Fglide.einstack.ai%2F)](https://glide.einstack.ai/)
[![LICENSE](https://img.shields.io/github/license/EinStack/glide.svg?style=flat-square&color=%233f90c8)](https://github.com/EinStack/glide/blob/main/LICENSE)
[![FOSSA Status](https://app.fossa.com/api/projects/git%2Bgithub.com%2FEinStack%2Fglide.svg?type=shield)](https://app.fossa.com/projects/git%2Bgithub.com%2FEinStack%2Fglide?ref=badge_shield)

---

Glide is your go-to cloud-native LLM gateway, delivering high-performance LLMOps in a lightweight, all-in-one package.

Expand All @@ -14,9 +20,11 @@ so you can dive into tackling your core challenges.
Glide sits between your application and model providers to seamlessly handle various LLMOps tasks like
model failover, caching, key management, etc.

<img src="docs/images/marketecture.svg" />

Take a look at the develop branch.

Check out our [documentation](https://backlandlabs.mintlify.app/introduction)!
Check out our [documentation](https://glide.einstack.ai)!

> [!Warning]
> Glide is under active development right now. Give us a star to support the project ✨
Expand All @@ -28,7 +36,7 @@ Check out our [documentation](https://backlandlabs.mintlify.app/introduction)!
- Support **popular LLM providers**.
- **High performance**. Performance is our priority. We want to keep Glide "invisible" for your latency-wise, while providing rich functionality.
- **Production-ready observability** via OpenTelemetry, emit metrics on models health, allows whitebox monitoring.
- Straightforward and simple maintenance and configuration, centrilized API key control & management & rotation, etc.
- Straightforward and simple maintenance and configuration, centralized API key control & management & rotation, etc.

## Supported Providers

Expand All @@ -48,7 +56,7 @@ Check out our [documentation](https://backlandlabs.mintlify.app/introduction)!

Routers are a core functionality of Glide. Think of routers as a group of models with some predefined logic. For example, the resilience router allows a user to define a set of backup models should the initial model fail. Another example, would be to leverage the least-latency router to make latency sensitive LLM calls in the most efficient manner.

Detailed info on routers can be found [here](https://backlandlabs.mintlify.app/essentials/routers).
Detailed info on routers can be found [here](https://glide.einstack.ai/essentials/routers).

#### Available Routers

Expand All @@ -62,43 +70,47 @@ Detailed info on routers can be found [here](https://backlandlabs.mintlify.app/e

## Get Started

#### Install
### Installation

The easiest way to deploy Glide is to build from source.
The easiest way to deploy Glide is to our [demo repository](https://github.com/EinStack/glide-demo.git) and [docker-compose](https://docs.docker.com/compose/).

Steps to build a container with Docker can be found [here](https://backlandlabs.mintlify.app/introduction#install-and-deploy).
### 1. Clone the demo repository

```bash
git clone https://github.com/EinStack/glide-demo.git
```

### 2. Init Configs

The demo repository comes with a basic config. Additionally, you need to init your secrets by running:

```bash
make init # from the demo root
```

#### Set Configuration File
This will create the `secrets` directory with one `.OPENAI_API_KEY` file that you need to put your key to.

Find detailed information on configuration [here](https://backlandlabs.mintlify.app/essentials/configuration).
### 3. Start Glide

```yaml
telemetry:
logging:
level: debug # debug, info, warn, error, fatal
encoding: console
After that, just use docker compose via this command to start your demo environment:

routers:
language:
- id: myrouter
models:
- id: openai
openai:
api_key: ""
```bash
make up
```

#### Sample API Request to `/chat` endpoint
### 4. Sample API Request to `/chat` endpoint

See [API Reference](https://backlandlabs.mintlify.app/api-reference/introduction) for more details.
See [API Reference](https://glide.einstack.ai/api-reference/introduction) for more details.

```json
{
"model": "gpt-3.5-turbo", # this is not required but can be used to specify different prompts to different models
"message":
{
"role": "user",
"content": "Where was it played?"
},
"messageHistory": [
"messageHistory": [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Who won the world series in 2020?"},
{"role": "assistant", "content": "The Los Angeles Dodgers won the World Series in 2020."}
Expand All @@ -108,13 +120,60 @@ See [API Reference](https://backlandlabs.mintlify.app/api-reference/introduction

### API Docs

Once deployed, Glide comes with OpenAPI documentation that is accessible via http://127.0.0.1:9099/v1/swagger/index.html
Finally, Glide comes with OpenAPI documentation that is accessible via http://127.0.0.1:9099/v1/swagger/index.html

That's it 🙌

Use [our documentation](https://glide.einstack.ai) to further learn about Glide capabilities and configs.

---

Other ways to install Glide are available:

### Homebrew (MacOS)

```bash
brew tap einstack/tap
brew install einstack/tap/glide
```

### Snapcraft (Linux)

Coming Soon

### Docker Images

Glide provides official images in our [GHCR](https://github.com/EinStack/glide/pkgs/container/glide):

- Alpine 3.19:
```bash
docker pull ghcr.io/einstack/glide:latest-alpine
```

- Ubuntu 22.04 LTS:
```bash
docker pull ghcr.io/einstack/glide:latest-ubuntu
```

- Google Distroless (non-root)
```bash
docker pull ghcr.io/einstack/glide:latest-distroless
```

- RedHat UBI 8.9 Micro
```bash
docker pull ghcr.io/einstack/glide:latest-redhat
```

### Helm Chart

Coming Soon

## Community

- Join [Discord](https://discord.gg/pt53Ej7rrc) for real-time discussion

Open [an issue](https://github.com/modelgateway/glide/issues) or start [a discussion](https://github.com/modelgateway/glide/discussions)
Open [an issue](https://github.com/EinStack/glide/issues) or start [a discussion](https://github.com/EinStack/glide/discussions)
if there is a feature or an enhancement you'd like to see in Glide.

## Contribute
Expand All @@ -126,6 +185,12 @@ if there is a feature or an enhancement you'd like to see in Glide.

Thanks everyone for already put their effort to make Glide better and more feature-rich:

<a href="https://github.com/modelgateway/glide/graphs/contributors">
<a href="https://github.com/EinStack/glide/graphs/contributors">
<img src="https://contributors-img.web.app/image?repo=modelgateway/glide" />
</a>

## License

Apache 2.0

[![FOSSA Status](https://app.fossa.com/api/projects/git%2Bgithub.com%2FEinStack%2Fglide.svg?type=large)](https://app.fossa.com/projects/git%2Bgithub.com%2FEinStack%2Fglide?ref=badge_large)
6 changes: 3 additions & 3 deletions ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,9 @@ This document describes the current status and the upcoming milestones of the Gl
| :---: | :--- | :---: |
| 🍏 | **Unified Chat Endpoint Support** | 4 / 4 |
| 🍏 | **Fallback Routing Strategy** | 1 / 1 |
| 🍏 | **Priority, Round Robin, Weighted Round Robin, Least Latency** | 2 / 4 |
| 🍎 | **Documentation** | 1 / 1 |
| 🍎 | **Private Preview** | 4 / 5 |
| 🍏 | **Priority, Round Robin, Weighted Round Robin, Least Latency** | 4 / 4 |
| 🍏 | **Documentation** | 1 / 1 |
| 🍎 | **Private Preview** | 4.5 / 5 |
| 🍎 | **Streaming Support** | 0 / 4 |
| 🍎 | **Embedding Support** | 0 / 4 |
| 🍎 | **Caching** | 0 / 1 |
Expand Down
2 changes: 1 addition & 1 deletion SECURITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,4 @@
We want to keep Glide safe for everyone.

If you've discovered a security vulnerability in Glide,
we appreciate your help in disclosing it to us in a responsible manner, using this email: [email protected]
we appreciate your help in disclosing it to us in a responsible manner, using this email: [[email protected]](mailto:[email protected])
33 changes: 26 additions & 7 deletions docs/docs.go
Original file line number Diff line number Diff line change
Expand Up @@ -10,12 +10,13 @@ const docTemplate = `{
"description": "{{escape .Description}}",
"title": "{{.Title}}",
"contact": {
"name": "Glide Community",
"url": "https://github.com/modelgateway/glide"
"name": "EinStack Community",
"url": "https://github.com/EinStack/glide/",
"email": "[email protected]"
},
"license": {
"name": "Apache 2.0",
"url": "https://github.com/modelgateway/glide/blob/develop/LICENSE"
"url": "https://github.com/EinStack/glide/blob/develop/LICENSE"
},
"version": "{{.Version}}"
},
Expand Down Expand Up @@ -626,6 +627,17 @@ const docTemplate = `{
}
}
},
"schemas.OverrideChatRequest": {
"type": "object",
"properties": {
"message": {
"$ref": "#/definitions/schemas.ChatMessage"
},
"model_id": {
"type": "string"
}
}
},
"schemas.ProviderResponse": {
"type": "object",
"properties": {
Expand All @@ -639,11 +651,11 @@ const docTemplate = `{
}
},
"tokenCount": {
"$ref": "#/definitions/schemas.TokenCount"
"$ref": "#/definitions/schemas.TokenUsage"
}
}
},
"schemas.TokenCount": {
"schemas.TokenUsage": {
"type": "object",
"properties": {
"promptTokens": {
Expand All @@ -668,6 +680,9 @@ const docTemplate = `{
"items": {
"$ref": "#/definitions/schemas.ChatMessage"
}
},
"override": {
"$ref": "#/definitions/schemas.OverrideChatRequest"
}
}
},
Expand Down Expand Up @@ -700,16 +715,20 @@ const docTemplate = `{
}
}
}
},
"externalDocs": {
"description": "Documentation",
"url": "https://glide.einstack.ai/"
}
}`

// SwaggerInfo holds exported Swagger Info so clients can modify it
var SwaggerInfo = &swag.Spec{
Version: "1.0",
Version: "0.0.1",
Host: "localhost:9099",
BasePath: "/",
Schemes: []string{"http"},
Title: "Glide Gateway",
Title: "Glide",
Description: "API documentation for Glide, an open-source lightweight high-performance model gateway",
InfoInstanceName: "swagger",
SwaggerTemplate: docTemplate,
Expand Down
Loading

0 comments on commit b02c949

Please sign in to comment.