Skip to content

Commit

Permalink
Merge remote-tracking branch 'origin/master' into integrations-audit-…
Browse files Browse the repository at this point in the history 
…logs
  • Loading branch information
@shay23b
shay23b committed Oct 1, 2023
2 parents 4aa5f26 + 7c87bfa commit 33d9108
Show file tree
Hide file tree
Showing 41 changed files with 781 additions and 1,176 deletions.
216 changes: 125 additions & 91 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,63 +3,73 @@

</div>

<div align="center">

<h4>

**[Memphis](https://memphis.dev)** is an intelligent, frictionless message broker.<br>Made to enable developers to build real-time and streaming apps fast.
<p align="center">
<a href="https://memphis.dev/discord"><img src="https://img.shields.io/discord/963333392844328961?color=6557ff&label=discord" alt="Discord"></a>
<a href="https://github.com/memphisdev/memphis/issues?q=is%3Aissue+is%3Aclosed"><img src="https://img.shields.io/github/issues-closed/memphisdev/memphis?color=6557ff"></a>
<img src="https://img.shields.io/npm/dw/memphis-dev?color=ffc633&label=installations">
<a href="https://github.com/memphisdev/memphis/blob/master/CODE_OF_CONDUCT.md"><img src="https://img.shields.io/badge/Code%20of%20Conduct-v1.0-ff69b4.svg?color=ffc633" alt="Code Of Conduct"></a>
<img alt="GitHub release (latest by date)" src="https://img.shields.io/github/v/release/memphisdev/memphis?color=61dfc6">
<img src="https://img.shields.io/github/last-commit/memphisdev/memphis?color=61dfc6&label=last%20commit">
</p>

</h4>
<div align="center">

<img width="200" alt="CNCF Silver Member" src="https://github.com/cncf/artwork/raw/master/other/cncf-member/silver/color/cncf-member-silver-color.svg#gh-light-mode-only">
<a href="https://landscape.cncf.io/?selected=memphis"><img width="200" alt="CNCF Silver Member" src="https://github.com/cncf/artwork/raw/master/other/cncf-member/silver/white/cncf-member-silver-white.svg#gh-dark-mode-only"></a>

</div>

<b><p align="center">
<a href="https://memphis.dev/pricing/">Cloud - </a><a href="https://memphis.dev/docs/">Docs</a> - <a href="https://twitter.com/Memphis_Dev">X</a> - <a href="https://www.youtube.com/channel/UCVdMDLCSxXOqtgrBaRUHKKg">YouTube</a>
</p></b>

<div align="center">

<img width="200" alt="CNCF Silver Member" src="https://github.com/cncf/artwork/raw/master/other/cncf-member/silver/color/cncf-member-silver-color.svg#gh-light-mode-only">

<h4>

**[Memphis.dev](https://memphis.dev)** is a highly scalable, painless, and effortless data streaming platform.<br>
Made to enable developers and data teams to collaborate and build<br>
real-time and streaming apps fast.

</h4>

</div>

<p align="center">
<a href="https://memphis.dev/pricing/">Cloud - </a><a href="https://memphis.dev/docs/">Docs</a> - <a href="https://twitter.com/Memphis_Dev">Twitter</a> - <a href="https://www.youtube.com/channel/UCVdMDLCSxXOqtgrBaRUHKKg">YouTube</a>
</p>

<p align="center">
<a href="https://memphis.dev/discord"><img src="https://img.shields.io/discord/963333392844328961?color=6557ff&label=discord" alt="Discord"></a>
<a href="https://github.com/memphisdev/memphis/issues?q=is%3Aissue+is%3Aclosed"><img src="https://img.shields.io/github/issues-closed/memphisdev/memphis?color=6557ff"></a>
<img src="https://img.shields.io/npm/dw/memphis-dev?color=ffc633&label=installations">
<a href="https://github.com/memphisdev/memphis/blob/master/CODE_OF_CONDUCT.md"><img src="https://img.shields.io/badge/Code%20of%20Conduct-v1.0-ff69b4.svg?color=ffc633" alt="Code Of Conduct"></a>
<img alt="GitHub release (latest by date)" src="https://img.shields.io/github/v/release/memphisdev/memphis?color=61dfc6">
<img src="https://img.shields.io/github/last-commit/memphisdev/memphis?color=61dfc6&label=last%20commit">
</p>
## 🚀 Getting Started
[Tutorials](https://docs.memphis.dev/memphis/getting-started/tutorials) | [Videos](https://www.youtube.com/playlist?list=PL_7iYjqhtXpWpZT2U0zDYo2eGOoGmg2mm)<br>
### ☸ Helm for Kubernetes
```shell
helm repo add memphis https://k8s.memphis.dev/charts/ --force-update && \
helm install my-memphis memphis/memphis --create-namespace --namespace memphis
```
**Establish a local access**
```shell
To access Memphis using UI/CLI/SDK from localhost, run the below commands:

- kubectl port-forward service/memphis 6666:6666 9000:9000 7770:7770 --namespace memphis > /dev/null &

Memphis.dev is more than a broker. It's a new streaming stack.<br><br>
It significantly accelerates the development of real-time applications that require a streaming platform with<br>
high throughput, low latency, easy troubleshooting, fast time-to-value,<br>minimal platform operations, and all the observability you can think of.<br>

## 🫣 A world without Memphis
When your application requires a message broker or a queue,<br>
Implementing one will require you to -
- Build a dead-letter queue, create observability, and a retry mechanism
- Build a scalable environment
- Create client wrappers
- Tag events to achieve multi-tenancy
- Enforce schemas and handle transformations
- Handle back pressure. Client or queue side
- Configure monitoring and real-time alerts
- Create a cloud-agnostic implementation
- Create config alignment between production to a dev environment
- Spent weeks and months learning the internals through archival documentation, ebooks, and courses
- Onboard your developers<br>
And the list continues...<br>

**Or, you can just use [Memphis](https://memphis.dev)** and focus your resources on tasks that matter 😎
<br>
For interacting with the broker via HTTP:

- kubectl port-forward service/memphis-rest-gateway 4444:4444 --namespace memphis > /dev/null &

Dashboard/CLI: http://localhost:9000
Broker: localhost:6666 (Client Connections)
REST gateway: localhost:4444 (Data + Mgmt)
```
### 🐳 Docker Compose
```shell
curl -s https://memphisdev.github.io/memphis-docker/docker-compose.yml -o docker-compose.yml && \
docker compose -f docker-compose.yml -p memphis up
```
**Establish a local access**
```shell
Dashboard/CLI: http://localhost:9000
Broker: localhost:6666
```

## ✨ Key Features [v1.3.0](https://docs.memphis.dev/memphis/release-notes/releases/v1.3.0-latest)

[**Roadmap**](https://github.com/orgs/memphisdev/projects/2/views/1)
[**Take part in our roadmap!**](https://memphis.dev/roadmap)

![20](https://user-images.githubusercontent.com/70286779/220196529-abb958d2-5c58-4c33-b5e0-40f5446515ad.png) Production-ready message broker in under 3 minutes<br>
![20](https://user-images.githubusercontent.com/70286779/220196529-abb958d2-5c58-4c33-b5e0-40f5446515ad.png) Easy-to-use UI, CLI, and SDKs<br>
Expand All @@ -72,35 +82,84 @@ And the list continues...<br>
![20](https://user-images.githubusercontent.com/70286779/220196529-abb958d2-5c58-4c33-b5e0-40f5446515ad.png) Kubernetes-native<br>
![20](https://user-images.githubusercontent.com/70286779/220196529-abb958d2-5c58-4c33-b5e0-40f5446515ad.png) Community driven<br>

<div align="center">
<img src="https://user-images.githubusercontent.com/70286779/225559098-726d57bd-96f6-40d4-bc35-c2648a5c463a.png" style="width:800px;">
</div>
<table>

<tr>
<th>
<a href="https://docs.memphis.dev">End-to-end Observability</a>
</th>
<th>
<a href="https://docs.memphis.dev">"Feel" your producers, consumers, and records</a>
</th>

</tr>

<tr>
<td width="50%">
<a href="https://docs.memphis.dev">
<img src="https://github.com/memphisdev/memphis/assets/70286779/5763319d-1c06-4478-b0ee-582f37acf2b8" />
</a>
</td>
<td width="50%">
<a href="https://docs.memphis.dev">
<img width="1512" alt="Screenshot 2023-09-29 at 10 17 01" src="https://github.com/memphisdev/memphis/assets/70286779/1fd5358d-adb8-4aa5-9b19-9640bdfd2417">
</a>
</td>
</tr>

<tr>
<th>
<a href="https://docs.memphis.dev">Quickly analyze system health using a graph overview</a>
</th>
<th>
<a href="https://docs.memphis.dev">Never lose a message with automatic dead-letter</a>
</th>
</tr>

<tr>
<td width="50%">
<a href="https://docs.memphis.dev">
<img src="https://github.com/memphisdev/memphis/assets/70286779/1be9768b-b658-4bb3-89bd-ee814a8813b9" />
</a>
</td>
<td width="50%">
<a href="https://docs.memphis.dev">
<img width="1512" alt="Screenshot 2023-09-29 at 10 17 12" src="https://github.com/memphisdev/memphis/assets/70286779/2d6638d8-d1f1-4aba-8e36-7b7a57e76f16">
</a>
</td>
</tr>

<tr>
<th>
<a href="https://docs.memphis.dev/memphis/memphis-broker/concepts/storage-and-redundancy">Save up 96% storage costs with Storage tiering</a>
</th>
<th>
<a href="https://docs.memphis.dev/memphis/memphis-schemaverse/schemaverse-schema-management">Increase data quality with schemas</a>
</th>
</tr>

<tr>
<td width="50%">
<a href="https://docs.memphis.dev">
<img width="1512" alt="Screenshot 2023-09-29 at 10 17 58" src="https://github.com/memphisdev/memphis/assets/70286779/beeaec9f-c8a5-4f6a-b520-a810eb68aa13">
</a>
</td>
<td width="50%">
<a href="https://docs.memphis.dev">
<img width="1512" alt="Screenshot 2023-09-29 at 10 16 40" src="https://github.com/memphisdev/memphis/assets/70286779/7c62977a-2d83-4c54-b233-084958be1411">
</a>
</td>
</tr>

</table>

## Public case studies
- [Dstny - Building the next-gen in-house communication using Memphis.dev](https://memphis.dev/blog/how-dstny-building-the-future-of-in-house-communication-using-memphis-dev/)
- [Gastromatic - Synchronizing data using Memphis.dev](https://medium.com/gastromatic/synchronizing-data-using-memphis-dev-a-case-study-2e6e9a7b5512)
- [KELA - Real-time cyber threats identification](https://memphis.dev/blog/how-kela-is-using-memphis-dev-for-real-time-cyber-threats-identification/)
- [Handling millions of discord messages](https://memphis.dev/blog/how-cactusfire-handles-millions-of-daily-discord-messages-using-memphis-dev/)

## 🚀 Getting Started
Helm for Kubernetes☸
```shell
helm repo add memphis https://k8s.memphis.dev/charts/ --force-update && \
helm install my-memphis memphis/memphis --create-namespace --namespace memphis
```
Docker🐳 Compose
```shell
curl -s https://memphisdev.github.io/memphis-docker/docker-compose.yml -o docker-compose.yml && \
docker compose -f docker-compose.yml -p memphis up
```

<p align="center">
<a href="https://youtu.be/-5YmxYRQsdw"><img align="center" alt="connect your first app" src="https://img.youtube.com/vi/-5YmxYRQsdw/0.jpg"></a>
</p>

[Tutorials](https://docs.memphis.dev/memphis/getting-started/tutorials)<br>
[Installation videos](https://www.youtube.com/playlist?list=PL_7iYjqhtXpWpZT2U0zDYo2eGOoGmg2mm)<br><br>

## High-Level Architecture
## Network diagram

<a href="https://docs.memphis.dev/memphis/memphis/architecture">
<p align="center">
Expand All @@ -109,31 +168,6 @@ docker compose -f docker-compose.yml -p memphis up
</p>
</a>

## Local access
### Via Kubernetes
```shell
To access Memphis using UI/CLI/SDK from localhost, run the below commands:

- kubectl port-forward service/memphis 6666:6666 9000:9000 7770:7770 --namespace memphis > /dev/null &

For interacting with the broker via HTTP:

- kubectl port-forward service/memphis-rest-gateway 4444:4444 --namespace memphis > /dev/null &

Dashboard/CLI: http://localhost:9000
Broker: localhost:6666 (Client Connections)
REST gateway: localhost:4444 (Data + Mgmt)
```

**For Production Environments**
Please expose the UI, Cluster, and Control-plane via k8s ingress / load balancer / nodeport

### Via Docker
```shell
Dashboard/CLI: http://localhost:9000
Broker: localhost:6666
```

## SDKs supported features
Feature | Go | Python | JS | .NET | Java | Rust
Expand Down
21 changes: 12 additions & 9 deletions models/functions.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -14,15 +14,18 @@ package models
import "time"

type FunctionsResult struct {
FunctionName string `json:"function_name"`
Description string `json:"description"`
Tags []string `json:"tags"`
Language string `json:"language"`
LastCommit time.Time `json:"last_commit"`
Link string `json:"link"`
Repository string `json:"repository"`
Branch string `json:"branch"`
Owner string `json:"owner"`
FunctionName string `json:"function_name"`
Description string `json:"description"`
Tags []string `json:"tags"`
RunTime string `json:"runtime"`
Memory int `json:"memory"`
Storgae int `json:"storgae"`
LastCommit time.Time `json:"last_commit"`
Link string `json:"link"`
Repository string `json:"repository"`
Branch string `json:"branch"`
Owner string `json:"owner"`
EnvironmentVars map[string]string `json:"environment_vars"`
}

type FunctionsRes struct {
Expand Down
6 changes: 6 additions & 0 deletions server/memphis_cloud.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2190,3 +2190,9 @@ func validateProducersCount(stationId int, tenantName string) error {

func InitializeCloudFunctionRoutes(functionsHandler FunctionsHandler, functionsRoutes *gin.RouterGroup) {
}

// Integrations

func (it IntegrationsHandler) GetSourecCodeBranches(c *gin.Context) {
c.IndentedJSON(401, "Unautorized")
}
38 changes: 27 additions & 11 deletions server/memphis_handlers_functions.go → server/memphis_handlers_functions_cloud.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ func (fh FunctionsHandler) GetAllFunctions(c *gin.Context) {
return
}

c.JSON(200, gin.H{"scm_integrated": functionsResult.ScmIntegrated, "functions": functionsResult.Functions})
c.IndentedJSON(200, gin.H{"scm_integrated": functionsResult.ScmIntegrated, "functions": functionsResult.Functions})
}

func (fh FunctionsHandler) GetFunctions(tenantName string) (models.FunctionsRes, error) {
Expand All @@ -54,7 +54,7 @@ func (fh FunctionsHandler) GetFunctions(tenantName string) (models.FunctionsRes,
}

func validateYamlContent(yamlMap map[string]interface{}) error {
requiredFields := []string{"function_name", "language"}
requiredFields := []string{"function_name", "runtime", "dependencies"}
missingFields := make([]string, 0)
for _, field := range requiredFields {
if _, exists := yamlMap[field]; !exists {
Expand Down Expand Up @@ -91,21 +91,37 @@ func GetFunctionsDetails(functionsDetails []functionDetails) ([]models.Functions
}
}

var environmentVarsStrings map[string]string
environmentVarsInterfaceSlice, ok := fucntionContentMap["environment_vars"].([]interface{})
if ok {
environmentVarsStrings = make(map[string]string, len(fucntionContentMap["environment_vars"].([]interface{})))
for _, environmentVar := range environmentVarsInterfaceSlice {
environmentVarMap := environmentVar.(map[interface{}]interface{})
for k, v := range environmentVarMap {
if str, ok := v.(string); ok {
environmentVarsStrings[k.(string)] = str
}
}
}
}
description, ok := fucntionContentMap["description"].(string)
if !ok {
description = ""
}

functionDetails := models.FunctionsResult{
FunctionName: fucntionContentMap["function_name"].(string),
Description: description,
Tags: tagsStrings,
Language: fucntionContentMap["language"].(string),
LastCommit: *commit.Commit.Committer.Date,
Link: *fileContent.HTMLURL,
Repository: repo,
Branch: branch,
Owner: owner,
FunctionName: fucntionContentMap["function_name"].(string),
Description: description,
Tags: tagsStrings,
RunTime: fucntionContentMap["runtime"].(string),
LastCommit: *commit.Commit.Committer.Date,
Link: *fileContent.HTMLURL,
Repository: repo,
Branch: branch,
Owner: owner,
Memory: fucntionContentMap["memory"].(int),
Storgae: fucntionContentMap["storage"].(int),
EnvironmentVars: environmentVarsStrings,
}

functions = append(functions, functionDetails)
Expand Down
Loading

0 comments on commit 33d9108

Please sign in to comment.