Skip to content

Commit

Permalink
feat: generic event source (argoproj#895)
Browse files Browse the repository at this point in the history
* feat: added generic event source

* feat: updated types for generic eventsource

* fix: remove site folder

* fix: added eventsource in eventing

* fix: added logic to auto-connect for generic eventsource

* fix: added logic to auto-connect for generic eventsource

* chore: updated mkdocs for generic eventsource

* fix: typo

* fix: logging in minio

* fix: conn closing condition

* chore: generic es logging

Signed-off-by: Derek Wang <[email protected]>
  • Loading branch information
VaibhavPage authored and whynowy committed Nov 17, 2020
1 parent 1eee338 commit a238bfd
Show file tree
Hide file tree
Showing 23 changed files with 1,400 additions and 336 deletions.
3 changes: 2 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -9,4 +9,5 @@ hack/**/debug
debug.test
*.iml
.coverage
*.out
*.out
site/
53 changes: 50 additions & 3 deletions api/event-source.html

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

94 changes: 90 additions & 4 deletions api/event-source.md

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

28 changes: 24 additions & 4 deletions api/openapi-spec/swagger.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Binary file added docs/assets/generic-eventsource.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
1 change: 1 addition & 0 deletions docs/eventsources/deployment-strategies.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,7 @@ EventSource object, defaults to 1.
- Redis
- Resource
- Calendar
- Generic

### Replicas Of Recreate Types

Expand Down
82 changes: 82 additions & 0 deletions docs/eventsources/generic.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
# Generic EventSource

Generic eventsource extends Argo-Events eventsources via a simple gRPC contract. This is specifically useful when you want to onboard a custom eventsource implementation.

## Contract

In order to qualify as generic eventsource, the eventsource server needs to implement following gRPC contract,

syntax = "proto3";
package generic;
service Eventing {
rpc StartEventSource(EventSource) returns (stream Event);
}
message EventSource {
// The event source name.
string name = 1;
// The event source configuration value.
bytes config = 2;
}
/**
* Represents an event
*/
message Event {
// The event source name.
string name = 1;
// The event payload.
bytes payload = 2;
}

The proto file is available [here](https://github.com/argoproj/argo-events/blob/master/eventsources/sources/generic/generic.proto).

## Architecture

<br/>
<br/>

![arch](../assets/generic-eventsource.png)

<br/>

Consider a generic eventsource,

apiVersion: argoproj.io/v1alpha1
kind: EventSource
metadata:
name: generic
spec:
generic:
example:
insecure: true
url: "generic-event-source-server.argo-events.svc:8080"
config: |-
key1: value1
key2: value2

The values placed under `config` field follows a free-form style and Argo-Events eventsource client is not
opinionated about them. Although, it is expected that the eventsource server implemented by the user is able to parse the configuration.

## Flow

1. The eventsource client connects to the server via the `url` defined under eventsource `spec` and sends over the configuration defined
under `config` over an RPC call.

2. The eventsource server then parses the configuration and connects to any external source if required to consume the events.
The eventsource server can produce events without connecting to any external source, e.g. a special implementation of calendar events.

3. The events from eventsource server are streamed back to the client.

4. Client then writes the events to the eventbus which are read by the sensor to trigger the workflows.

## Connection Strategy

The eventsource client performs indefinite retries to connect to the eventsource server and receives events over a stream upon successful
connection. This also applies when the eventsource server goes down.

## Deployment Strategy

The deployment strategy for the generic eventsource is `Recreate`. More info is available [here](../eventsources/deployment-strategies.md).
8 changes: 8 additions & 0 deletions eventsources/eventing.go
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ import (
"github.com/argoproj/argo-events/eventsources/sources/emitter"
"github.com/argoproj/argo-events/eventsources/sources/file"
"github.com/argoproj/argo-events/eventsources/sources/gcppubsub"
"github.com/argoproj/argo-events/eventsources/sources/generic"
"github.com/argoproj/argo-events/eventsources/sources/github"
"github.com/argoproj/argo-events/eventsources/sources/gitlab"
"github.com/argoproj/argo-events/eventsources/sources/hdfs"
Expand Down Expand Up @@ -223,6 +224,13 @@ func GetEventingServers(eventSource *v1alpha1.EventSource) map[apicommon.EventSo
}
result[apicommon.PulsarEvent] = servers
}
if len(eventSource.Spec.Generic) != 0 {
servers := []EventingServer{}
for k, v := range eventSource.Spec.Generic {
servers = append(servers, &generic.EventListener{EventSourceName: eventSource.Name, EventName: k, GenericEventSource: v})
}
result[apicommon.GenericEvent] = servers
}
return result
}

Expand Down
Loading

0 comments on commit a238bfd

Please sign in to comment.