Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: generic event source #895

Merged
merged 11 commits into from
Oct 12, 2020
3 changes: 2 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -9,4 +9,5 @@ hack/**/debug
debug.test
*.iml
.coverage
*.out
*.out
site/
53 changes: 50 additions & 3 deletions api/event-source.html

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

94 changes: 90 additions & 4 deletions api/event-source.md

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

28 changes: 24 additions & 4 deletions api/openapi-spec/swagger.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Binary file added docs/assets/generic-eventsource.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
1 change: 1 addition & 0 deletions docs/eventsources/deployment-strategies.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,7 @@ EventSource object, defaults to 1.
- Redis
- Resource
- Calendar
- Generic

### Replicas Of Recreate Types

Expand Down
82 changes: 82 additions & 0 deletions docs/eventsources/generic.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
# Generic EventSource

Generic eventsource extends Argo-Events eventsources via a simple gRPC contract. This is specifically useful when you want to onboard a custom eventsource implementation.

## Contract

In order to qualify as generic eventsource, the eventsource server needs to implement following gRPC contract,

syntax = "proto3";

package generic;

service Eventing {
rpc StartEventSource(EventSource) returns (stream Event);
}

message EventSource {
// The event source name.
string name = 1;
// The event source configuration value.
bytes config = 2;
}

/**
* Represents an event
*/
message Event {
// The event source name.
string name = 1;
// The event payload.
bytes payload = 2;
}

The proto file is available [here](https://github.com/argoproj/argo-events/blob/master/eventsources/sources/generic/generic.proto).

## Architecture

<br/>
<br/>

![arch](../assets/generic-eventsource.png)

<br/>

Consider a generic eventsource,

apiVersion: argoproj.io/v1alpha1
kind: EventSource
metadata:
name: generic
spec:
generic:
example:
insecure: true
url: "generic-event-source-server.argo-events.svc:8080"
config: |-
key1: value1
key2: value2

The values placed under `config` field follows a free-form style and Argo-Events eventsource client is not
opinionated about them. Although, it is expected that the eventsource server implemented by the user is able to parse the configuration.

## Flow

1. The eventsource client connects to the server via the `url` defined under eventsource `spec` and sends over the configuration defined
under `config` over an RPC call.

2. The eventsource server then parses the configuration and connects to any external source if required to consume the events.
The eventsource server can produce events without connecting to any external source, e.g. a special implementation of calendar events.

3. The events from eventsource server are streamed back to the client.

4. Client then writes the events to the eventbus which are read by the sensor to trigger the workflows.

## Connection Strategy

The eventsource client performs indefinite retries to connect to the eventsource server and receives events over a stream upon successful
connection. This also applies when the eventsource server goes down.

## Deployment Strategy

The deployment strategy for the generic eventsource is `Recreate`. More info is available [here](../eventsources/deployment-strategies.md).
8 changes: 8 additions & 0 deletions eventsources/eventing.go
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ import (
"github.com/argoproj/argo-events/eventsources/sources/emitter"
"github.com/argoproj/argo-events/eventsources/sources/file"
"github.com/argoproj/argo-events/eventsources/sources/gcppubsub"
"github.com/argoproj/argo-events/eventsources/sources/generic"
"github.com/argoproj/argo-events/eventsources/sources/github"
"github.com/argoproj/argo-events/eventsources/sources/gitlab"
"github.com/argoproj/argo-events/eventsources/sources/hdfs"
Expand Down Expand Up @@ -223,6 +224,13 @@ func GetEventingServers(eventSource *v1alpha1.EventSource) map[apicommon.EventSo
}
result[apicommon.PulsarEvent] = servers
}
if len(eventSource.Spec.Generic) != 0 {
VaibhavPage marked this conversation as resolved.
Show resolved Hide resolved
servers := []EventingServer{}
for k, v := range eventSource.Spec.Generic {
servers = append(servers, &generic.EventListener{EventSourceName: eventSource.Name, EventName: k, GenericEventSource: v})
}
result[apicommon.GenericEvent] = servers
}
return result
}

Expand Down
Loading