Pushing Cloud Events to PostreSQL (and TimescaleDB)

Extracts information from JSON based cloud events and pushes them to PostgreSQL. It is intended to be used with TimescaleDB.

Input

Cloud event:

• Data Content Type: Mime type of the payload, must be application/json
• Payload: JSON payload from which to extract values.

Output

There is no output. The result will be written to the configured PostgreSQL instance.

Payload

The application expects a JSON payload structure, from which it extracts fields and tags using JSON path expressions.

Configuration

You can use the following environment variables to configure its behavior:

Name	Required	Default	Description
DISABLE_TRY_PARSE		false	Disable trying to parse expected value from String format
RUST_LOG		none	The configuration of the logger, also see https://docs.rs/env_logger/latest/env_logger/
ENDPOINT__BIND_ADDR		127.0.0.1:8080	The address the HTTP server binds to
ENDPOINT__MAX_JSON_PAYLOAD_SIZE		65536	Maximum payload size for JSON
ENDPOINT__TOKEN		none	A bearer token the caller has to provide
ENDPOINT__USERNAME		none	A username the caller has to provide (requires "password" too)
ENDPOINT__PASSWORD		none	The password for the username
POSTGRESQL__TABLE	x	none	The table to write to
POSTGRESQL__TIME_COLUMN	x	none	The column to receive the timestamp
POSTGRESQL__CONNECTION__HOST	x	none	The hostname (or IP address) of the PostgreSQL instance
POSTGRESQL__CONNECTION__USER	x	none	The username to use for authenticating to the database
POSTGRESQL__CONNECTION__PASSWORD	x	none	The password to use for authenticating to the database
POSTGRESQL__CONNECTION__DBNAME	x	none	The database to use

Tags and fields

Additionally, you need to configure a set of fields and (optionally) some tags, which make up the write query. Both are configured using environment variables. Fields are prefixed with FIELD_ and tags are prefixed with TAG_.

JSON paths for both fields and tags must result in a single element. Queries which end up with no fields will not be executed.

Paths for fields are rooted to the data section of the cloud event. Paths for tags are rooted at the JSON representation of the cloud event.

Value types

You can also add a TYPE_FIELD_ (and TYPE_TAG_) prefixed variables, which define the expected type for the field or tag.

The following types are available:

none (the default)

Try auto-conversion. For numbers, this will try a float first, then fall back to signed, and then to unsigned integers.

float, number

Floating point value (`DOUBLE PRECISION`)

string, text

Text value (`VARCHAR`)

bool, boolean

Boolean value (`BOOLEAN`)

int, integer

Signed integer value (`BIGINT`)

uint, unsigned

Unsigned integer value (`NUMERIC`)

If a value cannot be converted, and error is raised.

PostgreSQL specifics

For PostgresSQL, tags and fields will end up in the same SQL statement, simply adding them as an SQL field in the insert statement. The only difference is, that tags have access to the full cloud events JSON for extracting information, and fields have not.

Examples

The following example defines a field (named temperature), which will take the value from the field temp of the data section of the cloud events:

- name: FIELD_TEMPERATURE
  value: $.temp

For each field, you can also configure the expected type, the default is to try and auto-convert the value:

- name: TYPE_FIELD_TEMPERATURE
  value: float

The following example defines a tag (named device_id), which will take the value from the cloud events attribute subject:

- name: TAG_DEVICE_ID
  value: $.subject

Building

You can build the container image using:

cargo build --release
docker build . -t drogue-influxdb-pusher