Skip to content

Latest commit

 

History

History
124 lines (112 loc) · 7.62 KB

README.md

File metadata and controls

124 lines (112 loc) · 7.62 KB

Conduit Connector for HTTP

The HTTP connector is a Conduit plugin. It provides both, a source and a destination HTTP connectors.

How to build?

Run make build to build the connector's binary.

Testing

Run make test to run all the unit tests.

Source

The HTTP source connector pulls data from the HTTP URL every pollingPeriod, the source adds the params and headers to the request, and sends it to the URL with the specified method from the Configuration. The returned data is used to create an openCDC record and return it.

Note: when using the OPTIONS method, the resulted options will be added to the record's metadata.

Configuration

name description required default value example
url HTTP URL to send requests to. true https://example.com/api/v1
method HTTP method to use in the request, supported methods are (GET,HEAD,OPTIONS). false GET POST
headers HTTP headers to use in the request, comma separated list of : separated pairs. false Authorization:Bearer TOKEN_VALUE,Content-Type:application/xml
params.* parameters to use in the request, use params.* as the config key and specify its value, ex: set "params.id" as "1". false params.query="foobar"
pollingperiod how often the connector will get data from the url, formatted as a time.Duration. false "5m" "5m"
script.parseResponse

The path to a .js file containing the code to parse the response.

The signature of the function needs to be:

function parseResponse(bytes)

where bytes is the original response's raw bytes (i.e. unparsed).

The function needs to return a Response object.

 false /path/to/get_request_data.js

An example script can be found in test/get_request_data.js
script.getRequestData

The path to a .js file containing the code to prepare the request data.

The signature of the function needs to be:

function getRequestData(cfg, previousResponse, position)

where:

  • cfg (a map) is the connector configuration
  • previousResponse (a map) contains data from the previous response (if any), returned by parseResponse
  • position (a byte array) contains the starting position of the connector.

The function needs to return a Request object.

 false /path/to/parse_response.js

An example script can be found in test/parse_response.js

Destination

The HTTP destination connector pushes data from upstream resources to an HTTP URL via Conduit. the destination adds the params and headers to the request, and sends it to the URL with the specified method from the Configuration.

Note: The request Body that will be sent is the value under record.Payload.After, if you want to change the format of that or manipulate the field in any way, please check our Builtin Processors Docs , or check Standalone Processors Docs if you'd like to build your own processor .

Configuration

name description required default value
url Is a Go template expression for the URL used in the HTTP request, using Go templates. The value provided to the template is opencdc.Record, so the template has access to all its fields (e.g. .Position, .Key, .Metadata, and so on). We also inject all template functions provided by sprig to make it easier to write templates. true
method Http method to use in the request, supported methods are (POST,PUT,DELETE,PATCH). false POST
headers Http headers to use in the request, comma separated list of : separated pairs. false
params.* parameters to use in the request, use params.* as the config key and specify its value, ex: set "params.id" as "1". false