feat: URL Parameters #115
feat: URL Parameters #115
Conversation
|
Good suggestion. However, content transformation use can be infinite. Instead, intent is to have content to be processed using transformation approach like here: https://infinyon.com/blog/2023/09/github-stars-to-slack. So approach is create SmartModule to process url and send out appropriate transformation |
|
@Day-OS if you cherry pick the commit on https://github.com/infinyon/http-sink-connector/tree/ac/params-reuse-request I refactored so the client and most of the request is not reinitialized each loop. (I think there was also a little buglet fixed in this refactor where the loop was sending the current record to the previous records URL with query parameters) |
It looks good, I've cherry-picked it |
| } | ||
|
|
||
| #[async_trait] | ||
| impl Sink<String> for HttpSink { |
There was a problem hiding this comment.
Why the Sink trait impl was removed? This is a contract that all sink connectors must follow.
|
|
||
| #[derive(Debug)] | ||
| pub(crate) struct HttpSink { | ||
| #[allow(dead_code)] | ||
| client: Client, |
There was a problem hiding this comment.
If it is not used anywhere, why do we need?
| @@ -32,6 +33,19 @@ pub(crate) struct HttpConfig { | |||
| /// Http connect timeout in milliseconds | |||
| #[serde(with = "humantime_serde", default = "default_http_connect_timeout")] | |||
| pub http_connect_timeout: Duration, | |||
|
|
|||
| //HTTP Parameters that can be gattered from a Message if the message is a json file | |||
| #[serde(default = "default_http_params")] | |||
There was a problem hiding this comment.
| #[serde(default = "default_http_params")] | |
| #[serde(default)] |
| if let Some(ref suffix) = param.suffix { | ||
| value = value.clone() + &suffix; | ||
| } | ||
| builder = builder.query(&[(encode(&url_key), &value)]); |
There was a problem hiding this comment.
If we use references at the end, can we do the same without all those clonings above?
(FROM README)
URL Parameter Configuration
Example: Inserting an ID from a JSON record into a URL Parameter
Let's assume a scenario that the following endpoint requires a SQL condition to update a piece of information:
https://someurl.com/It accepts
updatecondition=<SQL CONDITION HERE>as a parameter to set the condition for the exact row that we want to update.We have the following JSON record that we want to get updated:
{ "id": 2901, "name": "Luiz Barros Rocha", "age": 26 }We could write a url_parameter containing the following information:
This would be the ending result:
https://someurl.com?updatecondition=user_id+%3D+2901