httpjson: chain step request.url cannot be a replacement

[efd6]

We have a user who was attempting to implement a chained API ingestion using the custom API integration. The API that they were experimenting with in development (https://rickandmortyapi.com/api/) returns complete URLs to link between entities in the dataset. For example, the episode array.

$ curl https://rickandmortyapi.com/api/character/5 | jq
{
  "id": 5,
  "name": "Jerry Smith",
  "status": "Alive",
  "species": "Human",
  "type": "",
  "gender": "Male",
  "origin": {
    "name": "Earth (Replacement Dimension)",
    "url": "https://rickandmortyapi.com/api/location/20"
  },
  "location": {
    "name": "Earth (Replacement Dimension)",
    "url": "https://rickandmortyapi.com/api/location/20"
  },
  "image": "https://rickandmortyapi.com/api/character/avatar/5.jpeg",
  "episode": [
    "https://rickandmortyapi.com/api/episode/6",
    "https://rickandmortyapi.com/api/episode/7",
    "https://rickandmortyapi.com/api/episode/8",
    "https://rickandmortyapi.com/api/episode/9",
...

The user correctly attempted to have a step that would follow the set of episodes in their testing setup.

- step:
    replace: $.episode[:]
    request.method: GET
    request.url: $.episode[:]

This fails during config check with [failed to reloading inputs: 1 error: Error creating runner from config: parse "$.episode[:]": first path segment in URL cannot contain colon accessing ''chain.0.step.request.url'']. This happens because at the step.request.url must be validated as a URL, which it is not in its static form (though it would be after the replacement happens).

We can either document that this is a limitation of chain replacement (and point to CEL which is able to handle this kind of API) or add logic to elide URL checking in step request URLs. The second would probably just be a flag in the httpjson.requestConfig to conditionally validate, though this does complicate the code since currently validation is specified in the struct tag for the URL field. So I would be inclined to document and say that it is a non-supported feature.

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[alaudazzi]

@efd6

We can either document that this is a limitation of chain replacement (and point to CEL which is able to handle this kind of API)

I'm happy to help with the doc fix if you can help me clarify:

• on which doc page this limitation should be documented
• what's CEL
  Then we can draft the wording.

[ShourieG]

@efd6 We can use a small trick to achieve this via httpjson using the replace_with attribute

- step:
     request.url: https://rickandmortyapi.com/$.episode[:]
     request.method: GET
     replace: $.episode[:] 
     replace_with: 'https://rickandmortyapi.com/https://rickandmortyapi.com/,https://rickandmortyapi.com/'

This will basically create an invalid url initially https://rickandmortyapi.com/https://rickandmortyapi.com/api/episode/6, then fixing it with replace_with in the next step will produce https://rickandmortyapi.com/api/episode/6. It will work on individual array elements iteratively so all the requests will get replaced with proper urls by the end of the op. I know this is a bit hacky but it gets the job done and the point was just to show that this is possible in httpjson atm. Ofc one limitation is this is only possible when we have a constant replace pattern and know the value to replace with ahead of time, which is the case in this scenario.

NOTE: the reason we set request.url: https://rickandmortyapi.com/$.episode[:] instead of request.url:$.episode[:] is so that the template parser does not throw an invalid url error.

[efd6]

Thanks @ShourieG. So the question becomes, "Do we want to document that or make it work as a user would expect?"

[ShourieG]

Thanks @ShourieG. So the question becomes, "Do we want to document that or make it work as a user would expect?"

Let's make it work as the user would expect documenting this will not be a good idea as its more of a hack than an actual solution.

[alaudazzi]

Thank you @ShourieG for clarifying, as doc is no longer impacted I'll remove myself and the doc label from this ticket.

[efd6]

I have done some exploration of the required changes for this using a type-parametric approach (which I think would be less difficult/onerous than the code copying approach) and I believe that this is not realistically feasible to fix in the general case. The reason for this is that a functional URL is needed prior to placeholder expansion in client construction. This is not for the actual client, but for the metrics collection construction. What the work-around approach above does with the already-valid-url-prefix is to provide a hint that the client construction code needs to work.