Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Filebeat httpjson v2 input should set Content-Type header to match to specified encoder #24331

Closed
dcode opened this issue Mar 3, 2021 · 1 comment · Fixed by #24336
Closed
Assignees
Labels

Comments

@dcode
Copy link
Contributor

dcode commented Mar 3, 2021

Filebeat should match the Content-Type header to the encoding selected.

Steps to reproduce:

  1. Configure Filebeat to use httpjson v2 input (example configuration below)
  2. Try to specify form-encoded parameters
  3. Run the configuration with DEBUG logging
  4. Observe that the Content-Type is still set to application/json.

Example

Minimum viable config:

filebeat.inputs:
  - type: httpjson
    config_version: "2"
    interval: 1m
    enabled: true

    request.method: POST
    request.url: https://mb-api.abuse.ch/api/v1/
    request.encode_as: application/x-www-form-urlencoded

    request.transforms:
    - set:
        target: url.params.query
        value: get_recent
    - set:
        target: url.params.selector
        value: time

    response.split:
      target: body.data

output.console:
  enabled: true

Produces the following DEBUG output indicating Content-Type: application/json when run as ./filebeat run -e --d 'input.httpjson-stateless' --once -c test.yml

2021-03-03T16:44:49.442-0600	DEBUG	[input.httpjson-stateless]	v2/request.go:77	new request: v2.transformable{"header":http.Header{"Accept":[]string{"application/json"}, "Content-Type":[]string{"application/json"}, "User-Agent":[]string{"Elastic-Filebeat/8.0.0 (darwin; amd64; 08602f026af8d120a9e16a5b58fbd38cda07b94d; 2021-03-03 22:16:58 +0000 UTC)"}}, "url":url.URL{Scheme:"https", Opaque:"", User:(*url.Userinfo)(nil), Host:"mb-api.abuse.ch", Path:"/api/v1/", RawPath:"", ForceQuery:false, RawQuery:"query=get_recent&selector=time", Fragment:"", RawFragment:""}}	{"input_url": "https://mb-api.abuse.ch/api/v1/"}

The workaround is to specify the header directly in a request transform

filebeat.inputs:
  - type: httpjson
    config_version: "2"
    interval: 1m
    enabled: true

    request.method: POST
    request.url: https://mb-api.abuse.ch/api/v1/
    request.encode_as: application/x-www-form-urlencoded

    request.transforms:
    - set:
        target: header.Content-Type
        value: application/x-www-form-urlencoded
    - set:
        target: url.params.query
        value: get_recent
    - set:
        target: url.params.selector
        value: time

    response.split:
      target: body.data

output.console:
  enabled: true
@dcode dcode added the Filebeat Filebeat label Mar 3, 2021
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Mar 3, 2021
@elasticmachine
Copy link
Collaborator

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Mar 3, 2021
@leehinman leehinman self-assigned this Mar 3, 2021
leehinman added a commit to leehinman/beats that referenced this issue Mar 4, 2021
leehinman added a commit to leehinman/beats that referenced this issue Mar 8, 2021
leehinman added a commit to leehinman/beats that referenced this issue Mar 8, 2021
leehinman added a commit that referenced this issue Mar 8, 2021
leehinman added a commit that referenced this issue Mar 8, 2021
leweafan pushed a commit to leweafan/beats that referenced this issue Apr 28, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants