HTTP Body without Response Raw

[dahendel]

Can I get the body of the http response without enabling response_raw? servers all sending response raw is giving me 80GB indexes just for http data. because of an api call we have been troubleshooting it would be really helpful to have the json response logged. Thanks in advance for any help.

I am on 0.5.0. 3

Config excerpt--
[protocols.http]
ports = [80, 8080, 7200]
send_response = true

[http]
include_body_for=["application/json"]
send_headers = ["User-Agent", "Host", "Referer", "Set-Cookie", "Cookie"]
split_coookie = true
real_ip_header = "X-Forwarded-For"

[monicasarbu]

As it's configured now, Packetbeat exports all the headers for all http transactions and the body only for the http transaction with a json attachment. I am not sure I understand what fills up the 80GB. Are the headers the problem as they are stored for all transactions? Would you like to export the selected headers only for some http transactions?

[dahendel]

If I set send_response=false and have include_body_for=["application/json"] the body of the message does not show up in kibana.

Part of my problem before was that I had include_body_for=["txt/html", "application/json"]. That was probably a large part of the 80GB index. Now however, I just want the body to be in its own field, for example:
http.response.body: { "Status":{ "Status":"Success", "StatusCode":200 }, "resultInfo":{ "totalNumberOfResults":2, "pageNumber":1, "totalNumberOfPages":1, "resultsPerPage":2 }, "result":{ "genders":[ "Female", "Male" ] } }

instead of:
response_raw:
HTTP/1.1 200 OK
X-Backside-Transport: OK OK,OK OK,OK OK
Connection: Keep-Alive
Transfer-Encoding: chunked
Age: 67790
Date: Tue, 14 Jul 2015 01:37:10 GMT
Set-Cookie: BrowserId=X0yxkPGBTEmX6KraacypYQ;Path=/;Domain=.salesforce.com;Expires=Sat, 12-Sep-2015 01:37:10 GMT
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Content-Type: application/json
X-Client-IP: 127.0.0.1,127.0.0.1,10.26.109.5
X-Archived-Client-IP: 127.0.0.1,127.0.0.1,127.0.0.1

{ "Status":{ "Status":"Success", "StatusCode":200 }, "resultInfo":{ "totalNumberOfResults":2, "pageNumber":1, "totalNumberOfPages":1, "resultsPerPage":2 }, "result":{ "genders":[ "Female", "Male" ] } }

[monicasarbu]

In the current version, the body is added to the "response" field for the http transactions that have a json attachment if you set both:

send_response = true
include_body_for=["application/json"]

I agree it's not a good solution what we have now and it needs improvements. Adding the body in a extra field makes sense to me.

[monicasarbu]

@dahendel Starting with 5.0.0-alpha3, we introduced generic filtering that you might find it useful. With generic filtering you can define what fields to include or to drop from the event, or you can even drop the entire event if a condition is true. Here are more details.

[monicasarbu]

@dahendel I hope generic filtering solves better your use case. For more questions please the Beats forum.

[dahendel]

This is awesome. Thank you very much! Will certainly help.
On Jun 14, 2016 2:53 PM, "Monica Sarbu" [email protected] wrote:

Closed #179 #179.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#179 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/ALLVfY4BOBCF3ab7EdDsHw7zxzCA-vGwks5qLwbEgaJpZM4FYPlM
.