request help: should the plugin of kafka-logger add filter options and the body in request and response?

[cyxinda]

Issue description

There are three point,as follow:
1.
I hope the kafka-logger plugin would write the request body and the response body to kafka additionally.

I hope the kafka-logger plugin can add the filter features ,so that we can only record the necessary url .

These two functions should be optional by setting parameters.

Looking forward to your reply ,Thanks.

Environment

• apisix version (cmd: apisix version):
• OS:

[moonming]

@sshniro what do you think?

[sshniro]

I assume we can implement this filtering logic in the Log Util itself. Therefore, all the loggers can have this filtering logic enabled by default.

How about this format?

curl http://127.0.0.1:9080/apisix/admin/consumers -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
    "username": "foo",
    "plugins": {
       "kafka-logger": {
           "broker_list" :
             {
               "127.0.0.1":9092
             },
           "kafka_topic" : "test2",
           "key" : "key1",
           "log_filter":{
              response_body: true,
              request_body: true
           }
       }
    }
}'

By default request and response, body is disabled.
If agreed then I can implement the filtering logic in the logutil.

[cyxinda]

2. I hope the kafka-logger plugin can add the filter features ,so that we can only record the necessary url .
This means that I can filter out some URLs that I don't care about at all, such as CSS, JS, picture, etc.

[moonming]

you can enable kafka plugin in some routes, besides static files. one more think, why you deal with css, js files with API gateway? 月夜枫 <[email protected]> 于 2020年4月28日周二 下午5:53写道：

…

2. I hope the kafka-logger plugin can add the filter features ,so that we can only record the necessary url . This means that I can filter out some URLs that I don't care about at all, such as CSS, JS, picture, etc. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1512 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AGJZBK7DKDWCJOHCSMZZRH3RO2RS7ANCNFSM4MRSHW7A> .

[cyxinda]

you can enable kafka plugin in some routes, besides static files. one more think, why you deal with css, js files with API gateway? 月夜枫 [email protected] 于 2020年4月28日周二 下午5:53写道：
…
2. I hope the kafka-logger plugin can add the filter features ,so that we can only record the necessary url . This means that I can filter out some URLs that I don't care about at all, such as CSS, JS, picture, etc. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1512 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AGJZBK7DKDWCJOHCSMZZRH3RO2RS7ANCNFSM4MRSHW7A .

Sometimes, I need to proxy some web services with apisix, and the route contains wildcards. It naturally contains some resource files, such as CSS, JS, pictures, etc.

[sshniro]

2. I hope the kafka-logger plugin can add the filter features ,so that we can only record the necessary url .
This means that I can filter out some URLs that I don't care about at all, such as CSS, JS, picture, etc.

Then we need to include a regex in log until and send nil results if it does not satisfy the URL.

[sshniro]

BTW @moonming is there an easier way to obtain the response body?
https://github.com/openresty/lua-nginx-module#ngxrespget_headers
It seems ngx variable only exposes the response headers, but no the response body.

[highfly22]

BTW @moonming is there an easier way to obtain the response body?
https://github.com/openresty/lua-nginx-module#ngxrespget_headers
It seems ngx variable only exposes the response headers, but no the response body.

The response body can be retrieved by using body_filter_by_lua.

Please refer to https://gist.github.com/ejlp12/b3949bb40e748ae8367e17c193fa9602

Could you add response body to kafka logger plugin? @sshniro