[Filebeat] Add support to unix sockets in Nginx module

[sayden]

Tries to cover this: #10309

As there's not example in the issue, I have configured an Nginx myself to listen unix sockets and then access it using netcat to generate a log output, which is included in the PR.

[kaiyan-sheng]

Looks good!

[sayden]

jenkins, test this

[jsoriano]

We should use source.address also when the source is a unix socket, so no need to add a field for this.

If you would need to add this field, it'd be:

- name: origin
  type: keyword
  description: >
     Some description

path is intended to be used with aliases, and migration for migrated fields.

[jsoriano]

For the use of keyword instead of array, take a look to https://www.elastic.co/guide/en/elasticsearch/reference/6.6/array.html

[sayden]

Those fields have been moved to ECS so no need to update fields here finally

[jsoriano]

It is a bit difficult to see changes here 😬, but, why setting source.ip to null if later it is removed in that case?

[sayden]

Good question and sorry! I remember to think about "I'm gonna copy-paste pretty code on PR" and I finally forgot it

boolean isPrivate(def dot, def ip) {
    try {
        StringTokenizer tok = new StringTokenizer(ip, dot);
        int firstByte = Integer.parseInt(tok.nextToken());
        int secondByte = Integer.parseInt(tok.nextToken());
        if (firstByte == 10) {         return true;       }
        if (firstByte == 192 && secondByte == 168) {         return true;       }
        if (firstByte == 172 && secondByte >= 16 && secondByte <= 31) {         return true;       }
        if (firstByte == 127) {         return true;       }
        return false;
    } catch (Exception e) {
        return false;
    }
}

try {
    ctx.source.ip = null;
    if (ctx.nginx.access.remote_ip_list == null) { return; }
    def found = false;
    for (def item : ctx.nginx.access.remote_ip_list) {
        if (!isPrivate(params.dot, item)) {
            ctx.source.ip = item;
            found = true;
            break;
        }
    }
    
    if (!found) {
        ctx.source.ip = ctx.nginx.access.remote_ip_list[0];
    }
} catch (Exception e) {
    ctx.source.ip = null;
}

The thing is that "convert" step in the pipeline was failing by not recognizing "" as a valid string with the error:

{"type":"mapper_parsing_exception","reason":"failed to parse field [source.ip] of type [ip] in document with id '6SrYOWkBxc4oeub4XID4'","caused_by":{"type":"illegal_argument_exception","reason":"'' is not an IP string literal."}

But an ignore_missing works if the field doesn't exists or it's null. It doesn't work if field is just empty and ignore_failure seemed to just omit the entire line (not the processing step).

I have to admit that it was terribly difficult

[jsoriano]

Now it seems that we can use yaml here 😌 #11209

[jsoriano]

Now it seems that we can use yaml here 😌 #11209