Support for IPv6 address with scope ID

[killolol]

ES do not supprt IPv6 addressess with scope ID like : fe80::71a3:2b00:ddd3:753f%16

[jasontedor]

Would you please be more specific about the context and what you're expecting?

[killolol]

Here is error from log : ... "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [ip]", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"'xxxx::xxxx:xxxx:xxxx:xxxx%12' is not an IP string literal."}}}}}

It is valid IPv6 address since %12 is scope ID of IPv6 address.
I want to ask if you are planning support for scope ID in ES, or to parse it myself.
Thanks.

[clintongormley]

It looks like that not a scope ID but a zone index, which is used only for routing outgoing traffic via link local. None of the Address Formats use the zone index, so I very much doubt we'll be supporting it.

[vbohata]

Elasticsearch should automatically just ignore that part of IPv6 address.
Btw. :: is not accepted by elasticsearch, which should because this is a short format of 0000:0000:0000:0000:0000:0000:0000:0000.

[nick4u]

I have tons (about 1,5GB/day) of errors like that :

[DEBUG][o.e.a.b.TransportShardBulkAction] [ct102-15-log-fr1] [www-2017.10.09][0] failed to execute bulk item (index) BulkShardRequest [[www-2017.10.09][0]] containing [
org.elasticsearch.index.mapper.MapperParsingException: failed to parse [clientip]
.....
Caused by: java.lang.IllegalArgumentException: 'fe80::d1bb:dad5:6c5:95a4%12' is not an IP string literal.

coming from IIS logs

could anyone help me make appropriate logstash mutate/filter removing from field "clientip" string ("fe80::d1bb:dad5:6c5:95a4%12") everything past "%" including "%" sign?

[jdnewhouse]

@nick4u Here is how I fix the scope id problem in IPv6:

filter {
    if "%" in [clientip] { 
        grok {
            match => { "clientip" => "%{GREEDYDATA:clientip}%" }
            overwrite => [ "clientip" ]
            id => "grok_filter_ipv6"
        }
    }
}

[rdsworks]

I got around this in the Ingest Pipeline by defining a custom pattern_definition which removed the (%.+) from the already defined IPV6 pattern, then used that to strip scope.

It's throwing the exception when it's trying to store the value as type: ip.

"processors": [
{
"grok": {
"field": "message",
"patterns": ["%{TIMESTAMP_ISO8601:date_time} %{WORD:s_sitename} %{HOSTNAME:s_computername} %{IPC:s_ip} %{WORD:cs_method} %{URIPATH:cs_uri_stem} %{NOTSPACE:cs_uri_query} %{NUMBER:s_port} %{NOTSPACE:cs_username} %{IPC:c_ip}(%.+)? %{NOTSPACE:cs_version} %{NOTSPACE:cs_user_agent} %{NOTSPACE:cs_cookie} %{NOTSPACE:cs_referer} %{NOTSPACE:cs_host} %{NUMBER:sc_status} %{NUMBER:sc_substatus} %{NUMBER:sc_win32_status} %{NUMBER:sc_bytes} %{NUMBER:cs_bytes} %{NUMBER:time_taken}"],
"pattern_definitions": {
"IPV6C" : "((([0-9A-Fa-f]{1,4}:){7}([0-9A-Fa-f]{1,4}|:))|(([0-9A-Fa-f]{1,4}:){6}(:[0-9A-Fa-f]{1,4}|((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){5}(((:[0-9A-Fa-f]{1,4}){1,2})|:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){4}(((:[0-9A-Fa-f]{1,4}){1,3})|((:[0-9A-Fa-f]{1,4})?:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){3}(((:[0-9A-Fa-f]{1,4}){1,4})|((:[0-9A-Fa-f]{1,4}){0,2}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){2}(((:[0-9A-Fa-f]{1,4}){1,5})|((:[0-9A-Fa-f]{1,4}){0,3}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){1}(((:[0-9A-Fa-f]{1,4}){1,6})|((:[0-9A-Fa-f]{1,4}){0,4}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(:(((:[0-9A-Fa-f]{1,4}){1,7})|((:[0-9A-Fa-f]{1,4}){0,5}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:)))",
"IPC" : "(?:%{IPV6C}|%{IPV4})"
}
}