[Fleet]: Log level dropdown is blank under Logs tab.

[amolnater-qasource]

Kibana version: 7.15.0 Snapshot Kibana cloud environment

Host OS and Browser version: All, All

Build details:

Build: 43322
Commit: f448fcd00b319a3be0d1a1ae356956446e4d7ef8
Artifact Link: https://snapshots.elastic.co/7.15.0-af472020/downloads/beats/elastic-agent/elastic-agent-7.15.0-SNAPSHOT-linux-x86_64.tar.gz

Preconditions:

1. 7.15.0 Snapshot Kibana cloud environment should be available.
2. An agent must be installed with Default policy having System integration.

Steps to reproduce:

1. Login to Kibana environment.
2. Navigate to Agents>Logs tab.
3. Click Log level dropdown and observe no log level filters available.

Expected Result:
Log level dropdown should give user options to select log level say- error, info, debug, warn etc.

Screenshot:

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[amolnater-qasource]

@manishgupta-qasource Please review.

[manishgupta-qasource]

Reviewed & Assigned to @jen-huang

CC: @EricDavisX

[jen-huang]

Looks like this is caused by lack of mappings on log.* fields. I also noticed that mappings are missing for a bunch of other fields like monitoring.* too:

This screenshot is from a logs-elastic_agent.filebeat-* from a 7.14 agent:

Here is a sample response of

GET logs-elastic_agent-*,logs-elastic_agent.*-*/_mapping

on my local machine, this includes mappings for logs-elastic_agent-*, logs-elastic_agent.filebeat-* and logs-elastic_agent.metricbeat-*:

Notice the lack of mappings for fields which are present on the documents like log.*, agent.*, event.dataset, host.os.type:

Response

{
  ".ds-logs-elastic_agent-default-2021.08.16-000001" : {
    "mappings" : {
      "dynamic" : "false",
      "_meta" : {
        "package" : {
          "name" : "elastic_agent"
        },
        "managed_by" : "ingest-manager",
        "managed" : true
      },
      "_data_stream_timestamp" : {
        "enabled" : true
      },
      "dynamic_templates" : [
        {
          "strings_as_keyword" : {
            "match_mapping_type" : "string",
            "mapping" : {
              "ignore_above" : 1024,
              "type" : "keyword"
            }
          }
        }
      ],
      "date_detection" : false,
      "properties" : {
        "@timestamp" : {
          "type" : "date"
        },
        "cloud" : {
          "properties" : {
            "account" : {
              "properties" : {
                "id" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                }
              }
            },
            "availability_zone" : {
              "type" : "keyword",
              "ignore_above" : 1024
            },
            "image" : {
              "properties" : {
                "id" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                }
              }
            },
            "instance" : {
              "properties" : {
                "id" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                },
                "name" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                }
              }
            },
            "machine" : {
              "properties" : {
                "type" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                }
              }
            },
            "project" : {
              "properties" : {
                "id" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                }
              }
            },
            "provider" : {
              "type" : "keyword",
              "ignore_above" : 1024
            },
            "region" : {
              "type" : "keyword",
              "ignore_above" : 1024
            }
          }
        },
        "container" : {
          "properties" : {
            "id" : {
              "type" : "keyword",
              "ignore_above" : 1024
            },
            "image" : {
              "properties" : {
                "name" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                }
              }
            },
            "labels" : {
              "type" : "object"
            },
            "name" : {
              "type" : "keyword",
              "ignore_above" : 1024
            }
          }
        },
        "data_stream" : {
          "properties" : {
            "dataset" : {
              "type" : "constant_keyword",
              "value" : "elastic_agent"
            },
            "namespace" : {
              "type" : "constant_keyword",
              "value" : "default"
            },
            "type" : {
              "type" : "constant_keyword",
              "value" : "logs"
            }
          }
        },
        "elastic_agent" : {
          "properties" : {
            "id" : {
              "type" : "keyword",
              "ignore_above" : 1024
            },
            "process" : {
              "type" : "keyword",
              "ignore_above" : 1024
            },
            "snapshot" : {
              "type" : "boolean"
            },
            "version" : {
              "type" : "keyword",
              "ignore_above" : 1024
            }
          }
        },
        "event" : {
          "properties" : {
            "agent_id_status" : {
              "type" : "keyword",
              "ignore_above" : 1024
            },
            "ingested" : {
              "type" : "date",
              "format" : "strict_date_time_no_millis||strict_date_optional_time||epoch_millis"
            }
          }
        },
        "host" : {
          "properties" : {
            "architecture" : {
              "type" : "keyword",
              "ignore_above" : 1024
            },
            "containerized" : {
              "type" : "boolean"
            },
            "domain" : {
              "type" : "keyword",
              "ignore_above" : 1024
            },
            "hostname" : {
              "type" : "keyword",
              "ignore_above" : 1024
            },
            "id" : {
              "type" : "keyword",
              "ignore_above" : 1024
            },
            "ip" : {
              "type" : "ip"
            },
            "mac" : {
              "type" : "keyword",
              "ignore_above" : 1024
            },
            "name" : {
              "type" : "keyword",
              "ignore_above" : 1024
            },
            "os" : {
              "properties" : {
                "build" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                },
                "codename" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                },
                "family" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                },
                "kernel" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                },
                "name" : {
                  "type" : "keyword",
                  "ignore_above" : 1024,
                  "fields" : {
                    "text" : {
                      "type" : "text"
                    }
                  }
                },
                "platform" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                },
                "version" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                }
              }
            },
            "type" : {
              "type" : "keyword",
              "ignore_above" : 1024
            }
          }
        },
        "message" : {
          "type" : "text"
        }
      }
    }
  },
  ".ds-logs-elastic_agent.filebeat-default-2021.08.16-000001" : {
    "mappings" : {
      "dynamic" : "false",
      "_meta" : {
        "package" : {
          "name" : "elastic_agent"
        },
        "managed_by" : "ingest-manager",
        "managed" : true
      },
      "_data_stream_timestamp" : {
        "enabled" : true
      },
      "dynamic_templates" : [
        {
          "strings_as_keyword" : {
            "match_mapping_type" : "string",
            "mapping" : {
              "ignore_above" : 1024,
              "type" : "keyword"
            }
          }
        }
      ],
      "date_detection" : false,
      "properties" : {
        "@timestamp" : {
          "type" : "date"
        },
        "cloud" : {
          "properties" : {
            "account" : {
              "properties" : {
                "id" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                }
              }
            },
            "availability_zone" : {
              "type" : "keyword",
              "ignore_above" : 1024
            },
            "image" : {
              "properties" : {
                "id" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                }
              }
            },
            "instance" : {
              "properties" : {
                "id" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                },
                "name" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                }
              }
            },
            "machine" : {
              "properties" : {
                "type" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                }
              }
            },
            "project" : {
              "properties" : {
                "id" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                }
              }
            },
            "provider" : {
              "type" : "keyword",
              "ignore_above" : 1024
            },
            "region" : {
              "type" : "keyword",
              "ignore_above" : 1024
            }
          }
        },
        "container" : {
          "properties" : {
            "id" : {
              "type" : "keyword",
              "ignore_above" : 1024
            },
            "image" : {
              "properties" : {
                "name" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                }
              }
            },
            "labels" : {
              "type" : "object"
            },
            "name" : {
              "type" : "keyword",
              "ignore_above" : 1024
            }
          }
        },
        "data_stream" : {
          "properties" : {
            "dataset" : {
              "type" : "constant_keyword",
              "value" : "elastic_agent.filebeat"
            },
            "namespace" : {
              "type" : "constant_keyword",
              "value" : "default"
            },
            "type" : {
              "type" : "constant_keyword",
              "value" : "logs"
            }
          }
        },
        "elastic_agent" : {
          "properties" : {
            "id" : {
              "type" : "keyword",
              "ignore_above" : 1024
            },
            "process" : {
              "type" : "keyword",
              "ignore_above" : 1024
            },
            "snapshot" : {
              "type" : "boolean"
            },
            "version" : {
              "type" : "keyword",
              "ignore_above" : 1024
            }
          }
        },
        "event" : {
          "properties" : {
            "agent_id_status" : {
              "type" : "keyword",
              "ignore_above" : 1024
            },
            "ingested" : {
              "type" : "date",
              "format" : "strict_date_time_no_millis||strict_date_optional_time||epoch_millis"
            }
          }
        },
        "host" : {
          "properties" : {
            "architecture" : {
              "type" : "keyword",
              "ignore_above" : 1024
            },
            "containerized" : {
              "type" : "boolean"
            },
            "domain" : {
              "type" : "keyword",
              "ignore_above" : 1024
            },
            "hostname" : {
              "type" : "keyword",
              "ignore_above" : 1024
            },
            "id" : {
              "type" : "keyword",
              "ignore_above" : 1024
            },
            "ip" : {
              "type" : "ip"
            },
            "mac" : {
              "type" : "keyword",
              "ignore_above" : 1024
            },
            "name" : {
              "type" : "keyword",
              "ignore_above" : 1024
            },
            "os" : {
              "properties" : {
                "build" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                },
                "codename" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                },
                "family" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                },
                "kernel" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                },
                "name" : {
                  "type" : "keyword",
                  "ignore_above" : 1024,
                  "fields" : {
                    "text" : {
                      "type" : "text"
                    }
                  }
                },
                "platform" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                },
                "version" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                }
              }
            },
            "type" : {
              "type" : "keyword",
              "ignore_above" : 1024
            }
          }
        },
        "message" : {
          "type" : "text"
        }
      }
    }
  },
  ".ds-logs-elastic_agent.metricbeat-default-2021.08.16-000001" : {
    "mappings" : {
      "dynamic" : "false",
      "_meta" : {
        "package" : {
          "name" : "elastic_agent"
        },
        "managed_by" : "ingest-manager",
        "managed" : true
      },
      "_data_stream_timestamp" : {
        "enabled" : true
      },
      "dynamic_templates" : [
        {
          "strings_as_keyword" : {
            "match_mapping_type" : "string",
            "mapping" : {
              "ignore_above" : 1024,
              "type" : "keyword"
            }
          }
        }
      ],
      "date_detection" : false,
      "properties" : {
        "@timestamp" : {
          "type" : "date"
        },
        "cloud" : {
          "properties" : {
            "account" : {
              "properties" : {
                "id" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                }
              }
            },
            "availability_zone" : {
              "type" : "keyword",
              "ignore_above" : 1024
            },
            "image" : {
              "properties" : {
                "id" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                }
              }
            },
            "instance" : {
              "properties" : {
                "id" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                },
                "name" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                }
              }
            },
            "machine" : {
              "properties" : {
                "type" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                }
              }
            },
            "project" : {
              "properties" : {
                "id" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                }
              }
            },
            "provider" : {
              "type" : "keyword",
              "ignore_above" : 1024
            },
            "region" : {
              "type" : "keyword",
              "ignore_above" : 1024
            }
          }
        },
        "container" : {
          "properties" : {
            "id" : {
              "type" : "keyword",
              "ignore_above" : 1024
            },
            "image" : {
              "properties" : {
                "name" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                }
              }
            },
            "labels" : {
              "type" : "object"
            },
            "name" : {
              "type" : "keyword",
              "ignore_above" : 1024
            }
          }
        },
        "data_stream" : {
          "properties" : {
            "dataset" : {
              "type" : "constant_keyword",
              "value" : "elastic_agent.metricbeat"
            },
            "namespace" : {
              "type" : "constant_keyword",
              "value" : "default"
            },
            "type" : {
              "type" : "constant_keyword",
              "value" : "logs"
            }
          }
        },
        "elastic_agent" : {
          "properties" : {
            "id" : {
              "type" : "keyword",
              "ignore_above" : 1024
            },
            "process" : {
              "type" : "keyword",
              "ignore_above" : 1024
            },
            "snapshot" : {
              "type" : "boolean"
            },
            "version" : {
              "type" : "keyword",
              "ignore_above" : 1024
            }
          }
        },
        "event" : {
          "properties" : {
            "agent_id_status" : {
              "type" : "keyword",
              "ignore_above" : 1024
            },
            "ingested" : {
              "type" : "date",
              "format" : "strict_date_time_no_millis||strict_date_optional_time||epoch_millis"
            }
          }
        },
        "host" : {
          "properties" : {
            "architecture" : {
              "type" : "keyword",
              "ignore_above" : 1024
            },
            "containerized" : {
              "type" : "boolean"
            },
            "domain" : {
              "type" : "keyword",
              "ignore_above" : 1024
            },
            "hostname" : {
              "type" : "keyword",
              "ignore_above" : 1024
            },
            "id" : {
              "type" : "keyword",
              "ignore_above" : 1024
            },
            "ip" : {
              "type" : "ip"
            },
            "mac" : {
              "type" : "keyword",
              "ignore_above" : 1024
            },
            "name" : {
              "type" : "keyword",
              "ignore_above" : 1024
            },
            "os" : {
              "properties" : {
                "build" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                },
                "codename" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                },
                "family" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                },
                "kernel" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                },
                "name" : {
                  "type" : "keyword",
                  "ignore_above" : 1024,
                  "fields" : {
                    "text" : {
                      "type" : "text"
                    }
                  }
                },
                "platform" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                },
                "version" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                }
              }
            },
            "type" : {
              "type" : "keyword",
              "ignore_above" : 1024
            }
          }
        },
        "message" : {
          "type" : "text"
        }
      }
    }
  }
}

[elasticmachine]

Pinging @elastic/agent (Team:Agent)

[jen-huang]

@ruflin Do you think this could have been caused by recent changes we made to elastic_agent package data stream definitions? Did the package prior to those changes have mappings for log.* fields?

The dropdown in the UI uses a terms agg on log.level to populate the values, I think it's failing now (no terms are returned) due to the lack of keyword mappings. This used to work so I'm wondering what changed.

[ruflin]

Yes, this is very likely to my change to the package. By default I turned dynamic mappings off, that is why it worked before as it just mapped everything. I see @nchaulet is working on a PR for elastic agent package, will this also be addressed?

[nchaulet]

Yes, this is very likely to my change to the package. By default I turned dynamic mappings off, that is why it worked before as it just mapped everything. I see @nchaulet is working on a PR for elastic agent package, will this also be addressed?

Yes I can add the mapping back in the same PR

[jen-huang]

Thanks @nchaulet, I'll assign this one to you.