Unable to load kibana dashboards from zip file

[ycombinator]

Observed on master / 8.0.0.

When using the setup.dashboards.file option, setting it to a zip file containing Kibana dashboard files, Packetbeat does not actually load the dashboards in the zip file into Kibana.

Relevant settings in packetbeat.yml:

setup.dashboards.file: /path/to/dashboards.zip
setup.kibana.host: "localhost:5601"

Note that if the zip file is extracted and the setup.dashboards.directory option, pointed to that folder, is used instead of the setup.dashboards.file option, the dashboards do get loaded into Kibana as expected.

[eedugon]

Zip example used when reproducing this:
packetbeat_test.zip

[dplavcic]

UPDATE 4

(previous updates are available in revision history):

TLDR (not a bug);
There is a predefined combination of directory structure and setup.dashboards.beat property that has to be defined to successfully import dashboards forom archive file to the Kibana. Directory structure must contain four levels of directory structure (firstLevelDir/secondLevelDir/thirdLevelDir/fourthLevelDir which is not the case with provided test file (packetbeat_test.zip)

Combinations of setup.dashboards.beat and directory structure together with status (Imported to Kibana? YES/NO/ERROR) are show in the table below:

imp.cfg.Beat name	setup.dashboards.beat	firstLevelDir	secondLevelDir	thirdLevelDir (kibana version)	fourthLevelDir	Imported to Kibana?
CASE 1:
filebeat	#setup.dashboards.beat: ""	kibana	7	dashboard	N/A	NO
filebeat	setup.dashboards.beat: ""	kibna	7	dashboard	N/A	ERROR
filebeat	setup.dashboards.beat: filebeat	kibana	7	dashboard	N/A	NO
CASE 2:
filebeat	#setup.dashboards.beat: ""	kibana	filebeat	7	dashboard	YES
filebeat	setup.dashboards.beat: ""	kibana	filebeat	7	dashboard	YES
filebeat	setup.dashboards.beat: filebeat	kibana	filebeat	7	dashboard	YES
metricbeat	setup.dashboards.beat: metricbeat	kibana	filebeat	7	dashboard	NO
CASE 3:
filebeat	#setup.dashboards.beat: ""	kibana	filebeat	7	dashboard	YES
			metricbeat	7	dashboard	NO
filebeat	setup.dashboards.beat: ""	kibana	filebeat	7	dashboard	YES
			metricbeat	7	dashboard	YES
filebeat	setup.dashboards.beat: filebeat	kibana	filebeat	7	dashboard	YES
			metricbeat	7	dashboard	NO
metricbeat	setup.dashboards.beat: metricbeat	kibana	filebeat	7	dashboard	NO
			metricbeat	7	dashboard	YES

Title: Combination of working directory structure with corresponding setup.dashboards.beat value

Legend:
Directory structure of archive file is: {firstLevelDir} / {secondLevelDir} / {thirdLevelDir} / {fourthLevelDir}

Name	Description
imp.cfg.Beat name	Beat name (set automatically or by "setup.dashboards.beat" property
setup.dashboards.beat	Value of "setup.dashboards.beat" property
firstLevelDir	First level directory value (anything)
secondLevelDir	Second level directory value (beatname e.g. filebeat, metricbeat)
thirdLevelDir	Kibana version (e.g. 7)
fourthLevelDir	Dashboard
Imported to Kibana?	Is combination of previous values successfully imported to Kibana?

Description

Libbeat implementation requires predefined combination of directory structure and setup.dashboards.beat property that has to be defined to successfully import dashboards to the Kibana.

From documentation in filebeat.reference.yml:

# In case the archive contains the dashboards from multiple Beats, this lets you
# select which one to load. You can load all the dashboards in the archive by
# setting this to the empty string.
#setup.dashboards.beat: ""

Part of code relevant to importing dashboard from extracted archive file:

------------------------------------------------------------
libbeat/dashboards/importer.go:194
------------------------------------------------------------
194  func (imp Importer) ImportArchive() error {
     	...
220  	err = imp.unzip(archive, target)
221  	if err != nil {
222  		return fmt.Errorf("Failed to unzip the archive: %s: %v", archive, err)
223  	}
224  	dirs, err := getDirectories(target)
225  	if err != nil {
226  		return err
227  	}
228  	if len(dirs) != 1 {
229  		return fmt.Errorf("Too many directories under %s", target)
230  	}
231  
232  	dirs, err = getDirectories(dirs[0])
233  	if err != nil {
234  		return err
235  	}
236  
237  	for _, dir := range dirs {
238  		imp.loader.statusMsg("Importing Kibana from %s", dir)
239  		if imp.cfg.Beat == "" || filepath.Base(dir) == imp.cfg.Beat {
240  			err = imp.ImportKibanaDir(dir)
241  			if err != nil {
242  				return err
243  			}
244  		}
245  	}
246  	return nil
247  }

Example file is dashboard.zip and has the following structure (please ignore *.json file names):

.
└── kibana
    └── filebeat
        └── 7
            └── dashboard
                └── Packetbeat-cassandra.json

4 directories, 1 file

On line 220 dashboard.zip is extracted to /tmp/tmp181549145:

archive variable value: dashboard.zip
target variable value: /tmp/tmp181549145

On line 224 the dir variable represents firstLevelDir and has a value of:

dirs = {[]string} len:1, cap:1
   0 = {string} "/tmp/tmp181549145/kibana"

On line 232 the dir variable represents secondLevelDir and has a value of:

dirs = {[]string} len:1, cap:1
   0 = {string} "/tmp/tmp181549145/kibana/filebeat" 

Previous directory structure is hard coded and represents firstLevelDir and secondLevel dir values from the table.

On line 239 is a condition that check if either of following is true:

1. imp.cfg.Beat == "" checks if beat name is "" (set by setup.dashboards.beat property or given automatically based on running beat)
2. filepath.Base(dir) == imp.cfg.Beat checks if base dir name corresponds to the given beat name (set by setup.dashboards.beat property or given automatically based on running beat)

If any of previous condition is true, execution continues with importing dashboards to the Kibana.

There is another part of the code that assumes two another levels of directory structure, which are also hard coded and consists of Kibana version, in this case 7 (libbeat/dashboards/importer.go:309) and hardcoded value of 'dashboards' (libbeat/dashboards/importer.go:112). At the same time those value represents thirdLevelDir and fourthLevelDir from the table.

------------------------------------------------------------
libbeat/dashboards/importer.go:304
------------------------------------------------------------

303 // import Kibana dashboards and index-pattern or only one of these
304 func (imp Importer) ImportKibanaDir(dir string) error {
306 
307 	versionPath := "7"
308 
309 	dir = path.Join(dir, versionPath) 
...
313 	if _, err := os.Stat(dir); err != nil {
314 		return newErrNotFound("No directory %s", dir)
315 	}
...
360 }

------------------------------------------------------------
(libbeat/dashboards/importer.go:309
------------------------------------------------------------
309  func (imp Importer) ImportDir(dirType string, dir string) error {
310  	imp.loader.statusMsg("Import directory %s", dir)
311  
312 	dir = path.Join(dir, dirType)
313 	var errors []string
...

Proposal:

1. Update documentation where 'setup.dashboards.file' is mentioned to include required directory structure example.
2. Increase logging on beats/libbeat/dashboards/importer.go:239 if
   'imp.cfg.Beat == "" || filepath.Base(dir) == imp.cfg.Beat' is false to indicate that no Kibana dashboards will be created for specified combination of dir structure and setup.dashboards.beat property.

@ycombinator: Could you please try to import Kibana dashboard from the archive once again, this time, using one of the available combinations from the table (directory structure + setup.dashboards.beat)?