[Enhancement] Improve search performance by using external index engine

[To-om]

Request Type

Enhancement

Description

Currently, TheHive4 suffers from a performance problem. The reason is that TheHive uses basic index mechanism (embedded in JanusGraph). This indexes are simple to use and to manage but they have limitations: they support only equality lookups and cannot be used for sort. In TheHive, almost all lists (list of cases, list of alerts, ...) are sorted. This implies a scan of all the elements of the list, which have a heavy performance impact (especially if the list is long).

In order to solve this issue, TheHive 4.1 will come with a new index engine.

If TheHive4 is used in a cluster mode (more than one server that run TheHive), all servers must connect to common index engine (which can be in a cluster mode or not). In this case a new component must be installed for index management. For this kind of architecture, an Elasticsearch will be used as index engine. In contrary of the use of Elasticsearch in TheHive3, in TheHive4 it will use Elasticsearch only for indexes (it wont store data). Once this new component is installed, it must be configured on all TheHive nodes (in application.conf).

db.janusgraph {
  storage { ... }
  index.search {
    backend = elasticsearch
    hostname = ["es1.local", "es2.local"] // IP or hostname of elasticsearch nodes
    index-name = thehive
  }
}

If you use TheHive4 in a single server mode, you can use Elasticsearch or a file based index engine (Lucene). The latter solution has less infrastructure impact (no component to install) and only requires configuration that indicate where index data is stored in the filesystem. The platform administrator must ensure that there is enough space to hold data.

db.janusgraph {
  storage { ... }
  index.search {
    backend = lucene
    directory = /opt/thehive/index
  }
}

If the indices are not yet installed, TheHive indexes all data at startup. This could be a long process, depending on the volume of data.

[cugu]

Will this also speed up insertion / creation of cases?

[To-om]

@cugu No, indexes only speed up data retrieval, not modification