 |
 |
|---|
This document is licensed under the Creative Commons Attribution-ShareAlike 4.0 International License. To view a copy of this license, visit https://creativecommons.org/licenses/by-sa/4.0/
The Analysis module is an experimental module, planned to be released as part of X-Road Metrics. Currently X-Road Metrics includes following modules:
- Database module
- Collector module
- Corrector module
- Reports module
- Anonymizer module
- Opendata module
- Networking/Visualizer module
The Analysis module is responsible for detecting and presenting anomalies in the usage of different X-Road components.
Overall system, its users and rights, processes and directories are designed in a way, that all modules can reside in one server (different users but in same group 'opmon') but also in separate servers.
Overall system is also designed in a way, that allows to monitor data from different X-Road instances (e.g. in
Estonia there are three instances: ee-dev, ee-test and EE.)
Overall system is also designed in a way, that can be used by X-Road Centre for all X-Road members as well as for Member own monitoring (includes possibilities to monitor also members data exchange partners).
The Analysis module consists of two parts:
- Analyzer: the back-end of the analysis module, responsible for detecting anomalies based on requests made via the X-Road platform.
- User Interface: the front-end of the analysis module, responsible for presenting the found anomalies to the user and recording user feedback.
- The Analyzer sub-module needs access to the Database_Module.
- The User Interface runs an Apache webserver. Default Apache configuration accepts incoming HTTP requests on port 80.
The analyzer program has two stages of operation, "train and update" and "find anomalies".
TODO: Add a short definition for service call.
As mentioned on the diagram, service calls can be in different phases. The phase determines how the analyzer fetches the service call data from the database.
During the train/update stage a service call can be in one of the following phases:
- pre-training: less than 3 months have passed since the first request was made by that service call. No data are retrieved.
- first-time training: 3 months have just passed since the first request. All the data will be retrieved for training.
- second-time training: the first model was trained at least 10 days ago and the first incidents have just expired. Data are retrieved since the beginning until the time of the expired incidents (excluding requests that are part of a "true" incident). The model is retrained.
- regular: data are retrieved since the last update to the model until the time of the expired incidents (excluding requests that are part of a "true" incident). The model is updated based on these data.
During the anomaly detection stage a service call can be in one of the following phases:
- pre-training: less than 3 months have passed since the first request was made by that service call. No data are retrieved.
- first-time anomaly finding: 3 months have passed since the first request and the first version of the model has just been trained. All the data will be retrieved for anomaly finding.
- regular: data are retrieved since the last anomaly finding time until the "present" moment ("present" means the last valid date considering the corrector buffer, i.e. 10 days ago). Anomalies are found based on these data.
It is important to note that it can take up to 7 days for the Collector module to receive X-Road operational data from Security Server(s) and up to 3 days for the Corrector_module to clean the raw data and derive monitoring metrics in a clean database collection. This means that Analyzer results are available at least 10 days after data received.
The Analyzer and Interface are distributed as .deb packages for Ubuntu 20.04 and Ubuntu 22.04. Detailed installation instructions are in these documents:
Descriptions of different secondary configuration / customization files and parameters can be found ==> here <==
Use Case-based functionality descriptions with illustrative screenshots can be found ==> here <==