Skip to content

Commit

Permalink
Elastic documentation
Browse files Browse the repository at this point in the history
  • Loading branch information
plirglo committed May 18, 2021
1 parent fcd3fe2 commit d8d7306
Showing 1 changed file with 25 additions and 9 deletions.
34 changes: 25 additions & 9 deletions docs/home/howto/LOGGING.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,16 +55,16 @@ specification:
repo: /var/lib/elasticsearch-snapshots
logs: /var/log/elasticsearch
```
## How to manage data in Opendistro for Elasticsearch
## How to manage Opendistro for Elasticsearch data
Elasticsearch stores data using JSON documents, and an Index is a collection of documents. It is crucial to correctly maintain data inside this database (as in every database). It's almost impossible to deliver database configuration which will fit to every type of project and data stored inside. Epiphany deploys preconfigured Opendistro Elasticsearch, but this configuration may not meet user requirements. Before going to production some configurations need to be adjust. All configuration tips and tricks are available in [official documentation](https://opendistro.github.io/for-elasticsearch-docs/).
Elasticsearch stores data using JSON documents, and an Index is a collection of documents. As in every database it's crutial to correctly maintain data in this one. It's almost impossible to deliver database configuration which will fit to every type of project and data stored in. Epiphany deploys preconfigured Opendistro Elasticsearch, but this configuration may not meet user requirements. Before going to production configuration shoud be tailor to project needs. All configuration tips and tricks are available in [official documentation](https://opendistro.github.io/for-elasticsearch-docs/).
The main and most importand decisions to take before you deploy cluster are:
1) How many Nodes are needed
2) How big machines and/or storage data disks need to be used
These parameters are adjustable in yaml file and it's crutial to create enough big cluster.
These parameters are defined in yaml file and it's important to create big enough cluster.
```
specification:
Expand All @@ -81,9 +81,9 @@ specification:
If it's required to have Elasticsearch which works in cluster formation configuration, except setting up more than one machine in yaml config file please acquaint dedicated support [article](https://opendistro.github.io/for-elasticsearch-docs/docs/elasticsearch/cluster/) and adjust Elasticseach configuration file.
Even with multi node configuration and big disks size there is still possibility to harm machine by fulfilling disk space. Correct ``ISM`` configuration should secure machine(s) from this.
At this moment Opendistro for Elasticsearch does not support [ILM](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-lifecycle-management.html) (like it was in OSS Elasticsearch), log rotation is possible only by configuration created in Index State Management.
`ISM - Index State Management` - is a plugin that lets automate periodic, administrative operations by triggering them besed on index age, size, or number of documents. Using the ISM plugin, can define policies that automatically handle index rollovers or deletions to fit your use case. ISM is installed with Opendistro by default - user does not have to enable this.
`ISM - Index State Management` - is a plugin that provides users and administrative panel to monitor the indices and apply policies at different index stages. ISM lets users automate periodic, administrative operations by triggering them besed on index age, size, or number of documents. Using the ISM plugin, can define policies that automatically handle index rollovers or deletions. ISM is installed with Opendistro by default - user does not have to enable this.
Official documentation is available in [Opendistro for Elasticsearch website](https://opendistro.github.io/for-elasticsearch-docs/docs/im/ism/).

To reduce the consumption of disk resources, every index you created should use well designed [policy](https://opendistro.github.io/for-elasticsearch-docs/docs/im/ism/policies/).
Expand All @@ -94,8 +94,9 @@ Among others these two index actions might save machine from filling up disk spa

[`Index Deletion`](https://opendistro.github.io/for-elasticsearch-docs/docs/im/ism/policies/#delete) - deletes indexes managed by policy

Combining these two actions and knowig data amount and specification there is a possibility to create policy which will secure node from fullfilling disk space.
Be aware that this is only example, and it needs to be adjust to enviroment needs.
Combining these actions, adapting them to data amount and specification users are able to create policy which will maintain data in cluster for example: to secure node from fullfilling disk space.

There is example of policy below. Be aware that this is only example, and it needs to be adjust to environment needs.

```
{
Expand All @@ -114,7 +115,7 @@ Be aware that this is only example, and it needs to be adjust to enviroment need
{
"state_name": "delete",
"conditions": {
"min_index_age": "15d"
"min_index_age": "14d"
}
},
{
Expand Down Expand Up @@ -162,7 +163,22 @@ Be aware that this is only example, and it needs to be adjust to enviroment need
}
}
```
Example above shows configuration with rollover daily or when index achieve 1GB size. Indexes older than 15 days will be deleted. States and condionals could be cobined. Please see [policies](https://opendistro.github.io/for-elasticsearch-docs/docs/im/ism/policies/) documentation for more details.
Example above shows configuration with rollover daily or when index achieve 1GB size. Indexes older than 14 days will be deleted. States and condionals could be cobined. Please see [policies](https://opendistro.github.io/for-elasticsearch-docs/docs/im/ism/policies/) documentation for more details.
`Apply Policy`
To apply policy use similar API request as presented below:
```
PUT _template/template_01
{
"index_patterns": ["filebeat*"],
"settings": {
"opendistro.index_state_management.rollover_alias": "filebeat"
"opendistro.index_state_management.policy_id": "epi_policy"
}
}
```
After applying this policy every new index created under this one will apply to it. There is also possibility to apply policy to already existing policies by assigning them to policy in Index Management Kibana panel.
## How to manage Opendistro for Elasticsearch data
Expand Down

0 comments on commit d8d7306

Please sign in to comment.