Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Modules][Dashboard] The toast to reload the page due to the index pattern fields were refreshed appear constatly despite to refresh the page previously. #3918

Closed
Desvelao opened this issue Mar 25, 2022 · 7 comments · Fixed by #3924 or #3946
Assignees
Labels
type/bug Bug issue

Comments

@Desvelao
Copy link
Member

Wazuh Elastic Rev Security
4.x 7.x 4xxx Basic, ODFE, Xpack
Browser
Chrome, Firefox, Safari, etc

Description
When accessing some dashboards of modules, a toast appears saying the index pattern fields were refreshed and it is required to reload the page. After reloading the page, the toast appears again.

Steps to reproduce

  1. Navigate to Modules/Google Cloud Platform/Dashboard
  2. See the toast
  3. Refresh the page by clicking on the Reload page button or refreshing the page with the browser button or shortcut.
  4. The toast appears again.

Expected Result

  1. The toast should appear constantly.

Screenshots
image

Additional context
Add any other context about the problem here. Here you can paste log entries too or any other useful information that may help with the issue.

@Desvelao Desvelao added the type/bug Bug issue label Mar 25, 2022
@Desvelao Desvelao changed the title [Modules][Dashboard] The toast to reload the page due to the index pattern fields were refreshed appear constatly despite to refresh the page previosly. [Modules][Dashboard] The toast to reload the page due to the index pattern fields were refreshed appear constatly despite to refresh the page previously. Mar 25, 2022
@davidjiglesias davidjiglesias moved this to To do in Release 4.3.0 Mar 25, 2022
@asteriscos asteriscos self-assigned this Mar 25, 2022
@Desvelao
Copy link
Member Author

Desvelao commented Mar 28, 2022

Research

When building a visualization requires a non-existent field in the index pattern, an error is caught. At the current source code status, the field name is extracted from the error message and is checked if this exists in the known fields and used to refresh the index pattern fields as an extra field with builtin metadata and the existent fields in the indices.

For some modules such as Office 365 or GitHub, the fields used in the visualizations are not included in the builtin known fields and this causes the field can't be found there and can't be added to the index pattern fields despite the field really doesn't exist in the indices. This causes when there is no data related to alerts of these modules, the toast message of The index pattern was refreshed successfully to be displayed every time despite refreshing the browser tab.

Possible solutions

  • Sure the fields used by the visualization are available in the indices before creating the visualizations. This approach changes the current logic to render the visualizations. They are built without a previous checking.
  • Sure all fields used in visualizations related to alerts are included in the known fields fields. This approach could cause the toast appears again if the index pattern fields were refreshed by the plugin, when checking the fields in the indices or if the user uses the index pattern management to apply a manual refresh of fields (Stack management/Index pattern) doing the fields are removed.
  • After catching the error of a missing field, refresh all fields of the index pattern if the missing field exists in the indices.

@Desvelao
Copy link
Member Author

Desvelao commented Mar 28, 2022

Possible solution

For the approach:

After catching the error of a missing field, refresh all fields of the index pattern if the missing field exists in the indices.

I created the next patch that can be applied in the current status of 4.3-7.10 branch:

diff --git a/public/components/visualize/wz-visualize.js b/public/components/visualize/wz-visualize.js
index 2b578f2a9..065aa5a06 100644
--- a/public/components/visualize/wz-visualize.js
+++ b/public/components/visualize/wz-visualize.js
@@ -122,20 +122,16 @@ export const WzVisualize = compose(
       this.setState({ expandedVis: this.state.expandedVis === id ? false : id });
     };
 
-    refreshKnownFields = async (newField = null) => {
-      if (newField && newField.name) {
-        this.newFields[newField.name] = newField;
-      }
+    refreshKnownFields = async () => {
       if (!this.state.hasRefreshedKnownFields) {
         // Known fields are refreshed only once per dashboard loading
         try {
           this.setState({ hasRefreshedKnownFields: true, isRefreshing: true });
           if(satisfyPluginPlatformVersion('<7.11')){
-            await PatternHandler.refreshIndexPattern(this.newFields);
+            await PatternHandler.refreshIndexPattern();
           };
           this.setState({ isRefreshing: false });
           this.reloadToast();
-          this.newFields = {};
         } catch (error) {
           this.setState({ isRefreshing: false });
           const options = {
diff --git a/public/kibana-integrations/kibana-vis.js b/public/kibana-integrations/kibana-vis.js
index ea6460099..2dfe700d3 100644
--- a/public/kibana-integrations/kibana-vis.js
+++ b/public/kibana-integrations/kibana-vis.js
@@ -18,7 +18,7 @@ import { connect } from 'react-redux';
 import { LoadedVisualizations } from '../factories/loaded-visualizations';
 import { RawVisualizations } from '../factories/raw-visualizations';
 import { VisHandlers } from '../factories/vis-handlers';
-import { AppState } from '../react-services';
+import { AppState, SavedObject } from '../react-services';
 import { TabVisualizations } from '../factories/tab-visualizations';
 import store from '../redux/store';
 import { updateMetric } from '../redux/actions/visualizationsActions';
@@ -42,7 +42,6 @@ import {
   getOverlays,
   getPlugins,
 } from '../kibana-services';
-import { KnownFields } from '../utils/known-fields';
 import { union } from 'lodash';
 import { getFilterWithAuthorizedAgents } from '../react-services/filter-authorization-agents';
 import { AUTHORIZED_AGENTS } from '../../common/constants';
@@ -312,7 +311,7 @@ class KibanaVis extends Component {
           this.tabVisualizations.addDeadVis();
           return this.renderComplete();
         }
-        const match = error.message.match(/id:(.*)\)/);
+        const match = error.message.match(/id: (.*)\)/);
         this.deadField = match[1] || true;
         if (this.props.refreshKnownFields && !this.hasRefreshed) {
           this.hasRefreshed = true;
@@ -322,11 +321,13 @@ class KibanaVis extends Component {
           this.renderInProgress = false;
           this.rendered = false;
 
-          // if there's a field name it looks for known fields structures
-          const foundField =
-            match[1] && KnownFields.find((field) => field.name === match[1].trim());
-
-          await this.props.refreshKnownFields(foundField);
+          const currentPattern = AppState.getCurrentPattern();
+          const pattern = await getDataPlugin().indexPatterns.get(currentPattern);
+          const fields = await SavedObject.getIndicesFields(pattern.title);
+          const foundField = fields.find(field => field.name === match[1]);
+          if(foundField){
+            await this.props.refreshKnownFields();
+          };
         }
         this.renderInProgress = false;
         return this.myRender(raw);

@Desvelao Desvelao self-assigned this Mar 28, 2022
@Desvelao
Copy link
Member Author

I noticed that the index pattern includes fields related to aws, for example, data.aws.bytes, despite I have no alerts related to aws. I was researching and this could be caused due to some or multiple indices that could have defined these fields in their mappings. These mappings are coming from the template added in Filebeat to use with the Wazuh.

So, I guess if we add the fields used by the visualizations in the affected modules, Office365 and GitHub, to the template used by Wazuh and this is set to Elasticsearch, the new indices will contain their fields in the mapping, and the endpoint /api/index-pattern/_fields_for_widcard?pattern=<PATTERN_TITLE> will return the fields despite the indices has no data related to these modules.

@Desvelao
Copy link
Member Author

Desvelao commented Mar 29, 2022

@asteriscos and I were discussing this issue and researching the options to solve.

Current behavior

The visualizations in the Modules/<Module>/Dashboard are built when accessing the section. They use some expected fields to be present in the fields of the index pattern. If the field is not found, then an error is thrown and caught where the plugin gets the field name and uses a method managed by WzVisualize component to add the field to the fields of the index pattern. This method is executed only one time, despite there were more visualizations with missing fields.

Desirable

Our intention was that each visualization with the error of a missing field, report the field to the parent component to centralize the logic to update the fields of index pattern taking in account all the fields that are missing.

We discussed how this could be done and managed some options to centralize the missing fields in the WzVisualize component using the component state or using the Redux store. We review how the drilldown for the GitHub or Office 365 modules are made and we saw that the WzVisualize component is not used because is used the component that build the individual visualization. This means that we would have to create similar logic for the two different uses cases (if there is not more cases). As this approach, increases the technical debt, we think that we should fix this problem if unify the behavior.

Proposal

We were discussing this with @gdiazlo and we agree to update the template used by Filebeat and the know fields that use the plugin to create the index pattern of alerts when there are no related indices. This is a partial solution that could avoid the toast appearing each time due to Kibana getting the fields of index patterns. Related to #3918 (comment).

We should check:

  • If a new index is created after updating the template with the new fields, they could appear as the response to /api/index-patterns/_fields_for_wildcard?pattern=<PATTERN_TITLE> despite the previous indices have not the mapping of new fields. If the plugin updates the fields for the index pattern, the new fields should appear and causes that accessing to the modules with the problem doesn't appear the toast message.

@Desvelao
Copy link
Member Author

Research

The plugin platform exposes methods to get the index pattern. The object of the index pattern has methods to manage the fields. They could help to add/modify the existing known fields by Kibana and avoid some errors related to unknown fields, in visualizations or fields displayed in the table of Modules Events` tab.

// Add a field to the index pattern
indexPattern.fields.add(fieldSpec);
// example of fieldSpec 
/*
const fieldSpec = {
  name: "@sampledata",
  type: "boolean",
  esTypes: [
  "boolean"
  ],
  searchable: true,
  aggregatable: true,
  readFromDocValues: true
},
*/

another methods of indexPattern.fields:
image

In the case of Kibana 7.10.2, the button to refresh the fields of index pattern uses exposed methods of data plugin (data.indexPatterns). This could be useful to update the index pattern fields without requiring to refresh the browser tab: https://github.com/elastic/kibana/blob/v7.10.2/src/plugins/index_pattern_management/public/components/edit_index_pattern/edit_index_pattern.tsx#L121-L129

@Desvelao
Copy link
Member Author

Desvelao commented Mar 31, 2022

@asteriscos did a PR to update the template with the missing fields of the visualizations: wazuh/wazuh#12932

Using the new template

When the user updates the template, the new indices will have the new fields in the index mapping, not apply to the older ones. The user could reindex the older indices to take the new index settings: mappings and quey default fields, but it is not required. This causes that when the plugin gets the fields of the index pattern, they will be the fields defined in the mappings of the indices plus the fields extracted from the indexed documents.

It is required that at least one index of the index pattern contains the new fields in its mappings, so when the fields of the index pattern are updated, the toast message should not appear due to a missing field in the index pattern that is used in some visualization of some dashboard.

@Desvelao Desvelao linked a pull request Apr 5, 2022 that will close this issue
@gdiazlo gdiazlo added this to the Release 4.3.0 RC 5 milestone Apr 12, 2022
@gdiazlo gdiazlo moved this from To do to Known issues in Release 4.3.0 Apr 12, 2022
@gdiazlo gdiazlo removed this from the Release 4.3.0 RC 5 milestone May 5, 2022
@AlexRuiz7
Copy link
Member

Related issue #4091

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type/bug Bug issue
Projects
No open projects
Status: Done
Status: Done
4 participants