Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support multiple category fields #66

Merged
merged 35 commits into from
Aug 20, 2021
Merged

Support multiple category fields #66

Show file tree
Hide file tree
Changes from 29 commits
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
ca42b95
add comments
ohltyler May 24, 2021
0c2a495
Add multi category fields in combo box
ohltyler May 25, 2021
296df97
Set character limit on heatmap chart y axis
ohltyler May 25, 2021
83244ae
Support multi category for bucket agg query and result parsing
ohltyler May 26, 2021
64749f1
Persist modelId to support multi category in parseTopEntityAnomalySum…
ohltyler May 27, 2021
7afabf4
Clean up filtering for all HC result queries
ohltyler May 27, 2021
46876d7
Handle entity string edge case
ohltyler Jun 1, 2021
767110b
Change data models to support entity list instead of single entity
ohltyler Jun 2, 2021
7ace456
Store all entity list data in heatmap cell; add ability to see result…
ohltyler Jun 3, 2021
c595caf
Add filters for each entity name/value pair
ohltyler Jun 3, 2021
9b23f16
Support multi category in preview heatmap
ohltyler Jun 3, 2021
a4a2455
Support multi category in results table
ohltyler Jun 3, 2021
3a44ec9
Minor wording changes; add list of category fields in heatmap title
ohltyler Jun 3, 2021
530693a
Tune anomaly occurence chart title
ohltyler Jun 4, 2021
da3f85f
Make multi entity delimiters consistent; remove feature entity prefix
ohltyler Jun 4, 2021
c21002d
Sort category field list on the fly; add max category field num constant
ohltyler Jun 4, 2021
a63c1ea
Support entity list in config page; tune wording on category field
ohltyler Jun 7, 2021
40c0aff
Clean up detector config changes
ohltyler Jun 7, 2021
0119573
Update and add UT
ohltyler Jun 7, 2021
bf37d5e
Change y axis delim to \n
ohltyler Jun 9, 2021
62cb824
Fix bug of selecting indiv entity; clean up helpers
ohltyler Jun 9, 2021
cd5986f
Clean up anomaly history rebase changes
ohltyler Jul 28, 2021
7e619b0
Add multi category field selection in create flow; clean up formattin…
ohltyler Jul 28, 2021
d56aece
Refactor AnomalyHistory to show anomaly + feature charts simultaneously
ohltyler Jul 28, 2021
df081c4
Fix bug of showing RT HC non-multi-category results on historical hea…
ohltyler Jul 28, 2021
f2bd2ca
Clean up tests
ohltyler Jul 28, 2021
8fc9c5a
Add UT for AnomaliesChart
ohltyler Jul 29, 2021
0771e7a
Remove leftover EditFeatures
ohltyler Jul 29, 2021
3c5f100
Show partial results in historical HC case
ohltyler Aug 9, 2021
f66b09a
Remove log statement
ohltyler Aug 9, 2021
8704639
Fix bug of single-entity bucketized results not showing
ohltyler Aug 10, 2021
35c6a27
Render newline on results table correctly
ohltyler Aug 10, 2021
c03ca08
Remove unused historical detector code; show RT + historical detector…
ohltyler Aug 11, 2021
de4d6a9
Tune wording to specify 2 category fields are supported
ohltyler Aug 12, 2021
f60e5d4
Change default refresh rate to 30s for historical HC
ohltyler Aug 20, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 2 additions & 4 deletions public/pages/AnomalyCharts/components/FeatureChart/FeatureChart.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,6 @@ interface FeatureChartProps {
showFeatureMissingDataPointAnnotation?: boolean;
detectorEnabledTime?: number;
rawFeatureData: FeatureAggregationData[];
titlePrefix?: string;
}

export const FeatureChart = (props: FeatureChartProps) => {
Expand Down Expand Up @@ -165,10 +164,9 @@ export const FeatureChart = (props: FeatureChartProps) => {
return (
<ContentPanel
title={
(props.titlePrefix ? props.titlePrefix + ' - ' : '') +
(props.feature.featureEnabled
props.feature.featureEnabled
? props.feature.featureName
: `${props.feature.featureName} (disabled)`)
: `${props.feature.featureName} (disabled)`
}
bodyStyles={
!props.feature.featureEnabled
Expand Down
33 changes: 21 additions & 12 deletions public/pages/AnomalyCharts/containers/AnomaliesChart.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ import {
EuiSuperDatePicker,
EuiTitle,
} from '@elastic/eui';
import { get } from 'lodash';
import { get, orderBy } from 'lodash';
import moment, { DurationInputArg2 } from 'moment';
import React, { useState } from 'react';
import { EntityAnomalySummaries } from '../../../../server/models/interfaces';
Expand All @@ -58,6 +58,7 @@ import {
getConfidenceWording,
getFeatureBreakdownWording,
getFeatureDataWording,
getHCTitle,
} from '../utils/anomalyChartUtils';
import {
DATE_PICKER_QUICK_OPTIONS,
Expand All @@ -67,7 +68,7 @@ import { AnomalyOccurrenceChart } from './AnomalyOccurrenceChart';
import { FeatureBreakDown } from './FeatureBreakDown';
import { convertTimestampToString } from '../../../utils/utils';

interface AnomaliesChartProps {
export interface AnomaliesChartProps {
onDateRangeChange(
startDate: number,
endDate: number,
Expand Down Expand Up @@ -245,9 +246,13 @@ export const AnomaliesChart = React.memo((props: AnomaliesChartProps) => {
<AnomalyHeatmapChart
detectorId={get(props.detector, 'id', '')}
detectorName={get(props.detector, 'name', '')}
detectorTaskProgress={get(
props.detector,
'taskProgress',
0
)}
isHistorical={props.isHistorical}
dateRange={props.dateRange}
//@ts-ignore
title={props.detectorCategoryField[0]}
anomalies={anomalies}
isLoading={props.isLoading}
showAlerts={props.showAlerts}
Expand All @@ -266,6 +271,12 @@ export const AnomaliesChart = React.memo((props: AnomaliesChartProps) => {
onDisplayOptionChanged={props.onDisplayOptionChanged}
heatmapDisplayOption={props.heatmapDisplayOption}
isNotSample={props.isNotSample}
// Category fields in HC results are always sorted alphabetically. To make all chart
ylwu-amzn marked this conversation as resolved.
Show resolved Hide resolved
// wording consistent with the returned results, we sort the given category
// fields in alphabetical order as well.
categoryField={orderBy(props.detectorCategoryField, [
(categoryField) => categoryField.toLowerCase(),
])}
/>,
props.isNotSample !== true
? [
Expand All @@ -284,15 +295,13 @@ export const AnomaliesChart = React.memo((props: AnomaliesChartProps) => {
<EuiSpacer size="s" />
<EuiTitle size="s">
<h3>
{`${get(
props,
'detectorCategoryField.0'
)} `}
<b>
{
props.selectedHeatmapCell
?.entityValue
}
{props.selectedHeatmapCell
? getHCTitle(
props.selectedHeatmapCell
.entityList
)
: '-'}
</b>
</h3>
</EuiTitle>
Expand Down
143 changes: 100 additions & 43 deletions public/pages/AnomalyCharts/containers/AnomalyHeatmapChart.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,8 +24,7 @@
* permissions and limitations under the License.
*/

import React, { useState } from 'react';

import React, { useState, useEffect } from 'react';
import moment from 'moment';
import Plotly, { PlotData } from 'plotly.js-dist';
import plotComponentFactory from 'react-plotly.js/factory';
Expand Down Expand Up @@ -54,13 +53,25 @@ import {
filterHeatmapPlotDataByY,
getEntitytAnomaliesHeatmapData,
} from '../utils/anomalyChartUtils';
import { MIN_IN_MILLI_SECS } from '../../../../server/utils/constants';
import { EntityAnomalySummaries } from '../../../../server/models/interfaces';
import {
MIN_IN_MILLI_SECS,
ENTITY_LIST_DELIMITER,
} from '../../../../server/utils/constants';
import {
EntityAnomalySummaries,
Entity,
} from '../../../../server/models/interfaces';
import { HEATMAP_CHART_Y_AXIS_WIDTH } from '../utils/constants';
import {
convertToEntityList,
convertToCategoryFieldString,
} from '../../utils/anomalyResultUtils';

interface AnomalyHeatmapChartProps {
title: string;
detectorId: string;
detectorName: string;
detectorTaskProgress?: number;
isHistorical?: boolean;
anomalies?: any[];
dateRange: DateRange;
isLoading: boolean;
Expand All @@ -72,11 +83,13 @@ interface AnomalyHeatmapChartProps {
heatmapDisplayOption?: HeatmapDisplayOption;
entityAnomalySummaries?: EntityAnomalySummaries[];
isNotSample?: boolean;
categoryField?: string[];
}

export interface HeatmapCell {
dateRange: DateRange;
entityValue: string;
entityList: Entity[];
modelId?: string;
}

export interface HeatmapDisplayOption {
Expand Down Expand Up @@ -123,14 +136,15 @@ export const AnomalyHeatmapChart = React.memo(
//@ts-ignore
individualEntities = inputHeatmapData[0].y.filter(
//@ts-ignore
(entityValue) => entityValue && entityValue.trim().length > 0
(entityListAsString) =>
entityListAsString && entityListAsString.trim().length > 0
);
}
const individualEntityOptions = [] as any[];
//@ts-ignore
individualEntities.forEach((entityValue) => {
individualEntities.forEach((entityListAsString: string) => {
individualEntityOptions.push({
label: entityValue,
label: entityListAsString.replace(ENTITY_LIST_DELIMITER, ', '),
});
});

Expand Down Expand Up @@ -165,7 +179,7 @@ export const AnomalyHeatmapChart = React.memo(
getEntitytAnomaliesHeatmapData(
props.dateRange,
props.entityAnomalySummaries,
props.heatmapDisplayOption.entityOption.value
props.heatmapDisplayOption?.entityOption.value
)
: // use anomalies data in case of sample result
getAnomaliesHeatmapData(
Expand All @@ -184,7 +198,7 @@ export const AnomalyHeatmapChart = React.memo(
AnomalyHeatmapSortType
>(
props.isNotSample
? props.heatmapDisplayOption.sortType
? props.heatmapDisplayOption?.sortType
: SORT_BY_FIELD_OPTIONS[0].value
);

Expand Down Expand Up @@ -215,9 +229,26 @@ export const AnomalyHeatmapChart = React.memo(
return false;
};

// Custom hook to refresh all of the heatmap data when running a historical task
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will it cause heavy load if query too much? Maybe we can increase interval between refresh query for HC?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's a fair point. I'm ok to increase the timeout - how about 30s?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good, maybe confirm with UX design to check if this is reasonable and should we add some tooltip

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, confirming with UX.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed 30s is ok - no extra notices/wording is necessary.

useEffect(() => {
if (props.isHistorical) {
console.log('updating partial HC results');
kaituo marked this conversation as resolved.
Show resolved Hide resolved
const updateHeatmapPlotData = getAnomaliesHeatmapData(
props.anomalies,
props.dateRange,
sortByFieldValue,
get(COMBINED_OPTIONS.options[0], 'value')
);
setOriginalHeatmapData(updateHeatmapPlotData);
setHeatmapData(updateHeatmapPlotData);
setNumEntities(updateHeatmapPlotData[0].y.length);
setEntityViewOptions(getViewEntityOptions(updateHeatmapPlotData));
}
}, [props.detectorTaskProgress]);

const handleHeatmapClick = (event: Plotly.PlotMouseEvent) => {
const selectedCellIndices = get(event, 'points[0].pointIndex', []);
const selectedEntity = get(event, 'points[0].y', '');
const selectedEntityString = get(event, 'points[0].y', '');
if (!isEmpty(selectedCellIndices)) {
let anomalyCount = get(event, 'points[0].text', 0);
if (
Expand Down Expand Up @@ -262,7 +293,11 @@ export const AnomalyHeatmapChart = React.memo(
startDate: selectedStartDate,
endDate: selectedEndDate,
},
entityValue: selectedEntity,
entityList: convertToEntityList(
selectedEntityString,
get(props, 'categoryField', []),
ENTITY_LIST_DELIMITER
),
} as HeatmapCell);
}
}
Expand Down Expand Up @@ -345,7 +380,7 @@ export const AnomalyHeatmapChart = React.memo(

setNumEntities(nonCombinedOptions.length);
const selectedYs = nonCombinedOptions.map((option) =>
get(option, 'label', '')
get(option, 'label', '').replace(', ', ENTITY_LIST_DELIMITER)
);

let selectedHeatmapData = filterHeatmapPlotDataByY(
Expand Down Expand Up @@ -407,40 +442,53 @@ export const AnomalyHeatmapChart = React.memo(
</EuiFlexItem>
</EuiFlexGroup>
<EuiFlexGroup style={{ padding: '0px' }}>
<EuiFlexItem grow={false} style={{ minWidth: '80px' }}>
<EuiFlexGroup alignItems="center" justifyContent="flexEnd">
<EuiText textAlign="right">
<h4>{props.title}</h4>
</EuiText>
</EuiFlexGroup>
</EuiFlexItem>
<EuiFlexItem style={{ paddingLeft: '5px' }}>
<EuiFlexGroup
style={{ padding: '0px' }}
justifyContent="spaceBetween"
>
<EuiFlexItem grow={false} style={{ marginLeft: '0px' }}>
<EuiFlexGroup gutterSize="s" alignItems="center">
<EuiFlexItem style={{ minWidth: 300 }}>
<EuiComboBox
placeholder="Select options"
options={entityViewOptions}
selectedOptions={currentViewOptions}
onChange={(selectedOptions) =>
handleViewEntityOptionsChange(selectedOptions)
}
/>
</EuiFlexItem>
<EuiFlexItem style={{ minWidth: 150 }}>
<EuiSuperSelect
options={SORT_BY_FIELD_OPTIONS}
valueOfSelected={sortByFieldValue}
onChange={(value) => handleSortByFieldChange(value)}
hasDividers
/>
</EuiFlexItem>
</EuiFlexGroup>
</EuiFlexItem>
<EuiFlexGroup
direction="column"
justifyContent="spaceBetween"
style={{ padding: '12px', marginBottom: '6px' }}
>
<EuiFlexItem style={{ marginBottom: '0px' }}>
<EuiText>
<h4>
View by:&nbsp;
<b>
{convertToCategoryFieldString(
get(props, 'categoryField', []) as string[],
', '
)}
</b>
</h4>
</EuiText>
</EuiFlexItem>

<EuiFlexItem grow={false}>
<EuiFlexGroup gutterSize="s" alignItems="center">
<EuiFlexItem style={{ minWidth: 300 }}>
<EuiComboBox
placeholder="Select options"
options={entityViewOptions}
selectedOptions={currentViewOptions}
onChange={(selectedOptions) =>
handleViewEntityOptionsChange(selectedOptions)
}
/>
</EuiFlexItem>
<EuiFlexItem style={{ minWidth: 150 }}>
<EuiSuperSelect
options={SORT_BY_FIELD_OPTIONS}
valueOfSelected={sortByFieldValue}
onChange={(value) => handleSortByFieldChange(value)}
hasDividers
/>
</EuiFlexItem>
</EuiFlexGroup>
</EuiFlexItem>
</EuiFlexGroup>
<EuiFlexItem grow={false}>
<EuiFlexGroup alignItems="center">
<EuiFlexItem>
Expand Down Expand Up @@ -542,6 +590,15 @@ export const AnomalyHeatmapChart = React.memo(
showline: true,
showgrid: false,
fixedrange: true,
automargin: true,
tickmode: 'array',
tickvals: heatmapData[0].y,
ticktext: heatmapData[0].y.map((label: string) =>
label.length <= HEATMAP_CHART_Y_AXIS_WIDTH
? label
: label.substring(0, HEATMAP_CHART_Y_AXIS_WIDTH - 3) +
'...'
),
},
margin: {
l: 100,
Expand Down
28 changes: 17 additions & 11 deletions public/pages/AnomalyCharts/containers/FeatureBreakDown.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,14 +40,17 @@ import {
Anomalies,
DateRange,
FEATURE_TYPE,
EntityData,
} from '../../../models/interfaces';
import { NoFeaturePrompt } from '../components/FeatureChart/NoFeaturePrompt';
import { focusOnFeatureAccordion } from '../../ConfigureModel/utils/helpers';
import moment from 'moment';
import { HeatmapCell } from './AnomalyHeatmapChart';
import { filterWithHeatmapFilter } from '../../utils/anomalyResultUtils';
import {
filterWithHeatmapFilter,
entityListsMatch,
} from '../../utils/anomalyResultUtils';
import { getDateRangeWithSelectedHeatmapCell } from '../utils/anomalyChartUtils';
import { Entity } from '../../../../server/models/interfaces';

interface FeatureBreakDownProps {
title?: string;
Expand Down Expand Up @@ -80,10 +83,19 @@ export const FeatureBreakDown = React.memo((props: FeatureBreakDownProps) => {
const filteredFeatureData = [];
for (let i = 0; i < anomaliesFound.length; i++) {
const currentAnomalyData = anomaliesResult.anomalies[i];
const dataEntityList = get(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the difference between dataEntityList and cellEntityList?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

dataEntityList is the associated entity list with the current anomaly data in the for loop. cellEntityList is the entity list pulled from the selected heatmap cell. Note that no logic changes were made here. Instead, entityList used to be get(currentAnomalyData, 'entity', [] as EntityData[])[0].value and cellEntityList used to be props.selectedHeatmapCell.entityValue.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cellEntityList is the entity list pulled from the selected heatmap cell

Does that mean user can choose multiple heatmap cells?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, only one can be selected at a time. But the cell itself can contain a list of entities if it is a multi-category detector

currentAnomalyData,
'entity',
[]
) as Entity[];
const cellEntityList = get(
props,
'selectedHeatmapCell.entityList',
[]
) as Entity[];
if (
!isEmpty(get(currentAnomalyData, 'entity', [] as EntityData[])) &&
get(currentAnomalyData, 'entity', [] as EntityData[])[0].value ===
props.selectedHeatmapCell.entityValue &&
!isEmpty(dataEntityList) &&
entityListsMatch(dataEntityList, cellEntityList) &&
get(currentAnomalyData, 'plotTime', 0) >=
props.selectedHeatmapCell.dateRange.startDate &&
get(currentAnomalyData, 'plotTime', 0) <=
Expand Down Expand Up @@ -200,12 +212,6 @@ export const FeatureBreakDown = React.memo((props: FeatureBreakDownProps) => {
props.showFeatureMissingDataPointAnnotation
}
detectorEnabledTime={props.detector.enabledTime}
titlePrefix={
props.selectedHeatmapCell &&
props.title !== 'Sample feature breakdown'
? props.selectedHeatmapCell.entityValue
: undefined
}
/>
{index + 1 ===
get(props, 'detector.featureAttributes', []).length ? null : (
Expand Down
Loading