Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Modify to use loader to get index pattern & Add dataSourceRef #20

Merged
merged 1 commit into from
Jul 21, 2022
Merged

Modify to use loader to get index pattern & Add dataSourceRef #20

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 19 additions & 2 deletions src/plugins/data/common/index_patterns/index_patterns/index_pattern.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,13 @@ import { IndexPatternField, IIndexPatternFieldList, fieldList } from '../fields'
import { formatHitProvider } from './format_hit';
import { flattenHitWrapper } from './flatten_hit';
import { FieldFormatsStartCommon, FieldFormat } from '../../field_formats';
import { IndexPatternSpec, TypeMeta, SourceFilter, IndexPatternFieldMap } from '../types';
import {
IndexPatternSpec,
TypeMeta,
SourceFilter,
IndexPatternFieldMap,
DataSourceRef,
} from '../types';
import { SerializedFieldFormat } from '../../../../expressions/common';

interface IndexPatternDeps {
Expand Down Expand Up @@ -86,7 +92,8 @@ export class IndexPattern implements IIndexPattern {
// savedObject version
public version: string | undefined;
public sourceFilters?: SourceFilter[];
public dataSourcesJSON: string | undefined;
public dataSourcesJSON?: string; // todo: cleanup, only keep one
Copy link

@zhongnansu zhongnansu Jul 21, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you share more about what's blocking you to clean up this field? I mean we don't need to be rush, I can wait for you to complete the clean up and then merge my PR.

Copy link
Author

@kristenTian kristenTian Jul 21, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The dataSourcesJSON is currently used by the UI side as well, so to clean it up, extra work is involved, however, for your side, just rely on dataSourceRef will do the trick. -- I will merge this PR, so we move fast to testing the read from dataSourceRef on visualization side. Just wanna callout that the cleanup is transparent to your side :)

public dataSourceRef?: DataSourceRef;

private originalSavedObjectBody: SavedObjectBody = {};
private shortDotsEnable: boolean = false;
Expand Down Expand Up @@ -132,8 +139,18 @@ export class IndexPattern implements IIndexPattern {
});

this.dataSourcesJSON = spec.dataSourcesJSON;
this.dataSourceRef = this.getDataSourceRef(spec.dataSourcesJSON);
}

getDataSourceRef = (dataSourcesJSON?: string) => {
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If it's just extracting some info from another field dataSourcesJSON. It can be done anywhere, I can do it from my side too. I wonder is it possible to get rid of the dataSourcesJSON in the data model itself.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you are following similar pattern to add xxxJSON to any saved object mapping and interfaces(E.g. see below dashboard.ts). But I didn't see this JSON field being added to the actual index-pattern mapping defined in index_patterns.ts? I am still trying to understand the codebase, so correct me if I am wrong.

OpenSearch-Dashboards/src/plugins/dashboard/server/saved_objects/dashboard.ts

Lines 57 to 65 in e41efcc

mappings: {
properties: {
description: { type: 'text' },
hits: { type: 'integer', index: false, doc_values: false },
kibanaSavedObjectMeta: {
properties: { searchSourceJSON: { type: 'text', index: false } },
},
optionsJSON: { type: 'text', index: false },
panelsJSON: { type: 'text', index: false },

OpenSearch-Dashboards/src/plugins/data/server/saved_objects/index_patterns.ts

Lines 59 to 65 in e41efcc

mappings: {
dynamic: false,
properties: {
title: { type: 'text' },
type: { type: 'keyword' },
},
},

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a good question, index pattern indeed have lots fields not added to map, e,g, fields, timeFieldName etc. My guess is it's due to they are optional fields.

if (dataSourcesJSON) {
const refs = JSON.parse(dataSourcesJSON);
if (!_.isEmpty(refs)) {
return refs[0]; // only support 1-1 mapping now
}
}
};

/**
* Get last saved saved object fields
*/
Expand Down
62 changes: 52 additions & 10 deletions src/plugins/data/common/index_patterns/index_patterns/index_patterns.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -127,7 +127,7 @@ export class IndexPatternsService {
fields: ['title'],
perPage: 10000,
});
}
} // todo: does this find method imcompatible with using loader?

/**
* Get list of index pattern ids
Expand Down Expand Up @@ -384,6 +384,46 @@ export class IndexPatternsService {
};
};

/**
* Converts index pattern saved object to index pattern spec
* @param savedObject
*/

indexPatternSavedObjectToSpec = (savedObject: IndexPatternSavedObject): IndexPatternSpec => {
const {
id,
version,
title,
timeFieldName,
intervalName,
fields,
sourceFilters,
fieldFormatMap,
typeMeta,
type,
dataSourcesJSON,
} = savedObject;

const parsedSourceFilters = sourceFilters ? JSON.parse(sourceFilters) : undefined;
const parsedTypeMeta = typeMeta ? JSON.parse(typeMeta) : undefined;
const parsedFieldFormatMap = fieldFormatMap ? JSON.parse(fieldFormatMap) : {};
const parsedFields: FieldSpec[] = fields ? JSON.parse(fields) : [];

this.addFormatsToFields(parsedFields, parsedFieldFormatMap);
return {
id,
version,
title,
intervalName,
timeFieldName,
sourceFilters: parsedSourceFilters,
fields: this.fieldArrayToMap(parsedFields),
typeMeta: parsedTypeMeta,
type,
dataSourcesJSON,
};
};

/**
* Get an index pattern by id. Cache optimized
* @param id
Expand All @@ -395,24 +435,26 @@ export class IndexPatternsService {
return cache;
}

const savedObject = await this.savedObjectsClient.get<IndexPatternAttributes>(
savedObjectType,
id
);
// const savedObject = await this.savedObjectsClient.get<IndexPatternAttributes>(
// savedObjectType,
// id
// );

const savedIndexPattern: IndexPatternSavedObject = await this.savedIndexPattern.get(id);
const { version, fieldFormatMap } = savedIndexPattern;

if (!savedObject.version) {
if (!version) {
throw new SavedObjectNotFound(
savedObjectType,
id,
'management/opensearch-dashboards/indexPatterns'
);
}

const spec = this.savedObjectToSpec(savedObject);
// const spec = this.savedObjectToSpec(savedObject);
const spec = this.indexPatternSavedObjectToSpec(savedIndexPattern);
const { title, type, typeMeta } = spec;
const parsedFieldFormats: FieldFormatMap = savedObject.attributes.fieldFormatMap
? JSON.parse(savedObject.attributes.fieldFormatMap)
: {};
const parsedFieldFormats: FieldFormatMap = fieldFormatMap ? JSON.parse(fieldFormatMap) : {};

const isFieldRefreshRequired = this.isFieldRefreshRequired(spec.fields);
let isSaveRequired = isFieldRefreshRequired;
Expand Down
8 changes: 7 additions & 1 deletion src/plugins/data/common/index_patterns/types.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@ export interface IIndexPattern {
getFormatterForField?: (
field: IndexPatternField | IndexPatternField['spec'] | IFieldType
) => FieldFormat;
dataSourceRef?: DataSourceRef;
}

export interface IndexPatternAttributes {
Expand Down Expand Up @@ -194,9 +195,14 @@ export interface IndexPatternSpec {
fields?: IndexPatternFieldMap;
typeMeta?: TypeMeta;
type?: string;
dataSourcesJSON?: string;
dataSourcesJSON?: string; // todo: refactor to dataSourceRefs : DataSourceRef[]
}

export interface SourceFilter {
value: string;
}

export interface DataSourceRef {
id: string;
type: string;
}
2 changes: 1 addition & 1 deletion src/plugins/data/public/index_patterns/saved_index_patterns/_saved_index_pattern.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ export function createSavedIndexPatternClass(services: SavedObjectOpenSearchDash
sourceFilters: 'text',
fields: 'text',
fieldFormatMap: 'text',
type: 'text',
type: 'keyword',
typeMeta: 'text',
dataSourcesJSON: 'text',
// todo
Expand Down
6 changes: 3 additions & 3 deletions src/plugins/data/public/index_patterns/types.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,9 @@ export interface IndexPatternObject {
title: string;
intervalName?: string;
timeFieldName?: string;
sourceFilters?: SourceFilter[];
fields?: IndexPatternFieldMap;
typeMeta?: TypeMeta;
sourceFilters?: string;
fields?: string;
typeMeta?: string;
type?: string;
fieldFormatMap?: string;
dataSourcesJSON?: string;
Expand Down