[POC] init impl to add datasource service/client lifecycle #19

zhongnansu · 2022-07-20T22:39:03Z

Description

Add dataSourceClient
- a map dataSourceClientsPool
- a flag that maps to global feature flag in osd.yml
- asDataSource() to find/create client, then return to caller
Add opensearch_data_service
- depends on savedObject service to init a savedObjectClient
- init DataSourceClient instance
Register opensearch_data_service to CoreSetup, CoreStart, CoreRouteHandlerContext
Refactor search_source to add dataSourceId field, and add it to ISearchOptions to provide to search() interface
Refactor opensearchagg.ts, to attach data source id if exists
Refactor opensearch_seearch_strategy to select between default cluster client and data source client

TODO

See comment for more. The critical TODOs for the end-to-end use case to run are:
- Align with the data modeling of index pattern and dataSource, pending PR from @kristenTian

Test

Only manual test for now
Steps:

add ecommerce sample data
delete its index
create a valid datasource and credential
Manually attach dataSourceId to ecommerce index pattern by something like:

POST /.kibana/_update/index-pattern:ff959d40-b880-11e8-a6d9-e546fe2bba5f
{
"script": "ctx._source['index-pattern'].dataSourceId = '794b5590-088b-11ed-8694-134ac2453186'"
}

Now you should be able to see a ecommerce visualization with the data from data source
Discover/TSVB/Vega/Timelion is not working for now, I'll post other PRs to address those.

Issues Resolved

TBA

Check List

src/core/server/core_route_handler_context.ts

seraphjiang · 2022-07-21T01:10:07Z

src/core/server/opensearch_data/client/data_source_client.ts

+    const url = dataSourceObj.endpoint.url;
+    /**
+     * TODO:
+     * credential manager will provide "decrypt(authId: string)" to return auth


@noCharger do we have action item for this.

Add Decrypt function

As discussed offline, we can use https://github.com/elastic-analytics/OpenSearch-Dashboards/blob/db8c74eacd7a78830f[…]c/plugins/credential_management/server/crypto/cli/crypto_cli.ts for the end-to-end PoC. Support decrypt(authId: string) is a non-blocking follow-up.

Also from the principle of single responsibility, fetch credential using saved object cli and decrypt with cryto cli could be decoupled. Let's name the function as fetch_and_decrypt(id: string)

@noCharger do we have action item for this.

opensearch-project#1926

@noCharger Made the change to first fetch encrypted password in opensearch_data_service, and then use existing CrptoCLI -> decrypt() function for the end-to-end use case to run. Any subsequent change will be posted in separate PR. But I don't think a functionality of getCredentialById is a multi-responsibility. We can discuss more

src/core/server/opensearch_data/opensearch_data_service.ts

seraphjiang · 2022-07-21T01:16:25Z

Please add one TODO:

pick up one viz to demonstrate usage of the new data source client.

Signed-off-by: Zhongnan Su <[email protected]>

zhongnansu · 2022-07-21T09:38:42Z

Please add one TODO:

pick up one viz to demonstrate usage of the new data source client.

@seraphjiang End to end use case verified. Steps added to the PR description

zengyan-amazon

do not commit yarn.lock

src/core/server/opensearch_data/client/data_source_client.ts

zengyan-amazon · 2022-07-21T22:10:24Z

src/core/server/opensearch_data/client/data_source_client.ts

+  }
+  async asDataSource(dataSourceId: string) {
+    const { endpoint, credentialId } = await this.getDataSourceInfo(dataSourceId);
+    const { username, password } = await this.getCredentialInfo(credentialId);


need to consider more auth types

What's are the current defined auth type in data source data model or credential manager? @kristenTian @noCharger

zengyan-amazon · 2022-07-21T22:17:50Z

src/core/server/opensearch_data/client/data_source_client.ts

+  }
+
+  // close anything in pool
+  public async close() {


please check the opensearch js client, see if it can keep the connection until being explicitly closed

https://github.com/elastic/elasticsearch-js/blob/4def742e4774dba44f31b8e1701345a8c29870ee/lib/Connection.js#L59-L63
keep-alive is default to true in ClientOptions

zengyan-amazon · 2022-07-21T22:18:46Z

src/core/server/opensearch_data/opensearch_data_service.ts

+  private readonly log: Logger;
+  private readonly config$: Observable<OpenSearchConfig>;
+  private auditorFactory?: AuditorFactory;
+  private stop$ = new Subject();


how is this field being used?

it's used in the Compatibility check(that I commented out for now). Compatibility checks happen constantly until "this.stop" is called (which it will be called in stop() stage in our case)

src/plugins/data/server/saved_objects/index_patterns.ts

zhongnansu · 2022-07-22T17:05:17Z

do not commit yarn.lock

removed

… on osd.yml (#18) 1. Move generate_crypto_materials to /script 2. Add cm configs to osd.yml 3. Add multipleDataSource global configuration support to osd.yml 4. Create default crypto material during plugin setup 5. Fix merge issues from #19

init commit to add datasource client lifecycle

282e578

zhongnansu changed the title ~~init commit to add datasource client lifecycle~~ init commit to add datasource service/client lifecycle Jul 20, 2022

zhongnansu marked this pull request as ready for review July 20, 2022 23:46

zhongnansu requested review from kristenTian, noCharger, seraphjiang and zengyan-amazon July 20, 2022 23:47

seraphjiang reviewed Jul 21, 2022

View reviewed changes

src/core/server/core_route_handler_context.ts Outdated Show resolved Hide resolved

seraphjiang reviewed Jul 21, 2022

View reviewed changes

src/core/server/opensearch_data/opensearch_data_service.ts Show resolved Hide resolved

zhongnansu changed the title ~~init commit to add datasource service/client lifecycle~~ [not ready to merge]init commit to add datasource service/client lifecycle Jul 21, 2022

zhongnansu changed the title ~~[not ready to merge]init commit to add datasource service/client lifecycle~~ [POC] init impl to add datasource service/client lifecycle Jul 21, 2022

use credential manager to decrypt password

1a8e74b

Signed-off-by: Zhongnan Su <[email protected]>

seraphjiang approved these changes Jul 21, 2022

View reviewed changes

zengyan-amazon reviewed Jul 21, 2022

View reviewed changes

zhongnansu force-pushed the search-source-poc branch 2 times, most recently from 7c33c9d to 7f0bfa3 Compare July 22, 2022 02:22

Merge remote-tracking branch 'origin/2.x' into search-source-poc

2567a2e

zhongnansu force-pushed the search-source-poc branch from 7f0bfa3 to 2567a2e Compare July 22, 2022 02:36

zhongnansu merged commit 4994527 into 2.x Jul 22, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[POC] init impl to add datasource service/client lifecycle #19

[POC] init impl to add datasource service/client lifecycle #19

zhongnansu commented Jul 20, 2022 •

edited

Loading

seraphjiang Jul 21, 2022

noCharger Jul 21, 2022 •

edited

Loading

noCharger Jul 21, 2022

zhongnansu Jul 21, 2022 •

edited

Loading

seraphjiang commented Jul 21, 2022

zhongnansu commented Jul 21, 2022

zengyan-amazon left a comment

zengyan-amazon Jul 21, 2022

zhongnansu Jul 22, 2022

zengyan-amazon Jul 21, 2022

zhongnansu Jul 22, 2022

zengyan-amazon Jul 21, 2022

zhongnansu Jul 22, 2022

zhongnansu commented Jul 22, 2022

[POC] init impl to add datasource service/client lifecycle #19

[POC] init impl to add datasource service/client lifecycle #19

Conversation

zhongnansu commented Jul 20, 2022 • edited Loading

Description

TODO

Test

Issues Resolved

Check List

Choose a reason for hiding this comment

noCharger Jul 21, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zhongnansu Jul 21, 2022 • edited Loading

Choose a reason for hiding this comment

seraphjiang commented Jul 21, 2022

zhongnansu commented Jul 21, 2022

zengyan-amazon left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zhongnansu commented Jul 22, 2022

zhongnansu commented Jul 20, 2022 •

edited

Loading

noCharger Jul 21, 2022 •

edited

Loading

zhongnansu Jul 21, 2022 •

edited

Loading