[ML] New Platform server shim: update analytics routes to use new platform router #53521

alvarezmelissa87 · 2019-12-18T19:00:57Z

Summary

Updates all data frame analytics routes to use new platform router.

Checklist

Use ~~strikethroughs~~ to remove checklist items you don't feel are applicable to this PR.

This was checked for cross-browser compatibility, including a check against IE11
~~- [ ] Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support~~
~~- [ ] Documentation was added for features that require explanation or tutorials
~~- [ ] Unit or functional tests were updated or added to match the most common scenarios~~
~~- [ ] This was checked for keyboard-only and screenreader accessibility~~

For maintainers

~~- [ ] This was checked for breaking API changes and was labeled appropriately~~
~~- [ ] This includes a feature addition or change that requires a release note and was labeled appropriately~~

elasticmachine · 2019-12-18T19:01:00Z

Pinging @elastic/ml-ui (:ml)

peteharverson

Tested the data frame analytics pages, and all LGTM

rudolf · 2019-12-19T14:51:03Z

x-pack/legacy/plugins/ml/server/new_platform/plugin.ts

@@ -180,17 +189,28 @@ export class Plugin {
      };
    });

+    // Can access via new platform router's handler function 'context' parameter - context.ml.mlClient
+    const mlClient = core.elasticsearch.createClient('ml', { plugins: [elasticsearchJsPlugin] });
+    // @ts-ignore // 'ml' not 'search' or 'security' or 'licensing


You can type 'ml' the same way search did it:

kibana/src/plugins/data/server/search/search_service.ts

Lines 37 to 41 in 8395596

declare module 'kibana/server' {

interface RequestHandlerContext {

search?: IRouteHandlerSearchContext;

}

}

This will allow any consumers of this context to get type information as well.

Good point - added here: f9a7b5b5c5cd88050be8512f2e59a79c512daa68

looks like this @ts-ignore isn't needed now

rudolf · 2019-12-19T15:06:45Z

x-pack/legacy/plugins/ml/server/routes/data_frame_analytics.js

+      });
+    } catch (e) {
+      // Case: default
+      return response.internalError({ body: e });


This will expose the stack trace of e which could reveal information about the server like the location of files. This should rather be response.internalError({body: e.message}); (also in other places in the PR)

Thanks for taking a look! 😄
Updated here: 66d9d97048c385885a158804899124f469390672

rudolf · 2019-12-19T15:10:50Z

x-pack/legacy/plugins/ml/server/routes/data_frame_analytics.js

+        params: schema.object({
+          analyticsId: schema.string(),
+        }),
+        body: schema.object({ ...dataAnalyticsJobConfigSchema }, { allowUnknowns: true }),


Validation is our best protection against prototype pollution attacks, allowUnknowns: true disables a lot of that protection. Is it possible to enumerate all the expected properties in the schema?

Good call - updated here: 66d9d97048c385885a158804899124f469390672

pgayvallet · 2019-12-20T11:09:23Z

x-pack/legacy/plugins/ml/server/routes/data_frame_analytics.js

    },
-    config: {
-      ...commonRouteConfig,
+    licensePreRoutingFactory(xpackMainPlugin, dfAnalyticsDeleteJobHandler)


NIT: subjective opinion, but you could inline the handlers as they are all used only once. This will also helps when converting to TS as the context, request, response types will be inferred.

licensePreRoutingFactory(xpackMainPlugin, (context, request, response) { try { const { analyticsId } = request.params; const results = await context.ml.mlClient.callAsCurrentUser('ml.deleteDataFrameAnalytics', { analyticsId, }); return response.ok({ body: { ...results }, }); } catch (e) { // Case: default return response.internalError({ body: e.message }); } })

Great suggestion! Changed in 1be4d620155fc0186cd7fedcaebb6c7ba5a51c9c

pgayvallet · 2019-12-20T11:16:00Z

x-pack/legacy/plugins/ml/server/new_platform/licence_check_pre_routing_factory.ts

+export const licensePreRoutingFactory = (
+  xpackMainPlugin: any,
+  handler: RequestHandler<any, any, any>
+): RequestHandler<any, any, any> => {


Should use proper type forwarding here, will be useful when your handlers will be migrated to TS (see src/core/server/http/router/error_wrapper.ts for an example)

export const licensePreRoutingFactory = < P extends ObjectType, Q extends ObjectType, B extends ObjectType >( xpackMainPlugin: any, handler: RequestHandler<P, Q, B> ): RequestHandler<P, Q, B> => { ... }

pgayvallet · 2019-12-20T11:24:54Z

x-pack/legacy/plugins/ml/server/new_platform/licence_check_pre_routing_factory.ts

+      return response.customError({
+        body: {
+          message: licenseCheckResults.message,
+        },
+        statusCode: 403,


You don't need to use customError for common return types. Use response.forbidden instead

Updated to use forbidden in 8c93d6b4a20f1de2821b6ab6f8835703bdb09abf

pgayvallet · 2019-12-20T11:25:45Z

x-pack/legacy/plugins/ml/server/routes/data_frame_analytics.js

+        body: { ...results },
+      });
+    } catch (e) {
+      // Case: default


NIT: comment seems strange?

Removed in 1be4d620155fc0186cd7fedcaebb6c7ba5a51c9c

jgowdyelastic · 2019-12-23T14:23:33Z

x-pack/legacy/plugins/ml/server/routes/data_frame_analytics.ts

+          body: { ...results },
+        });
+      } catch (e) {
+        return response.forbidden({ body: e.message });


is this going to return a 403 for every error that gets thrown.
Do we not want to do something similar to how the old routes work where the errors thrown from the es endpoints are wrapped in a boom error? or something similar if we don't want to use boom anymore.

Yep - find and replace gone wrong which I thought I removed but slipped through. Thanks. Fixed in 8c93d6b4a20f1de2821b6ab6f8835703bdb09abf

jgowdyelastic · 2019-12-23T14:25:45Z

x-pack/legacy/plugins/ml/server/new_platform/plugin.ts

+  }
+}
+
+export const PLUGIN_ID = 'ml';


nit, it's good practice to keep const vars like this are the top of the file

👌 Moved to top in 8c93d6b4a20f1de2821b6ab6f8835703bdb09abf

kibanamachine · 2019-12-28T20:11:41Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 3c173a00ef48044fde2bbb35cae1179bdbef8477

History

💚 Build #17031 succeeded f9a7b5b5c5cd88050be8512f2e59a79c512daa68
💚 Build #17000 succeeded 9e871bee1f834cabda13ec8871555016c6112bdc
💔 Build #16983 failed b6ac51daa63d364207294b42f36b1bd1b306755d
💔 Build #16827 failed 66d9d97048c385885a158804899124f469390672
💔 Build #16535 failed 8f866e17ff1df5cf9815104aded77dddbab843ce

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

alvarezmelissa87 · 2019-12-30T12:24:20Z

This has been updated and is ready for a final look when you get a chance. cc @walterra, @peteharverson, @darnautov, @pgayvallet.

walterra

Code itself LGTM, caveat: I don't know much about all the server side APIs involved but I see you got good feedback from others in that regard.

jgowdyelastic

LGTM!!

jgowdyelastic · 2020-01-02T13:15:20Z

I just noticed that i'm seeing server errors when viewing the results for a regression job. the api/ml/data_frame/_evaluate endpoint is throwing a 500.

Calling the underlying es endpoint manually returns this error:

{
  "error": {
    "root_cause": [
      {
        "type": "status_exception",
        "reason": "No documents found containing both [httpversion, ml.httpversion_prediction] fields"
      }
    ],
    "type": "status_exception",
    "reason": "No documents found containing both [httpversion, ml.httpversion_prediction] fields"
  },
  "status": 400
}

It looks like the use of boomify inside wrapError requires options containing the status code to be passed to it. This is how the original wrapError worked.
Without this status code, the error is assumed to be a 500.

The old version of wrapError for reference:

export function wrapError(error) {
  return boomify(error, { statusCode: error.status });
}

@jgowdyelastic - yep that status did need to get set as an option in the new error wrapper. The ui still gives the right callout so I missed this. Thanks for catching it. Added the status option here: b8c3bd08ea8efa46ea8c4496e80193987215e9a1

peteharverson

Tested latest edits and LGTM

jgowdyelastic

Latest changes LGTM

kibanamachine · 2020-01-02T17:01:28Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 5174b07

History

💚 Build #17364 succeeded 8c93d6b4a20f1de2821b6ab6f8835703bdb09abf

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

…tform router (elastic#53521) * update dfAnalytics routes to use np router * add route schemas and only show error message * convert route file to ts and set handlers inline * update df analytics param type * update mlClient type and assert mlClient is not null * handle errors correctly * ensure error status gets passed correctly to wrapper

…tform router (#53521) (#53897) * update dfAnalytics routes to use np router * add route schemas and only show error message * convert route file to ts and set handlers inline * update df analytics param type * update mlClient type and assert mlClient is not null * handle errors correctly * ensure error status gets passed correctly to wrapper

* master: Rename `/api/security/oidc` to `/api/security/oidc/callback`. (elastic#53886) Updating transitive dependencies to use [email protected] (elastic#53899) [Reporting/Tests] consolidate functional test configs (elastic#52671) [Reporting] Correct the docvalue_fields params in the search query Download CSV from Dashboard Panel (elastic#52833) [Test/Newsfeed] Re-enable test and add news item to be filtered (elastic#53905) cleanup server-log action (elastic#53326) [Uptime] Delete uptime eslint rule skip (elastic#50912) [skip-ci] Expression Lifecycle Docs (elastic#51494) [Endpoint] add react router to endpoint app (elastic#53808) [SIEM][Detection Engine] Silence 409 errors on signal creation (elastic#53859) [Maps] get max_result_window and max_inner_result_window from index settings (elastic#53500) [ML] New Platform server shim: update analytics routes to use new platform router (elastic#53521) fixes typo on engine detection page (elastic#53877)

* master: Rename `/api/security/oidc` to `/api/security/oidc/callback`. (elastic#53886) Updating transitive dependencies to use [email protected] (elastic#53899) [Reporting/Tests] consolidate functional test configs (elastic#52671) [Reporting] Correct the docvalue_fields params in the search query Download CSV from Dashboard Panel (elastic#52833) [Test/Newsfeed] Re-enable test and add news item to be filtered (elastic#53905) cleanup server-log action (elastic#53326) [Uptime] Delete uptime eslint rule skip (elastic#50912) [skip-ci] Expression Lifecycle Docs (elastic#51494) [Endpoint] add react router to endpoint app (elastic#53808) [SIEM][Detection Engine] Silence 409 errors on signal creation (elastic#53859) [Maps] get max_result_window and max_inner_result_window from index settings (elastic#53500) [ML] New Platform server shim: update analytics routes to use new platform router (elastic#53521) fixes typo on engine detection page (elastic#53877) [Maps] push mapbox value extraction from VectorStyle and into DynamicStyleProperty (elastic#53806) Fix suggested value for time_zone in range query (elastic#53841) Clean up generic hooks, use react-use instead (elastic#53822)

…ris/kibana into alerting/created_at-and-updated_at * 'alerting/created_at-and-updated_at' of github.com:gmmorris/kibana: Rename `/api/security/oidc` to `/api/security/oidc/callback`. (elastic#53886) Updating transitive dependencies to use [email protected] (elastic#53899) [Reporting/Tests] consolidate functional test configs (elastic#52671) [Reporting] Correct the docvalue_fields params in the search query Download CSV from Dashboard Panel (elastic#52833) [Test/Newsfeed] Re-enable test and add news item to be filtered (elastic#53905) cleanup server-log action (elastic#53326) [Uptime] Delete uptime eslint rule skip (elastic#50912) [skip-ci] Expression Lifecycle Docs (elastic#51494) [Endpoint] add react router to endpoint app (elastic#53808) [SIEM][Detection Engine] Silence 409 errors on signal creation (elastic#53859) [Maps] get max_result_window and max_inner_result_window from index settings (elastic#53500) [ML] New Platform server shim: update analytics routes to use new platform router (elastic#53521) fixes typo on engine detection page (elastic#53877) [Maps] push mapbox value extraction from VectorStyle and into DynamicStyleProperty (elastic#53806) Fix suggested value for time_zone in range query (elastic#53841) Clean up generic hooks, use react-use instead (elastic#53822)

…t-types * alerting/created_at-and-updated_at: updatedAt should equal createdAt on creation Move index patterns: src/legacy/core_plugins/data 👉 src/plugins/data (elastic#53794) moved Task Manager server code under "server" directory (elastic#53777) Rename `/api/security/oidc` to `/api/security/oidc/callback`. (elastic#53886) Updating transitive dependencies to use [email protected] (elastic#53899) [Reporting/Tests] consolidate functional test configs (elastic#52671) [Reporting] Correct the docvalue_fields params in the search query Download CSV from Dashboard Panel (elastic#52833) [Test/Newsfeed] Re-enable test and add news item to be filtered (elastic#53905) cleanup server-log action (elastic#53326) [Uptime] Delete uptime eslint rule skip (elastic#50912) [skip-ci] Expression Lifecycle Docs (elastic#51494) [Endpoint] add react router to endpoint app (elastic#53808) [SIEM][Detection Engine] Silence 409 errors on signal creation (elastic#53859) [Maps] get max_result_window and max_inner_result_window from index settings (elastic#53500) [ML] New Platform server shim: update analytics routes to use new platform router (elastic#53521) fixes typo on engine detection page (elastic#53877) [Maps] push mapbox value extraction from VectorStyle and into DynamicStyleProperty (elastic#53806) Fix suggested value for time_zone in range query (elastic#53841) Clean up generic hooks, use react-use instead (elastic#53822)

alvarezmelissa87 added :ml v8.0.0 release_note:skip Skip the PR/issue when compiling release notes Feature:NP Migration v7.6.0 labels Dec 18, 2019

alvarezmelissa87 requested review from walterra, pgayvallet, darnautov, peteharverson and jgowdyelastic December 18, 2019 19:00

alvarezmelissa87 requested a review from a team as a code owner December 18, 2019 19:00

alvarezmelissa87 self-assigned this Dec 18, 2019

peteharverson approved these changes Dec 19, 2019

View reviewed changes

alvarezmelissa87 requested review from joshdover and a team and removed request for pgayvallet and joshdover December 19, 2019 14:36

rudolf reviewed Dec 19, 2019

View reviewed changes

pgayvallet reviewed Dec 20, 2019

View reviewed changes

alvarezmelissa87 force-pushed the ml-new-platform-server-shim-update branch from 1be4d62 to b6ac51d Compare December 20, 2019 18:31

jgowdyelastic reviewed Dec 23, 2019

View reviewed changes

alvarezmelissa87 force-pushed the ml-new-platform-server-shim-update branch from f9a7b5b to 3c173a0 Compare December 28, 2019 18:55

elastic deleted a comment from elasticmachine Dec 28, 2019

walterra approved these changes Jan 2, 2020

View reviewed changes

jgowdyelastic approved these changes Jan 2, 2020

View reviewed changes

jgowdyelastic self-requested a review January 2, 2020 13:34

peteharverson approved these changes Jan 2, 2020

View reviewed changes

jgowdyelastic approved these changes Jan 2, 2020

View reviewed changes

alvarezmelissa87 added 7 commits January 2, 2020 08:41

update dfAnalytics routes to use np router

5131c6e

add route schemas and only show error message

7defac0

convert route file to ts and set handlers inline

8255e72

update df analytics param type

867faf9

update mlClient type and assert mlClient is not null

13d66e1

handle errors correctly

3c3b423

ensure error status gets passed correctly to wrapper

5174b07

alvarezmelissa87 force-pushed the ml-new-platform-server-shim-update branch from b8c3bd0 to 5174b07 Compare January 2, 2020 15:41

alvarezmelissa87 merged commit 9372100 into elastic:master Jan 2, 2020

alvarezmelissa87 mentioned this pull request Jan 2, 2020

[7.x] [ML] New Platform server shim: update analytics routes to use new platform router (#53521) #53897

Merged

alvarezmelissa87 deleted the ml-new-platform-server-shim-update branch January 3, 2020 04:11

peteharverson mentioned this pull request Feb 6, 2020

[ML] Errors creating analytics job if array of indices or query is configured in editor #56960

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ML] New Platform server shim: update analytics routes to use new platform router #53521

[ML] New Platform server shim: update analytics routes to use new platform router #53521

alvarezmelissa87 commented Dec 18, 2019 •

edited

Loading

elasticmachine commented Dec 18, 2019

peteharverson left a comment

rudolf Dec 19, 2019

alvarezmelissa87 Dec 20, 2019

jgowdyelastic Dec 23, 2019 •

edited

Loading

rudolf Dec 19, 2019

alvarezmelissa87 Dec 19, 2019

rudolf Dec 19, 2019

alvarezmelissa87 Dec 19, 2019

pgayvallet Dec 20, 2019

alvarezmelissa87 Dec 20, 2019

pgayvallet Dec 20, 2019

pgayvallet Dec 20, 2019

alvarezmelissa87 Dec 28, 2019

pgayvallet Dec 20, 2019

alvarezmelissa87 Dec 20, 2019

jgowdyelastic Dec 23, 2019 •

edited

Loading

alvarezmelissa87 Dec 28, 2019

jgowdyelastic Dec 23, 2019

alvarezmelissa87 Dec 28, 2019

kibanamachine commented Dec 28, 2019

alvarezmelissa87 commented Dec 30, 2019

walterra left a comment

jgowdyelastic left a comment

jgowdyelastic commented Jan 2, 2020 •

edited by alvarezmelissa87

Loading

peteharverson left a comment

jgowdyelastic left a comment

kibanamachine commented Jan 2, 2020

	declare module 'kibana/server' {
	interface RequestHandlerContext {
	search?: IRouteHandlerSearchContext;
	}
	}

[ML] New Platform server shim: update analytics routes to use new platform router #53521

[ML] New Platform server shim: update analytics routes to use new platform router #53521

Conversation

alvarezmelissa87 commented Dec 18, 2019 • edited Loading

Summary

Checklist

For maintainers

elasticmachine commented Dec 18, 2019

peteharverson left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jgowdyelastic Dec 23, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jgowdyelastic Dec 23, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kibanamachine commented Dec 28, 2019

💚 Build Succeeded

History

alvarezmelissa87 commented Dec 30, 2019

walterra left a comment

Choose a reason for hiding this comment

jgowdyelastic left a comment

Choose a reason for hiding this comment

jgowdyelastic commented Jan 2, 2020 • edited by alvarezmelissa87 Loading

peteharverson left a comment

Choose a reason for hiding this comment

jgowdyelastic left a comment

Choose a reason for hiding this comment

kibanamachine commented Jan 2, 2020

💚 Build Succeeded

History

alvarezmelissa87 commented Dec 18, 2019 •

edited

Loading

jgowdyelastic Dec 23, 2019 •

edited

Loading

jgowdyelastic Dec 23, 2019 •

edited

Loading

jgowdyelastic commented Jan 2, 2020 •

edited by alvarezmelissa87

Loading