[http] api_integration tests handle internal route restriction #192407

TinaHeiligers · 2024-09-09T19:30:40Z

fix #192052

Summary

Internal APIs will be restricted from public access as of 9.0.0. In non-serverless environments, this breaking change will result in a 400 error if an external request is made to an internal Kibana API (route access option as "internal" or "public").
This PR allows API owners of non-xpack plugins to run their ftr API integration tests against the restriction and adds examples of how to handle it.

Checklist

Unit or functional tests were updated or added to match the most common scenarios

Note to reviewers: The header needed to allow access to internal apis shouldn't change your test output, with or without the restriction enabled.

How to test the changes work:

Non x-pack:

Set server.restrictInternalApis: true in test/common/config.js
Ensure your tests pass

x-pack:

Set server.restrictInternalApis: true in x-pack/test/api_integration/apis/security/config.ts
Ensure the spaces tests pass

…ng in kibana.yml

TinaHeiligers · 2024-09-09T22:51:02Z

test/analytics/config.ts

@@ -36,6 +36,7 @@ export default async function ({ readConfigFile }: FtrConfigProviderContext) {
      serverArgs: [
        ...functionalConfig.get('kbnTestServer.serverArgs'),
        '--telemetry.optIn=true',
+        '--server.restrictInternalApis=false',


set to true to test changes

TinaHeiligers · 2024-09-09T23:59:22Z

x-pack/test/ftr_apis/security_and_spaces/config.ts

@@ -30,6 +30,8 @@ export default async function ({ readConfigFile }: FtrConfigProviderContext) {
      serverArgs: [
        ...apiIntegrationConfig.get('kbnTestServer.serverArgs'),
        '--server.xsrf.disableProtection=true',
+        // disable internal API restriction. See https://github.com/elastic/kibana/issues/163654
+        '--server.restrictInternalApis=false',


Set to true to enable restriction and run spaces api_integration tests. The tests should still pass.

TinaHeiligers · 2024-09-10T00:00:01Z

x-pack/test/functional/config.base.js

@@ -51,6 +51,8 @@ export default async function ({ readConfigFile }) {
        '--xpack.discoverEnhanced.actions.exploreDataInContextMenu.enabled=true',
        '--savedObjects.maxImportPayloadBytes=10485760', // for OSS test management/_import_objects,
        '--savedObjects.allowHttpApiAccess=false', // override default to not allow hiddenFromHttpApis saved objects access to the http APIs see https://github.com/elastic/dev/issues/2200
+        // explicitly disable internal API restriction. See https://github.com/elastic/kibana/issues/163654
+        '--server.restrictInternalApis=false',


Setting this to true will cause tests that haven't been updated to fail.

TinaHeiligers

Notes for reviewers

jloleysens

Nice work! Overall prepration for restricting internal APIs LGTM. Left a q about some lines that are commented out, but once that's addressed looks good!

test/api_integration/apis/console/proxy_route.ts

jughosta

Data Discovery changes LGTM, thanks!

jeramysoucy · 2024-09-11T10:04:21Z

Just leaving a note, after modifying x-pack/test/ftr_apis/security_and_spaces/config.ts with server.restrictInternalApis set to true, running the FTR from x-pack/test/ftr_apis/security_and_spaces/config.ts will have failures, but running from x-pack/test/api_integration/apis/security/config.ts will not. Maybe running from x-pack/test/ftr_apis/security_and_spaces/config.ts is not intended, from main either config runs in the FTR without failures.

I don't think this should hold up this PR, but we (@elastic/kibana-security ) will want to make sure that all intended execution paths will pass with the restrictions enabled.

test/api_integration/apis/kql_telemetry/kql_telemetry.ts

test/analytics/services/kibana_ebt.ts

test/analytics/tests/analytics_from_the_server.ts

TinaHeiligers · 2024-09-11T16:58:26Z

@elasticmachine merge upstream

…nst over hard coded string

TinaHeiligers · 2024-09-11T17:50:49Z

Just leaving a note, after modifying x-pack/test/ftr_apis/security_and_spaces/config.ts with server.restrictInternalApis set to true, running the FTR from x-pack/test/ftr_apis/security_and_spaces/config.ts will have failures, but running from x-pack/test/api_integration/apis/security/config.ts will not. Maybe running from x-pack/test/ftr_apis/security_and_spaces/config.ts is not intended, from main either config runs in the FTR without failures.

I don't think this should hold up this PR, but we (@elastic/kibana-security ) will want to make sure that all intended execution paths will pass with the restrictions enabled.

@jeramysoucy Thanks for mentioning this. x-pack/test/ftr_apis/security_and_spaces were added in #149188, to support the FTR Saved Object Client we had to introduce for accessing and handling all saved objects types.

The config is for the testing client and shouldn't be used to run the API integration tests. I fixed the typo in the PR description. There's nothing for the platform security team to do here, besides be aware of it.

TinaHeiligers · 2024-09-11T19:10:44Z

@elasticmachine merge upstream

kibana-ci · 2024-09-11T20:36:22Z

💚 Build Succeeded

Buildkite Build
Commit: 4501b1f

Metrics [docs]

✅ unchanged

History

💛 Build #233543 was flaky 555870e
💚 Build #232896 succeeded 19e55c6

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

fix #163654 This PR enforces internal API restrictions in our standard offering. Internal APIs are subject to rapid change and are intentionally not public. By restricting their access, we protect consumers from these rapid changes. This PR does not change any public APIs and they remain available for external consumption. ## Note to reviewers: I chose the most practical way of resolving the failures (add the header or disable the restriction). ## Details Requests to internal Kibana APIs will be restricted globally. This allows more flexibility in making breaking changes to internal APIs, without a risk to external consumers. ## Why are we doing this? The restriction is there to help mitigate the risk of breaking external integrations consuming APIs. Internal APIs are subject to frequent changes, necessary for feature development. ## What this means to plugin authors : Kibana core applies the restriction when enabled through HTTP config. ## What this means to Kibana consumers: Explicitly restricting access to internal APIs has advantages for external consumers: - Consumers can safely integrate with Kibana's stable, public APIs - Consumers are protected from Internal route development, which may involve breaking changes - Relevant information in Kibana's external documentation that is user-friendly and complete. KB article explaining the change (tracked as part of elastic/kibana-team#1044) ## Release note Starting with this release, requests to internal Kibana APIs are globally restricted by default. This change is designed to provide more flexibility in making breaking changes to internal APIs while protecting external consumers from unexpected disruptions. **Key Changes**: • _Internal API Access_: External consumers no longer have access to Kibana’s internal APIs, which are now strictly reserved for internal development and subject to frequent changes. This helps ensure that external integrations only interact with stable, public APIs. • _Error Handling_: When a request is made to an internal API without the proper internal identifier (header or query parameter), Kibana will respond with a 400 Bad Request error, indicating that the route exists but is not allowed under the current Kibana configuration. ## How to test this ### Running kibana 1. Set `server.restrictInternalApis: true` in `kibana.yml` 2. Start es with any license 3. Start kibana 4. Make an external request to an internal API: <details> <summary>curl request to get the config for 9.0:</summary> ```unset curl --location 'localhost:5601/abc/api/kibana/management/saved_objects/_bulk_get' \ --header 'Content-Type: application/json' \ --header 'kbn-xsrf: kibana' \ --header 'x-elastic-internal-origin: kibana' \ --header 'Authorization: Basic ZWxhc3RpYzpjaGFuZ2VtZQ==' \ --data '[ { "type": "config", "id": "9.0.0" } ]' ``` </details> The request should be successful. 5. Make the same curl request without the internal origin header <details> <summary>curl:</summary> ```unset curl --location 'localhost:5601/abc/api/kibana/management/saved_objects/_bulk_get' \ --header 'Content-Type: application/json' \ --header 'kbn-xsrf: kibana' \ --header 'Authorization: Basic ZWxhc3RpYzpjaGFuZ2VtZQ==' \ --data '[ { "type": "config", "id": "9.0.0" } ]' ``` </details> The response should be an error similar to: `{"statusCode":400,"error":"Bad Request","message":"uri [/api/kibana/management/saved_objects/_bulk_get] with method [post] exists but is not available with the current configuration"}` 6. Remove `server.restrictInternalApis` from `kibana.yml` or set it to `false`. 7. Repeat both curl requests above. Both should respond without an error. ### Checklist Delete any items that are not applicable to this PR. - [X] [Documentation] was added for features that require explanation or tutorials (In PR #191943) - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios (and PR #192407) - [x] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) (docker list updated in #156935, cloud stack packs for 9.0 kibana to follow) ### Risk Matrix | Risk | Probability | Severity | Mitigation/Notes | |---------------------------|-------------|----------|-------------------------| | The restriction is knowingly bypassed by end-users adding the required header to use `internal` APIs | Low | High | Kibana's internal APIs are not documented in the OAS specifications. External consumption will be prevented unless explicitly bypassed. | | Upstream services don't include the header and requests to `internal` APIs fail | Medium | Medium | The Core team needs to ensure intra-stack components are updated too | ### For maintainers - [x] This was checked for breaking API changes and was [labeled appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) --------- Co-authored-by: Elastic Machine <[email protected]>

TinaHeiligers added 7 commits September 6, 2024 10:14

Adds traditional to restricInternalApis in http_config to allow setti…

2dea9f0

…ng in kibana.yml

Updates unit test

979331b

Updates non-xpack ftr api_integration tests

4e71f1a

Updates server_integration tests, x-pack spaccces, x-pack ftr_apis

efa5b2f

Merge branch 'main' into kbn-192052-update-ftr-test-for-restriction

e04eef2

Merge branch 'main' into kbn-192052-update-ftr-test-for-restriction

47871d8

Disable internal api restriction for standard offerings

19e55c6

TinaHeiligers added Feature:http release_note:skip Skip the PR/issue when compiling release notes test-api-integration v8.16.0 labels Sep 9, 2024

TinaHeiligers marked this pull request as ready for review September 9, 2024 21:18

TinaHeiligers requested review from a team as code owners September 9, 2024 21:18

TinaHeiligers commented Sep 9, 2024

View reviewed changes

TinaHeiligers commented Sep 10, 2024

View reviewed changes

elena-shostak approved these changes Sep 10, 2024

View reviewed changes

jloleysens approved these changes Sep 11, 2024

View reviewed changes

test/api_integration/apis/console/proxy_route.ts Outdated Show resolved Hide resolved

jughosta approved these changes Sep 11, 2024

View reviewed changes

markov00 reviewed Sep 11, 2024

View reviewed changes

elasticmachine and others added 2 commits September 11, 2024 18:58

Merge branch 'main' into kbn-192052-update-ftr-test-for-restriction

9297290

Uncomments code, reverts test assertion changes, use global header co…

555870e

…nst over hard coded string

TinaHeiligers enabled auto-merge (squash) September 11, 2024 19:10

Merge branch 'main' into kbn-192052-update-ftr-test-for-restriction

4501b1f

TinaHeiligers requested review from markov00 and a team September 11, 2024 19:11

markov00 approved these changes Sep 12, 2024

View reviewed changes

TinaHeiligers merged commit 3a68f8b into elastic:main Sep 12, 2024
19 checks passed

kibanamachine added the backport:skip This commit does not require backporting label Sep 12, 2024

TinaHeiligers deleted the kbn-192052-update-ftr-test-for-restriction branch September 12, 2024 19:23

szaffarano mentioned this pull request Sep 13, 2024

[Telemetry][Security Solution] Fix integration tests on branch 9.0 #192842

Merged

1 task

TinaHeiligers mentioned this pull request Sep 23, 2024

Globally enforce internal API restriction #193792

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[http] api_integration tests handle internal route restriction #192407

[http] api_integration tests handle internal route restriction #192407

TinaHeiligers commented Sep 9, 2024 •

edited

Loading

TinaHeiligers Sep 9, 2024

TinaHeiligers Sep 9, 2024

TinaHeiligers Sep 10, 2024

TinaHeiligers left a comment

jloleysens left a comment

jughosta left a comment

jeramysoucy commented Sep 11, 2024 •

edited

Loading

TinaHeiligers commented Sep 11, 2024

TinaHeiligers commented Sep 11, 2024 •

edited

Loading

TinaHeiligers commented Sep 11, 2024

kibana-ci commented Sep 11, 2024

[http] api_integration tests handle internal route restriction #192407

[http] api_integration tests handle internal route restriction #192407

Conversation

TinaHeiligers commented Sep 9, 2024 • edited Loading

Summary

Checklist

How to test the changes work:

Non x-pack:

x-pack:

TinaHeiligers Sep 9, 2024

Choose a reason for hiding this comment

TinaHeiligers Sep 9, 2024

Choose a reason for hiding this comment

TinaHeiligers Sep 10, 2024

Choose a reason for hiding this comment

TinaHeiligers left a comment

Choose a reason for hiding this comment

jloleysens left a comment

Choose a reason for hiding this comment

jughosta left a comment

Choose a reason for hiding this comment

jeramysoucy commented Sep 11, 2024 • edited Loading

TinaHeiligers commented Sep 11, 2024

TinaHeiligers commented Sep 11, 2024 • edited Loading

TinaHeiligers commented Sep 11, 2024

kibana-ci commented Sep 11, 2024

💚 Build Succeeded

Metrics [docs]

History

TinaHeiligers commented Sep 9, 2024 •

edited

Loading

jeramysoucy commented Sep 11, 2024 •

edited

Loading

TinaHeiligers commented Sep 11, 2024 •

edited

Loading