[Serverless] Select project type via config #155754

afharo · 2023-04-25T17:13:52Z

Summary

It adds the serverless: <es>|<oblt>|<security> config parameter so it can be set in the kibana.yml file.

This is a requirement by CP (cc @pebrc @sebgl) instead of relying on the existing CLI flag.

I went with serverless as the config parameter so we can remove the CLI flag and still work as expected (options can be provided to Kibana via CLI if they are provided like --{config.key}=value). The way the configuration schema is defined, we can iterate into more complex structures in the future if needed while maintaining BWC.

Note: Due to the Project Type switcher helper, we need to keep the CLI flag. For non-distributable versions of Kibana (running from source), the CLI flag will intercept the config, then write the config serverless in the file config/serverless.recent.yml and add the file to the list of files to read.
In distributable versions of Kibana, the CLI flag won't intercept --serverless, and it will be treated as a config parameter.

For maintainers

This was checked for breaking API changes and was labeled appropriately

elasticmachine · 2023-04-25T17:13:55Z

Pinging @elastic/kibana-core (Team:Core)

afharo · 2023-04-25T17:21:57Z

config/serverless.es.yml

@@ -0,0 +1 @@
+uiSettings.overrides.defaultRoute: /app/enterprise_search/content/search_indices


I added this setting to change the landing page. This would make it super obvious that we are loading different project type files.

This is in the PoC, we'll be doing this for IC2, so this works.

rudolf

I think we'd need to cleanup the existing --serverless cli option and add a test (could be similar to src/cli/serve/integration_tests/reload_logging_config.test.ts. ?) but the approach looks good to me 👍

packages/core/root/core-root-server-internal/src/bootstrap.ts

packages/core/root/core-root-server-internal/src/root/serverless_config.ts

afharo · 2023-04-25T22:05:45Z

Thank you @rudolf. I didn't remove the existing CLI option because I wanted to confirm this approach made sense.

I'll push the changes tomorrow morning.

…m:afharo/kibana into serverless/select-project-type-via-config

packages/core/root/core-root-server-internal/src/bootstrap.ts

src/cli/serve/integration_tests/serverless_config_flag.test.ts

afharo · 2023-04-26T17:29:31Z

packages/core/root/core-root-server-internal/src/bootstrap.ts

+
+    env = Env.createDefault(REPO_ROOT, {
+      configs: extendedConfigs,
+      cliArgs: { ...cliArgs, serverless: true },


Adding this for BWC. I noticed we are leveraging it in #155142.

IMO, we should either change Env to expose mode.serverless: true (instead of referring to the cliArgs), or, even better: expose the behavior under configuration that can be set in serverless.yml.

I created #155912 to tackle that

…ct-project-type-via-config

afharo

@clintandrewhall, I had to make some changes to the way the projectSwitcher works.

Please, let me know if this is fine.

afharo · 2023-04-26T21:28:12Z

src/cli/serve/serve.js

@@ -124,17 +114,24 @@ function maybeAddConfig(name, configs, method) {
 function maybeSetRecentConfig(file, mode, configs, method) {


I've changed this method's behaviour to rewrite serverless.recent.yml only to include the settings

xpack.serverless.plugin.developer.projectSwitcher.enabled: true serverless: ${selectedProjectType}

@clintandrewhall, do you think this is a good approach?

afharo · 2023-04-26T21:30:23Z

src/cli/serve/serve.js

      )
      .option(
-        '--serverless <oblt|security|es>',
-        'Start Kibana in a specific serverless project mode'
+        '--serverless [oblt|security|es]',


the plain --serverless option was handling all commands, and we were never making it to the values here.

Looking at commander's documentation, using [value] allows us to specify that they can be optional.

afharo · 2023-04-26T21:31:05Z

src/cli/serve/serve.js

@@ -322,18 +312,13 @@ export default function (program) {
    const configs = [getConfigPath(), ...getEnvConfigs(), ...(opts.config || [])];
    const serverlessMode = getServerlessProjectMode(opts);

-    // we "unshift" .serverless. config so that it only overrides defaults
-    if (serverlessMode) {
-      maybeAddConfig(`serverless.yml`, configs, 'push');


I'm intentionally not adding serverless.yml because it's added later on when bootstrap.ts finds out in the config that a specific serverless mode is required.

afharo · 2023-04-26T21:34:24Z

src/cli/serve/serve.js

    }

    // .dev. configs are "pushed" so that they override all other config files
    if (opts.dev && opts.devConfig !== false) {
      maybeAddConfig('kibana.dev.yml', configs, 'push');
      if (serverlessMode) {
        maybeAddConfig(`serverless.dev.yml`, configs, 'push');
-        maybeSetRecentConfig('serverless.recent.dev.yml', serverlessMode, configs, 'unshift');
+        maybeAddConfig('serverless.recent.dev.yml', configs, 'push');


Using maybeAddConfig because otherwise serverless.recent.dev.yml and serverless.recent.yml will be the exact copies. IIUC, this file is used to manually override something in dev mode.

Awesome catch.

afharo · 2023-04-26T21:35:41Z

src/cli/serve/serve.js

-      );
-  }
-
-  if (isServerlessCapableDistribution()) {


Back to --serverless only being available for non-distributables (mind the if(!isKibanaDistributable()) a few lines above.

afharo · 2023-04-26T21:37:51Z

x-pack/plugins/serverless/server/plugin.ts

+            writeFileSync(
+              resolve(getConfigDirectory(), 'serverless.recent.yml'),
+              `xpack.serverless.plugin.developer.projectSwitcher.enabled: true\nserverless: ${selectedProjectType}\n`
+            );


Same as in serve.js: the behavior now is to set serverless: {myChosenProjectType} and enabling the projectSwitcher.

kibana-ci · 2023-04-27T00:57:30Z

💚 Build Succeeded

Buildkite Build
Commit: 722f20a

Metrics [docs]

Unknown metric groups

ESLint disabled line counts

id	before	after	diff
`enterpriseSearch`	17	19	+2
`securitySolution`	399	402	+3
total			+5

Total ESLint disabled count

id	before	after	diff
`enterpriseSearch`	18	20	+2
`securitySolution`	479	482	+3
total			+5

History

💔 Build #123723 failed 03917d7
💚 Build #123601 succeeded aeb3089
💔 Build #123590 failed ed7b220
💚 Build #123386 succeeded 9e02e5e
💚 Build #123142 succeeded ccdc1ee

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @afharo

clintandrewhall

Tested locally, and we're in good shape. Fantastic compromise, and awesome improvements, too! 🚢

clintandrewhall · 2023-04-27T02:00:56Z

src/cli/serve/serve.js

    }

    // .dev. configs are "pushed" so that they override all other config files
    if (opts.dev && opts.devConfig !== false) {
      maybeAddConfig('kibana.dev.yml', configs, 'push');
      if (serverlessMode) {
        maybeAddConfig(`serverless.dev.yml`, configs, 'push');
-        maybeSetRecentConfig('serverless.recent.dev.yml', serverlessMode, configs, 'unshift');
+        maybeAddConfig('serverless.recent.dev.yml', configs, 'push');


Awesome catch.

clintandrewhall · 2023-04-27T02:01:11Z

x-pack/plugins/serverless/server/plugin.ts

+            writeFileSync(
+              resolve(getConfigDirectory(), 'serverless.recent.yml'),
+              `xpack.serverless.plugin.developer.projectSwitcher.enabled: true\nserverless: ${selectedProjectType}\n`
+            );


lukeelmers

Overall approach makes sense to me. The one piece of functionality I think we lost is the ability to have a serverless.{mode}.dev.yml which can be merged in as an override similar to kibana.dev.yml. Not sure if that happened in this PR or maybe #155582, but I'm okay addressing in a follow up as this PR is time-sensitive.

kibanamachine · 2023-04-27T11:02:00Z

💔 All backports failed

Status	Branch	Result
❌	8.8	Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

node scripts/backport --pr 155754

Questions ?

Please refer to the Backport tool documentation

(cherry picked from commit de64ff5) # Conflicts: # config/serverless.es.yml # config/serverless.security.yml # config/serverless.yml # src/cli/serve/serve.js # x-pack/plugins/serverless/server/plugin.ts

afharo · 2023-04-27T11:07:31Z

💚 All backports created successfully

Status	Branch	Result
✅	8.8

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

# Backport This will backport the following commits from `main` to `8.8`: - [[Serverless] Select project type via config (#155754)](#155754)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport)  Co-authored-by: Kibana Machine <[email protected]>

afharo self-assigned this Apr 25, 2023

afharo requested a review from a team as a code owner April 25, 2023 17:13

afharo commented Apr 25, 2023

View reviewed changes

[Serverless] Select project type via config

8731670

afharo force-pushed the serverless/select-project-type-via-config branch from 539c971 to 8731670 Compare April 25, 2023 17:23

Merge branch 'main' into serverless/select-project-type-via-config

ccdc1ee

rudolf reviewed Apr 25, 2023

View reviewed changes

packages/core/root/core-root-server-internal/src/bootstrap.ts Show resolved Hide resolved

packages/core/root/core-root-server-internal/src/root/serverless_config.ts Show resolved Hide resolved

afharo added 3 commits April 26, 2023 10:16

Remove --serverless CLI flag to rely on the config param

5afc3d1

Merge branch 'serverless/select-project-type-via-config' of github.co…

f20ed2f

…m:afharo/kibana into serverless/select-project-type-via-config

Add integration test for --serverless

9e02e5e

afharo enabled auto-merge (squash) April 26, 2023 11:56

rudolf reviewed Apr 26, 2023

View reviewed changes

packages/core/root/core-root-server-internal/src/bootstrap.ts Show resolved Hide resolved

src/cli/serve/integration_tests/serverless_config_flag.test.ts Show resolved Hide resolved

afharo added 3 commits April 26, 2023 17:30

Add some prevalidation before loading the config files (just in case)

2fc4e88

Add tests

ed7b220

Keep Env.cliArg.serverless flag for BWC

aeb3089

afharo commented Apr 26, 2023

View reviewed changes

afharo mentioned this pull request Apr 26, 2023

[Serverless] Do not reach out to env.cliArgs.serverless #155912

Closed

Merge branch 'main' of github.com:elastic/kibana into serverless/sele…

90bc8e0

…ct-project-type-via-config

afharo requested a review from a team as a code owner April 26, 2023 21:25

afharo force-pushed the serverless/select-project-type-via-config branch from 4817761 to b0d33db Compare April 26, 2023 21:32

afharo commented Apr 26, 2023

View reviewed changes

Adapt Project Switcher logic

bf22c4c

afharo force-pushed the serverless/select-project-type-via-config branch from b0d33db to bf22c4c Compare April 26, 2023 21:50

afharo added 2 commits April 27, 2023 00:22

Comment xpack.canvas.enabled: false

06ffecc

Add sanity "Kibana starts" test

908bcfa

afharo force-pushed the serverless/select-project-type-via-config branch from 1f13e64 to 908bcfa Compare April 26, 2023 22:48

afharo added 3 commits April 27, 2023 00:49

Remove unnecessary comment

03917d7

Use getConfigDirectory() instead of REPO_ROOT

0b08f2f

Skip test

722f20a

clintandrewhall approved these changes Apr 27, 2023

View reviewed changes

lukeelmers approved these changes Apr 27, 2023

View reviewed changes

afharo merged commit de64ff5 into elastic:main Apr 27, 2023

afharo deleted the serverless/select-project-type-via-config branch April 27, 2023 04:49

kibanamachine added the v8.9.0 label Apr 27, 2023

afharo added backport:prev-minor Backport to (8.x) the previous minor version (i.e. one version back from main) and removed backport:skip This commit does not require backporting labels Apr 27, 2023

afharo mentioned this pull request Apr 27, 2023

[8.8] [Serverless] Select project type via config (#155754) #155997

Merged

kibanamachine added the v8.8.0 label Apr 27, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Serverless] Select project type via config #155754

[Serverless] Select project type via config #155754

afharo commented Apr 25, 2023 •

edited by kibanamachine

Loading

elasticmachine commented Apr 25, 2023

afharo Apr 25, 2023

clintandrewhall Apr 26, 2023

rudolf left a comment

afharo commented Apr 25, 2023

afharo Apr 26, 2023

afharo Apr 26, 2023

afharo left a comment

afharo Apr 26, 2023

afharo Apr 26, 2023

afharo Apr 26, 2023

afharo Apr 26, 2023

clintandrewhall Apr 27, 2023

afharo Apr 26, 2023

afharo Apr 26, 2023

clintandrewhall Apr 27, 2023

kibana-ci commented Apr 27, 2023

ESLint disabled line counts

Total ESLint disabled count

clintandrewhall left a comment

clintandrewhall Apr 27, 2023

clintandrewhall Apr 27, 2023

lukeelmers left a comment

kibanamachine commented Apr 27, 2023

afharo commented Apr 27, 2023

		@@ -0,0 +1 @@
		uiSettings.overrides.defaultRoute: /app/enterprise_search/content/search_indices

		@@ -124,17 +114,24 @@ function maybeAddConfig(name, configs, method) {
		function maybeSetRecentConfig(file, mode, configs, method) {

[Serverless] Select project type via config #155754

[Serverless] Select project type via config #155754

Conversation

afharo commented Apr 25, 2023 • edited by kibanamachine Loading

Summary

For maintainers

elasticmachine commented Apr 25, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rudolf left a comment

Choose a reason for hiding this comment

afharo commented Apr 25, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

afharo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kibana-ci commented Apr 27, 2023

💚 Build Succeeded

Metrics [docs]

ESLint disabled line counts

Total ESLint disabled count

History

clintandrewhall left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lukeelmers left a comment

Choose a reason for hiding this comment

kibanamachine commented Apr 27, 2023

💔 All backports failed

Manual backport

Questions ?

afharo commented Apr 27, 2023

💚 All backports created successfully

Questions ?

afharo commented Apr 25, 2023 •

edited by kibanamachine

Loading