[Ingest] Enable adding system monitoring (datasource) during Agent Config create

[paul-tavares]

❗️DO NOT MERGE❗️ until #59376 is merged. Change this PR's base to master once it is.

Summary

Resolves #59564

Enables the "Collect system metrics" switch on the Agent Configuration Create flyout. When checked (default), a Datasource will be created from the System package and added to the newly created Agent Configuration

Checklist

Manually tested:

• Throwing an exception while attempting to create the System datasource will still result in a successful Agent Creation result
• When switch in checked in UI, validated Datasource is created and added to Agent Config
• When switch is un-checked in UI, validated Datasource is not created.

[elasticmachine]

Pinging @elastic/ingest-management (Feature:EPM)

[paul-tavares]

It feels like this function belongs in one of the services as a method - maybe Datasources?
Let me know if its best for it to live elsewhere.

[jen-huang]

I think this logic could be split up into two services.

1. A new service inside x-pack/plugins/ingest_manager/common/services/package_to_config.ts that builds a complete new datasource from package info:

export const packageToConfigDatasource = (
  packageInfo: PackageInfo,
  configId: string,
  outputId: string,
  datasourceName?: string
): NewDatasource => {
      return {
        name: datasourceName || `${packageInfo.name}-1`,
        package: {
          name: packageInfo.name,
          title: packageInfo.title,
          version: packageInfo.version,
        },
        enabled: true,
        config_id: configId,
        output_id: outputId,
        inputs: packageToConfigDatasourceInputs(packageInfo),
      };
}

2. A new datasourceService that retrieves the information needed to use that service inside x-pack/plugins/ingest_manager/server/services/datasource.ts:

(almost identical to what you currently have here, just removing sys and changing return line to use the generic service)

const createDatasourceFromPackage = async (
  soClient: SavedObjectsClientContract,
  pkgName: string,
): Promise<NewDatasource | undefined> => {
  const pkgInstall = await findInstalledPackageByName({
    savedObjectsClient: soClient,
    pkgName,
  });
  if (pkgInstall) {
    const [pkgInfo, defaultOutputId] = await Promise.all([
      getPackageInfo({
        savedObjectsClient: soClient,
        pkgkey: `${pkgInstall.name}-${pkgInstall.version}`,
      }),
      outputService.getDefaultOutputId(soClient),
    ]);
    if (pkgInfo) {
      return packageToConfigDatasource(pkgInfo, '', defaultOutputId);
    }
  }
};

What do you think? I can see the generic service inside /common being useful elsewhere too, hence my not putting everything inside datasourceService. It also makes it easier to write unit tests for.

[paul-tavares]

thanks for the feedback @jen-huang . This makes sense to me and I agree that that the common service to top-level /common/services will be helpful in a few places (maybe even for Endpoint as we will need to create datasources for Policy). I have also noticed that we generate the package key manually in several places - maybe a service to handle that as well so that the generation of the value can be centralized?

[jen-huang]

I have a few suggestions - let me know if they make sense.

[jen-huang]

the agentConfig that is returned here does not contain the most up-to-date information for the newly created config (aka it won't have the system datasource inside). I think we should retrieve this agent config again after creating system datasource.

[paul-tavares]

Done :) - thanks

[jen-huang]

I think this logic could be split up into two services.

1. A new service inside x-pack/plugins/ingest_manager/common/services/package_to_config.ts that builds a complete new datasource from package info:

export const packageToConfigDatasource = (
  packageInfo: PackageInfo,
  configId: string,
  outputId: string,
  datasourceName?: string
): NewDatasource => {
      return {
        name: datasourceName || `${packageInfo.name}-1`,
        package: {
          name: packageInfo.name,
          title: packageInfo.title,
          version: packageInfo.version,
        },
        enabled: true,
        config_id: configId,
        output_id: outputId,
        inputs: packageToConfigDatasourceInputs(packageInfo),
      };
}

2. A new datasourceService that retrieves the information needed to use that service inside x-pack/plugins/ingest_manager/server/services/datasource.ts:

(almost identical to what you currently have here, just removing sys and changing return line to use the generic service)

const createDatasourceFromPackage = async (
  soClient: SavedObjectsClientContract,
  pkgName: string,
): Promise<NewDatasource | undefined> => {
  const pkgInstall = await findInstalledPackageByName({
    savedObjectsClient: soClient,
    pkgName,
  });
  if (pkgInstall) {
    const [pkgInfo, defaultOutputId] = await Promise.all([
      getPackageInfo({
        savedObjectsClient: soClient,
        pkgkey: `${pkgInstall.name}-${pkgInstall.version}`,
      }),
      outputService.getDefaultOutputId(soClient),
    ]);
    if (pkgInfo) {
      return packageToConfigDatasource(pkgInfo, '', defaultOutputId);
    }
  }
};

What do you think? I can see the generic service inside /common being useful elsewhere too, hence my not putting everything inside datasourceService. It also makes it easier to write unit tests for.

[paul-tavares]

Closing this. New PR opened against master - #60111

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: 78dfbcd

History

• 💔 Build #32596 failed 78dfbcd

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream