[Automatic Import] Adding base cel generation as experimental feature

[kgeller]

Summary

This PR adds base level support for CEL input configuration generation for Automatic Import.

How this works

For this phase of the CEL generation, we will produce three things:

1. A simple CEL program. This will contain logic for querying an endpoint and mapping its response to events for processing based on an OpenAPI spec file. It does not contain more complex functionality for things like authentication.
2. An initial state. This will be based on the program and contain defaults based on the openapi spec file.
3. A list of state variables that need redaction from the logs.

These three pieces will be available for user review, and then plumbed directly into the manifest file as default values for their corresponding settings where the user can modify as needed.

Note: It is not yet expected that the generated output will be fully functional without any tweaking or add-on's from the user for things like authentication.

(Temporary) UI Flow

If a user selects CEL during the datastream step, after completion of the review, the user will then be able to upload and review the new CEL steps.

The generated results shown to the user, and are then plumbed as defaults to the input settings, where a user is able to modify during configuration of the integration.

(Note: this flow will be changed with forthcoming UX designs)

Feature flag

This feature will be behind an experimental feature flag for now, as the design is still a work in progress. To enable this feature, add xpack.integration_assistant.enableExperimental: ['generateCel'] to kibana.yml

Maintainer's notes

• UI tests were intentionally omitted for now, as the UI implemented is only temporary until we have a UX design.
• Some OpenAPI specs are too large to be uploaded at this time. I am working on adding support for that and have added another item to the meta issue as such

Relates: #193074

---

Screenshots

After selecting CEL during datastream configuration and reviewing those results, the user will be brought to a new screen to upload an open api spec

The user can upload the spec file (as long as it isn't over the file upload limit)

The user waits while the LLM runs

The user can view results

The results are automatically pasted into the config, where the user may further edit and configure the input

Sample results

source: MISP

program:

(
  request("POST", state.url + "/events/restSearch?" + {
    "page": [string(state.page)],
    "limit": [string(state.limit)],
    "sort": ["date"],
    "direction": ["asc"]
  }.format_query()).with({
    "Header": {
      "Content-Type": ["application/json"]
    }
  }).do_request().as(resp,
    resp.StatusCode == 200 ?
      bytes(resp.Body).decode_json().as(body, {
        "events": body.map(e, {
          "message": e.encode_json()
        }),
        "want_more": body.size() == state.limit,
        "page": state.page + 1,
        "limit": state.limit
      })
    :
      {
        "events": [{
          "error": {
            "code": string(resp.StatusCode),
            "id": string(resp.Status),
            "message": string(resp.Body)
          }
        }],
        "want_more": false
      }
  )
)

intiial state:

page : 1
limit : 50

redact vars:

[ ]

[elasticmachine]

Pinging @elastic/security-scalability (Team:Security-Scalability)

[semd]

The overall code and architecture look great. I only added a couple of comments.
LGTM!

[bhapas]

Overall Great Work @kgeller. Haven't tested this locally yet , will do that but it looks great.

Minor comments and nits overall.

[bhapas]

Tested locally and it looks great.. LGTM.

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 92a7324

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
integrationAssistant	565	578	+13

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
integrationAssistant	49	55	+6

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
integrationAssistant	949.3KB	959.8KB	+10.5KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
integrationAssistant	9.5KB	10.3KB	+861.0B

Unknown metric groups

API count

id	before	after	diff
integrationAssistant	59	66	+7

History

• 💔 Build #241635 failed 5efb39b
• 💚 Build #240851 succeeded 7f8b3f0
• 💚 Build #240850 succeeded 39455ec
• 💚 Build #240360 succeeded e829b87
• 💛 Build #240045 was flaky 1306a3f
• 💔 Build #240020 failed a7a7e8f

cc @kgeller

[kibanamachine]

Starting backport for target branches: 8.x

https://github.com/elastic/kibana/actions/runs/11297752057

[kibanamachine]

💚 All backports created successfully

Status	Branch	Result
✅	8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation