[8.x] [Automatic Import] Adding base cel generation as experimental feature (#195309) #195977
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…elastic#195309) ## Summary This PR adds base level support for CEL input configuration generation for Automatic Import. ## How this works For this phase of the CEL generation, we will produce three things: 1. A simple CEL program. This will contain logic for querying an endpoint and mapping its response to events for processing based on an OpenAPI spec file. It does **not** contain more complex functionality for things like authentication. 2. An initial state. This will be based on the program and contain defaults based on the openapi spec file. 3. A list of state variables that need redaction from the logs. These three pieces will be available for user review, and then plumbed directly into the manifest file as default values for their corresponding settings where the user can modify as needed. Note: It is not yet expected that the generated output will be fully functional without any tweaking or add-on's from the user for things like authentication. ## (Temporary) UI Flow If a user selects CEL during the datastream step, after completion of the review, the user will then be able to upload and review the new CEL steps. The generated results shown to the user, and are then plumbed as defaults to the input settings, where a user is able to modify during configuration of the integration. (Note: this flow will be changed with forthcoming UX designs) ## Feature flag This feature will be behind an experimental feature flag for now, as the design is still a work in progress. To enable this feature, add `xpack.integration_assistant.enableExperimental: ['generateCel']` to kibana.yml ## Maintainer's notes - UI tests were intentionally omitted for now, as the UI implemented is only temporary until we have a UX design. - Some OpenAPI specs are too large to be uploaded at this time. I am working on adding support for that and have added another item to the [meta issue](elastic#193074) as such Relates: elastic#193074 ___ <details> <summary>Screenshots</summary> After selecting CEL during datastream configuration and reviewing those results, the user will be brought to a new screen to upload an open api spec <img width="650" alt="upload" src="https://github.com/user-attachments/assets/efdace4a-cc26-4f33-8b67-35c08df5f640"> The user can upload the spec file (as long as it isn't over the file upload limit) <img width="650" alt="spec uploaded" src="https://github.com/user-attachments/assets/9fd1b868-f8da-4d3c-b975-522bf66e05a5"> The user waits while the LLM runs <img width="650" alt="Screenshot 2024-10-09 at 1 37 59 PM" src="https://github.com/user-attachments/assets/3eca6b97-4525-4496-89b0-3002a97fa27d"> The user can view results <img width="650" alt="review" src="https://github.com/user-attachments/assets/ee44fb16-fd3a-48c4-975f-706e6d381339"> The results are automatically pasted into the config, where the user may further edit and configure the input <img width="635" alt="Screenshot 2024-10-08 at 11 17 46 AM" src="https://github.com/user-attachments/assets/45151e13-0fd9-4f9a-bbfe-68e6f9b0e671"> </details> <details> <summary>Sample results </summary> source: [MISP](https://raw.githubusercontent.com/MISP/MISP/develop/app/webroot/doc/openapi.yaml) program: ``` ( request("POST", state.url + "/events/restSearch?" + { "page": [string(state.page)], "limit": [string(state.limit)], "sort": ["date"], "direction": ["asc"] }.format_query()).with({ "Header": { "Content-Type": ["application/json"] } }).do_request().as(resp, resp.StatusCode == 200 ? bytes(resp.Body).decode_json().as(body, { "events": body.map(e, { "message": e.encode_json() }), "want_more": body.size() == state.limit, "page": state.page + 1, "limit": state.limit }) : { "events": [{ "error": { "code": string(resp.StatusCode), "id": string(resp.Status), "message": string(resp.Body) } }], "want_more": false } ) ) ``` intiial state: ``` page : 1 limit : 50 ``` redact vars: ``` [ ] ``` </details> (cherry picked from commit 7f24e38)
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
💚 Build Succeeded
Metrics [docs]Module Count
Public APIs missing comments
Async chunks
Page load bundle
cc @kgeller |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Backport
This will backport the following commits from
mainto8.x:Questions ?
Please refer to the Backport tool documentation