Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Automatic Import][meta] Add support for creating CEL input config #193074

Open
ebeahan opened this issue Sep 16, 2024 · 1 comment
Open

[Automatic Import][meta] Add support for creating CEL input config #193074

ebeahan opened this issue Sep 16, 2024 · 1 comment
Assignees
@kgeller
Labels
8.16 candidate enhancement New value added to drive a business result Feature:AutomaticImport Team:Security-Scalability Team label for Security Integrations Scalability Team

Comments

@ebeahan
Copy link
Member

ebeahan commented Sep 16, 2024 •
edited by kgeller
Loading

Summary

Track work to add chain(s) to create the Common Expression Language (CEL) input config.

Base: ✅

The goal of this phase is to support the addition of CEL configs of minimal complexity around paging, chaining API requests and auth.

Requirements:

  • Setup new CEL chain
  • Add prompt for summarizing the necessary paging query parameter requirements of the API
  • Add prompt for building the CEL program
  • Build template for CEL input

Auth: [In progress]

This is where we'd want to ensure all auth scenarios are supported. In the base phase, we don't add any auth to the program.

Requirements:

  • Add any necessary prompts for collecting auth information (basic auth, oauth, api tokens)
  • Update CEL builder prompt to allow auth configuration in the program

OAS optimization [In progress]

Some OpenAPI spec files are quite large. We should take advantage of something like https://github.com/readmeio/oas (source: https://openapi.tools/#parsers) to better work with the spec file.

Validation:

The goal of this would be to expand upon validation. In the base phase, we have basic support for running mito, but now we'd need to focus on how to update the program if any errors occur.

Requirements:

  • Add validation prompt(s) to update the program upon finding any errors in mito output

Complexity:

This involves adding support for more complex types of APIs. This includes asynchronous event handling, chaining API requests, and any other common patterns we have.

Requirements:

  • Add prompt for identifying any of these common patterns
  • Add prompt(s) for each pattern to update the base CEL config generated by the base phase

Input formats:

This is where we'd want to expand our API input beyond only OpenAPI specs, to include PDFs or online doc pages, etc.

Requirements:

  • Add support for other formats beyond OpenAPI spec
  • Update any necessary prompts to support other input types

Mito

  • Add support for running the mito tool necessary for validation
  • Add template for dockerfile mock service
  • Add template for system tests
@ebeahan ebeahan added 8.16 candidate enhancement New value added to drive a business result Feature:AutomaticImport Team:Security-Scalability Team label for Security Integrations Scalability Team labels Sep 16, 2024
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-scalability (Team:Security-Scalability)

@kgeller kgeller mentioned this issue Oct 8, 2024
Merged
kgeller added a commit that referenced this issue Oct 11, 2024
@kgeller
[Automatic Import] Adding base cel generation as experimental feature (…
7f24e38 
…#195309)

## Summary

This PR adds base level support for CEL input configuration generation
for Automatic Import.

## How this works

For this phase of the CEL generation, we will produce three things:

1. A simple CEL program. This will contain logic for querying an
endpoint and mapping its response to events for processing based on an
OpenAPI spec file. It does **not** contain more complex functionality
for things like authentication.
2. An initial state. This will be based on the program and contain
defaults based on the openapi spec file.
3. A list of state variables that need redaction from the logs. 

These three pieces will be available for user review, and then plumbed
directly into the manifest file as default values for their
corresponding settings where the user can modify as needed.

Note: It is not yet expected that the generated output will be fully
functional without any tweaking or add-on's from the user for things
like authentication.

## (Temporary) UI Flow

If a user selects CEL during the datastream step, after completion of
the review, the user will then be able to upload and review the new CEL
steps.

The generated results shown to the user, and are then plumbed as
defaults to the input settings, where a user is able to modify during
configuration of the integration.

(Note: this flow will be changed with forthcoming UX designs)

## Feature flag

This feature will be behind an experimental feature flag for now, as the
design is still a work in progress. To enable this feature, add
`xpack.integration_assistant.enableExperimental: ['generateCel']` to
kibana.yml

## Maintainer's notes

- UI tests were intentionally omitted for now, as the UI implemented is
only temporary until we have a UX design.
- Some OpenAPI specs are too large to be uploaded at this time. I am
working on adding support for that and have added another item to the
[meta issue](#193074) as such

Relates: #193074
___ 

<details>
  <summary>Screenshots</summary>
  
After selecting CEL during datastream configuration and reviewing those
results, the user will be brought to a new screen to upload an open api
spec
<img width="650" alt="upload"
src="https://github.com/user-attachments/assets/efdace4a-cc26-4f33-8b67-35c08df5f640">

The user can upload the spec file (as long as it isn't over the file
upload limit)
<img width="650" alt="spec uploaded"
src="https://github.com/user-attachments/assets/9fd1b868-f8da-4d3c-b975-522bf66e05a5">

The user waits while the LLM runs
<img width="650" alt="Screenshot 2024-10-09 at 1 37 59 PM"
src="https://github.com/user-attachments/assets/3eca6b97-4525-4496-89b0-3002a97fa27d">

The user can view results 
<img width="650" alt="review"
src="https://github.com/user-attachments/assets/ee44fb16-fd3a-48c4-975f-706e6d381339">

The results are automatically pasted into the config, where the user may
further edit and configure the input
<img width="635" alt="Screenshot 2024-10-08 at 11 17 46 AM"
src="https://github.com/user-attachments/assets/45151e13-0fd9-4f9a-bbfe-68e6f9b0e671">

</details>

<details>
  <summary>Sample results </summary>

source:
[MISP](https://raw.githubusercontent.com/MISP/MISP/develop/app/webroot/doc/openapi.yaml)
  
program:
```
(
  request("POST", state.url + "/events/restSearch?" + {
    "page": [string(state.page)],
    "limit": [string(state.limit)],
    "sort": ["date"],
    "direction": ["asc"]
  }.format_query()).with({
    "Header": {
      "Content-Type": ["application/json"]
    }
  }).do_request().as(resp,
    resp.StatusCode == 200 ?
      bytes(resp.Body).decode_json().as(body, {
        "events": body.map(e, {
          "message": e.encode_json()
        }),
        "want_more": body.size() == state.limit,
        "page": state.page + 1,
        "limit": state.limit
      })
    :
      {
        "events": [{
          "error": {
            "code": string(resp.StatusCode),
            "id": string(resp.Status),
            "message": string(resp.Body)
          }
        }],
        "want_more": false
      }
  )
)
```

intiial state:
```
page : 1
limit : 50
```

redact vars:
```
[ ]
```

</details>
kibanamachine pushed a commit to kibanamachine/kibana that referenced this issue Oct 11, 2024
@kgeller
[Automatic Import] Adding base cel generation as experimental feature (…
f4e9bd9 
…elastic#195309)

## Summary

This PR adds base level support for CEL input configuration generation
for Automatic Import.

## How this works

For this phase of the CEL generation, we will produce three things:

1. A simple CEL program. This will contain logic for querying an
endpoint and mapping its response to events for processing based on an
OpenAPI spec file. It does **not** contain more complex functionality
for things like authentication.
2. An initial state. This will be based on the program and contain
defaults based on the openapi spec file.
3. A list of state variables that need redaction from the logs.

These three pieces will be available for user review, and then plumbed
directly into the manifest file as default values for their
corresponding settings where the user can modify as needed.

Note: It is not yet expected that the generated output will be fully
functional without any tweaking or add-on's from the user for things
like authentication.

## (Temporary) UI Flow

If a user selects CEL during the datastream step, after completion of
the review, the user will then be able to upload and review the new CEL
steps.

The generated results shown to the user, and are then plumbed as
defaults to the input settings, where a user is able to modify during
configuration of the integration.

(Note: this flow will be changed with forthcoming UX designs)

## Feature flag

This feature will be behind an experimental feature flag for now, as the
design is still a work in progress. To enable this feature, add
`xpack.integration_assistant.enableExperimental: ['generateCel']` to
kibana.yml

## Maintainer's notes

- UI tests were intentionally omitted for now, as the UI implemented is
only temporary until we have a UX design.
- Some OpenAPI specs are too large to be uploaded at this time. I am
working on adding support for that and have added another item to the
[meta issue](elastic#193074) as such

Relates: elastic#193074
___

<details>
  <summary>Screenshots</summary>

After selecting CEL during datastream configuration and reviewing those
results, the user will be brought to a new screen to upload an open api
spec
<img width="650" alt="upload"
src="https://github.com/user-attachments/assets/efdace4a-cc26-4f33-8b67-35c08df5f640">

The user can upload the spec file (as long as it isn't over the file
upload limit)
<img width="650" alt="spec uploaded"
src="https://github.com/user-attachments/assets/9fd1b868-f8da-4d3c-b975-522bf66e05a5">

The user waits while the LLM runs
<img width="650" alt="Screenshot 2024-10-09 at 1 37 59 PM"
src="https://github.com/user-attachments/assets/3eca6b97-4525-4496-89b0-3002a97fa27d">

The user can view results
<img width="650" alt="review"
src="https://github.com/user-attachments/assets/ee44fb16-fd3a-48c4-975f-706e6d381339">

The results are automatically pasted into the config, where the user may
further edit and configure the input
<img width="635" alt="Screenshot 2024-10-08 at 11 17 46 AM"
src="https://github.com/user-attachments/assets/45151e13-0fd9-4f9a-bbfe-68e6f9b0e671">

</details>

<details>
  <summary>Sample results </summary>

source:
[MISP](https://raw.githubusercontent.com/MISP/MISP/develop/app/webroot/doc/openapi.yaml)

program:
```
(
  request("POST", state.url + "/events/restSearch?" + {
    "page": [string(state.page)],
    "limit": [string(state.limit)],
    "sort": ["date"],
    "direction": ["asc"]
  }.format_query()).with({
    "Header": {
      "Content-Type": ["application/json"]
    }
  }).do_request().as(resp,
    resp.StatusCode == 200 ?
      bytes(resp.Body).decode_json().as(body, {
        "events": body.map(e, {
          "message": e.encode_json()
        }),
        "want_more": body.size() == state.limit,
        "page": state.page + 1,
        "limit": state.limit
      })
    :
      {
        "events": [{
          "error": {
            "code": string(resp.StatusCode),
            "id": string(resp.Status),
            "message": string(resp.Body)
          }
        }],
        "want_more": false
      }
  )
)
```

intiial state:
```
page : 1
limit : 50
```

redact vars:
```
[ ]
```

</details>

(cherry picked from commit 7f24e38)
kibanamachine added a commit that referenced this issue Oct 11, 2024
@kibanamachine @kgeller
[8.x] [Automatic Import] Adding base cel generation as experimental f…
5eafc27 
…eature (#195309) (#195977)

# Backport

This will backport the following commits from `main` to `8.x`:
- [[Automatic Import] Adding base cel generation as experimental feature
(#195309)](#195309)

<!--- Backport version: 9.4.3 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Kylie
Meli","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-10-11T18:50:21Z","message":"[Automatic
Import] Adding base cel generation as experimental feature
(#195309)\n\n## Summary\r\n\r\nThis PR adds base level support for CEL
input configuration generation\r\nfor Automatic Import.\r\n\r\n## How
this works\r\n\r\nFor this phase of the CEL generation, we will produce
three things:\r\n\r\n1. A simple CEL program. This will contain logic
for querying an\r\nendpoint and mapping its response to events for
processing based on an\r\nOpenAPI spec file. It does **not** contain
more complex functionality\r\nfor things like authentication.\r\n2. An
initial state. This will be based on the program and contain\r\ndefaults
based on the openapi spec file.\r\n3. A list of state variables that
need redaction from the logs. \r\n\r\nThese three pieces will be
available for user review, and then plumbed\r\ndirectly into the
manifest file as default values for their\r\ncorresponding settings
where the user can modify as needed.\r\n\r\nNote: It is not yet expected
that the generated output will be fully\r\nfunctional without any
tweaking or add-on's from the user for things\r\nlike
authentication.\r\n\r\n## (Temporary) UI Flow\r\n\r\nIf a user selects
CEL during the datastream step, after completion of\r\nthe review, the
user will then be able to upload and review the new
CEL\r\nsteps.\r\n\r\nThe generated results shown to the user, and are
then plumbed as\r\ndefaults to the input settings, where a user is able
to modify during\r\nconfiguration of the integration.\r\n\r\n(Note: this
flow will be changed with forthcoming UX designs)\r\n\r\n## Feature
flag\r\n\r\nThis feature will be behind an experimental feature flag for
now, as the\r\ndesign is still a work in progress. To enable this
feature, add\r\n`xpack.integration_assistant.enableExperimental:
['generateCel']` to\r\nkibana.yml\r\n\r\n## Maintainer's notes\r\n\r\n-
UI tests were intentionally omitted for now, as the UI implemented
is\r\nonly temporary until we have a UX design.\r\n- Some OpenAPI specs
are too large to be uploaded at this time. I am\r\nworking on adding
support for that and have added another item to the\r\n[meta
issue](#193074) as
such\r\n\r\nRelates:
https://github.com/elastic/kibana/issues/193074\r\n___
\r\n\r\n<details>\r\n <summary>Screenshots</summary>\r\n \r\nAfter
selecting CEL during datastream configuration and reviewing
those\r\nresults, the user will be brought to a new screen to upload an
open api\r\nspec\r\n<img width=\"650\"
alt=\"upload\"\r\nsrc=\"https://github.com/user-attachments/assets/efdace4a-cc26-4f33-8b67-35c08df5f640\">\r\n\r\nThe
user can upload the spec file (as long as it isn't over the
file\r\nupload limit)\r\n<img width=\"650\" alt=\"spec
uploaded\"\r\nsrc=\"https://github.com/user-attachments/assets/9fd1b868-f8da-4d3c-b975-522bf66e05a5\">\r\n\r\nThe
user waits while the LLM runs\r\n<img width=\"650\" alt=\"Screenshot
2024-10-09 at 1 37
59 PM\"\r\nsrc=\"https://github.com/user-attachments/assets/3eca6b97-4525-4496-89b0-3002a97fa27d\">\r\n\r\nThe
user can view results \r\n<img width=\"650\"
alt=\"review\"\r\nsrc=\"https://github.com/user-attachments/assets/ee44fb16-fd3a-48c4-975f-706e6d381339\">\r\n\r\nThe
results are automatically pasted into the config, where the user
may\r\nfurther edit and configure the input\r\n<img width=\"635\"
alt=\"Screenshot 2024-10-08 at 11 17
46 AM\"\r\nsrc=\"https://github.com/user-attachments/assets/45151e13-0fd9-4f9a-bbfe-68e6f9b0e671\">\r\n\r\n</details>\r\n\r\n<details>\r\n
<summary>Sample results
</summary>\r\n\r\nsource:\r\n[MISP](https://raw.githubusercontent.com/MISP/MISP/develop/app/webroot/doc/openapi.yaml)\r\n
\r\nprogram:\r\n```\r\n(\r\n request(\"POST\", state.url +
\"/events/restSearch?\" + {\r\n \"page\": [string(state.page)],\r\n
\"limit\": [string(state.limit)],\r\n \"sort\": [\"date\"],\r\n
\"direction\": [\"asc\"]\r\n }.format_query()).with({\r\n \"Header\":
{\r\n \"Content-Type\": [\"application/json\"]\r\n }\r\n
}).do_request().as(resp,\r\n resp.StatusCode == 200 ?\r\n
bytes(resp.Body).decode_json().as(body, {\r\n \"events\": body.map(e,
{\r\n \"message\": e.encode_json()\r\n }),\r\n \"want_more\":
body.size() == state.limit,\r\n \"page\": state.page + 1,\r\n \"limit\":
state.limit\r\n })\r\n :\r\n {\r\n \"events\": [{\r\n \"error\": {\r\n
\"code\": string(resp.StatusCode),\r\n \"id\": string(resp.Status),\r\n
\"message\": string(resp.Body)\r\n }\r\n }],\r\n \"want_more\":
false\r\n }\r\n )\r\n)\r\n```\r\n\r\nintiial state:\r\n```\r\npage :
1\r\nlimit : 50\r\n```\r\n\r\nredact vars:\r\n```\r\n[
]\r\n```\r\n\r\n</details>","sha":"7f24e388829933be17f1ed7b690dc0562a354cd8","branchLabelMapping":{"^v9.0.0$":"main","^v8.16.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["enhancement","release_note:skip","v9.0.0","backport:prev-minor","Team:Security-Scalability","Feature:AutomaticImport"],"title":"[Automatic
Import] Adding base cel generation as experimental
feature","number":195309,"url":"https://github.com/elastic/kibana/pull/195309","mergeCommit":{"message":"[Automatic
Import] Adding base cel generation as experimental feature
(#195309)\n\n## Summary\r\n\r\nThis PR adds base level support for CEL
input configuration generation\r\nfor Automatic Import.\r\n\r\n## How
this works\r\n\r\nFor this phase of the CEL generation, we will produce
three things:\r\n\r\n1. A simple CEL program. This will contain logic
for querying an\r\nendpoint and mapping its response to events for
processing based on an\r\nOpenAPI spec file. It does **not** contain
more complex functionality\r\nfor things like authentication.\r\n2. An
initial state. This will be based on the program and contain\r\ndefaults
based on the openapi spec file.\r\n3. A list of state variables that
need redaction from the logs. \r\n\r\nThese three pieces will be
available for user review, and then plumbed\r\ndirectly into the
manifest file as default values for their\r\ncorresponding settings
where the user can modify as needed.\r\n\r\nNote: It is not yet expected
that the generated output will be fully\r\nfunctional without any
tweaking or add-on's from the user for things\r\nlike
authentication.\r\n\r\n## (Temporary) UI Flow\r\n\r\nIf a user selects
CEL during the datastream step, after completion of\r\nthe review, the
user will then be able to upload and review the new
CEL\r\nsteps.\r\n\r\nThe generated results shown to the user, and are
then plumbed as\r\ndefaults to the input settings, where a user is able
to modify during\r\nconfiguration of the integration.\r\n\r\n(Note: this
flow will be changed with forthcoming UX designs)\r\n\r\n## Feature
flag\r\n\r\nThis feature will be behind an experimental feature flag for
now, as the\r\ndesign is still a work in progress. To enable this
feature, add\r\n`xpack.integration_assistant.enableExperimental:
['generateCel']` to\r\nkibana.yml\r\n\r\n## Maintainer's notes\r\n\r\n-
UI tests were intentionally omitted for now, as the UI implemented
is\r\nonly temporary until we have a UX design.\r\n- Some OpenAPI specs
are too large to be uploaded at this time. I am\r\nworking on adding
support for that and have added another item to the\r\n[meta
issue](#193074) as
such\r\n\r\nRelates:
https://github.com/elastic/kibana/issues/193074\r\n___
\r\n\r\n<details>\r\n <summary>Screenshots</summary>\r\n \r\nAfter
selecting CEL during datastream configuration and reviewing
those\r\nresults, the user will be brought to a new screen to upload an
open api\r\nspec\r\n<img width=\"650\"
alt=\"upload\"\r\nsrc=\"https://github.com/user-attachments/assets/efdace4a-cc26-4f33-8b67-35c08df5f640\">\r\n\r\nThe
user can upload the spec file (as long as it isn't over the
file\r\nupload limit)\r\n<img width=\"650\" alt=\"spec
uploaded\"\r\nsrc=\"https://github.com/user-attachments/assets/9fd1b868-f8da-4d3c-b975-522bf66e05a5\">\r\n\r\nThe
user waits while the LLM runs\r\n<img width=\"650\" alt=\"Screenshot
2024-10-09 at 1 37
59 PM\"\r\nsrc=\"https://github.com/user-attachments/assets/3eca6b97-4525-4496-89b0-3002a97fa27d\">\r\n\r\nThe
user can view results \r\n<img width=\"650\"
alt=\"review\"\r\nsrc=\"https://github.com/user-attachments/assets/ee44fb16-fd3a-48c4-975f-706e6d381339\">\r\n\r\nThe
results are automatically pasted into the config, where the user
may\r\nfurther edit and configure the input\r\n<img width=\"635\"
alt=\"Screenshot 2024-10-08 at 11 17
46 AM\"\r\nsrc=\"https://github.com/user-attachments/assets/45151e13-0fd9-4f9a-bbfe-68e6f9b0e671\">\r\n\r\n</details>\r\n\r\n<details>\r\n
<summary>Sample results
</summary>\r\n\r\nsource:\r\n[MISP](https://raw.githubusercontent.com/MISP/MISP/develop/app/webroot/doc/openapi.yaml)\r\n
\r\nprogram:\r\n```\r\n(\r\n request(\"POST\", state.url +
\"/events/restSearch?\" + {\r\n \"page\": [string(state.page)],\r\n
\"limit\": [string(state.limit)],\r\n \"sort\": [\"date\"],\r\n
\"direction\": [\"asc\"]\r\n }.format_query()).with({\r\n \"Header\":
{\r\n \"Content-Type\": [\"application/json\"]\r\n }\r\n
}).do_request().as(resp,\r\n resp.StatusCode == 200 ?\r\n
bytes(resp.Body).decode_json().as(body, {\r\n \"events\": body.map(e,
{\r\n \"message\": e.encode_json()\r\n }),\r\n \"want_more\":
body.size() == state.limit,\r\n \"page\": state.page + 1,\r\n \"limit\":
state.limit\r\n })\r\n :\r\n {\r\n \"events\": [{\r\n \"error\": {\r\n
\"code\": string(resp.StatusCode),\r\n \"id\": string(resp.Status),\r\n
\"message\": string(resp.Body)\r\n }\r\n }],\r\n \"want_more\":
false\r\n }\r\n )\r\n)\r\n```\r\n\r\nintiial state:\r\n```\r\npage :
1\r\nlimit : 50\r\n```\r\n\r\nredact vars:\r\n```\r\n[
]\r\n```\r\n\r\n</details>","sha":"7f24e388829933be17f1ed7b690dc0562a354cd8"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/195309","number":195309,"mergeCommit":{"message":"[Automatic
Import] Adding base cel generation as experimental feature
(#195309)\n\n## Summary\r\n\r\nThis PR adds base level support for CEL
input configuration generation\r\nfor Automatic Import.\r\n\r\n## How
this works\r\n\r\nFor this phase of the CEL generation, we will produce
three things:\r\n\r\n1. A simple CEL program. This will contain logic
for querying an\r\nendpoint and mapping its response to events for
processing based on an\r\nOpenAPI spec file. It does **not** contain
more complex functionality\r\nfor things like authentication.\r\n2. An
initial state. This will be based on the program and contain\r\ndefaults
based on the openapi spec file.\r\n3. A list of state variables that
need redaction from the logs. \r\n\r\nThese three pieces will be
available for user review, and then plumbed\r\ndirectly into the
manifest file as default values for their\r\ncorresponding settings
where the user can modify as needed.\r\n\r\nNote: It is not yet expected
that the generated output will be fully\r\nfunctional without any
tweaking or add-on's from the user for things\r\nlike
authentication.\r\n\r\n## (Temporary) UI Flow\r\n\r\nIf a user selects
CEL during the datastream step, after completion of\r\nthe review, the
user will then be able to upload and review the new
CEL\r\nsteps.\r\n\r\nThe generated results shown to the user, and are
then plumbed as\r\ndefaults to the input settings, where a user is able
to modify during\r\nconfiguration of the integration.\r\n\r\n(Note: this
flow will be changed with forthcoming UX designs)\r\n\r\n## Feature
flag\r\n\r\nThis feature will be behind an experimental feature flag for
now, as the\r\ndesign is still a work in progress. To enable this
feature, add\r\n`xpack.integration_assistant.enableExperimental:
['generateCel']` to\r\nkibana.yml\r\n\r\n## Maintainer's notes\r\n\r\n-
UI tests were intentionally omitted for now, as the UI implemented
is\r\nonly temporary until we have a UX design.\r\n- Some OpenAPI specs
are too large to be uploaded at this time. I am\r\nworking on adding
support for that and have added another item to the\r\n[meta
issue](#193074) as
such\r\n\r\nRelates:
https://github.com/elastic/kibana/issues/193074\r\n___
\r\n\r\n<details>\r\n <summary>Screenshots</summary>\r\n \r\nAfter
selecting CEL during datastream configuration and reviewing
those\r\nresults, the user will be brought to a new screen to upload an
open api\r\nspec\r\n<img width=\"650\"
alt=\"upload\"\r\nsrc=\"https://github.com/user-attachments/assets/efdace4a-cc26-4f33-8b67-35c08df5f640\">\r\n\r\nThe
user can upload the spec file (as long as it isn't over the
file\r\nupload limit)\r\n<img width=\"650\" alt=\"spec
uploaded\"\r\nsrc=\"https://github.com/user-attachments/assets/9fd1b868-f8da-4d3c-b975-522bf66e05a5\">\r\n\r\nThe
user waits while the LLM runs\r\n<img width=\"650\" alt=\"Screenshot
2024-10-09 at 1 37
59 PM\"\r\nsrc=\"https://github.com/user-attachments/assets/3eca6b97-4525-4496-89b0-3002a97fa27d\">\r\n\r\nThe
user can view results \r\n<img width=\"650\"
alt=\"review\"\r\nsrc=\"https://github.com/user-attachments/assets/ee44fb16-fd3a-48c4-975f-706e6d381339\">\r\n\r\nThe
results are automatically pasted into the config, where the user
may\r\nfurther edit and configure the input\r\n<img width=\"635\"
alt=\"Screenshot 2024-10-08 at 11 17
46 AM\"\r\nsrc=\"https://github.com/user-attachments/assets/45151e13-0fd9-4f9a-bbfe-68e6f9b0e671\">\r\n\r\n</details>\r\n\r\n<details>\r\n
<summary>Sample results
</summary>\r\n\r\nsource:\r\n[MISP](https://raw.githubusercontent.com/MISP/MISP/develop/app/webroot/doc/openapi.yaml)\r\n
\r\nprogram:\r\n```\r\n(\r\n request(\"POST\", state.url +
\"/events/restSearch?\" + {\r\n \"page\": [string(state.page)],\r\n
\"limit\": [string(state.limit)],\r\n \"sort\": [\"date\"],\r\n
\"direction\": [\"asc\"]\r\n }.format_query()).with({\r\n \"Header\":
{\r\n \"Content-Type\": [\"application/json\"]\r\n }\r\n
}).do_request().as(resp,\r\n resp.StatusCode == 200 ?\r\n
bytes(resp.Body).decode_json().as(body, {\r\n \"events\": body.map(e,
{\r\n \"message\": e.encode_json()\r\n }),\r\n \"want_more\":
body.size() == state.limit,\r\n \"page\": state.page + 1,\r\n \"limit\":
state.limit\r\n })\r\n :\r\n {\r\n \"events\": [{\r\n \"error\": {\r\n
\"code\": string(resp.StatusCode),\r\n \"id\": string(resp.Status),\r\n
\"message\": string(resp.Body)\r\n }\r\n }],\r\n \"want_more\":
false\r\n }\r\n )\r\n)\r\n```\r\n\r\nintiial state:\r\n```\r\npage :
1\r\nlimit : 50\r\n```\r\n\r\nredact vars:\r\n```\r\n[
]\r\n```\r\n\r\n</details>","sha":"7f24e388829933be17f1ed7b690dc0562a354cd8"}}]}]
BACKPORT-->

Co-authored-by: Kylie Meli <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@kgeller kgeller

Labels
8.16 candidate enhancement New value added to drive a business result Feature:AutomaticImport Team:Security-Scalability Team label for Security Integrations Scalability Team
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@ebeahan @kgeller @elasticmachine