Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support multiple input/output maps for ML processors #244

Merged
merged 5 commits into from
Jul 26, 2024

Conversation

ohltyler
Copy link
Member

Description

This PR expands the form inputs for the input map / output map fields for ML processors, to be a list of maps, instead of a single map. This is so multiple predict/inference calls can happen within a single processor's execution. At a high level, this means changing the form, schema, and components to maintain an arr of maps instead of a single one.

Additionally, In the advanced transform modals, users can now select which predict/inference call they want to test out and see test transformation results of.

More specific details:

  • add new mapArray config field type
  • add new MapArrayField which takes in a Formik's arr field and maintains an arr of individual MapFields
  • update the transform modals to maintain a specific map/prediction to test with. Adds in a selector for users to choose which map/prediction they want
  • minor refactoring updates to the conversion fns for handling the new mapArray field type, used in the ML processor's inputMap/outputMap fields

Testing

  • confirmed arrays of input_maps and output_maps work as expected in the simulate pipeline calls within the transform modals, as well as the actual ingest calls and checking the indexed data
  • confirmed edge cases (some empty/mismatching arr sizes) for both input and output transforms work as expected

Demo video, showing configuration of multiple prediction/inference mappings configured in the input and output maps, validation of mismatching, and transformation testing within the input transform modal and output transform modals:

screen-capture.12.webm

Issues Resolved

Makes progress on #23

Check List

  • Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@ohltyler ohltyler merged commit e908023 into opensearch-project:main Jul 26, 2024
6 checks passed
@ohltyler ohltyler deleted the nested-input-map branch July 26, 2024 23:17
opensearch-trigger-bot bot pushed a commit that referenced this pull request Jul 26, 2024
Signed-off-by: Tyler Ohlsen <[email protected]>
(cherry picked from commit e908023)
ohltyler added a commit that referenced this pull request Jul 26, 2024
Signed-off-by: Tyler Ohlsen <[email protected]>
(cherry picked from commit e908023)

Co-authored-by: Tyler Ohlsen <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants