-
Notifications
You must be signed in to change notification settings - Fork 503
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add documentation about setting a default model for neural search #5121
Conversation
Signed-off-by: Fanit Kolchina <[email protected]>
Signed-off-by: Fanit Kolchina <[email protected]>
Signed-off-by: Fanit Kolchina <[email protected]>
Signed-off-by: Fanit Kolchina <[email protected]>
Signed-off-by: Fanit Kolchina <[email protected]>
Signed-off-by: Fanit Kolchina <[email protected]>
"query_text": "Hello world", | ||
"model_id": "xzy76xswsd", | ||
"passage_embedding": { | ||
"query_text": "Hi world", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I believe here "Hi planet" would come.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actually, I didn't want to search for the exact text here because vector search would match both documents anyway. Do you think it makes more sense to search for the exact text?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we should always give user a sucessful case so he goes in right direction
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Both GET requests return results. I have added the responses to show the results.
"match": { "passage_text": "Hello world" } | ||
"match": { | ||
"passage_text": "Hi world" | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
"Hi planet"
_search-plugins/neural-search.md
Outdated
|
||
To eliminate passing the model ID with each neural query request, you can set a default model on a k-NN index or a field. | ||
|
||
First, create a [search pipeline]({{site.url}}{{site.baseurl}}/search-plugins/search-pipelines/index/) with a [`neural_query_enricher`]({{site.url}}{{site.baseurl}}/search-plugins/search-pipelines/neural-query-enricher/) request processor. To set a default model on an index, provide the model ID in the `default_model_id` parameter. To set a default model on a specific field, provide the field name and the corresponding model ID in the `neural_field_default_id` map: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In this sentence, can you please mention that if both default_model_id and neural_field_default_id is provided then priority would be given to neural_field_default_id .
"query": { | ||
"neural": { | ||
"passage_embedding": { | ||
"query_text": "Hi world", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi planet
Field | Data type | Description | ||
:--- | :--- | :--- | ||
`default_model_id` | String | The model ID of the default model for an index. Optional. You must specify at least one of `default_model_id` and `neural_field_default_id`. | ||
`neural_field_default_id` | Object | A map of key-value pairs representing document field names and their associated default model IDs. Optional. You must specify at least one of `default_model_id` and `neural_field_default_id`. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If both are given then neural field default Id would be prioritized
Signed-off-by: Fanit Kolchina <[email protected]>
Signed-off-by: Fanit Kolchina <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Great writing! Few comments that should be quick to address/review.
|
||
## Updating a search pipeline | ||
|
||
To update a search pipeline dynamically, replace the search pipeline using the Search Pipeline API. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should we include a link to the search pipeline API documentation?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There's really not a separate API page for this and I think in this case I'm just referring to the /_search/pipeline endpoint so it should be understandable.
#### Example request | ||
|
||
The following request creates a search pipeline with a `filter_query` request processor that uses a term query to return only public messages and a response processor that renames the field `message` to `notification`: | ||
To create a search pipeline, send a request to the search pipeline endpoint, specifying an ordered list of processors, which will be applied sequentially: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Great job writing this sentence succinctly. Your wording is perfect.
grand_parent: Search pipelines | ||
--- | ||
|
||
# Neural query enrich processor |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should "enrich" be "enricher"?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Technically, yes. I went back and forth on this and decided to go with a more English-like version :) Let's ask @natebower's opinion.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's err on the side of technical accuracy and go with "enricher".
@@ -0,0 +1,47 @@ | |||
--- | |||
layout: default | |||
title: Neural query enrich |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should title mirror the wording used in line 10?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since this page is under the "Search processors" heading in the left nav, I'm just omitting "processor" here so the string is shorter. I do the same with the other processors.
|
||
# Retrieving search pipelines | ||
|
||
To retrieve the details of an existing search pipeline, use the Search Pipeline API. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Include link to API documentation?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Same as above: not really anything to link to :)
Co-authored-by: Melissa Vagi <[email protected]> Signed-off-by: kolchfa-aws <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@kolchfa-aws Please see my comments and changes and let me know if you have any questions. Thanks!
_search-plugins/neural-search.md
Outdated
OpenSearch responds with information about your new index: | ||
## Step 3: Ingest documents into the index | ||
|
||
To ingest documents into the index created in the previous section, send a POST request for each document: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
"step" instead of "section"?
_search-plugins/neural-search.md
Outdated
|
||
First, create a [search pipeline]({{site.url}}{{site.baseurl}}/search-plugins/search-pipelines/index/) with a [`neural_query_enricher`]({{site.url}}{{site.baseurl}}/search-plugins/search-pipelines/neural-query-enricher/) request processor. To set a default model on an index, provide the model ID in the `default_model_id` parameter. To set a default model on a specific field, provide the field name and the corresponding model ID in the `neural_field_default_id` map. If you provide both `default_model_id` and `neural_field_default_id`, `neural_field_default_id` takes precedence: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
To set a default model "for" an index? To set a default model "for" a specific field?
Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: kolchfa-aws <[email protected]>
) * Add documentation about setting a default model for neural search Signed-off-by: Fanit Kolchina <[email protected]> * Add new processor to the processor list Signed-off-by: Fanit Kolchina <[email protected]> * More tweaks Signed-off-by: Fanit Kolchina <[email protected]> * Refactor search pipeline documentation Signed-off-by: Fanit Kolchina <[email protected]> * Refactor retrieving search pipelines Signed-off-by: Fanit Kolchina <[email protected]> * Add working examples Signed-off-by: Fanit Kolchina <[email protected]> * Implement tech review comments Signed-off-by: Fanit Kolchina <[email protected]> * Add responses to documentation Signed-off-by: Fanit Kolchina <[email protected]> * Update _search-plugins/search-pipelines/neural-query-enricher.md Co-authored-by: Melissa Vagi <[email protected]> Signed-off-by: kolchfa-aws <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: kolchfa-aws <[email protected]> --------- Signed-off-by: Fanit Kolchina <[email protected]> Signed-off-by: kolchfa-aws <[email protected]> Co-authored-by: Melissa Vagi <[email protected]> Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Melissa Vagi <[email protected]>
…ensearch-project#5121) * Add documentation about setting a default model for neural search Signed-off-by: Fanit Kolchina <[email protected]> * Add new processor to the processor list Signed-off-by: Fanit Kolchina <[email protected]> * More tweaks Signed-off-by: Fanit Kolchina <[email protected]> * Refactor search pipeline documentation Signed-off-by: Fanit Kolchina <[email protected]> * Refactor retrieving search pipelines Signed-off-by: Fanit Kolchina <[email protected]> * Add working examples Signed-off-by: Fanit Kolchina <[email protected]> * Implement tech review comments Signed-off-by: Fanit Kolchina <[email protected]> * Add responses to documentation Signed-off-by: Fanit Kolchina <[email protected]> * Update _search-plugins/search-pipelines/neural-query-enricher.md Co-authored-by: Melissa Vagi <[email protected]> Signed-off-by: kolchfa-aws <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: kolchfa-aws <[email protected]> --------- Signed-off-by: Fanit Kolchina <[email protected]> Signed-off-by: kolchfa-aws <[email protected]> Co-authored-by: Melissa Vagi <[email protected]> Co-authored-by: Nathan Bower <[email protected]>
) * Add documentation about setting a default model for neural search Signed-off-by: Fanit Kolchina <[email protected]> * Add new processor to the processor list Signed-off-by: Fanit Kolchina <[email protected]> * More tweaks Signed-off-by: Fanit Kolchina <[email protected]> * Refactor search pipeline documentation Signed-off-by: Fanit Kolchina <[email protected]> * Refactor retrieving search pipelines Signed-off-by: Fanit Kolchina <[email protected]> * Add working examples Signed-off-by: Fanit Kolchina <[email protected]> * Implement tech review comments Signed-off-by: Fanit Kolchina <[email protected]> * Add responses to documentation Signed-off-by: Fanit Kolchina <[email protected]> * Update _search-plugins/search-pipelines/neural-query-enricher.md Co-authored-by: Melissa Vagi <[email protected]> Signed-off-by: kolchfa-aws <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: kolchfa-aws <[email protected]> --------- Signed-off-by: Fanit Kolchina <[email protected]> Signed-off-by: kolchfa-aws <[email protected]> Co-authored-by: Melissa Vagi <[email protected]> Co-authored-by: Nathan Bower <[email protected]>
Fixes #5060
Checklist
For more information on following Developer Certificate of Origin and signing off your commits, please check here.