Semantic text dense vector support #105515

carlosdelest · 2024-02-14T17:28:22Z

Adds support for dense vector models.

To make sure the field mappers receive enough information to perform indexing, model information is included into each document that is ingested under model_settings field:

{
    "infer_field": "these are not the droids you're looking for. He's free to go around",
    "non_infer_field": "that's no moon. It's a space station",
    "_semantic_text_inference": {
      "infer_field": {
        "model_settings": {
          "inference_id": "my_e5_small",
          "task_type": "text_embedding",
          "dimensions": 384
        },
        "inference_results": [
          {
            "inference": [
              0.006414609029889107,
              0.00162538792937994,
              -0.05126339942216873,
              -0.05343875288963318,
              0.009770095348358154,
              0.03464227914810181
            ],
            "text": "these are not the droids you're looking for. He's free to go around"
          }
        ]
      }
    }
  }

…s class

…results

…ference results" This reverts commit bd4e19e.

…dense-vector-support # Conflicts: # server/src/main/java/org/elasticsearch/inference/ServiceSettings.java

carlosdelest · 2024-02-20T11:09:47Z

server/src/main/java/org/elasticsearch/action/bulk/BulkShardRequestInferenceProvider.java

@@ -90,7 +92,13 @@ public void onResponse(ModelRegistry.UnparsedModel unparsedModel) {
                        var service = inferenceServiceRegistry.getService(unparsedModel.service());
                        if (service.isEmpty() == false) {
                            InferenceProvider inferenceProvider = new InferenceProvider(
-                                service.get().parsePersistedConfig(inferenceId, unparsedModel.taskType(), unparsedModel.settings()),
+                                service.get()
+                                    .parsePersistedConfigWithSecrets(


Secrets are needed in order to perform inference on external services

carlosdelest · 2024-02-20T11:10:15Z

server/src/main/java/org/elasticsearch/action/bulk/BulkShardRequestInferenceProvider.java

+                        for (InferenceResults inferenceResults : results.transformToCoordinationFormat()) {
+                            String inferenceFieldName = inferenceFieldNames.get(i++);
+                            Map<String, Object> inferenceFieldResult = new LinkedHashMap<>();
+                            inferenceFieldResult.putAll(new ModelSettings(inferenceProvider.model).asMap());


Add model settings information to make it available to field mapping

carlosdelest · 2024-02-20T11:13:52Z

server/src/test/java/org/elasticsearch/action/bulk/BulkOperationTests.java

Some refactorings to provide mocks for the new methods used to retrieve config and inference

…dense-vector-support # Conflicts: # x-pack/plugin/inference/qa/test-service-plugin/src/main/java/org/elasticsearch/xpack/inference/mock/AbstractTestInferenceService.java # x-pack/plugin/inference/qa/test-service-plugin/src/main/java/org/elasticsearch/xpack/inference/mock/TestDenseInferenceServiceExtension.java # x-pack/plugin/inference/qa/test-service-plugin/src/main/java/org/elasticsearch/xpack/inference/mock/TestSparseInferenceServiceExtension.java

elasticsearchmachine · 2024-03-05T11:24:15Z

Pinging @elastic/es-search (Team:Search)

…dense-vector-support

On Serverless it is not possible to configure deprecation indexing (it is always off). This commit updates the behaviour of `ElasticsearchCluster` to no longer attempt to configure deprecation indexing on stateless nodes.

…tures` conditions (elastic#105763)

increase ILM logging to TRACE. Relates to elastic#103981

mute for elastic#105952

kderusso

Nice work!

server/src/main/java/org/elasticsearch/inference/ModelSettings.java

kderusso · 2024-03-05T13:30:45Z

...in/java/org/elasticsearch/xpack/inference/mapper/SemanticTextInferenceResultFieldMapper.java

+                }
+            }
+            Integer dimensions = modelSettings.dimensions();
+            if (dimensions == null) {


Couldn't dims be calculated automatically if they weren't provided?

They could! But it would signal that we're not getting what we expect from the inference results, which will always include them for dense vectors.

kderusso · 2024-03-05T13:31:56Z

...lugin/inference/src/yamlRestTest/java/org/elasticsearch/xpack/inference/InferenceRestIT.java

@@ -21,7 +21,7 @@ public class InferenceRestIT extends ESClientYamlSuiteTestCase {
    public static ElasticsearchCluster cluster = ElasticsearchCluster.local()
        .setting("xpack.security.enabled", "false")
        .setting("xpack.security.http.ssl.enabled", "false")
-        .plugin("org.elasticsearch.xpack.inference.mock.TestInferenceServicePlugin")
+        .plugin("inference-service-test")


Oh cool, you can just refer to the name here?

TIL as well 😆

...rence/src/yamlRestTest/resources/rest-api-spec/test/inference/10_semantic_text_inference.yml

…ests (elastic#105731) This setting requires expensive processing due to verification the integrity of many important files during a shard recovery or relocation. Therefore, it takes lots of time for the files to clean up and the assertShardFolder check may not complete in 30s. Fixes elastic#105202

Addresses test bug introduced in elastic#105721: we must consume all the `SnapshotInfo` instances before completing the final listener. Closes elastic#105922

server/src/main/java/org/elasticsearch/inference/ModelSettings.java

jimczi · 2024-03-05T16:01:35Z

...ce/src/yamlRestTest/resources/rest-api-spec/test/inference/20_semantic_text_field_mapper.yml

+                  - text: "inference test"
+                    inference: [0.1, 0.2, 0.3, 0.4, 0.5]
+                  - text: "another inference test"
+                    inference: [-0.1, -0.2, -0.3, -0.4, -0.5]


We should check that the mapping is as expected?

We won't be doing mapping updates, just the raw indexing into the Lucene doc. How can we check that?

I've added some tests for checking the mapping does not create additional fields, is that what you were thinking of?

I was thinking of checking that the field is updated with the right parameters (dimensions and similarity).

It is not - there is no change to the mappings that reflect this change:

The semantic_text field keeps having just inference_id and type.

There are no other fields created, as the metadata field mapper deals with the results without creating mapping changes.

Is there any mapping change you'd expect from this operation?

oh I see, the mapping change is on the inner field and they are not exposed in the mapping?

That is correct. There is no mapping change, the metadata field mapper performs the indexing directly, without a mapping change.

AFAIU we can't do updates in the metadata field mapper for holding the changes. And we can't create a mapping with the same field path as a metadata field.

Is there something I'm missing?

no that's just me trying to understand how to best test this (checking that we're creating the right field type in Lucene). We probably need a unit test that checks that we're doing the right thing but that can be a follow up.

There's some unit tests provided by @Mikep86 that check the Lucene doc structure.

Also, we can check the end-to-end scenario once we have the semantic_query ready, so we can check retrieving docs work

jimczi · 2024-03-05T16:03:03Z

...ce/src/yamlRestTest/resources/rest-api-spec/test/inference/20_semantic_text_field_mapper.yml

+                      feature_1: 0.1
+                      feature_2: 0.2
+                      feature_3: 0.3
+                      feature_4: 0.4


We should check the response too? And then verify that the mapping is as expected?

These tests provide the actual inference results so we can check that the parsing is done as expected, and the inference format is correct.

What would you like to check in the responses?

I misread, I thought it was a bulk index but it's just one document sorry.

…5930) This change ensures that the matches implementation of the `SourceConfirmedTextQuery` only checks the current document instead of calling advance on the two phase iterator. The latter tries to find the first doc that matches the query instead of restricting the search to the current doc. This can lead to abnormally slow highlighting if the query is very restrictive and the highlight is done on a non-matching document. Closes elastic#103298

…c#105912) * Bugfix for CCR queries using text expansion * Fix test * PR feedback * Fix test * Minor cleanup * Edit comment * One more comment clarification --------- Co-authored-by: Elastic Machine <[email protected]>

jimczi

I left a minor comment, LGTM otherwise

jimczi · 2024-03-06T12:34:33Z

server/src/main/java/org/elasticsearch/inference/ModelSettings.java

@@ -8,7 +8,6 @@



Not a fan of the renaming, why not keeping the original name? ModelSettings is too generic imo and we only need these settings for the field mapping.

Makes sense, renaming it back to SemanticTextModelSettings

benwtrent

Good step in the right direction. I like the reuse of the mapper builders. Maybe even allowing MORE model types in the future.

We do need to figure out how to prevent inference results formats from changing once we have parsed a particular kind of field, but that discussion is outside the scope of this pr.

carlosdelest added 16 commits February 14, 2024 18:16

Move SimilarityMeasure to server code

8a2dbd4

Add dimensions and similarity to ServiceSettings, create ModelSetting…

fc76918

…s class

Change implementation of asMap() to avoid extra nesting in inference …

bd4e19e

…results

Change inference results structure

896ec49

Field mapper uses new inference results structure

af763f0

Fix BulkOperationTests

fbefa0b

Fix tests of SemanticTextInferenceResultFieldMapper

59194d7

Revert "Change implementation of asMap() to avoid extra nesting in in…

8b5489b

…ference results" This reverts commit bd4e19e.

Use coordination format instead of changing the asMap() implementation

ebd49b0

Fix tests for new results structure

7f1dfb4

Add service extension for dense vector embeddings

22daa5d

Add tests for dense vector embeddings

724f8d1

Fix bug in model settings

ce4125a

Initial work on inference field results mapping for validation

02b7cc4

Fix spotless

7e98736

Refactored inference services with common abstract class

436b40f

carlosdelest mentioned this pull request Feb 15, 2024

Extract common ServiceSettings methods #105553

Merged

Merge branch 'feature/semantic-text' into carlosdelest/semantic-text-…

b2769f3

…dense-vector-support # Conflicts: # server/src/main/java/org/elasticsearch/inference/ServiceSettings.java

carlosdelest commented Feb 20, 2024

View reviewed changes

carlosdelest mentioned this pull request Feb 20, 2024

Add dense vector inference mock service for testing #105655

Merged

carlosdelest added 4 commits February 21, 2024 14:04

Add javadoc

3500c9c

Fixing tests after merge

2e46037

Fix tests

9d7be42

Mikep86 mentioned this pull request Feb 22, 2024

WIP - Semantic text query Mikep86/elasticsearch#1

Closed

Change back javadoc

fd553a9

carlosdelest changed the title ~~WIP - Semantic text dense vector support~~ Semantic text dense vector support Mar 5, 2024

carlosdelest marked this pull request as ready for review March 5, 2024 11:22

carlosdelest requested a review from a team March 5, 2024 11:22

elasticsearchmachine added the needs:triage Requires assignment of a team area label label Mar 5, 2024

carlosdelest added :Search Foundations/Mapping Index mappings, including merging and defining field types Team:Search Meta label for search team :Search Relevance/Vectors Vector search >non-issue labels Mar 5, 2024

elasticsearchmachine removed the needs:triage Requires assignment of a team area label label Mar 5, 2024

carlosdelest and others added 6 commits March 5, 2024 12:26

Merge branch 'feature/semantic-text' into carlosdelest/semantic-text-…

cdc579f

…dense-vector-support

Fix node_selector in new esql test (elastic#105943)

139c94b

Fix gradle run on Serverless (elastic#105938)

e9ff896

On Serverless it is not possible to configure deprecation indexing (it is always off). This commit updates the behaviour of `ElasticsearchCluster` to no longer attempt to configure deprecation indexing on stateless nodes.

YAML test framework: re-introduce requires section and `cluster_fea…

1e76b18

…tures` conditions (elastic#105763)

Unmute testRollupNonTSIndex() and (elastic#105949)

7191758

increase ILM logging to TRACE. Relates to elastic#103981

Test mute for elastic#105952 (elastic#105953)

4f2c8ca

mute for elastic#105952

kderusso reviewed Mar 5, 2024

View reviewed changes

benwtrent and others added 3 commits March 5, 2024 08:44

Add note about optional times and epochs (elastic#105786)

61b3d98

Fix TransportSLMGetExpiredSnapshotsActionTests (elastic#105950)

7c6120b

Addresses test bug introduced in elastic#105721: we must consume all the `SnapshotInfo` instances before completing the final listener. Closes elastic#105922

jimczi reviewed Mar 5, 2024

View reviewed changes

jimczi and others added 3 commits March 5, 2024 16:06

Merge branch 'main' into carlosdelest/semantic-text-dense-vector-support

a3bdabf

carlosdelest requested a review from a team as a code owner March 5, 2024 16:40

carlosdelest added the v8.14.0 label Mar 5, 2024

carlosdelest added 3 commits March 5, 2024 18:49

Remove duplicate class

0f5e7a3

Check mappings are as expected

7eddba0

I hate YAML

360256d

carlosdelest requested a review from jimczi March 6, 2024 09:24

jimczi approved these changes Mar 6, 2024

View reviewed changes

benwtrent approved these changes Mar 6, 2024

View reviewed changes

carlosdelest merged commit b1a3ee8 into elastic:feature/semantic-text Mar 6, 2024
13 of 14 checks passed

carlosdelest added a commit that referenced this pull request Mar 6, 2024

This was supposed to be merged into #105515 but didn't make it

2039fb3

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Semantic text dense vector support #105515

Semantic text dense vector support #105515

carlosdelest commented Feb 14, 2024 •

edited

Loading

carlosdelest Feb 20, 2024

carlosdelest Feb 20, 2024

carlosdelest Feb 20, 2024

elasticsearchmachine commented Mar 5, 2024

kderusso left a comment

kderusso Mar 5, 2024

carlosdelest Mar 5, 2024

kderusso Mar 5, 2024

carlosdelest Mar 5, 2024

jimczi Mar 5, 2024

carlosdelest Mar 5, 2024

carlosdelest Mar 5, 2024

jimczi Mar 6, 2024

carlosdelest Mar 6, 2024

jimczi Mar 6, 2024

carlosdelest Mar 6, 2024

jimczi Mar 6, 2024

carlosdelest Mar 6, 2024

jimczi Mar 5, 2024

carlosdelest Mar 5, 2024

jimczi Mar 6, 2024

jimczi left a comment

jimczi Mar 6, 2024

carlosdelest Mar 6, 2024

benwtrent left a comment

Semantic text dense vector support #105515

Semantic text dense vector support #105515

Conversation

carlosdelest commented Feb 14, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elasticsearchmachine commented Mar 5, 2024

kderusso left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jimczi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

benwtrent left a comment

Choose a reason for hiding this comment

carlosdelest commented Feb 14, 2024 •

edited

Loading