Add an option to force the numeric type of a field sort #38095

jimczi · 2019-01-31T14:29:15Z

This change adds an option to the FieldSortBuilder that allows to transform the type
of a numeric field into another. Possible values for this option are long that transforms
the source field into an integer and double that transforms the source field into a floating point.
This new option is useful for cross-index search when the sort field is mapped differently on some
indices. For instance if a field is mapped as a floating point in one index and as an integer in another
it is possible to align the type for both indices using the numeric_type option:

{
   "sort": {
    "field": "my_field",
    "numeric_type": "double" <1>
   }
}

<1> Ensure that values for this field are transformed to a floating point if needed.

Only long and double are supported at the moment but the goal is to also handle date and date_nanos when #32601 is merged.

Naming should also be improved, I choose numeric_type to start with something but I am not happy with it. Propositions are welcome ;).

This change adds an option to the `FieldSortBuilder` that allows to transform the type of a numeric field into another. Possible values for this option are `long` that transforms the source field into an integer and `double` that transforms the source field into a floating point. This new option is useful for cross-index search when the sort field is mapped differently on some indices. For instance if a field is mapped as a floating point in one index and as an integer in another it is possible to align the type for both indices using the `numeric_type` option: ``` { "sort": { "field": "my_field", "numeric_type": "double" <1> } } ``` <1> Ensure that values for this field are transformed to a floating point if needed. Only `long` and `double` are supported at the moment but the goal is to also handle `date` and `date_nanos` when elastic#32601 is merged.

elasticmachine · 2019-01-31T14:29:17Z

Pinging @elastic/es-search

mayya-sharipova

Thanks Jim, makes sense!

One thing I was wondering (not related to your PR) if it is correct that we return 503 for wrong sorting (using float and double and not setting numeric_type)? Should it be 400 instead?

{
  "error": {
    "root_cause": [],
    "type": "search_phase_execution_exception",
    "reason": "",
    "phase": "fetch",
    "grouped": true,
    "failed_shards": [],
    "caused_by": {
      "type": "class_cast_exception",
      "reason": "class java.lang.Float cannot be cast to class java.lang.Double (java.lang.Float and java.lang.Double are in module java.base of loader 'bootstrap')"
    }
  },
  "status": 503
}

mayya-sharipova · 2019-02-06T19:52:10Z

docs/reference/search/request/sort.asciidoc

+
+Since `field` is mapped as a `double` in the first index and as a `long`
+in the second index, it is not possible to use this field to sort requests
+that query both indices bu default. However you can force the type to one


bu default -> by default

jpountz

I left some comments but the idea makes sense to me.

jpountz · 2019-03-11T19:02:02Z

docs/reference/search/request/sort.asciidoc

+It is also possible to transform a floating point field into a `long`
+but note that in this case floating points are replaced by the largest
+value that is less than or equal to the argument and is equal to a mathematical
+integer.


this is not correct for negative values

right, thanks

jpountz · 2019-03-11T19:06:39Z

server/src/main/java/org/elasticsearch/search/sort/FieldSortBuilder.java

-            SortField field = fieldData.sortField(missing, localSortMode, nested, reverse);
+            final SortField field;
+            if (numericType != null) {
+                if (fieldData instanceof SortedNumericDVIndexFieldData == false) {


why do you check against this class rather than IndexNumericFieldData?

This is a mistake, I pushed 24c961a

This change adds an option to the `FieldSortBuilder` that allows to transform the type of a numeric field into another. Possible values for this option are `long` that transforms the source field into an integer and `double` that transforms the source field into a floating point. This new option is useful for cross-index search when the sort field is mapped differently on some indices. For instance if a field is mapped as a floating point in one index and as an integer in another it is possible to align the type for both indices using the `numeric_type` option: ``` { "sort": { "field": "my_field", "numeric_type": "double" <1> } } ``` <1> Ensure that values for this field are transformed to a floating point if needed.

This change adapts the bwc version check for the new numeric_type option of the FieldSortBuilder. Since this change needs to be backported to 7x, all bwc tests are temporary disabled until elastic#40084 is merged. Relates elastic#38095

) This change adds an option to the `FieldSortBuilder` that allows to transform the type of a numeric field into another. Possible values for this option are `long` that transforms the source field into an integer and `double` that transforms the source field into a floating point. This new option is useful for cross-index search when the sort field is mapped differently on some indices. For instance if a field is mapped as a floating point in one index and as an integer in another it is possible to align the type for both indices using the `numeric_type` option: ``` { "sort": { "field": "my_field", "numeric_type": "double" <1> } } ``` <1> Ensure that values for this field are transformed to a floating point if needed.

This change adapts the bwc version check for the new numeric_type option of the FieldSortBuilder. Since this change needs to be backported to 7x, all bwc tests are temporary disabled until #40084 is merged. Relates #38095

jpountz · 2019-07-05T10:29:37Z

@jimczi I'm assuming there is nothing left to backport and removed the backport pending label.

russcam · 2019-08-30T02:32:24Z

Looks like this went into v7.2.0?

elasticsearch/server/src/main/java/org/elasticsearch/search/sort/FieldSortBuilder.java

Lines 306 to 321 in be513ce

    
           public FieldSortBuilder setNumericType(String numericType) { 
        
               String lowerCase = numericType.toLowerCase(Locale.ENGLISH); 
        
               switch (lowerCase) { 
        
                   case "long": 
        
                   case "double": 
        
                   case "date": 
        
                   case "date_nanos": 
        
                       break; 
        
                   default: 
        
                       throw new IllegalArgumentException("invalid value for [numeric_type], " + 
        
                           "must be [long, double, date, date_nanos], got " + lowerCase); 
        
               } 
        
               this.numericType = lowerCase; 
        
               return this; 
        
           }

jpountz · 2019-09-05T12:09:51Z

@russcam Indeed, I just fixed the version tag and the release notes. Thanks for spotting!

jimczi added >feature :Search/Search Search-related issues that do not fall into other categories v7.0.0 v6.7.0 labels Jan 31, 2019

jimczi added 3 commits January 31, 2019 15:36

checkstyle

0174ca5

checkstyle

75feb1a

Merge branch 'master' into sort_mix_numeric

c4cb499

jasontedor added v8.0.0 and removed v7.0.0 labels Feb 6, 2019

mayya-sharipova approved these changes Feb 6, 2019

View reviewed changes

danielmitterdorfer added v7.2.0 and removed v6.7.0 labels Feb 7, 2019

jpountz self-requested a review March 11, 2019 09:51

jpountz requested changes Mar 11, 2019

View reviewed changes

jimczi added 4 commits March 12, 2019 09:52

Merge branch 'master' into sort_mix_numeric

4114401

address review

24c961a

fix typo

a27eb7c

adapt serialization version

437fc64

jimczi requested a review from jpountz March 12, 2019 16:05

jpountz approved these changes Mar 13, 2019

View reviewed changes

fix conflict

0bc1b72

jimczi merged commit ddd34b1 into elastic:master Mar 15, 2019

jimczi deleted the sort_mix_numeric branch March 15, 2019 09:25

jimczi mentioned this pull request Mar 15, 2019

Add an option to force the numeric type of a field sort (#38095) #40084

Merged

jimczi mentioned this pull request Mar 15, 2019

Adapt bwc serialization in FieldSortBuilder #40095

Merged

jimczi added the backport pending label Mar 15, 2019

jakelandis added v7.3.0 and removed v7.2.0 labels Jun 17, 2019

jpountz removed the backport pending label Jul 5, 2019

Mpdreamz mentioned this pull request Aug 7, 2019

[meta] 7.3 Release elastic/elasticsearch-net#4001

Closed

16 tasks

jpountz added v7.2.0 and removed v7.3.0 labels Sep 4, 2019

jpountz added a commit that referenced this pull request Sep 5, 2019

Fix release version of #38095.

7f5bbd2

jpountz added a commit that referenced this pull request Sep 5, 2019

Fix release version of #38095.

5ce779c

jpountz added a commit that referenced this pull request Sep 5, 2019

Fix release version of #38095.

4c0849a

jpountz added a commit that referenced this pull request Sep 5, 2019

Fix release version of #38095.

3f5f028

jakelandis added v8.0.0-alpha1 and removed v8.0.0 labels Jul 26, 2021

nephel mentioned this pull request Nov 5, 2021

Sorting mixing "float" and "double" types doesn't work #78699

Open

bczifra mentioned this pull request Nov 11, 2021

ClassCastException when sorting while mixing long and float fields #80658

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add an option to force the numeric type of a field sort #38095

Add an option to force the numeric type of a field sort #38095

jimczi commented Jan 31, 2019

elasticmachine commented Jan 31, 2019

mayya-sharipova left a comment

mayya-sharipova Feb 6, 2019

jpountz left a comment

jpountz Mar 11, 2019

jimczi Mar 12, 2019

jpountz Mar 11, 2019

jimczi Mar 12, 2019

jpountz commented Jul 5, 2019

russcam commented Aug 30, 2019

jpountz commented Sep 5, 2019

Add an option to force the numeric type of a field sort #38095

Add an option to force the numeric type of a field sort #38095

Conversation

jimczi commented Jan 31, 2019

elasticmachine commented Jan 31, 2019

mayya-sharipova left a comment

Choose a reason for hiding this comment

mayya-sharipova Feb 6, 2019

Choose a reason for hiding this comment

jpountz left a comment

Choose a reason for hiding this comment

jpountz Mar 11, 2019

Choose a reason for hiding this comment

jimczi Mar 12, 2019

Choose a reason for hiding this comment

jpountz Mar 11, 2019

Choose a reason for hiding this comment

jimczi Mar 12, 2019

Choose a reason for hiding this comment

jpountz commented Jul 5, 2019

russcam commented Aug 30, 2019

jpountz commented Sep 5, 2019