Problem with sorting on a filtered query

[binary-data]

Hello. I am trying to sort search results by an integer field, but only see exceptions. I am using version 1.3.4 of elasticsearch-php and elasticsearch 1.6

Here is my mapping:

$mapping = [
            '_source' => [
                'enabled' => true
            ],
            'properties' => [
                'title' => ['type' => 'string'],
                'description' => ['type' => 'string'],
                'category_title' => ['type' => 'string'],
                'price' => ['type' => 'integer', 'include_in_all' => false],
                'is_hit' => ['type' => 'boolean', 'include_in_all' => false],
                'has_discount' => ['type' => 'boolean', 'include_in_all' => false],
                'updated_at' => ['type' => 'date', 'format' => 'YYYY-MM-dd H:m:s', 'include_in_all' => false],
            ]
        ];

And here is my basic search request that is working:

$es_query = ['match' =>
            [
                '_all' => [
                    'query' => $query,
                    'fuzziness' => 'AUTO',
                    'operator' => 'and',
                    'fuzzy_transpositions' => 'true',
                ],
            ],
        ];

$es_filter['term']['has_discount'] = true;

$params = [
            'index' => 'my_index',
            'type' => 'my_type',
            'size' => 12,
            'from' => 0,
            'body' => [
                'query' => [
                    'filtered' => [
                        'query' => $es_query,
                        'filter' => $es_filter,
                    ],
                ],
            ],
        ];

$product_results = $this->client->search($params);

But when I add sort parameter to the body of this query, I see BadRequest exception:

$params['body']['sort'] = [
            ['price' => ['order' => 'asc']],
        ];

Elasticsearch log says:

[2015-07-09 16:47:49,980][DEBUG][action.count ] [Lightbright] [my_index][0], node[nS1lsH9YSG-7ER01Zc5kzg], [P], s[STARTED]: failed to execute [[[my_index]][], source[{"query":{"filtered":{"query":{"match":{"_all":{"query":"sometext","fuzziness":"AUTO","operator":"and","fuzzy_transpositions":"true"}}},"filter":[]}},"sort":[{"price":{"order":"asc"}}]}]] org.elasticsearch.index.query.QueryParsingException: [my_index] request does not support [sort] at org.elasticsearch.index.query.IndexQueryParserService.parseQuery(IndexQueryParserService.java:360) at org.elasticsearch.action.count.TransportCountAction.shardOperation(TransportCountAction.java:187) at org.elasticsearch.action.count.TransportCountAction.shardOperation(TransportCountAction.java:66) at org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction$1.run(TransportBroadcastOperationAction.java:170) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

When I trying to add sort parameter to other parameters, like this

$params['sort'] = 'price:asc';

I see an error message, telling me that there is no such parameter available.

Fatal error: Uncaught exception 'Elasticsearch\Common\Exceptions\UnexpectedValueException' with message '"sort" is not a valid parameter. Allowed parameters are: "ignore_unavailable", "allow_no_indices", "expand_wildcards", "min_score", "preference", "routing", "source", "ignore", "custom", "curlOpts"' in /home/binarydata/work/project/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Endpoints/AbstractEndpoint.php:266

When I querying from browser, elasticsearch shows me correct sorting:

http://localhost:9200/my_index/_search?pretty&q=sometext&sort=price:asc

Could you tell me, what is the correct way of sorting such queries?

[polyfractal]

Can you paste your complete code? There must be something else going on, this works for me:

$client = new Elasticsearch\Client();

$es_query = ['match' =>
    [
        '_all' => [
            'query' => "abc",
            'fuzziness' => 'AUTO',
            'operator' => 'and',
            'fuzzy_transpositions' => 'true',
        ],
    ],
];

$es_filter['term']['has_discount'] = true;

$params = [
    'index' => 'my_index',
    'type' => 'my_type',
    'size' => 12,
    'from' => 0,
    'body' => [
        'query' => [
            'filtered' => [
                'query' => $es_query,
                'filter' => $es_filter,
            ],
        ],
    ],
];
$params['body']['sort'] = [
    ['price' => ['order' => 'asc']],
];


$product_results = $client->search($params);
print_r($product_results);

Array
(
    [took] => 4
    [timed_out] => 
    [_shards] => Array
        (
            [total] => 5
            [successful] => 5
            [failed] => 0
        )

    [hits] => Array
...

[binary-data]

I see nothing special in my other code. Here is gist: https://gist.github.com/binary-data/f1e5608650afea28d4ff

I can provide any other information that you might need

[polyfractal]

Hmm.... I don't see anything obvious right away. Could you add this right before the search:

print_r($params);

Just so we can see what the final params array looks like before it is sent to the client?

[binary-data]

Here is print_r():

Array
(
    [index] => my_index
    [type] => my_type
    [size] => 12
    [from] => 0
    [body] => Array
        (
            [query] => Array
                (
                    [filtered] => Array
                        (
                            [query] => Array
                                (
                                    [match] => Array
                                        (
                                            [_all] => Array
                                                (
                                                    [query] => abcdef
                                                    [fuzziness] => AUTO
                                                    [operator] => and
                                                    [fuzzy_transpositions] => true
                                                )

                                        )

                                )

                            [filter] => Array
                                (
                                )

                        )

                )

            [sort] => Array
                (
                    [0] => Array
                        (
                            [price] => Array
                                (
                                    [order] => asc
                                )

                        )

                )

        )

)

And here is var_export():

array (
  'index' => 'my_index',
  'type' => 'my_type',
  'size' => '12',
  'from' => 0,
  'body' => 
  array (
    'query' => 
    array (
      'filtered' => 
      array (
        'query' => 
        array (
          'match' => 
          array (
            '_all' => 
            array (
              'query' => 'abcdef',
              'fuzziness' => 'AUTO',
              'operator' => 'and',
              'fuzzy_transpositions' => 'true',
            ),
          ),
        ),
        'filter' => 
        array (
        ),
      ),
    ),
    'sort' => 
    array (
      0 => 
      array (
        'price' => 
        array (
          'order' => 'asc',
        ),
      ),
    ),
  ),
)

[modmac]

Hey, try to change integer types to long.

I think problemm occurs when ES try to compare int types of data with another (long?) types.

This is not a elasticsearch-php bug, but ES generally.

[modmac]

You shouldn't have different types for the same-named field in your index.

This should be fixed by elastic/elasticsearch#8870

[binary-data]

Hi, Modest. Thanks for your reply. But could you explain, what of my index fields have same name? I have only one type in one index. http://localhost:9200/_all/_mapping?pretty gives me:

{
  "my_index" : {
    "mappings" : {
      "my_type" : {
        "properties" : {
          "category_title" : {
            "type" : "string"
          },
          "description" : {
            "type" : "string"
          },
          "has_discount" : {
            "type" : "boolean"
          },
          "is_hit" : {
            "type" : "boolean"
          },
          "price" : {
            "type" : "integer",
            "include_in_all" : false
          },
          "title" : {
            "type" : "string"
          },
          "updated_at" : {
            "type" : "date",
            "format" : "YYYY-MM-dd H:m:s",
            "include_in_all" : false
          }
        }
      }
    }
  }
}

[modmac]

Hmm try to make simple search request with sort param:

http://localhost:9200/my_index/my_type/_search?sort=price

[binary-data]

I tried this before, as I described in my first post. I also tried it right now with adding type, and I am seeing correct results.

[polyfractal]

Good news and bad news.

Good news, I reproduced your error. I've been mulling over this and just couldn't reproduce it based on the code you provided. But then I noticed two strange things in your log error: [DEBUG][action.count ] and TransportCountAction. The TransportCountAction class is used for Count API requests, not search requests.

That execution path does not support sorting (or aggregations, suggest, highlighting, etc) because you are explicitly only asking for a count. ES throws an exception because you supplied something other than "query".

So that's the problem. Somewhere, something is invoking a Count API with a full search query.

The bad news is that the gist you provided is only using Search.. So it must be somewhere else in your code. Or perhaps you've modified the client source somewhere?

[binary-data]

Thanks a lot! It's true, there was a count() method in my code below. Before calling it I was unsetting unnecessary parameters, but forgot to unset a sort parameter. Now it all works fine! Sorry for distract you.

[polyfractal]

Np, happy to help! Glad you found the culprit, I was quickly running out of ideas :)