The Elasticsearch Learning to Rank plugin uses machine learning to improve search relevance ranking. It's powering search at places like Wikimedia Foundation and Snagajob!

What this plugin does...

This plugin:

• Allows you to store features (Elasticsearch query templates) in Elasticsearch
• Logs features scores (relevance scores) to create a training set for offline model development
• Stores linear, xgboost, or ranklib ranking models in Elasticsearch that use features you've stored
• Ranks search results using a stored model

Where's the docs?

We recommend taking time to read the docs. There's quite a bit of detailed information about learning to rank basics and how this plugin can ease learning to rank development.

I want to jump in!

If you want to just jump in, go straight to the demo. The demo uses Ranklib, a relatively straightforward Java Learning to Rank library, to train models. Follow the directions in the demo README, edit code, and have fun!

Installing

See the full list of prebuilt versions. If you don't see a version available, see the link below for building or file a request via issues.

To install, you'd run a command such as:

./bin/elasticsearch-plugin install http://es-learn-to-rank.labs.o19s.com/ltr-1.0.0-es6.1.2.zip

(It's expected you'll confirm some security exceptions, you can pass -b to elasticsearch-plugin to automatically install)

If you already are running Elasticsearch, don't forget to restart!

Know issues

As any other piece of software, this plugin is not exempt from issues. Please read the known issues to learn about the current issues that we are aware of. This file might include workarounds to mitigate them when possible.

Development

Notes if you want to dig into the code or build for a version there's no build for.

1. Build with Gradle Wrapper

./gradlew clean check

This runs the tasks in the esplugin gradle plugin that builds, tests, generates a Elasticsearch plugin zip file available in ./build/distributions.

When building to support another version of Elasticsearch, versions should be modified in ./build.gradle, and possibly ./gradle/wrapper/gradle-wrapper.properties. See https://mvnrepository.com/artifact/org.elasticsearch.gradle/build-tools for available build tool versions.

2. Install with ./bin/elasticsearch-plugin

./bin/elasticsearch-plugin install file:///path/to/project/build/distributions/ltr-<LTR-VER>-es<ES-VER>.zip

Open in IntelliJ

The normal gradle idea commands apply to this project, and should work. To open the project in IntelliJ:

./gradlew idea open

Docs Development

Docs are built using Sphinx and written in reStructuredText. After installing sphinx (pip install sphinx) rebuild the docs with:

cd docs
make html

In another tab, you can simply run:

python -m http.server

Visit localhost:8000 and browse to the _build/html directory in your browser to view the built docs.

Docs changes at master will be automatically built and deployed to readthedocs.

Who built this?

• Initially developed at OpenSource Connections.
• Significant contributions by Wikimedia Foundation, Snagajob Engineering, Bonsai, and Yelp Engineering
• Thanks to Jettro Coenradie for porting to ES 6.1

Other Acknowledgments & Stuff To Read

• Bloomberg's Learning to Rank work for Solr
• Our Berlin Buzzwords Talk, We built an Elasticsearch Learning to Rank plugin. Then came the hard part
• Blog article on How is Search Different from Other Machine Learning Problems
• Also check out our other relevance/search thingies: book Relevant Search, projects Elyzer, Splainer, and Quepid