diff --git a/_aggregations/bucket/geohash-grid.md b/_aggregations/bucket/geohash-grid.md index c69d5a5599..13f89799ba 100644 --- a/_aggregations/bucket/geohash-grid.md +++ b/_aggregations/bucket/geohash-grid.md @@ -255,7 +255,7 @@ Geohash grid aggregation requests support the following parameters. Parameter | Data type | Description :--- | :--- | :--- field | String | The field on which aggregation is performed. This field must be mapped as a `geo_point` or `geo_shape` field. If the field contains an array, all array values are aggregated. Required. -precision | Integer | The zoom level used to determine grid cells for bucketing results. Valid values are in the [0, 15] range. Optional. Default is 5. +precision | Integer | The granularity level used to determine grid cells for bucketing results. Cells cannot exceed the specified size (diagonal) of the required precision. Valid values are in the [0, 12] range. Optional. Default is 5. bounds | Object | The bounding box for filtering geopoints and geoshapes. The bounding box is defined by the upper-left and lower-right vertices. Only shapes that intersect with this bounding box or are completely enclosed by this bounding box are included in the aggregation output. The vertices are specified as geopoints in one of the following formats:
- An object with a latitude and longitude
- An array in the [`longitude`, `latitude`] format
- A string in the "`latitude`,`longitude`" format
- A geohash
- WKT
See the [geopoint formats]({{site.url}}{{site.baseurl}}/opensearch/supported-field-types/geo-point#formats) for formatting examples. Optional. size | Integer | The maximum number of buckets to return. When there are more buckets than `size`, OpenSearch returns buckets with more documents. Optional. Default is 10,000. shard_size | Integer | The maximum number of buckets to return from each shard. Optional. Default is max (10, `size` · number of shards), which provides a more accurate count of more highly prioritized buckets. diff --git a/_aggregations/bucket/geohex-grid.md b/_aggregations/bucket/geohex-grid.md index a60be16ef5..de11061055 100644 --- a/_aggregations/bucket/geohex-grid.md +++ b/_aggregations/bucket/geohex-grid.md @@ -387,7 +387,7 @@ Geohex grid aggregation requests support the following parameters. Parameter | Data type | Description :--- | :--- | :--- field | String | The field that contains the geopoints. This field must be mapped as a `geo_point` field. If the field contains an array, all array values are aggregated. Required. -precision | Integer | The zoom level used to determine grid cells for bucketing results. Valid values are in the [0, 15] range. Optional. Default is 5. +precision | Integer | The granularity level used to determine grid cells for bucketing results. Cells cannot exceed the specified size (diagonal) of the required precision. Valid values are in the [0, 15] range. Optional. Default is 5. bounds | Object | The bounding box for filtering geopoints. The bounding box is defined by the upper-left and lower-right vertices. The vertices are specified as geopoints in one of the following formats:
- An object with a latitude and longitude
- An array in the [`longitude`, `latitude`] format
- A string in the "`latitude`,`longitude`" format
- A geohash
- WKT
See the [geopoint formats]({{site.url}}{{site.baseurl}}/opensearch/supported-field-types/geo-point#formats) for formatting examples. Optional. size | Integer | The maximum number of buckets to return. When there are more buckets than `size`, OpenSearch returns buckets with more documents. Optional. Default is 10,000. shard_size | Integer | The maximum number of buckets to return from each shard. Optional. Default is max (10, `size` · number of shards), which provides a more accurate count of more highly prioritized buckets. \ No newline at end of file diff --git a/_aggregations/bucket/geotile-grid.md b/_aggregations/bucket/geotile-grid.md index cb4347288c..dd0c4f8a1f 100644 --- a/_aggregations/bucket/geotile-grid.md +++ b/_aggregations/bucket/geotile-grid.md @@ -544,7 +544,7 @@ Geotile grid aggregation requests support the following parameters. Parameter | Data type | Description :--- | :--- | :--- field | String | The field that contains the geopoints. This field must be mapped as a `geo_point` field. If the field contains an array, all array values are aggregated. Required. -precision | Integer | The zoom level used to determine grid cells for bucketing results. Valid values are in the [0, 15] range. Optional. Default is 5. +precision | Integer | The granularity level used to determine grid cells for bucketing results. Cells cannot exceed the specified size (diagonal) of the required precision. Valid values are in the [0, 29] range. Optional. Default is 7. bounds | Object | The bounding box for filtering geopoints. The bounding box is defined by the upper-left and lower-right vertices. The vertices are specified as geopoints in one of the following formats:
- An object with a latitude and longitude
- An array in the [`longitude`, `latitude`] format
- A string in the "`latitude`,`longitude`" format
- A geohash
- WKT
See the [geopoint formats]({{site.url}}{{site.baseurl}}/opensearch/supported-field-types/geo-point#formats) for formatting examples. Optional. size | Integer | The maximum number of buckets to return. When there are more buckets than `size`, OpenSearch returns buckets with more documents. Optional. Default is 10,000. shard_size | Integer | The maximum number of buckets to return from each shard. Optional. Default is max (10, `size` · number of shards), which provides a more accurate count of more highly prioritized buckets. \ No newline at end of file diff --git a/_clients/javascript/helpers.md b/_clients/javascript/helpers.md index b03af21f94..f88efd8e00 100644 --- a/_clients/javascript/helpers.md +++ b/_clients/javascript/helpers.md @@ -11,7 +11,7 @@ Helper methods simplify the use of complicated API tasks. For the client's compl ## Bulk helper -The bulk helper simplifies making complex bulk API requests. +The bulk helper simplifies making complex bulk API requests. The bulk helper supports operations of the same kind. Alternatively, you can use the `client.bulk` method to perform multiple types of bulk operations. For example, you can send `delete` and `index` operations in one bulk request. For more information, see the [Bulk guide](https://github.com/opensearch-project/opensearch-js/blob/main/guides/bulk.md). ### Usage @@ -199,4 +199,7 @@ client.helpers.bulk({ } }) ``` -{% include copy.html %} \ No newline at end of file +{% include copy.html %} + +## Related articles +https://github.com/opensearch-project/opensearch-js/tree/main/guides diff --git a/_dashboards/discover/index-discover.md b/_dashboards/discover/index-discover.md index bf1e4a739f..4e32c4f185 100644 --- a/_dashboards/discover/index-discover.md +++ b/_dashboards/discover/index-discover.md @@ -1,17 +1,17 @@ --- layout: default -title: Discover +title: Analyzing data nav_order: 20 has_children: true +redirect_from: + - /dashboards/discover/index-discover/ --- -# Discover +# Analyzing data -**Discover** is a tool for exploring your data in OpenSearch Dashboards. You can use **Discover** to visually represent your data on a dashboard and provide a high-level view of key metrics. +To analyze your data in OpenSearch and visualize key metrics, you can use the **Discover** application in OpenSearch Dashboards. An example of data analysis in **Discover** is shown in the following image. -The following image represents a typical **Discover** page using sample data. - -Discover start screen +A Discover default page ## Getting started @@ -19,70 +19,57 @@ In this tutorial, you'll learn about using **Discover** to: - Add data. - Interpret and visualize data. -- Share the data findings. +- Share data findings. - Set alerts. -## Prerequisites - -The following are prerequisites for using **Discover**: - -- Install [OpenSearch Dashboards 2.10 or later](https://opensearch.org/downloads.html). -- Add OpenSearch [sample data]({{site.url}}{{site.baseurl}}/dashboards/quickstart/) or import your own data into OpenSearch. -- Have a foundational understanding of OpenSearch [documents and indexes]({{site.url}}{{site.baseurl}}/im-plugin/index/). - -## Adding data - -Data must be added to OpenSearch before it can be analyzed. In this tutorial, you'll use the sample data. To learn about importing your own data, see [Managing indexes]({{site.url}}{{site.baseurl}}/im-plugin/index/). - -To add the sample data, follow these steps: - -1. On the OpenSearch Dashboards home page, choose **Add sample data**. -2. Choose the desired sample data and select the **Add data** button. A screenshot of the **Add sample data** interface is shown in the following image. - -Add sample data interface +Before getting started, make sure you: +- Install [OpenSearch Dashboards](https://opensearch.org/downloads.html). +- Add sample data or import your own data into OpenSearch. Go to the [OpenSearch Dashboards quickstart guide]({{site.url}}{{site.baseurl}}/dashboards/quickstart/) to learn about adding sample datasets. Go to [Managing indexes]({{site.url}}{{site.baseurl}}/im-plugin/index/) to learn about importing your own data. +- Have a foundational understanding of [OpenSearch documents and indexes]({{site.url}}{{site.baseurl}}/im-plugin/index/). + ## Defining the search To define a search, follow these steps: 1. On the OpenSearch Dashboards navigation menu, select **Discover**. 2. Choose the data you want to work with. In this case, choose `opensearch_dashboards_sample_data_flights` from the upper-left dropdown menu. -3. Select the calendar icon ({::nomarkdown}calendar icon{:/}) to change the time range of your search and then select **Refresh**. +3. Select the {::nomarkdown}calendar icon{:/} icon to change the time range of your search and then select **Refresh**. -You'll see a view similar to the one in the following image. +The resulting view is shown in the following image. Discover interface showing search of flight sample data for Last 7 days -## Adding data fields and viewing data details +## Analyzing document tables -The document table contains document data. Each row represents a single document, and each column contains a different document field representing metrics such as flight destination, average ticket price, and flight delay. You can add, delete, or modify data fields in a document table as needed to meet your data analysis requirements. +In OpenSearch, a document table stores unstructured data. In a document table, each row represents a single document, and each column contains document attributes. -To add or delete fields in a document table, follow these steps: +To examine document attributes, follow these steps: -1. View the data fields listed under **Available fields** and select the plus icon ({::nomarkdown}plus icon{:/}) to add the desired fields to the document table. The field will be automatically added to both **Selected fields** and the document table. For this example, choose the fields `Carrier`, `AvgTicketPrice`, and `Dest`. -2. To arrange or sort the columns, select **Sort fields** > **Pick fields to sort by** and then drag and drop the fields in the order you want them to be ordered. +1. From the data table's left column, choose the {::nomarkdown}inspect icon{:/} icon to open the **Document Details** window. Select the {::nomarkdown}minimize icon{:/} icon to close the **Document Details** window. +2. Examine the metadata. You can switch between the **Table** and **JSON** tabs to view the data in your preferred format. +3. Select **View surrounding documents** to view data for other log entries either preceding or following your current document or select **View single document** to view a particular log entry. -You'll see a view similar to the one in the following image. +The resulting view is shown in the following image. -Discover interface showing adding and sorting data fields +Document attributes -You can view individual or multiple fields in the document table. To gather information about the data in the document table, follow these steps: +To add or delete fields in a document table, follow these steps: -1. From the data table's left-side column, choose the inspect icon ({::nomarkdown}inspect icon{:/}) to open the **Document Details** window. Select the minimize icon ({::nomarkdown}minimize icon{:/}) to close the **Document Details** window. -2. Review the data details. You can switch between the **Table** and **JSON** tabs to view the data in your preferred format. -3. Select **View surrounding documents** to view data for other log entries either preceding or following your current document or select **View single document** to view a particular log entry. +1. View the data fields listed under **Available fields** and select the {::nomarkdown}plus icon{:/} icon to add the desired fields to the document table. The field will be automatically added to both **Selected fields** and the document table. For this example, choose the fields `Carrier`, `AvgTicketPrice`, and `Dest`. +2. Select **Sort fields** > **Pick fields to sort by**. Drag and drop the chosen fields in the desired sort order. -You'll see a view similar to the one in the following image. +The resulting view is shown in the following image. -Document details interface +Adding and deleting data fields -## Searching the data +## Searching data You can use the search toolbar to enter a [DQL]({{site.url}}{{site.baseurl}}/dashboards/discover/dql/) or [query string]({{site.url}}{{site.baseurl}}/query-dsl/full-text/query-string/) query. The search toolbar is best for basic queries; for full query and filter capability, use [query domain-specific language (DSL)]({{site.url}}{{site.baseurl}}/query-dsl/index/) in the [Dev Tools console]({{site.url}}{{site.baseurl}}/dashboards/dev-tools/index-dev/). For more information, see [Discover and Dashboard search toolbar]({{site.url}}{{site.baseurl}}/dashboards/index/#discover-and-dashboard-search-bar). -## Filtering the data +## Filtering data Filters allow you to narrow the results of a query by specifying certain criteria. You can filter by field, value, or range. The **Add filter** pop-up suggests the available fields and operators. @@ -91,33 +78,36 @@ To filter your data, follow these steps: 1. Under the DQL search bar, choose **Add filter**. 2. Select the desired options from the **Field**, **Operator**, and **Value** dropdown lists. For example, select `Cancelled`, `is`, and `true`. 3. Choose **Save**. -4. To remove the filter, choose the cross icon ({::nomarkdown}cross icon{:/}) next to the filter name. -5. Add more filters to further explore the data. +4. To remove a filter, choose the {::nomarkdown}cross icon{:/} icon to the right of the filter name. + +The resulting view is shown in the following image. + +Visualize data findings interface ## Saving a search To save your search, including the query text, filters, and current data view, follow these steps: -1. Select **Save** in the upper-right corner. -2. Give the search a title, and then choose **Save**. -3. Select **Open** to access the saved search. +1. Select **Save** on the upper-right toolbar. +2. Add a title, and then choose **Save**. +3. Select **Open** on the upper-right toolbar to access your saved searches. -## Creating data visualizations through Discover +## Visualizing data findings -To create visualizations of the data findings using the **Discover** app, follow these steps: +To visualize your data findings, follow these steps: -1. Select the inspect icon ({::nomarkdown}inspect icon{:/}) next to the field you want to visualize. +1. Select the {::nomarkdown}inspect icon{:/} icon to the right of the field you want to visualize. - You'll see a view similar to the following image. + The resulting view is shown in the following image. Visualize data findings interface -2. Select the **Visualize** button. The **Visualize** app opens and a visualization is displayed. Learn more about the **Visualize** app and data visualizations in [Building data visualizations]({{site.url}}{{site.baseurl}}/dashboards/visualize/viz-index/). +2. Select the **Visualize** button. When the **Visualize** application is launched, a visualization appears. - You'll see a view similar to the following image. + The resulting view is shown in the following image. Data visualization of flight sample data field destination ## Setting alerts -You can set alerts to notify you when your data changes beyond the thresholds you define. To learn more about using **Discover** to create and manage alerts, see [Alerting dashboards and visualizations]({{site.url}}{{site.baseurl}}/observing-your-data/alerting/dashboards-alerting/). +Set alerts to notify you when your data exceeds your specified thresholds. Go to [Alerting dashboards and visualizations]({{site.url}}{{site.baseurl}}/observing-your-data/alerting/dashboards-alerting/) to learn about creating and managing alerts. diff --git a/_dashboards/discover/time-filter.md b/_dashboards/discover/time-filter.md index 0241c623f5..8fad57cfba 100644 --- a/_dashboards/discover/time-filter.md +++ b/_dashboards/discover/time-filter.md @@ -1,10 +1,11 @@ --- layout: default title: Time filter -parent: Discover +parent: Analyzing data nav_order: 20 redirect_from: - /dashboards/get-started/time-filter/ + -/dashboards/discover/time-filter/ --- # Time filter @@ -16,7 +17,7 @@ The default time range is **Last 15 minutes**. You can change the time range at To change the time range at the dashboard level, follow these steps: -1. From an OpenSearch Dashboards application (Discover, Dashboards, or Visualize), select the calendar icon ({::nomarkdown}calendar icon{:/}) on the right of the search bar. +1. From an OpenSearch Dashboards application (Discover, Dashboards, or Visualize), select the {::nomarkdown}calendar icon{:/} icon on the right of the search bar. 2. Select one of the time filter options, as shown in the following image: - **Quick select:** Choose a time based on the last or next number of seconds, minutes, hours, days, or another time unit. - **Commonly used:** Choose a common time range like **Today**, **Last 7 days**, or **Last 30 days**. diff --git a/_dashboards/quickstart.md b/_dashboards/quickstart.md index 77b601f4d3..eccdeb7d6c 100644 --- a/_dashboards/quickstart.md +++ b/_dashboards/quickstart.md @@ -1,120 +1,118 @@ --- layout: default -title: Quickstart guide +title: OpenSearch Dashboards quickstart guide nav_order: 2 has_children: false redirect_from: - /dashboards/quickstart-dashboards/ --- -# Quickstart guide +# OpenSearch Dashboards quickstart guide -This quickstart guide covers the core concepts that you need to understand to get started with OpenSearch Dashboards. You'll learn how to: +This quickstart guide provides tutorials on using OpenSearch Dashboards applications and tools. You can use these tutorials, either in your own environment or on [OpenSearch Playground](https://playground.opensearch.org/app/home#/), to learn the following fundamental concepts: -- Add sample data. -- Explore and inspect data. -- Visualize data. +- **Adding sample data:** Use preloaded visualizations, dashboards, and other tools to explore OpenSearch Dashboards before adding your own data. +- **Using the Discover application:** Analyze your data to gain insights. +- **Using the Dashboards application:** Create and store data visualizations. +- **Turning dark mode on or off:** Change the Dashboards theme. -Here's a glance at the view you see when you open the **Dashboard** or **Discover** tool. +To dock or undock the navigation pane, select the {::nomarkdown}menu icon{:/} icon and then **Dock navigation** or **Undock navigation**. The OpenSearch Dashboards home page is shown in the following image. -Light and dark mode UI on Discover and Dashboard tools' home page +OpenSearch Dashboards home page {::nomarkdown}alert icon{:/} **Note**
Before you get started, make sure you've installed OpenSearch and OpenSearch Dashboards. For information about installation and configuration, see [Install and configure OpenSearch]({{site.url}}{{site.baseurl}}/install-and-configure/install-opensearch/index/) and [Install and configure OpenSearch Dashboards]({{site.url}}{{site.baseurl}}/install-and-configure/install-dashboards/index/). {: .note} ## Adding sample data -Sample datasets come with visualizations, dashboards, and other tools to help you explore Dashboards before you add your own data. To add sample data, perform the following steps: - -1. Verify access to OpenSearch Dashboards by connecting to [http://localhost:5601](http://localhost:5601) from a browser. The default username and password are `admin`. -1. On the OpenSearch Dashboards **Home** page, choose **Add sample data**. -2. Choose **Add data** to add the datasets, as shown in the following image. +The following tutorials use the [**Sample flight data**](https://playground.opensearch.org/app/home#/tutorial_directory) dataset. +{: .note} - Sample datasets +To add sample data, follow these steps: -## Exploring and inspecting data +1. On the OpenSearch Dashboards **Home** page, choose **Add sample data**. Alternatively, choose **Add data** on the upper-right toolbar. +2. On the **Add sample data** page, choose the dataset(s) you want to add to Dashboards. The following image shows the available sample datasets. -In [**Discover**]({{site.url}}{{site.baseurl}}/dashboards/discover/index-discover/), you can: +Adding sample data window -- Choose data to explore, set a time range for that data, search it using [Dashboards Query Language (DQL)]({{site.url}}{{site.baseurl}}/dashboards/dql/), and filter the results. -- Explore the data, view individual documents, and create tables summarizing the data. -- Visualize your findings. +## Using the Discover application -### Try it: Getting familiar with Discover +With [**Discover**]({{site.url}}{{site.baseurl}}/dashboards/discover/index-discover/), you can: -1. On the OpenSearch Dashboards **Home** page, choose **Discover**. -1. Change the [time filter]({{site.url}}{{site.baseurl}}/dashboards/discover/time-filter/) to **Last 7 days**, as shown in the following image. +- Choose data for analysis, set a time range for that data, search it using [Dashboards Query Language (DQL)]({{site.url}}{{site.baseurl}}/dashboards/dql/), and filter the results. +- Analyze your data by querying and filtering, viewing results in a table, and examining documents. +- Create histograms to display the distribution of your data. - Time filter interface +Follow these steps to use the Discover tool: -2. Search using the DQL query `FlightDelay:true AND DestCountry: US AND FlightDelayMin >= 60` and then choose **Update**. You should see results for US-bound flights delayed by 60 minutes or more, as shown in the following image. - - DQL search field example +1. From the OpenSearch Dashboards navigation menu, choose **Discover**. +2. On the **Discover** page, choose the index pattern `opensearch_dashboards_sample_data_flights` from the dropdown menu on the upper left. +3. Select the {::nomarkdown}calendar icon{:/} icon to change the [time filter]({{site.url}}{{site.baseurl}}/dashboards/discover/time-filter/) from the default of **Last 15 minutes** to **Last 7 days**. +4. In the DQL search bar, enter `FlightDelay:true AND DestCountry: US AND FlightDelayMin >= 60` and select **Update**. Results are shown for US-bound flights delayed by 60 minutes or more. +5. Filter data by selecting **Add filter** from the DQL search bar and then selecting a **Field**, **Operator**, and **Value** from the dropdown lists in the **Edit Filter** pop-up window. For example, select `FlightDelayType`, **is**, and **Weather Delay**. -3. To filter data, choose **Add filter** and then select an **Available field**. For example, select `FlightDelayType`, **is**, and **Weather delay** from the **Field**, **Operator**, and **Value** dropdown lists, as shown in the following image. +The resulting view is shown in the following image. - Filter data by FlightDelayType field +Discover output of steps 1 through 6 -## Visualizing data +## Using the Dashboards application -Raw data can be difficult to comprehend and use. Data visualizations help you prepare and present data in a visual form. In **Dashboard** you can: +With **Dashboards**, you can: - Display data in a single view. - Build dynamic dashboards. - Create and share reports. - Embed analytics to differentiate your applications. -### Try it: Getting familiar with Dashboard +The **Dashboards** application creates and stores visualizations generated from your data. Follow these steps to use the application: -1. On the OpenSearch Dashboards **Home** page, choose **Dashboard**. -1. Choose **[Flights] Global Flight Data** in the **Dashboards** window, as shown in the following image. +1. On the OpenSearch Dashboards **Home** page, choose **Dashboards**. A list of dashboards generated from the sample data appears. +2. In the search toolbar, search for and select **[Flights] Global Flight Dashboard**. You'll see a dashboard preloaded with visualizations, including charts, maps, and data tables. +3. To add other panels to the dashboard, select the **Edit** button and then choose **Add** from the toolbar. The **Add panels** window opens. +4. In the search toolbar in the **Add panels** window, search for and select the existing panel **[Flights] Delay Buckets**. A pop-up message confirms that you've added the panel. +5. Select close `x` to exit the **Add panels** window. +6. The newly added panel is now displayed on the dashboard. - Data visualization dashboard +The resulting view is shown in the following image. -1. To add panels to the dashboard, choose **Edit** and then **Add** from the toolbar. -1. In the **Add panels** window, choose the existing panel **[Flights] Delay Buckets**. You'll see a pop-up window on the lower right confirming that you've added the panel. -1. Select `x` to close the **Add panels** window. -1. View the added panel **[Flights] Delay Buckets**, which is added as the last panel on the dashboard, as shown in the following image. +Add panel tutorial screen view - Add panel to dashboard - -### Try it: Creating a visualization panel +For information about using a specific data visualization type, such as VisBuilder, go to [Building data visualizations]({{site.url}}{{site.baseurl}}/dashboards/visualize/viz-index/). For information about using dashboards and visualizations in **Observability**, go to [Observability]({{site.url}}{{site.baseurl}}/observing-your-data/). +{: .note} -Continuing with the preceding dashboard, you'll create a bar chart comparing the number of canceled flights and delayed flights to delay type and then add the panel to the dashboard: +### Interacting with data using dashboards -1. Change the default [time range]({{site.url}}{{site.baseurl}}/dashboards/discover/time-filter/) from **24 hours** to **Last 7 days**. -1. In the toolbar, choose **Edit**, then **Create new**. -1. Select **VisBuilder** in the **New Visualizations** window. -1. In the **Data Source** dropdown list, choose `opensearch_dashboards_sample_data_flights`. -1. Drag the fields **Cancelled** and **FlightDelay** to the y-axis column. -1. Drag the field **FlightDelayType** to the x-axis column. -1. Choose **Save** and name the visualization in the **Title** field. -2. Choose **Save and return**. The following bar chart is added as the last panel on the dashboard, as shown in the following image. +Interactive dashboards allow you to analyze data in more depth and filter it in several ways. With **Dashboards**, you can use dashboard-level filters to directly interact with data. -Creating a visualization panel +Using the **[Flights] Global Flight Dashboard** dashboard, follow these steps to further analyze and filter the sample flight data: -## Interacting with data +1. On the **[Flights] Airline Carrier** panel, choose **OpenSearch-Air**. The dashboard updates automatically. +2. Choose **Save** to save the dashboard. -Interactive dashboards allow you analyze data in more depth and filter it in several ways. In Dashboards, you can interact directly with data on a dashboard by using dashboard-level filters. For example, continuing with the preceding dashboard, you can filter to show delays and cancellations for a specific airline. +Alternatively, you can use the dashboard toolbar to apply filters by following these steps: -### Try it: Interacting with the sample flight data +1. In the dashboard toolbar, choose **Add filter**. +2. From the **Field**, **Operator**, and **Value** dropdown lists, choose **Carrier**, **is**, and **OpenSearch-Air**, respectively, as shown in the following image. +3. Choose **Save**. The dashboard updates automatically. -1. On the **[Flights] Airline Carrier** panel, choose **OpenSearch-Air**. The dashboard updates automatically. -1. Choose **Save** to save the customized dashboard. +The resulting view is shown in the following image. -Alternatively, you can apply filters using the dashboard toolbar: +Screenshot of Dashboard tutorial panel view -1. In the dashboard toolbar, choose **Add filter**. -1. From the **Field**, **Operator**, and **Value** dropdown lists, choose **Carrier**, **is**, and **OpenSearch-Air**, respectively, as shown in the following image. +## Turning dark mode on or off - Edit field interface +Changing the Dashboards theme requires admin permissions. If you are an admin, follow these steps: -1. Choose **Save**. The dashboard updates automatically, and the result is the dashboard shown in the following image. +1. Navigate to **Management** > **Dashboards Management** > **Advanced Settings**. +2. Scroll down to the **Appearance** section and locate the **Dark mode** option. +3. Use the toggle switch to turn dark mode on or off for all users of your Dashboards instance, as shown in the image following these steps. +4. Select the **Save changes** button and then the **Reload** button. The updated theme is applied immediately. - Dashboard view after applying Carrier filter +Dark mode view ## Next steps -- **Visualize data**. To learn more about data visualizations in OpenSearch Dashboards, see [**Building data visualizations**]({{site.url}}{{site.baseurl}}/dashboards/visualize/viz-index/). -- **Create dashboards**. To learn more about creating dashboards in OpenSearch Dashboards, see [**Creating dashboards**]({{site.url}}{{site.baseurl}}/dashboards/quickstart-dashboards/). -- **Explore data**. To learn more about exploring data in OpenSearch Dashboards, see [**Exploring data**]({{site.url}}{{site.baseurl}}/dashboards/discover/index-discover/). \ No newline at end of file +- Go to [Building data visualizations]({{site.url}}{{site.baseurl}}/dashboards/visualize/viz-index/) to learn more about Dashboards data visualizations. +- Go to [Creating dashboards]({{site.url}}{{site.baseurl}}/dashboards/quickstart-dashboards/) to learn more about creating dashboards. +- Go to [Analyzing data]({{site.url}}{{site.baseurl}}/dashboards/discover/index-discover/) to learn more about using Dashboards to analyze data. +- Go to [Ingest APIs]({{site.url}}{{site.baseurl}}/api-reference/ingest-apis/index/) and [Ingest pipelines]({{site.url}}{{site.baseurl}}/ingest-pipelines/) to learn more about using OpenSearch for data ingestion. diff --git a/_ml-commons-plugin/api/model-apis/register-model.md b/_ml-commons-plugin/api/model-apis/register-model.md index ed36db1c1f..c10292dba8 100644 --- a/_ml-commons-plugin/api/model-apis/register-model.md +++ b/_ml-commons-plugin/api/model-apis/register-model.md @@ -85,7 +85,7 @@ Field | Data type | Required/Optional | Description `name`| String | Required | The model name. | `version` | String | Required | The model version. | `model_format` | String | Required | The portable format of the model file. Valid values are `TORCH_SCRIPT` and `ONNX`. | -`function_name` | String | Required | Set this parameter to `SPARSE_ENCODING` or `SPARSE_TOKENIZE`. +`function_name` | String | Required | For text embedding models, set this parameter to `TEXT_EMBEDDING`. For sparse encoding models, set this parameter to `SPARSE_ENCODING` or `SPARSE_TOKENIZE`. For cross-encoder models, set this parameter to `TEXT_SIMILARITY`. `model_content_hash_value` | String | Required | The model content hash generated using the SHA-256 hashing algorithm. `url` | String | Required | The URL that contains the model. | `description` | String | Optional| The model description. | diff --git a/_ml-commons-plugin/custom-local-models.md b/_ml-commons-plugin/custom-local-models.md index d701dad3ff..f96f784196 100644 --- a/_ml-commons-plugin/custom-local-models.md +++ b/_ml-commons-plugin/custom-local-models.md @@ -157,7 +157,7 @@ POST /_plugins/_ml/models/_register ``` {% include copy.html %} -For a description of Register API parameters, see [Register a model]({{site.url}}{{site.baseurl}}/ml-commons-plugin/api/model-apis/register-model/). +For descriptions of Register API parameters, see [Register a model]({{site.url}}{{site.baseurl}}/ml-commons-plugin/api/model-apis/register-model/). The `model_task_type` corresponds to the model type. For text embedding models, set this parameter to `TEXT_EMBEDDING`. For sparse encoding models, set this parameter to `SPARSE_ENCODING` or `SPARSE_TOKENIZE`. For cross-encoder models, set this parameter to `TEXT_SIMILARITY`. OpenSearch returns the task ID of the register operation: @@ -321,148 +321,3 @@ The response contains the tokens and weights: ## Step 5: Use the model for search To learn how to use the model for vector search, see [Using an ML model for neural search]({{site.url}}{{site.baseurl}}/search-plugins/neural-search/#using-an-ml-model-for-neural-search). - -## Cross-encoder models - -Cross-encoder models support query reranking. - -To register a cross-encoder model, send a request in the following format. The `model_config` object is optional. For cross-encoder models, specify the `function_name` as `TEXT_SIMILARITY`. For example, the following request registers an `ms-marco-TinyBERT-L-2-v2` model: - -```json -POST /_plugins/_ml/models/_register -{ - "name": "ms-marco-TinyBERT-L-2-v2", - "version": "1.0.0", - "function_name": "TEXT_SIMILARITY", - "description": "test model", - "model_format": "TORCH_SCRIPT", - "model_group_id": "lN4AP40BKolAMNtR4KJ5", - "model_content_hash_value": "90e39a926101d1a4e542aade0794319404689b12acfd5d7e65c03d91c668b5cf", - "model_config": { - "model_type": "bert", - "embedding_dimension": 1, - "framework_type": "huggingface_transformers", - "total_chunks":2, - "all_config": "{\"total_chunks\":2}" - }, - "url": "https://github.com/opensearch-project/ml-commons/blob/main/ml-algorithms/src/test/resources/org/opensearch/ml/engine/algorithms/text_similarity/TinyBERT-CE-torch_script.zip?raw=true" -} -``` -{% include copy-curl.html %} - -Then send a request to deploy the model: - -```json -POST _plugins/_ml/models//_deploy -``` -{% include copy-curl.html %} - -To test a cross-encoder model, send the following request: - -```json -POST _plugins/_ml/models//_predict -{ - "query_text": "today is sunny", - "text_docs": [ - "how are you", - "today is sunny", - "today is july fifth", - "it is winter" - ] -} -``` -{% include copy-curl.html %} - -The model calculates the similarity score of `query_text` and each document in `text_docs` and returns a list of scores for each document in the order they were provided in `text_docs`: - -```json -{ - "inference_results": [ - { - "output": [ - { - "name": "similarity", - "data_type": "FLOAT32", - "shape": [ - 1 - ], - "data": [ - -6.077798 - ], - "byte_buffer": { - "array": "Un3CwA==", - "order": "LITTLE_ENDIAN" - } - } - ] - }, - { - "output": [ - { - "name": "similarity", - "data_type": "FLOAT32", - "shape": [ - 1 - ], - "data": [ - 10.223609 - ], - "byte_buffer": { - "array": "55MjQQ==", - "order": "LITTLE_ENDIAN" - } - } - ] - }, - { - "output": [ - { - "name": "similarity", - "data_type": "FLOAT32", - "shape": [ - 1 - ], - "data": [ - -1.3987057 - ], - "byte_buffer": { - "array": "ygizvw==", - "order": "LITTLE_ENDIAN" - } - } - ] - }, - { - "output": [ - { - "name": "similarity", - "data_type": "FLOAT32", - "shape": [ - 1 - ], - "data": [ - -4.5923924 - ], - "byte_buffer": { - "array": "4fSSwA==", - "order": "LITTLE_ENDIAN" - } - } - ] - } - ] -} -``` - -A higher document score means higher similarity. In the preceding response, documents are scored as follows against the query text `today is sunny`: - -Document text | Score -:--- | :--- -`how are you` | -6.077798 -`today is sunny` | 10.223609 -`today is july fifth` | -1.3987057 -`it is winter` | -4.5923924 - -The document that contains the same text as the query is scored the highest, and the remaining documents are scored based on the text similarity. - -To learn how to use the model for reranking, see [Reranking search results]({{site.url}}{{site.baseurl}}/search-plugins/search-relevance/reranking-search-results/). diff --git a/_ml-commons-plugin/pretrained-models.md b/_ml-commons-plugin/pretrained-models.md index cbcb11bf33..2563c6507c 100644 --- a/_ml-commons-plugin/pretrained-models.md +++ b/_ml-commons-plugin/pretrained-models.md @@ -12,17 +12,72 @@ nav_order: 120 OpenSearch provides a variety of open-source pretrained models that can assist with a range of machine learning (ML) search and analytics use cases. You can upload any supported model to the OpenSearch cluster and use it locally. -## Prerequisites +## Supported pretrained models + +OpenSearch supports the following models, categorized by type. Text embedding models are sourced from [Hugging Face](https://huggingface.co/). Sparse encoding models are trained by OpenSearch. Although models with the same type will have similar use cases, each model has a different model size and will perform differently depending on your cluster setup. For a performance comparison of some pretrained models, see the [SBERT documentation](https://www.sbert.net/docs/pretrained_models.html#model-overview). + +Running local models on the CentOS 7 operating system is not supported. Moreover, not all local models can run on all hardware and operating systems. +{: .important} + +### Sentence transformers + +Sentence transformer models map sentences and paragraphs across a dimensional dense vector space. The number of vectors depends on the type of model. You can use these models for use cases such as clustering or semantic search. + +The following table provides a list of sentence transformer models and artifact links you can use to download them. Note that you must prefix the model name with `huggingface/`, as shown in the **Model name** column. + +| Model name | Version | Vector dimensions | Auto-truncation | TorchScript artifact | ONNX artifact | +|:---|:---|:---|:---|:---|:---| +| `huggingface/sentence-transformers/all-distilroberta-v1` | 1.0.1 | 768-dimensional dense vector space. | Yes | - [model_url](https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/all-distilroberta-v1/1.0.1/torch_script/sentence-transformers_all-distilroberta-v1-1.0.1-torch_script.zip)
- [config_url](https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/all-distilroberta-v1/1.0.1/torch_script/config.json) | - [model_url](https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/all-distilroberta-v1/1.0.1/onnx/sentence-transformers_all-distilroberta-v1-1.0.1-onnx.zip)
- [config_url](https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/all-distilroberta-v1/1.0.1/onnx/config.json) | +| `huggingface/sentence-transformers/all-MiniLM-L6-v2` | 1.0.1 | 384-dimensional dense vector space. | Yes | - [model_url](https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/all-MiniLM-L6-v2/1.0.1/torch_script/sentence-transformers_all-MiniLM-L6-v2-1.0.1-torch_script.zip)
- [config_url](https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/all-MiniLM-L6-v2/1.0.1/torch_script/config.json) | - [model_url](https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/all-MiniLM-L6-v2/1.0.1/onnx/sentence-transformers_all-MiniLM-L6-v2-1.0.1-onnx.zip)
- [config_url](https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/all-MiniLM-L6-v2/1.0.1/onnx/config.json) | +| `huggingface/sentence-transformers/all-MiniLM-L12-v2` | 1.0.1 | 384-dimensional dense vector space. | Yes | - [model_url](https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/all-MiniLM-L12-v2/1.0.1/torch_script/sentence-transformers_all-MiniLM-L12-v2-1.0.1-torch_script.zip)
- [config_url](https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/all-MiniLM-L12-v2/1.0.1/torch_script/config.json) | - [model_url](https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/all-MiniLM-L12-v2/1.0.1/onnx/sentence-transformers_all-MiniLM-L12-v2-1.0.1-onnx.zip)
- [config_url](https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/all-MiniLM-L12-v2/1.0.1/onnx/config.json) | +| `huggingface/sentence-transformers/all-mpnet-base-v2` | 1.0.1 | 768-dimensional dense vector space. | Yes | - [model_url](https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/all-mpnet-base-v2/1.0.1/torch_script/sentence-transformers_all-mpnet-base-v2-1.0.1-torch_script.zip)
- [config_url](https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/all-mpnet-base-v2/1.0.1/torch_script/config.json) | - [model_url](https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/all-mpnet-base-v2/1.0.1/onnx/sentence-transformers_all-mpnet-base-v2-1.0.1-onnx.zip)
- [config_url](https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/all-mpnet-base-v2/1.0.1/onnx/config.json) | +| `huggingface/sentence-transformers/msmarco-distilbert-base-tas-b` | 1.0.2 | 768-dimensional dense vector space. Optimized for semantic search. | Yes | - [model_url](https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/msmarco-distilbert-base-tas-b/1.0.2/torch_script/sentence-transformers_msmarco-distilbert-base-tas-b-1.0.2-torch_script.zip)
- [config_url](https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/msmarco-distilbert-base-tas-b/1.0.2/torch_script/config.json) | - [model_url](https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/msmarco-distilbert-base-tas-b/1.0.2/onnx/sentence-transformers_msmarco-distilbert-base-tas-b-1.0.2-onnx.zip)
- [config_url](https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/msmarco-distilbert-base-tas-b/1.0.2/onnx/config.json) | +| `huggingface/sentence-transformers/multi-qa-MiniLM-L6-cos-v1` | 1.0.1 | 384-dimensional dense vector space. Designed for semantic search and trained on 215 million question/answer pairs. | Yes | - [model_url](https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/multi-qa-MiniLM-L6-cos-v1/1.0.1/torch_script/sentence-transformers_multi-qa-MiniLM-L6-cos-v1-1.0.1-torch_script.zip)
- [config_url](https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/multi-qa-MiniLM-L6-cos-v1/1.0.1/torch_script/config.json) | - [model_url](https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/multi-qa-MiniLM-L6-cos-v1/1.0.1/onnx/sentence-transformers_multi-qa-MiniLM-L6-cos-v1-1.0.1-onnx.zip)
- [config_url](https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/multi-qa-MiniLM-L6-cos-v1/1.0.1/onnx/config.json) | +| `huggingface/sentence-transformers/multi-qa-mpnet-base-dot-v1` | 1.0.1 | 384-dimensional dense vector space. | Yes | - [model_url](https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/multi-qa-mpnet-base-dot-v1/1.0.1/torch_script/sentence-transformers_multi-qa-mpnet-base-dot-v1-1.0.1-torch_script.zip)
- [config_url](https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/multi-qa-mpnet-base-dot-v1/1.0.1/torch_script/config.json) | - [model_url](https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/multi-qa-mpnet-base-dot-v1/1.0.1/onnx/sentence-transformers_multi-qa-mpnet-base-dot-v1-1.0.1-onnx.zip)
- [config_url](https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/multi-qa-mpnet-base-dot-v1/1.0.1/onnx/config.json) | +| `huggingface/sentence-transformers/paraphrase-MiniLM-L3-v2` | 1.0.1 | 384-dimensional dense vector space. | Yes | - [model_url](https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/paraphrase-MiniLM-L3-v2/1.0.1/torch_script/sentence-transformers_paraphrase-MiniLM-L3-v2-1.0.1-torch_script.zip)
- [config_url](https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/paraphrase-MiniLM-L3-v2/1.0.1/torch_script/config.json) | - [model_url](https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/paraphrase-MiniLM-L3-v2/1.0.1/onnx/sentence-transformers_paraphrase-MiniLM-L3-v2-1.0.1-onnx.zip)
- [config_url](https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/paraphrase-MiniLM-L3-v2/1.0.1/onnx/config.json) | +| `huggingface/sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2` | 1.0.1 | 384-dimensional dense vector space. | Yes | - [model_url](https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2/1.0.1/torch_script/sentence-transformers_paraphrase-multilingual-MiniLM-L12-v2-1.0.1-torch_script.zip)
- [config_url](https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2/1.0.1/torch_script/config.json) | - [model_url](https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2/1.0.1/onnx/sentence-transformers_paraphrase-multilingual-MiniLM-L12-v2-1.0.1-onnx.zip)
- [config_url](https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2/1.0.1/onnx/config.json) | +| `huggingface/sentence-transformers/paraphrase-mpnet-base-v2` | 1.0.0 | 768-dimensional dense vector space. | Yes | - [model_url](https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/paraphrase-mpnet-base-v2/1.0.0/torch_script/sentence-transformers_paraphrase-mpnet-base-v2-1.0.0-torch_script.zip)
- [config_url](https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/paraphrase-mpnet-base-v2/1.0.0/torch_script/config.json) | - [model_url](https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/paraphrase-mpnet-base-v2/1.0.0/onnx/sentence-transformers_paraphrase-mpnet-base-v2-1.0.0-onnx.zip)
- [config_url](https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/paraphrase-mpnet-base-v2/1.0.0/onnx/config.json) | +| `huggingface/sentence-transformers/distiluse-base-multilingual-cased-v1` | 1.0.1 | 512-dimensional dense vector space. | Yes | - [model_url](https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/distiluse-base-multilingual-cased-v1/1.0.1/torch_script/sentence-transformers_distiluse-base-multilingual-cased-v1-1.0.1-torch_script.zip)
- [config_url](https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/distiluse-base-multilingual-cased-v1/1.0.1/torch_script/config.json) | Not available | + + +### Sparse encoding models +**Introduced 2.11** +{: .label .label-purple } + +Sparse encoding models transfer text into a sparse vector and convert the vector to a list of `` pairs representing the text entry and its corresponding weight in the sparse vector. You can use these models for use cases such as clustering or sparse neural search. + +We recommend the following models for optimal performance: + +- Use the `amazon/neural-sparse/opensearch-neural-sparse-encoding-v1` model during both ingestion and search. +- Use the `amazon/neural-sparse/opensearch-neural-sparse-encoding-doc-v1` model during ingestion and the +`amazon/neural-sparse/opensearch-neural-sparse-tokenizer-v1` model during search. + +The following table provides a list of sparse encoding models and artifact links you can use to download them. + +| Model name | Version | Auto-truncation | TorchScript artifact | Description | +|:---|:---|:---|:---|:---| +| `amazon/neural-sparse/opensearch-neural-sparse-encoding-v1` | 1.0.1 | Yes | - [model_url](https://artifacts.opensearch.org/models/ml-models/amazon/neural-sparse/opensearch-neural-sparse-encoding-v1/1.0.1/torch_script/neural-sparse_opensearch-neural-sparse-encoding-v1-1.0.1-torch_script.zip)
- [config_url](https://artifacts.opensearch.org/models/ml-models/amazon/neural-sparse/opensearch-neural-sparse-encoding-v1/1.0.1/torch_script/config.json) | A neural sparse encoding model. The model transforms text into a sparse vector, identifies the indexes of non-zero elements in the vector, and then converts the vector into `` pairs, where each entry corresponds to a non-zero element index. | +| `amazon/neural-sparse/opensearch-neural-sparse-encoding-doc-v1` | 1.0.1 | Yes | - [model_url](https://artifacts.opensearch.org/models/ml-models/amazon/neural-sparse/opensearch-neural-sparse-encoding-doc-v1/1.0.1/torch_script/neural-sparse_opensearch-neural-sparse-encoding-doc-v1-1.0.1-torch_script.zip)
- [config_url](https://artifacts.opensearch.org/models/ml-models/amazon/neural-sparse/opensearch-neural-sparse-encoding-doc-v1/1.0.1/torch_script/config.json) | A neural sparse encoding model. The model transforms text into a sparse vector, identifies the indexes of non-zero elements in the vector, and then converts the vector into `` pairs, where each entry corresponds to a non-zero element index. | +| `amazon/neural-sparse/opensearch-neural-sparse-tokenizer-v1` | 1.0.1 | Yes | - [model_url](https://artifacts.opensearch.org/models/ml-models/amazon/neural-sparse/opensearch-neural-sparse-tokenizer-v1/1.0.1/torch_script/neural-sparse_opensearch-neural-sparse-tokenizer-v1-1.0.1-torch_script.zip)
- [config_url](https://artifacts.opensearch.org/models/ml-models/amazon/neural-sparse/opensearch-neural-sparse-tokenizer-v1/1.0.1/torch_script/config.json) | A neural sparse tokenizer model. The model tokenizes text into tokens and assigns each token a predefined weight, which is the token's inverse document frequency (IDF). If the IDF file is not provided, the weight defaults to 1. For more information, see [Preparing a model]({{site.url}}{{site.baseurl}}/ml-commons-plugin/custom-local-models/#preparing-a-model). | + +### Cross-encoder models +**Introduced 2.12** +{: .label .label-purple } + +Cross-encoder models support query reranking. -To get started, select one of the [supported pretrained models](#supported-pretrained-models). +The following table provides a list of cross-encoder models and artifact links you can use to download them. -### Cluster settings +| Model name | Version | TorchScript artifact | ONNX artifact | +|:---|:---|:---|:---| +| `ms-marco-MiniLM-L-6-v2` | 1.0.2 | - [model_url](https://artifacts.opensearch.org/models/ml-models/huggingface/cross-encoders/ms-marco-MiniLM-L-6-v2/1.0.2/torch_script/cross-encoders_ms-marco-MiniLM-L-6-v2-1.0.2-torch_script.zip)
- [config_url](https://artifacts.opensearch.org/models/ml-models/huggingface/cross-encoders/ms-marco-MiniLM-L-6-v2/1.0.2/torch_script/config.json) | - [model_url](https://artifacts.opensearch.org/models/ml-models/huggingface/cross-encoders/ms-marco-MiniLM-L-6-v2/1.0.2/onnx/cross-encoders_ms-marco-MiniLM-L-6-v2-1.0.2-onnx.zip)
- [config_url](https://artifacts.opensearch.org/models/ml-models/huggingface/cross-encoders/ms-marco-MiniLM-L-6-v2/1.0.2/onnx/config.json) | +| `ms-marco-MiniLM-L-12-v2` | 1.0.2 | - [model_url](https://artifacts.opensearch.org/models/ml-models/huggingface/cross-encoders/ms-marco-MiniLM-L-12-v2/1.0.2/torch_script/cross-encoders_ms-marco-MiniLM-L-12-v2-1.0.2-torch_script.zip)
- [config_url](https://artifacts.opensearch.org/models/ml-models/huggingface/cross-encoders/ms-marco-MiniLM-L-12-v2/1.0.2/torch_script/config.json) | - [model_url](https://artifacts.opensearch.org/models/ml-models/huggingface/cross-encoders/ms-marco-MiniLM-L-12-v2/1.0.2/onnx/cross-encoders_ms-marco-MiniLM-L-12-v2-1.0.2-onnx.zip)
- [config_url](https://artifacts.opensearch.org/models/ml-models/huggingface/cross-encoders/ms-marco-MiniLM-L-12-v2/1.0.2/onnx/config.json) -This example uses a simple setup with no dedicated ML nodes and allows running a model on a non-ML node. +## Prerequisites -On clusters with dedicated ML nodes, specify `"only_run_on_ml_node": "true"` for improved performance. For more information, see [ML Commons cluster settings]({{site.url}}{{site.baseurl}}/ml-commons-plugin/cluster-settings/). +On clusters with dedicated ML nodes, specify `"only_run_on_ml_node": "true"` for improved performance. For more information, see [ML Commons cluster settings]({{site.url}}{{site.baseurl}}/ml-commons-plugin/cluster-settings/). -To ensure that this basic local setup works, specify the following cluster settings: +This example uses a simple setup with no dedicated ML nodes and allows running a model on a non-ML node. To ensure that this basic local setup works, specify the following cluster settings: ```json PUT _cluster/settings @@ -172,6 +227,8 @@ If a cluster or node is restarted, then you need to redeploy the model. To learn Use the [Predict API]({{site.url}}{{site.baseurl}}/ml-commons-plugin/api/train-predict/predict/) to test the model. +### Text embedding model + For a text embedding model, send the following request: ```json @@ -210,6 +267,8 @@ The response contains text embeddings for the provided sentence: } ``` +### Sparse encoding model + For a sparse encoding model, send the following request: ```json @@ -245,57 +304,121 @@ The response contains the tokens and weights: } ``` -## Step 5: Use the model for search +### Cross-encoder model -To learn how to set up a vector index and use text embedding models for search, see [Semantic search]({{site.url}}{{site.baseurl}}/search-plugins/semantic-search/). +For a cross-encoder model, send the following request: -To learn how to set up a vector index and use sparse encoding models for search, see [Neural sparse search]({{site.url}}{{site.baseurl}}/search-plugins/neural-sparse-search/). - - -## Supported pretrained models - -OpenSearch supports the following models, categorized by type. Text embedding models are sourced from [Hugging Face](https://huggingface.co/). Sparse encoding models are trained by OpenSearch. Although models with the same type will have similar use cases, each model has a different model size and will perform differently depending on your cluster setup. For a performance comparison of some pretrained models, see the [SBERT documentation](https://www.sbert.net/docs/pretrained_models.html#model-overview). - -Running local models on the CentOS 7 operating system is not supported. Moreover, not all local models can run on all hardware and operating systems. -{: .important} - -### Sentence transformers +```json +POST _plugins/_ml/models//_predict +{ + "query_text": "today is sunny", + "text_docs": [ + "how are you", + "today is sunny", + "today is july fifth", + "it is winter" + ] +} +``` +{% include copy-curl.html %} -Sentence transformer models map sentences and paragraphs across a dimensional dense vector space. The number of vectors depends on the type of model. You can use these models for use cases such as clustering or semantic search. +The model calculates the similarity score of `query_text` and each document in `text_docs` and returns a list of scores for each document in the order they were provided in `text_docs`: -The following table provides a list of sentence transformer models and artifact links you can use to download them. Note that you must prefix the model name with `huggingface/`, as shown in the **Model name** column. +```json +{ + "inference_results": [ + { + "output": [ + { + "name": "similarity", + "data_type": "FLOAT32", + "shape": [ + 1 + ], + "data": [ + -6.077798 + ], + "byte_buffer": { + "array": "Un3CwA==", + "order": "LITTLE_ENDIAN" + } + } + ] + }, + { + "output": [ + { + "name": "similarity", + "data_type": "FLOAT32", + "shape": [ + 1 + ], + "data": [ + 10.223609 + ], + "byte_buffer": { + "array": "55MjQQ==", + "order": "LITTLE_ENDIAN" + } + } + ] + }, + { + "output": [ + { + "name": "similarity", + "data_type": "FLOAT32", + "shape": [ + 1 + ], + "data": [ + -1.3987057 + ], + "byte_buffer": { + "array": "ygizvw==", + "order": "LITTLE_ENDIAN" + } + } + ] + }, + { + "output": [ + { + "name": "similarity", + "data_type": "FLOAT32", + "shape": [ + 1 + ], + "data": [ + -4.5923924 + ], + "byte_buffer": { + "array": "4fSSwA==", + "order": "LITTLE_ENDIAN" + } + } + ] + } + ] +} +``` -| Model name | Version | Vector dimensions | Auto-truncation | TorchScript artifact | ONNX artifact | -|:---|:---|:---|:---|:---| -| `huggingface/sentence-transformers/all-distilroberta-v1` | 1.0.1 | 768-dimensional dense vector space. | Yes | - [model_url](https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/all-distilroberta-v1/1.0.1/torch_script/sentence-transformers_all-distilroberta-v1-1.0.1-torch_script.zip)
- [config_url](https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/all-distilroberta-v1/1.0.1/torch_script/config.json) | - [model_url](https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/all-distilroberta-v1/1.0.1/onnx/sentence-transformers_all-distilroberta-v1-1.0.1-onnx.zip)
- [config_url](https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/all-distilroberta-v1/1.0.1/onnx/config.json) | -| `huggingface/sentence-transformers/all-MiniLM-L6-v2` | 1.0.1 | 384-dimensional dense vector space. | Yes | - [model_url](https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/all-MiniLM-L6-v2/1.0.1/torch_script/sentence-transformers_all-MiniLM-L6-v2-1.0.1-torch_script.zip)
- [config_url](https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/all-MiniLM-L6-v2/1.0.1/torch_script/config.json) | - [model_url](https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/all-MiniLM-L6-v2/1.0.1/onnx/sentence-transformers_all-MiniLM-L6-v2-1.0.1-onnx.zip)
- [config_url](https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/all-MiniLM-L6-v2/1.0.1/onnx/config.json) | -| `huggingface/sentence-transformers/all-MiniLM-L12-v2` | 1.0.1 | 384-dimensional dense vector space. | Yes | - [model_url](https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/all-MiniLM-L12-v2/1.0.1/torch_script/sentence-transformers_all-MiniLM-L12-v2-1.0.1-torch_script.zip)
- [config_url](https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/all-MiniLM-L12-v2/1.0.1/torch_script/config.json) | - [model_url](https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/all-MiniLM-L12-v2/1.0.1/onnx/sentence-transformers_all-MiniLM-L12-v2-1.0.1-onnx.zip)
- [config_url](https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/all-MiniLM-L12-v2/1.0.1/onnx/config.json) | -| `huggingface/sentence-transformers/all-mpnet-base-v2` | 1.0.1 | 768-dimensional dense vector space. | Yes | - [model_url](https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/all-mpnet-base-v2/1.0.1/torch_script/sentence-transformers_all-mpnet-base-v2-1.0.1-torch_script.zip)
- [config_url](https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/all-mpnet-base-v2/1.0.1/torch_script/config.json) | - [model_url](https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/all-mpnet-base-v2/1.0.1/onnx/sentence-transformers_all-mpnet-base-v2-1.0.1-onnx.zip)
- [config_url](https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/all-mpnet-base-v2/1.0.1/onnx/config.json) | -| `huggingface/sentence-transformers/msmarco-distilbert-base-tas-b` | 1.0.2 | 768-dimensional dense vector space. Optimized for semantic search. | Yes | - [model_url](https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/msmarco-distilbert-base-tas-b/1.0.2/torch_script/sentence-transformers_msmarco-distilbert-base-tas-b-1.0.2-torch_script.zip)
- [config_url](https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/msmarco-distilbert-base-tas-b/1.0.2/torch_script/config.json) | - [model_url](https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/msmarco-distilbert-base-tas-b/1.0.2/onnx/sentence-transformers_msmarco-distilbert-base-tas-b-1.0.2-onnx.zip)
- [config_url](https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/msmarco-distilbert-base-tas-b/1.0.2/onnx/config.json) | -| `huggingface/sentence-transformers/multi-qa-MiniLM-L6-cos-v1` | 1.0.1 | 384-dimensional dense vector space. Designed for semantic search and trained on 215 million question/answer pairs. | Yes | - [model_url](https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/multi-qa-MiniLM-L6-cos-v1/1.0.1/torch_script/sentence-transformers_multi-qa-MiniLM-L6-cos-v1-1.0.1-torch_script.zip)
- [config_url](https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/multi-qa-MiniLM-L6-cos-v1/1.0.1/torch_script/config.json) | - [model_url](https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/multi-qa-MiniLM-L6-cos-v1/1.0.1/onnx/sentence-transformers_multi-qa-MiniLM-L6-cos-v1-1.0.1-onnx.zip)
- [config_url](https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/multi-qa-MiniLM-L6-cos-v1/1.0.1/onnx/config.json) | -| `huggingface/sentence-transformers/multi-qa-mpnet-base-dot-v1` | 1.0.1 | 384-dimensional dense vector space. | Yes | - [model_url](https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/multi-qa-mpnet-base-dot-v1/1.0.1/torch_script/sentence-transformers_multi-qa-mpnet-base-dot-v1-1.0.1-torch_script.zip)
- [config_url](https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/multi-qa-mpnet-base-dot-v1/1.0.1/torch_script/config.json) | - [model_url](https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/multi-qa-mpnet-base-dot-v1/1.0.1/onnx/sentence-transformers_multi-qa-mpnet-base-dot-v1-1.0.1-onnx.zip)
- [config_url](https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/multi-qa-mpnet-base-dot-v1/1.0.1/onnx/config.json) | -| `huggingface/sentence-transformers/paraphrase-MiniLM-L3-v2` | 1.0.1 | 384-dimensional dense vector space. | Yes | - [model_url](https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/paraphrase-MiniLM-L3-v2/1.0.1/torch_script/sentence-transformers_paraphrase-MiniLM-L3-v2-1.0.1-torch_script.zip)
- [config_url](https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/paraphrase-MiniLM-L3-v2/1.0.1/torch_script/config.json) | - [model_url](https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/paraphrase-MiniLM-L3-v2/1.0.1/onnx/sentence-transformers_paraphrase-MiniLM-L3-v2-1.0.1-onnx.zip)
- [config_url](https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/paraphrase-MiniLM-L3-v2/1.0.1/onnx/config.json) | -| `huggingface/sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2` | 1.0.1 | 384-dimensional dense vector space. | Yes | - [model_url](https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2/1.0.1/torch_script/sentence-transformers_paraphrase-multilingual-MiniLM-L12-v2-1.0.1-torch_script.zip)
- [config_url](https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2/1.0.1/torch_script/config.json) | - [model_url](https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2/1.0.1/onnx/sentence-transformers_paraphrase-multilingual-MiniLM-L12-v2-1.0.1-onnx.zip)
- [config_url](https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2/1.0.1/onnx/config.json) | -| `huggingface/sentence-transformers/paraphrase-mpnet-base-v2` | 1.0.0 | 768-dimensional dense vector space. | Yes | - [model_url](https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/paraphrase-mpnet-base-v2/1.0.0/torch_script/sentence-transformers_paraphrase-mpnet-base-v2-1.0.0-torch_script.zip)
- [config_url](https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/paraphrase-mpnet-base-v2/1.0.0/torch_script/config.json) | - [model_url](https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/paraphrase-mpnet-base-v2/1.0.0/onnx/sentence-transformers_paraphrase-mpnet-base-v2-1.0.0-onnx.zip)
- [config_url](https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/paraphrase-mpnet-base-v2/1.0.0/onnx/config.json) | -| `huggingface/sentence-transformers/distiluse-base-multilingual-cased-v1` | 1.0.1 | 512-dimensional dense vector space. | Yes | - [model_url](https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/distiluse-base-multilingual-cased-v1/1.0.1/torch_script/sentence-transformers_distiluse-base-multilingual-cased-v1-1.0.1-torch_script.zip)
- [config_url](https://artifacts.opensearch.org/models/ml-models/huggingface/sentence-transformers/distiluse-base-multilingual-cased-v1/1.0.1/torch_script/config.json) | Not available | +A higher document score means higher similarity. In the preceding response, documents are scored as follows against the query text `today is sunny`: +Document text | Score +:--- | :--- +`how are you` | -6.077798 +`today is sunny` | 10.223609 +`today is july fifth` | -1.3987057 +`it is winter` | -4.5923924 -### Sparse encoding models -**Introduced 2.11** -{: .label .label-purple } +The document that contains the same text as the query is scored the highest, and the remaining documents are scored based on the text similarity. -Sparse encoding models transfer text into a sparse vector and convert the vector to a list of `` pairs representing the text entry and its corresponding weight in the sparse vector. You can use these models for use cases such as clustering or sparse neural search. +## Step 5: Use the model for search -We recommend the following models for optimal performance: +To learn how to set up a vector index and use text embedding models for search, see [Semantic search]({{site.url}}{{site.baseurl}}/search-plugins/semantic-search/). -- Use the `amazon/neural-sparse/opensearch-neural-sparse-encoding-v1` model during both ingestion and search. -- Use the `amazon/neural-sparse/opensearch-neural-sparse-encoding-doc-v1` model during ingestion and the -`amazon/neural-sparse/opensearch-neural-sparse-tokenizer-v1` model during search. +To learn how to set up a vector index and use sparse encoding models for search, see [Neural sparse search]({{site.url}}{{site.baseurl}}/search-plugins/neural-sparse-search/). -The following table provides a list of sparse encoding models and artifact links you can use to download them. +To learn how to use cross-encoder models for reranking, see [Reranking search results]({{site.url}}{{site.baseurl}}/search-plugins/search-relevance/reranking-search-results/). -| Model name | Version | Auto-truncation | TorchScript artifact | Description | -|---|---|---|---| -| `amazon/neural-sparse/opensearch-neural-sparse-encoding-v1` | 1.0.1 | Yes | - [model_url](https://artifacts.opensearch.org/models/ml-models/amazon/neural-sparse/opensearch-neural-sparse-encoding-v1/1.0.1/torch_script/neural-sparse_opensearch-neural-sparse-encoding-v1-1.0.1-torch_script.zip)
- [config_url](https://artifacts.opensearch.org/models/ml-models/amazon/neural-sparse/opensearch-neural-sparse-encoding-v1/1.0.1/torch_script/config.json) | A neural sparse encoding model. The model transforms text into a sparse vector, identifies the indexes of non-zero elements in the vector, and then converts the vector into `` pairs, where each entry corresponds to a non-zero element index. | -| `amazon/neural-sparse/opensearch-neural-sparse-encoding-doc-v1` | 1.0.1 | Yes | - [model_url](https://artifacts.opensearch.org/models/ml-models/amazon/neural-sparse/opensearch-neural-sparse-encoding-doc-v1/1.0.1/torch_script/neural-sparse_opensearch-neural-sparse-encoding-doc-v1-1.0.1-torch_script.zip)
- [config_url](https://artifacts.opensearch.org/models/ml-models/amazon/neural-sparse/opensearch-neural-sparse-encoding-doc-v1/1.0.1/torch_script/config.json) | A neural sparse encoding model. The model transforms text into a sparse vector, identifies the indexes of non-zero elements in the vector, and then converts the vector into `` pairs, where each entry corresponds to a non-zero element index. | -| `amazon/neural-sparse/opensearch-neural-sparse-tokenizer-v1` | 1.0.1 | Yes | - [model_url](https://artifacts.opensearch.org/models/ml-models/amazon/neural-sparse/opensearch-neural-sparse-tokenizer-v1/1.0.1/torch_script/neural-sparse_opensearch-neural-sparse-tokenizer-v1-1.0.1-torch_script.zip)
- [config_url](https://artifacts.opensearch.org/models/ml-models/amazon/neural-sparse/opensearch-neural-sparse-tokenizer-v1/1.0.1/torch_script/config.json) | A neural sparse tokenizer model. The model tokenizes text into tokens and assigns each token a predefined weight, which is the token's inverse document frequency (IDF). If the IDF file is not provided, the weight defaults to 1. For more information, see [Preparing a model]({{site.url}}{{site.baseurl}}/ml-commons-plugin/custom-local-models/#preparing-a-model). | diff --git a/_search-plugins/search-relevance/reranking-search-results.md b/_search-plugins/search-relevance/reranking-search-results.md index 47860a74e4..14c418020d 100644 --- a/_search-plugins/search-relevance/reranking-search-results.md +++ b/_search-plugins/search-relevance/reranking-search-results.md @@ -13,7 +13,7 @@ Introduced 2.12 You can rerank search results using a cross-encoder reranker in order to improve search relevance. To implement reranking, you need to configure a [search pipeline]({{site.url}}{{site.baseurl}}/search-plugins/search-pipelines/index/) that runs at search time. The search pipeline intercepts search results and applies the [`rerank` processor]({{site.url}}{{site.baseurl}}/search-plugins/search-pipelines/rerank-processor/) to them. The `rerank` processor evaluates the search results and sorts them based on the new scores provided by the cross-encoder model. **PREREQUISITE**
-Before configuring a reranking pipeline, you must set up a cross-encoder model. For more information, see [Cross-encoder models]({{site.url}}{{site.baseurl}}/ml-commons-plugin/custom-local-models/#cross-encoder-models). +Before configuring a reranking pipeline, you must set up a cross-encoder model. For information about using an OpenSearch-provided model, see [Cross-encoder models]({{site.url}}{{site.baseurl}}/ml-commons-plugin/pretrained-models/#cross-encoder-models). For information about using a custom model, see [Custom local models]({{site.url}}{{site.baseurl}}/ml-commons-plugin/custom-local-models/). {: .note} ## Running a search with reranking diff --git a/images/dashboards/add-panel2.png b/images/dashboards/add-panel2.png new file mode 100644 index 0000000000..f76d00d4a1 Binary files /dev/null and b/images/dashboards/add-panel2.png differ diff --git a/images/dashboards/add-sample-data-2.png b/images/dashboards/add-sample-data-2.png new file mode 100644 index 0000000000..2f1b5ec901 Binary files /dev/null and b/images/dashboards/add-sample-data-2.png differ diff --git a/images/dashboards/dark-mode.png b/images/dashboards/dark-mode.png new file mode 100644 index 0000000000..f25716d134 Binary files /dev/null and b/images/dashboards/dark-mode.png differ diff --git a/images/dashboards/dashboards-home.png b/images/dashboards/dashboards-home.png new file mode 100644 index 0000000000..b8223ec7d0 Binary files /dev/null and b/images/dashboards/dashboards-home.png differ diff --git a/images/dashboards/dashboards-overview-page.png b/images/dashboards/dashboards-overview-page.png new file mode 100644 index 0000000000..ba5382f891 Binary files /dev/null and b/images/dashboards/dashboards-overview-page.png differ diff --git a/images/dashboards/discover-basics.png b/images/dashboards/discover-basics.png new file mode 100644 index 0000000000..0f3418882d Binary files /dev/null and b/images/dashboards/discover-basics.png differ diff --git a/images/dashboards/discover-filter.png b/images/dashboards/discover-filter.png new file mode 100644 index 0000000000..632e0b6f9e Binary files /dev/null and b/images/dashboards/discover-filter.png differ diff --git a/images/dashboards/discover-view.png b/images/dashboards/discover-view.png new file mode 100644 index 0000000000..63699494f9 Binary files /dev/null and b/images/dashboards/discover-view.png differ diff --git a/images/dashboards/discover.png b/images/dashboards/discover.png new file mode 100644 index 0000000000..a00e49cfa2 Binary files /dev/null and b/images/dashboards/discover.png differ diff --git a/images/dashboards/opensearch-dashboards-discover.png b/images/dashboards/opensearch-dashboards-discover.png new file mode 100644 index 0000000000..b84d8e7b0d Binary files /dev/null and b/images/dashboards/opensearch-dashboards-discover.png differ diff --git a/images/dashboards/opensearch-dashboards-home.png b/images/dashboards/opensearch-dashboards-home.png new file mode 100644 index 0000000000..a4e8dbeea2 Binary files /dev/null and b/images/dashboards/opensearch-dashboards-home.png differ diff --git a/images/icons/menu-icon.png b/images/icons/menu-icon.png new file mode 100644 index 0000000000..d44367f298 Binary files /dev/null and b/images/icons/menu-icon.png differ