Support for integration with Point In Time Search

[eirsep]

Is your feature request related to a problem? Please describe.

Point in Time Search allows users to run different queries against the same fixed data set in time. Users can create a Point In Time which only takes data into account up until the moment it is created. Hence, none of the resources that are required to return the data from the initial request are modified or deleted. This feature allows users to maintain a state which can be re-used by different queries in order to achieve consistent results.

RFC : opensearch-project/OpenSearch#1147

Describe the solution you'd like

Dashboards can support the functionality to create a Point In Time. The Create Point In Time API returns a PIT Id.

This PIT Id can be passed in search requests. This would allow Dashboards users to set up a Point In Time view of the User data and let them run multiple queries in that consistent state without data modifications, additions or deletions.

A clear and concise description of what you want to happen.

Describe alternatives you've considered

N/A

Additional context

N/A

[r-kuhr]

This is an important feature to have. It gives flexibility in creating UIs for users.

This would remove the possibility of confusion for a customer paging through search results that could change over time by allowing the UI to only keep track of the initial search ID and then paging thus working with the initial search results.

This feature would eliminate the need for developers to get a large result set and page in code (reducing code complexity and shrinking data sets being returned).

This is especially useful when the initial search results contains 100(00)s of products.

[kavilla]

Hello, thank you for opening this! @kgcreative and @ahopp will take a look into this as soon as possible.

[ahopp]

@eirsep Taking a look now.

@eirsep & @r-kuhr Can you clarify a bit of the UI outcome you've considered or envisioned? There seems to be few applications across search and dashboarding want to make sure I understand what you're proposing.

[r-kuhr]

@ahopp
In my use case, the application would use the PIT search results to keep the search results consistent for paging back and forth and not causing confusion to the customer when there is a possibility of items to be updated in time window. This is most useful for having an infinite scroll functionality for our phone apps.

[ahopp]

Thanks @r-kuhr That's helpful!

@eirsep Based on the RFC there are quite a few points of integration we could consider. For example, there seems to be an administration use case that we might want to consider (e.g., central UI for deleting PIT search akin to "DELETE /_point_in_time/" in the API, view all PIT search in the UI akin to "GET /_point_in_time" in the API) in addition to the discussion of where and how we can use the PIT to enhance the experience in OpenSearch Dashboards.

Before we get into the types of UI integrations we'd want to support, I'd like to better understand the constraints or resource expectations so we can be a bit more paternalistic in our design. For example, it seems that a PIT that is open for a long while could create quite a bit of overhead if your index is changing frequently (for example, delaying the ability to perform a merge process to free up resources) - this would affect the design patterns we choose to propose/employ later.

Are there any other performance and resource constraints with PIT that we should consider before the UI + Dashboard enhancements?

[ahopp]

Additionally, I think there are number of user considerations that we should discuss/track before making a UI proposal. For example;

• How does a user "keep" a PIT search longer term (past the 12hr max current set)? Are there guardrails?
• Can a user set a friendly name on a PIT search? Can they share it?
• How does the API differentiate between single user PIT search or those meant to be shared? Should users be able to share in the UI? If so, how?
• Does every user have access to run PIT searches? Can any user delete a shared PIT search?
• Are there any limitations to aggregations or other query elements?

Finally, I think we will need to consider other features enhancements outside of Dashboards. For instance, it would seem to me that PIT could also serve to enhance reporting. Or anywhere which Saved Objects could use PIT.

[eirsep]

Additionally, I think there are number of user considerations that we should discuss/track before making a UI proposal. For example;

• How does a user "keep" a PIT search longer term (past the 12hr max current set)? Are there guardrails?

There is a setting in place to tune the max duration of a PIT . It can be set to 2 days or 3 days or 6 hours.

• Can a user set a friendly name on a PIT search? Can they share it?

User cannot set a friendly name for a PIT. This would need to be handled by the dashboard if deemed as a must-have.

• How does the API differentiate between single user PIT search or those meant to be shared? Should users be able to share in the UI? If so, how?

API doesn't differentiate by PIT Id passed in search request. User must not share a PIT ID if he doesn't wish for the point in time to be used by other users.

• Does every user have access to run PIT searches? Can any user delete a shared PIT search?

If the dashboard user has also the constructs of the security plugin, we could have a separate role with permission to create PIT and a separate role with permission to use PITs in a search.

• Are there any limitations to aggregations or other query elements?

No limitations to aggregations or any query elements. All elements of a search query will work the same way.

[Cindyqi8506]

working on it

[kgcreative]

See #1612 for UX flows