Part 1: Support for cancel_after_timeinterval parameter in search and msearch request #986

sohami · 2021-07-19T20:42:20Z

This PR introduces the new request level parameter to configure the timeout interval after which
a search request will be cancelled. For msearch request the parameter is supported both at parent
request and at sub child search requests. If it is provided at parent level and child search request
doesn't have it then the parent level value is set at such child request. The parent level msearch
is not used to cancel the parent request as it may be tricky to come up with correct value in cases
when child search request can have different runtimes

TEST: Added test for ser/de with new parameter

Signed-off-by: Sorabh Hamirwasia [email protected]

Description

Part-1: This commit introduces the new request level parameter (cancel_after_time_interval) to configure the timeout interval after which
a search request will be cancelled. For msearch request the parameter is supported both at parent
request and at sub child search requests. If it is provided at parent level and child search request
doesn't have it then the parent level value is set at such child request. The parent level msearch
is not used to cancel the parent request as it may be tricky to come up with correct value in cases
when child search request can have different runtimes

Part 2: Support for consuming cancel_after_time_interval parameter in search and msearch request. This commit adds the handling of the new request level parameter and schedule cancellation task. It also adds a cluster setting search.cancel_after_time_interval: To set a global cancellation timeout for search request which will be used in absence of request level timeout.

Below is the behavior in various cases:

request param	cluster param	Effective	Feature
null	10s	10s
null	-1	-1	disabled
-1	10s	-1	disabled
-1	-1	-1	disabled
20s	10s	20s
20s	-1	20s

Issues Resolved

#817

Check List

[Y] New functionality includes testing.
- [Y] All tests pass
New functionality has been documented.
- New functionality has javadoc added
[Y] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

opensearch-ci-bot · 2021-07-19T20:44:16Z

✅ Gradle Wrapper Validation success 0f3785fbd3856a72834e53493b50f463603f0d4d

opensearch-ci-bot · 2021-07-19T20:44:46Z

✅ DCO Check Passed 0f3785fbd3856a72834e53493b50f463603f0d4d

opensearch-ci-bot · 2021-07-19T20:51:45Z

✅ Gradle Precommit success 0f3785fbd3856a72834e53493b50f463603f0d4d

AmiStrn

Nice!

I am commenting as I am not a maintainer. please flag one of them in a comment to take a look as well:)

My only real issue is the name of the param - I think that cancel_after_time_interval would be better. the main indicator for this is that you wrote it in camel case like this: cancelAfterTimeInterval... separating between the time and the interval makes sense.
Other things I commented on are minor, to say the least.

Thanks for calling my attention to the PR I would be happy to review the next phase as well when you get to it!

server/src/main/java/org/opensearch/action/search/SearchRequest.java

server/src/main/java/org/opensearch/action/search/MultiSearchRequest.java

server/src/main/java/org/opensearch/rest/action/search/RestMultiSearchAction.java

dblock · 2021-07-27T18:37:21Z

@AmiStrn thanks for reviewing this! Looking forward to the next update @sohami

opensearch-ci-bot · 2021-07-28T05:39:10Z

✅ DCO Check Passed 433f91f

sohami · 2021-07-28T05:39:38Z

@AmiStrn - Thanks for the review. I have rebased and added commit for part 2 as well. Have also replied to your comment related to naming. Please take a look.
@dblock - Updated with part 2 changes

opensearch-ci-bot · 2021-07-28T05:39:50Z

✅ Gradle Wrapper Validation success 433f91f

opensearch-ci-bot · 2021-07-28T05:51:56Z

✅ Gradle Precommit success 433f91f

dblock

The change looks good to me, but we should spend some time talking about the name of this setting. I see two problems with the current name.

It does not tell me whether it's seconds, milliseconds or something else, while we do have absoluteStartMillis that is in milliseconds. So maybe the name should include millis, ms or whatever other convention is used for another setting?
You call this "cancellation timeout" in a bunch of places in tests. This seems like what this feature does, time out after a certain time. This feels like possibly a much more sensible name (RequestTimeout or RequestTimeoutMillis?).

server/src/internalClusterTest/java/org/opensearch/search/SearchCancellationIT.java

server/src/main/java/org/opensearch/action/search/MultiSearchRequest.java

AmiStrn

Cool!
I liked the listener wrapper solution in particular.
@dblock - regarding the use of the OriginSettingClient in TimeoutTaskCancellationUtility - will this be a problem with the changes going to be introduced in the security module?
Regarding the params - I suggest using just the one for the time interval itself without the boolean.
Regarding the cluster settings, there is concern over the surface area of the settings in general in the project getting too big. No good solution for that, however, these settings may not be a "must" for this feature at this point. Consider using only the time interval setting anyway, since adding another that basically says "use the other setting" may be confusing.
All in all I enjoyed reviewing this, Thanks @sohami :)

server/src/main/java/org/opensearch/action/search/MultiSearchRequest.java

server/src/internalClusterTest/java/org/opensearch/search/SearchCancellationIT.java

server/src/main/java/org/opensearch/action/search/SearchTask.java

server/src/main/java/org/opensearch/action/search/TransportSearchAction.java

server/src/main/java/org/opensearch/action/support/TimeoutTaskCancellationUtility.java

server/src/internalClusterTest/java/org/opensearch/search/SearchCancellationIT.java

server/src/main/java/org/opensearch/common/settings/ClusterSettings.java

sohami · 2021-08-02T19:18:54Z

The change looks good to me, but we should spend some time talking about the name of this setting. I see two problems with the current name.

It does not tell me whether it's seconds, milliseconds or something else, while we do have absoluteStartMillis that is in milliseconds. So maybe the name should include millis, ms or whatever other convention is used for another setting?

You call this "cancellation timeout" in a bunch of places in tests. This seems like what this feature does, time out after a certain time. This feels like possibly a much more sensible name (RequestTimeout or RequestTimeoutMillis?).

The parameter can accept the time in any unit since it is of TimeValue type. Input can be 10s or 10ms or any other format. Similar parameters are defined here: search.keep_alive_interval or default_search_timeout
The feature basically cancels the search request after a defined time interval to avoid any resource consumption by search request. There is already a timeout parameter in search request which is honored at shard level instead at search request level. To be more explicit and avoid any confusion, I am naming the parameter as cancel_after_time_interval. I am fine with @AmiStrn suggestion on splitting it to time_interval

opensearch-ci-bot · 2021-08-02T23:33:11Z

❌ DCO Check Failed 6f5208042b6ee2441904b469578f761d6dee4568
Run ./dev-tools/signoff-check.sh remotes/origin/main 6f5208042b6ee2441904b469578f761d6dee4568 to check locally
Use git commit with -s to add 'Signed-of-by: {EMAIL}' on impacted commits

opensearch-ci-bot · 2021-08-02T23:33:52Z

✅ Gradle Wrapper Validation success 6f5208042b6ee2441904b469578f761d6dee4568

sohami · 2021-08-02T23:34:17Z

Cool!
I liked the listener wrapper solution in particular.
@dblock - regarding the use of the OriginSettingClient in TimeoutTaskCancellationUtility - will this be a problem with the changes going to be introduced in the security module?
Regarding the params - I suggest using just the one for the time interval itself without the boolean.
Regarding the cluster settings, there is concern over the surface area of the settings in general in the project getting too big. No good solution for that, however, these settings may not be a "must" for this feature at this point. Consider using only the time interval setting anyway, since adding another that basically says "use the other setting" may be confusing.
All in all I enjoyed reviewing this, Thanks @sohami :)

@AmiStrn and @dblock : Thanks for the review and feedback. Have responded to majority of it and updated the PR with changes related to some of the items. For now have taken @AmiStrn feedback to name it as cancel_after_time_interval. Will wait for your next round of input :)

opensearch-ci-bot · 2021-08-02T23:38:16Z

✅ Gradle Wrapper Validation success c54f40fd5b3a2d187960fe2c8985df3cf1ced35f

opensearch-ci-bot · 2021-08-02T23:38:44Z

✅ DCO Check Passed c54f40fd5b3a2d187960fe2c8985df3cf1ced35f

opensearch-ci-bot · 2021-08-02T23:43:30Z

✅ Gradle Precommit success 6f5208042b6ee2441904b469578f761d6dee4568

opensearch-ci-bot · 2021-08-02T23:43:31Z

✅ Gradle Precommit success c54f40fd5b3a2d187960fe2c8985df3cf1ced35f

opensearch-ci-bot · 2021-08-03T19:26:41Z

✅ DCO Check Passed 9d48da524dced80a8b4b3844c0687426fefc2b61

opensearch-ci-bot · 2021-08-03T19:26:41Z

✅ Gradle Wrapper Validation success 9d48da524dced80a8b4b3844c0687426fefc2b61

sohami · 2021-08-03T19:30:12Z

@dblock and @AmiStrn : Thanks for your active review! To summarize, most of the comments are taken care of and following are still open:

Cluster setting to control enabling/disabling this feature. Being discussed here. My preference would be to keep it in v1 and slowly deprecate over next couple of releases. I can own that issue.
Concern over using OriginSettingClient in context of security model changes discussed here. I don't see issue with this usage though, given the same mechanism is used in other places (Example here). If it needs some changes then it can be done as separate PR across all the usages.

opensearch-ci-bot · 2021-08-03T19:36:11Z

✅ Gradle Precommit success 9d48da524dced80a8b4b3844c0687426fefc2b61

dblock

This LGTM!

dblock · 2021-08-04T13:16:29Z

start gradle check

opensearch-ci-bot · 2021-08-04T13:43:00Z

✅ Gradle Check success 9d48da524dced80a8b4b3844c0687426fefc2b61
Log 368

Reports 368

opensearch-ci-bot · 2021-08-05T03:36:50Z

✅ Gradle Wrapper Validation success eb39a6ab8b76e92d2ed30abb38ac401e7a9bba28

opensearch-ci-bot · 2021-08-11T19:50:40Z

✅ Gradle Precommit success cf11529

dblock · 2021-08-11T23:17:21Z

start gradle check

dblock · 2021-08-11T23:17:45Z

@malpani @AmiStrn can I please have a final pair of 👀 from you on this?

opensearch-ci-bot · 2021-08-11T23:46:30Z

✅ Gradle Check success cf11529
Log 385

Reports 385

malpani · 2021-08-12T05:30:50Z

@malpani @AmiStrn can I please have a final pair of 👀 from you on this?

lgtm! this is good to go!

AmiStrn · 2021-08-12T09:26:23Z

@malpani @AmiStrn can I please have a final pair of 👀 from you on this?

Done, LGTM as well. This was interesting and I learned a lot from this approach @sohami
Also, I like the refactoring of the tests in SearchCancellationIT for readability:)

Can't wait to use this feature!
@dblock @malpani is the next step for this feature to add support for this in the different clients?

dblock · 2021-08-12T15:03:22Z

I merged this. I imagine we want this in 1.1. @sohami can you please make a backport PR into the 1.x branch? (cherry pick the squashed commit and make a PR)

dblock · 2021-08-12T15:05:42Z

+++ this needs to be documented, open an issue or make PRs into https://github.com/opensearch-project/documentation-website please @sohami

… msearch request (opensearch-project#986) * Part 1: Support for cancel_after_timeinterval parameter in search and msearch request This commit introduces the new request level parameter to configure the timeout interval after which a search request will be cancelled. For msearch request the parameter is supported both at parent request and at sub child search requests. If it is provided at parent level and child search request doesn't have it then the parent level value is set at such child request. The parent level msearch is not used to cancel the parent request as it may be tricky to come up with correct value in cases when child search request can have different runtimes TEST: Added test for ser/de with new parameter Signed-off-by: Sorabh Hamirwasia <[email protected]> * Part 2: Support for cancel_after_timeinterval parameter in search and msearch request This commit adds the handling of the new request level parameter and schedule cancellation task. It also adds a cluster setting to set a global cancellation timeout for search request which will be used in absence of request level timeout. TEST: Added new tests in SearchCancellationIT Signed-off-by: Sorabh Hamirwasia <[email protected]> * Address Review feedback for Part 1 Signed-off-by: Sorabh Hamirwasia <[email protected]> * Address review feedback for Part 2 Signed-off-by: Sorabh Hamirwasia <[email protected]> * Update CancellableTask to remove the cancelOnTimeout boolean flag Signed-off-by: Sorabh Hamirwasia <[email protected]> * Replace search.cancellation.timeout cluster setting with search.enforce_server.timeout.cancellation to control if cluster level cancel_after_time_interval should take precedence over request level cancel_after_time_interval value Signed-off-by: Sorabh Hamirwasia <[email protected]> * Removing the search.enforce_server.timeout.cancellation cluster setting and just keeping search.cancel_after_time_interval setting with request level parameter taking the precedence. Signed-off-by: Sorabh Hamirwasia <[email protected]> Co-authored-by: Sorabh Hamirwasia <[email protected]>

sohami · 2021-08-12T16:36:45Z

I merged this. I imagine we want this in 1.1. @sohami can you please make a backport PR into the 1.x branch? (cherry pick the squashed commit and make a PR)

Created a PR on 1.1: #1085

sohami · 2021-08-12T16:37:40Z

+++ this needs to be documented, open an issue or make PRs into https://github.com/opensearch-project/documentation-website please @sohami

Corresponding documentation issue: opensearch-project/documentation-website#128

… msearch request (#986) (#1085) * Part 1: Support for cancel_after_timeinterval parameter in search and msearch request This commit introduces the new request level parameter to configure the timeout interval after which a search request will be cancelled. For msearch request the parameter is supported both at parent request and at sub child search requests. If it is provided at parent level and child search request doesn't have it then the parent level value is set at such child request. The parent level msearch is not used to cancel the parent request as it may be tricky to come up with correct value in cases when child search request can have different runtimes TEST: Added test for ser/de with new parameter Signed-off-by: Sorabh Hamirwasia <[email protected]> * Part 2: Support for cancel_after_timeinterval parameter in search and msearch request This commit adds the handling of the new request level parameter and schedule cancellation task. It also adds a cluster setting to set a global cancellation timeout for search request which will be used in absence of request level timeout. TEST: Added new tests in SearchCancellationIT Signed-off-by: Sorabh Hamirwasia <[email protected]> * Address Review feedback for Part 1 Signed-off-by: Sorabh Hamirwasia <[email protected]> * Address review feedback for Part 2 Signed-off-by: Sorabh Hamirwasia <[email protected]> * Update CancellableTask to remove the cancelOnTimeout boolean flag Signed-off-by: Sorabh Hamirwasia <[email protected]> * Replace search.cancellation.timeout cluster setting with search.enforce_server.timeout.cancellation to control if cluster level cancel_after_time_interval should take precedence over request level cancel_after_time_interval value Signed-off-by: Sorabh Hamirwasia <[email protected]> * Removing the search.enforce_server.timeout.cancellation cluster setting and just keeping search.cancel_after_time_interval setting with request level parameter taking the precedence. Signed-off-by: Sorabh Hamirwasia <[email protected]> Co-authored-by: Sorabh Hamirwasia <[email protected]> Co-authored-by: Sorabh Hamirwasia <[email protected]>

@sohami

I have nominated and maintainers have agreed to invite Sorabh Hamirwasia(@sohami) to be a co-maintainer. Sorabh has kindly accepted. Sorabh has led the design and implementation of multiple features like query cancellation, concurrent segment search for aggregations and snapshot inter-operability w/remote storage in OpenSearch. Some significant issues and PRs authored by Sorabh for this effort are as follows: Feature proposals Concurrent segment search for aggregations : #6798 Searchable Remote Index : #2900 Implementations Concurrent segment search for aggregations: #7514 Lucene changes to leaf slices for concurrent search: apache/lucene#12374 Moving concurrent search to core : #7203 Query cancellation support : #986 In total, Sorabh has authored 14 PRs going back to Aug 2021. He also frequently reviews contributions from others and has reviewed nearly 20 PRs in the same time frame. I believe Sorabh will be a valuable addition as a maintainer of OpenSearch and will continue to contribute to the success of the project going forward. Signed-off-by: Bukhtawar Khan <[email protected]>

@sohami

I have nominated and maintainers have agreed to invite Sorabh Hamirwasia(@sohami) to be a co-maintainer. Sorabh has kindly accepted. Sorabh has led the design and implementation of multiple features like query cancellation, concurrent segment search for aggregations and snapshot inter-operability w/remote storage in OpenSearch. Some significant issues and PRs authored by Sorabh for this effort are as follows: Feature proposals Concurrent segment search for aggregations : #6798 Searchable Remote Index : #2900 Implementations Concurrent segment search for aggregations: #7514 Lucene changes to leaf slices for concurrent search: apache/lucene#12374 Moving concurrent search to core : #7203 Query cancellation support : #986 In total, Sorabh has authored 14 PRs going back to Aug 2021. He also frequently reviews contributions from others and has reviewed nearly 20 PRs in the same time frame. I believe Sorabh will be a valuable addition as a maintainer of OpenSearch and will continue to contribute to the success of the project going forward. Signed-off-by: Bukhtawar Khan <[email protected]> (cherry picked from commit 7769682) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

@sohami

I have nominated and maintainers have agreed to invite Sorabh Hamirwasia(@sohami) to be a co-maintainer. Sorabh has kindly accepted. Sorabh has led the design and implementation of multiple features like query cancellation, concurrent segment search for aggregations and snapshot inter-operability w/remote storage in OpenSearch. Some significant issues and PRs authored by Sorabh for this effort are as follows: Feature proposals Concurrent segment search for aggregations : #6798 Searchable Remote Index : #2900 Implementations Concurrent segment search for aggregations: #7514 Lucene changes to leaf slices for concurrent search: apache/lucene#12374 Moving concurrent search to core : #7203 Query cancellation support : #986 In total, Sorabh has authored 14 PRs going back to Aug 2021. He also frequently reviews contributions from others and has reviewed nearly 20 PRs in the same time frame. I believe Sorabh will be a valuable addition as a maintainer of OpenSearch and will continue to contribute to the success of the project going forward. (cherry picked from commit 7769682) Signed-off-by: Bukhtawar Khan <[email protected]>

@sohami

I have nominated and maintainers have agreed to invite Sorabh Hamirwasia(@sohami) to be a co-maintainer. Sorabh has kindly accepted. Sorabh has led the design and implementation of multiple features like query cancellation, concurrent segment search for aggregations and snapshot inter-operability w/remote storage in OpenSearch. Some significant issues and PRs authored by Sorabh for this effort are as follows: Feature proposals Concurrent segment search for aggregations : opensearch-project#6798 Searchable Remote Index : opensearch-project#2900 Implementations Concurrent segment search for aggregations: opensearch-project#7514 Lucene changes to leaf slices for concurrent search: apache/lucene#12374 Moving concurrent search to core : opensearch-project#7203 Query cancellation support : opensearch-project#986 In total, Sorabh has authored 14 PRs going back to Aug 2021. He also frequently reviews contributions from others and has reviewed nearly 20 PRs in the same time frame. I believe Sorabh will be a valuable addition as a maintainer of OpenSearch and will continue to contribute to the success of the project going forward. Signed-off-by: Bukhtawar Khan <[email protected]>

@sohami

I have nominated and maintainers have agreed to invite Sorabh Hamirwasia(@sohami) to be a co-maintainer. Sorabh has kindly accepted. Sorabh has led the design and implementation of multiple features like query cancellation, concurrent segment search for aggregations and snapshot inter-operability w/remote storage in OpenSearch. Some significant issues and PRs authored by Sorabh for this effort are as follows: Feature proposals Concurrent segment search for aggregations : opensearch-project#6798 Searchable Remote Index : opensearch-project#2900 Implementations Concurrent segment search for aggregations: opensearch-project#7514 Lucene changes to leaf slices for concurrent search: apache/lucene#12374 Moving concurrent search to core : opensearch-project#7203 Query cancellation support : opensearch-project#986 In total, Sorabh has authored 14 PRs going back to Aug 2021. He also frequently reviews contributions from others and has reviewed nearly 20 PRs in the same time frame. I believe Sorabh will be a valuable addition as a maintainer of OpenSearch and will continue to contribute to the success of the project going forward. Signed-off-by: Bukhtawar Khan <[email protected]> Signed-off-by: Kaushal Kumar <[email protected]>

@sohami

I have nominated and maintainers have agreed to invite Sorabh Hamirwasia(@sohami) to be a co-maintainer. Sorabh has kindly accepted. Sorabh has led the design and implementation of multiple features like query cancellation, concurrent segment search for aggregations and snapshot inter-operability w/remote storage in OpenSearch. Some significant issues and PRs authored by Sorabh for this effort are as follows: Feature proposals Concurrent segment search for aggregations : opensearch-project#6798 Searchable Remote Index : opensearch-project#2900 Implementations Concurrent segment search for aggregations: opensearch-project#7514 Lucene changes to leaf slices for concurrent search: apache/lucene#12374 Moving concurrent search to core : opensearch-project#7203 Query cancellation support : opensearch-project#986 In total, Sorabh has authored 14 PRs going back to Aug 2021. He also frequently reviews contributions from others and has reviewed nearly 20 PRs in the same time frame. I believe Sorabh will be a valuable addition as a maintainer of OpenSearch and will continue to contribute to the success of the project going forward. Signed-off-by: Bukhtawar Khan <[email protected]> Signed-off-by: Ivan Brusic <[email protected]>

@sohami

I have nominated and maintainers have agreed to invite Sorabh Hamirwasia(@sohami) to be a co-maintainer. Sorabh has kindly accepted. Sorabh has led the design and implementation of multiple features like query cancellation, concurrent segment search for aggregations and snapshot inter-operability w/remote storage in OpenSearch. Some significant issues and PRs authored by Sorabh for this effort are as follows: Feature proposals Concurrent segment search for aggregations : opensearch-project#6798 Searchable Remote Index : opensearch-project#2900 Implementations Concurrent segment search for aggregations: opensearch-project#7514 Lucene changes to leaf slices for concurrent search: apache/lucene#12374 Moving concurrent search to core : opensearch-project#7203 Query cancellation support : opensearch-project#986 In total, Sorabh has authored 14 PRs going back to Aug 2021. He also frequently reviews contributions from others and has reviewed nearly 20 PRs in the same time frame. I believe Sorabh will be a valuable addition as a maintainer of OpenSearch and will continue to contribute to the success of the project going forward. Signed-off-by: Bukhtawar Khan <[email protected]> Signed-off-by: Shivansh Arora <[email protected]>

sohami mentioned this pull request Jul 19, 2021

Support timeout based search request cancellation #817

Open

AmiStrn reviewed Jul 21, 2021

View reviewed changes

sohami force-pushed the issue-817 branch from 0f3785f to 433f91f Compare July 28, 2021 05:37

dblock requested changes Aug 2, 2021

View reviewed changes

AmiStrn reviewed Aug 2, 2021

View reviewed changes

sohami force-pushed the issue-817 branch from 6f52080 to c54f40f Compare August 2, 2021 23:36

dblock approved these changes Aug 4, 2021

View reviewed changes

dblock approved these changes Aug 11, 2021

View reviewed changes

dblock merged commit 9b6e621 into opensearch-project:main Aug 12, 2021

dblock added the pending backport Identifies an issue or PR that still needs to be backported label Aug 12, 2021

dblock added the documentation pending Tracks issues which have PRs merged but documentation changes pending label Aug 12, 2021

sohami mentioned this pull request Aug 12, 2021

Part 1: Support for cancel_after_time_interval parameter in search and… #1085

Merged

2 tasks

This was referenced Aug 16, 2021

[BUG] Reproducible failure in SearchCancellationIT.testMSearchChildReqCancellationWithHybridTimeout #1099

Closed

Random Testing on main and version branches opensearch-project/opensearch-build#245

Closed

dblock removed the pending backport Identifies an issue or PR that still needs to be backported label Dec 6, 2021

AmiStrn mentioned this pull request Apr 19, 2023

[RFC] Detecting search queries which have been cancelled but continues to run for long. #6953

Closed

Bukhtawar mentioned this pull request Jul 22, 2023

Adding @sohami to OpenSearch maintainers #8828

Merged

6 tasks

Part 1: Support for cancel_after_timeinterval parameter in search and msearch request #986

Part 1: Support for cancel_after_timeinterval parameter in search and msearch request #986

Conversation

sohami commented Jul 19, 2021 • edited Loading

Description

Issues Resolved

Check List

opensearch-ci-bot commented Jul 19, 2021

opensearch-ci-bot commented Jul 19, 2021

opensearch-ci-bot commented Jul 19, 2021

AmiStrn left a comment

Choose a reason for hiding this comment

dblock commented Jul 27, 2021

opensearch-ci-bot commented Jul 28, 2021

sohami commented Jul 28, 2021

opensearch-ci-bot commented Jul 28, 2021

opensearch-ci-bot commented Jul 28, 2021

dblock left a comment • edited Loading

Choose a reason for hiding this comment

AmiStrn left a comment • edited Loading

Choose a reason for hiding this comment

sohami commented Aug 2, 2021

opensearch-ci-bot commented Aug 2, 2021

opensearch-ci-bot commented Aug 2, 2021

sohami commented Aug 2, 2021

opensearch-ci-bot commented Aug 2, 2021

opensearch-ci-bot commented Aug 2, 2021

opensearch-ci-bot commented Aug 2, 2021

opensearch-ci-bot commented Aug 2, 2021

opensearch-ci-bot commented Aug 3, 2021

opensearch-ci-bot commented Aug 3, 2021

sohami commented Aug 3, 2021

opensearch-ci-bot commented Aug 3, 2021

dblock left a comment

Choose a reason for hiding this comment

dblock commented Aug 4, 2021

opensearch-ci-bot commented Aug 4, 2021

opensearch-ci-bot commented Aug 5, 2021

opensearch-ci-bot commented Aug 11, 2021

dblock commented Aug 11, 2021

dblock commented Aug 11, 2021 • edited Loading

opensearch-ci-bot commented Aug 11, 2021

malpani commented Aug 12, 2021

AmiStrn commented Aug 12, 2021

dblock commented Aug 12, 2021

dblock commented Aug 12, 2021

sohami commented Aug 12, 2021

sohami commented Aug 12, 2021

sohami commented Jul 19, 2021 •

edited

Loading

dblock left a comment •

edited

Loading

AmiStrn left a comment •

edited

Loading

dblock commented Aug 11, 2021 •

edited

Loading