Implement Numeric Offset Parameter in Get Snapshots API #76233
Conversation
Add numeric offset parameter to this API. Relates elastic#74350
original-brownbear
added
>enhancement
:Distributed Coordination/Snapshot/Restore
elasticmachine
added
the
Team:Distributed (Obsolete)
|
Pinging @elastic/es-distributed (Team:Distributed) |
| @@ -475,7 +471,7 @@ private static SnapshotsInRepo buildSimpleSnapshotInfos( | |||
| ) | |||
| ); | |||
| } | |||
| return sortSnapshots(snapshotInfos, sortBy, after, size, order); | |||
| return sortSnapshots(snapshotInfos, sortBy, after, 0, GetSnapshotsRequest.NO_LIMIT, order); | |||
There was a problem hiding this comment.
For now I just dropped the size and offset limits everywhere but before resolving the final listener.
There isn't really an easy way of implementing a numeric offset with the current approach otherwise and the memory savings from dropping a few elements from the stream earlier are probably trivial anyway. For now this seems good enough to me and once this is in and the search parameter PR has been merged we can start optimizing the code here I think.
| @@ -134,12 +134,17 @@ Sort order. Valid values are `asc` for ascending and `desc` for descending order | |||
| (Optional, string) | |||
| Offset identifier to start pagination from as returned by the `next` field in the response body. | |||
|
|
|||
| `offset`:: | |||
| (Optional, integer) | |||
| Numeric offset to start pagination from. Using a non-zero value for this parameter is mutually exclusive with using the `after` parameter. | |||
There was a problem hiding this comment.
Perhaps worth clarifying the offset is evaluated after (name) filtering?
Also, I thought we had a case of needing to use both after and offset, but if not, I am fine with the restriction.
There was a problem hiding this comment.
Perhaps worth clarifying the offset is evaluated after (name) filtering?
Done
Also, I thought we had a case of needing to use both after and offset, but if not, I am fine with the restriction.
I don't think so, at least not today. I could see value in that if we were to open queries like "all snapshots after date so and so" to users, but as we don't have that right now and just use the after for iteration I don't think it's needed. But for now I think we're good here :)
| final List<SnapshotInfo> subsetSorted = getSnapshotsResponse.getSnapshots(); | ||
| assertEquals(subsetSorted, getSnapshotsResponseNumeric.getSnapshots()); | ||
| assertEquals(subsetSorted, allSorted.subList(j + 1, j + i + 1)); | ||
| assertEquals(allSnapshotNames.size(), getSnapshotsResponse.totalCount()); |
There was a problem hiding this comment.
I think we should also verify the total count and remaining from the response obtained using offset.
There was a problem hiding this comment.
++ added assertions for that
| @@ -130,6 +138,9 @@ public void writeTo(StreamOutput out) throws IOException { | |||
| out.writeEnum(sort); | |||
| out.writeVInt(size); | |||
| order.writeTo(out); | |||
| if (out.getVersion().onOrAfter(NUMERIC_PAGINATION_VERSION)) { | |||
There was a problem hiding this comment.
Would we not want to fail if offset is non-zero when talking to a too old version? Just like 3 lines down.
There was a problem hiding this comment.
++ added handling for that thanks!
| final List<SnapshotInfo> subsetSorted = getSnapshotsResponse.getSnapshots(); | ||
| assertEquals(subsetSorted, getSnapshotsResponseNumeric.getSnapshots()); | ||
| assertEquals(subsetSorted, allSorted.subList(j + 1, j + i + 1)); | ||
| assertEquals(allSnapshotNames.size(), getSnapshotsResponse.totalCount()); |
There was a problem hiding this comment.
I think we should also verify the total count and remaining from the response obtained using offset.
There was a problem hiding this comment.
++ added assertions for that
…s-by-numeric-offset
|
Thanks Henning! |
Add numeric offset parameter to this API. Relates elastic#74350
Add numeric offset parameter to this API.
Relates #74350