Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Subpartitioning: Adds support for Prefix Partition Key searches for sub partitioned containers #3109

Merged
merged 63 commits into from
Mar 7, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
63 commits
Select commit Hold shift + click to select a range
1228ccf
1st round of changes to support subpartitioning for ChangeFeed
philipthomas-MSFT Jan 18, 2022
d3c1b52
name change and moved a method to FeedRangePartitionKey
philipthomas-MSFT Jan 18, 2022
c41fa0e
support for prefix partition change feed and query including unit and…
philipthomas-MSFT Feb 10, 2022
12fd6c1
splitting out unit tests from emulator tests
philipthomas-MSFT Mar 22, 2022
f512155
1st round of changes to support subpartitioning for ChangeFeed
philipthomas-MSFT Jan 18, 2022
38c8c26
name change and moved a method to FeedRangePartitionKey
philipthomas-MSFT Jan 18, 2022
f8f56c9
support for prefix partition change feed and query including unit and…
philipthomas-MSFT Feb 10, 2022
17fd955
splitting out unit tests from emulator tests
philipthomas-MSFT Mar 22, 2022
086f23d
Merge branch 'users/philipthomas/subpartitioning' of https://github.c…
philipthomas-MSFT Mar 22, 2022
4b0ef1c
Merge branch 'master' into users/philipthomas/subpartitioning
philipthomas-MSFT Sep 1, 2022
6c2cdb2
moved logic to choose for prefix partition query and change feed
philipthomas-MSFT Sep 1, 2022
b7853b3
additional conditional to check for MultiHash partitionkeydefinition …
philipthomas-MSFT Sep 1, 2022
2f77385
removed unnecessary using
philipthomas-MSFT Sep 1, 2022
21214c5
cleanup
philipthomas-MSFT Sep 1, 2022
34b368c
unnecessary using
philipthomas-MSFT Sep 1, 2022
b837983
removed PREVIEW from proj
philipthomas-MSFT Sep 1, 2022
5e572f1
unit tests for ResolveFeedRangeBasedOnPrefixContainerAsync
philipthomas-MSFT Sep 1, 2022
21c5d41
change access modifier
philipthomas-MSFT Sep 1, 2022
3726bdb
since I changed to static, needed to fix tests
philipthomas-MSFT Sep 1, 2022
3e08d67
dealing with some testing nits
philipthomas-MSFT Sep 1, 2022
5b963c1
removed Console.Writeline
philipthomas-MSFT Sep 1, 2022
7c0d3b1
slight change in test use containerProperties and partitionKeyDefinition
philipthomas-MSFT Sep 2, 2022
90bf0c5
Merge branch 'master' into users/philipthomas/subpartitioning
NaluTripician Jan 9, 2023
b35d60f
Merge branch 'master' of https://github.com/Azure/azure-cosmos-dotnet…
NaluTripician Jan 9, 2023
ced7c65
Merge branch 'master' into users/philipthomas/subpartitioning
NaluTripician Jan 9, 2023
edca934
Merge branch 'users/philipthomas/subpartitioning' of https://github.c…
NaluTripician Jan 9, 2023
5186ff4
big fix for resolve feedrange, changes to error behavior, added addit…
NaluTripician Feb 2, 2023
ff71ffe
Merge branch 'master' into users/philipthomas/subpartitioning
NaluTripician Feb 2, 2023
9b531c6
test changes
NaluTripician Feb 3, 2023
3ad5309
Merge branch 'master' into users/philipthomas/subpartitioning
NaluTripician Feb 3, 2023
df167b7
ran updateContracts.ps1
NaluTripician Feb 3, 2023
48f7eb5
Merge branch 'users/philipthomas/subpartitioning' of https://github.c…
NaluTripician Feb 3, 2023
756a123
reverting to 3ad5309e9e22a7376e836cfdbe11fa2438a59133
NaluTripician Feb 3, 2023
739eb2a
Ran Update Contracts
NaluTripician Feb 7, 2023
22b3e6d
ran updateContracts.ps1
NaluTripician Feb 7, 2023
52c2d06
Merge branch 'master' into users/philipthomas/subpartitioning
NaluTripician Feb 7, 2023
a00e86e
UpdateContracts.ps1
NaluTripician Feb 7, 2023
bb2f9d3
updatecontracts
NaluTripician Feb 7, 2023
9701c3b
removed bad check from test, removed changes from updatecontracts.ps1
NaluTripician Feb 7, 2023
f47bcca
Merge branch 'users/philipthomas/subpartitioning' of https://github.c…
NaluTripician Feb 7, 2023
df318a9
revert UpdateContracts changes
NaluTripician Feb 7, 2023
27a81f0
Reverting to 756a123160d14c424c5e11c4f3520094115aa5d4
NaluTripician Feb 7, 2023
618c4af
removed accidental change from changelog
NaluTripician Feb 7, 2023
d11561b
removed unwanted change from Directory.build.props
NaluTripician Feb 7, 2023
92562bd
added test clean up
NaluTripician Feb 9, 2023
919076b
Merge branch 'master' into users/philipthomas/subpartitioning
NaluTripician Feb 10, 2023
2828c72
Merge branch 'master' into users/philipthomas/subpartitioning
NaluTripician Feb 13, 2023
7ccb272
Merge branch 'master' into users/philipthomas/subpartitioning
NaluTripician Feb 15, 2023
b56470e
removed console.writelines
NaluTripician Feb 16, 2023
49b61dd
Merge branch 'users/philipthomas/subpartitioning' of https://github.c…
NaluTripician Feb 16, 2023
bc737dc
updates to query test
NaluTripician Feb 16, 2023
0d81811
removed preview flag
NaluTripician Feb 27, 2023
b9ca602
Merge branch 'master' into users/philipthomas/subpartitioning
NaluTripician Feb 27, 2023
7301873
ran UpdateContrats.ps1
NaluTripician Feb 27, 2023
d24428b
Merge branch 'master' into users/philipthomas/subpartitioning
NaluTripician Feb 27, 2023
8fe7529
Merge branch 'master' into users/philipthomas/subpartitioning
NaluTripician Feb 27, 2023
63da8c8
Merge branch 'master' into users/philipthomas/subpartitioning
NaluTripician Mar 1, 2023
db06f92
Merge branch 'master' into users/philipthomas/subpartitioning
NaluTripician Mar 2, 2023
c098777
no longer recread feedRange for queries unless using prefix pk
NaluTripician Mar 3, 2023
87416c3
Merge branch 'master' into users/philipthomas/subpartitioning
NaluTripician Mar 3, 2023
73771ba
simplified if statement
NaluTripician Mar 6, 2023
6500636
fixed bug
NaluTripician Mar 6, 2023
416c823
Merge branch 'master' into users/philipthomas/subpartitioning
NaluTripician Mar 6, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 29 additions & 0 deletions Microsoft.Azure.Cosmos/src/Handler/RequestInvokerHandler.cs
Original file line number Diff line number Diff line change
Expand Up @@ -161,6 +161,14 @@ public virtual async Task<ResponseMessage> SendAsync(

if (feedRange != null)
{
philipthomas-MSFT marked this conversation as resolved.
Show resolved Hide resolved
if (!request.OperationType.IsPointOperation())
{
feedRange = await RequestInvokerHandler.ResolveFeedRangeBasedOnPrefixContainerAsync(
feedRange: feedRange,
cosmosContainerCore: cosmosContainerCore,
cancellationToken: cancellationToken);
}

if (feedRange is FeedRangePartitionKey feedRangePartitionKey)
{
if (cosmosContainerCore == null && object.ReferenceEquals(feedRangePartitionKey.PartitionKey, Cosmos.PartitionKey.None))
Expand Down Expand Up @@ -480,5 +488,26 @@ private static bool IsClientNoResponseSet(CosmosClientOptions clientOptions, Ope
&& clientOptions.EnableContentResponseOnWrite.HasValue
&& RequestInvokerHandler.IsItemNoRepsonseSet(clientOptions.EnableContentResponseOnWrite.Value, operationType);
}

internal static async Task<FeedRange> ResolveFeedRangeBasedOnPrefixContainerAsync(
FeedRange feedRange,
ContainerInternal cosmosContainerCore,
CancellationToken cancellationToken)
{
if (feedRange is FeedRangePartitionKey feedRangePartitionKey)
{
PartitionKeyDefinition partitionKeyDefinition = await cosmosContainerCore
.GetPartitionKeyDefinitionAsync(cancellationToken)
.ConfigureAwait(false);

if (partitionKeyDefinition != null && partitionKeyDefinition.Kind == PartitionKind.MultiHash
&& feedRangePartitionKey.PartitionKey.InternalKey?.Components?.Count < partitionKeyDefinition.Paths?.Count)
{
feedRange = new FeedRangeEpk(feedRangePartitionKey.PartitionKey.InternalKey.GetEPKRangeForPrefixPartitionKey(partitionKeyDefinition));
}
}

return feedRange;
}
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -959,12 +959,6 @@ private async Task<ResponseMessage> ExtractPartitionKeyAndProcessItemStreamAsync
// User specified PK value, no need to extract it
if (partitionKey.HasValue)
{
PartitionKeyDefinition pKeyDefinition = await this.GetPartitionKeyDefinitionAsync();
if (partitionKey.HasValue && partitionKey.Value != PartitionKey.None && partitionKey.Value.InternalKey.Components.Count != pKeyDefinition.Paths.Count)
{
throw new ArgumentException(RMResources.MissingPartitionKeyValue);
}

return await this.ProcessItemStreamAsync(
partitionKey,
itemId,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,12 @@
│ )
├── ItemSerialize(00000000-0000-0000-0000-000000000000) Transport-Component 00:00:00:000 0.00 milliseconds
├── Get Collection Cache(00000000-0000-0000-0000-000000000000) Routing-Component 00:00:00:000 0.00 milliseconds
│ └── Waiting for Initialization of client to complete(00000000-0000-0000-0000-000000000000) Unknown-Component 00:00:00:000 0.00 milliseconds
├── Read Collection(00000000-0000-0000-0000-000000000000) Transport-Component 00:00:00:000 0.00 milliseconds
│ (
│ [Client Side Request Stats]
│ Redacted To Not Change The Baselines From Run To Run
│ )
├── Batch Dispatch Async(00000000-0000-0000-0000-000000000000) Batch-Component 00:00:00:000 0.00 milliseconds
│ ├── Using Wait(00000000-0000-0000-0000-000000000000) Batch-Component 00:00:00:000 0.00 milliseconds
│ ├── Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler(00000000-0000-0000-0000-000000000000) RequestHandler-Component 00:00:00:000 0.00 milliseconds
Expand Down Expand Up @@ -72,7 +78,20 @@
},
{
"name": "Get Collection Cache",
"duration in milliseconds": 0
"duration in milliseconds": 0,
"children": [
{
"name": "Waiting for Initialization of client to complete",
"duration in milliseconds": 0
}
NaluTripician marked this conversation as resolved.
Show resolved Hide resolved
]
},
{
"name": "Read Collection",
"duration in milliseconds": 0,
"data": {
"Client Side Request Stats": "Redacted To Not Change The Baselines From Run To Run"
}
},
{
"name": "Batch Dispatch Async",
Expand Down Expand Up @@ -1634,6 +1653,12 @@
│ )
├── ItemSerialize(00000000-0000-0000-0000-000000000000) Transport-Component 00:00:00:000 0.00 milliseconds
├── Get Collection Cache(00000000-0000-0000-0000-000000000000) Routing-Component 00:00:00:000 0.00 milliseconds
│ └── Waiting for Initialization of client to complete(00000000-0000-0000-0000-000000000000) Unknown-Component 00:00:00:000 0.00 milliseconds
├── Read Collection(00000000-0000-0000-0000-000000000000) Transport-Component 00:00:00:000 0.00 milliseconds
│ (
│ [Client Side Request Stats]
│ Redacted To Not Change The Baselines From Run To Run
│ )
├── Batch Dispatch Async(00000000-0000-0000-0000-000000000000) Batch-Component 00:00:00:000 0.00 milliseconds
│ ├── Using Wait(00000000-0000-0000-0000-000000000000) Batch-Component 00:00:00:000 0.00 milliseconds
│ ├── Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler(00000000-0000-0000-0000-000000000000) RequestHandler-Component 00:00:00:000 0.00 milliseconds
Expand Down Expand Up @@ -1808,7 +1833,20 @@
},
{
"name": "Get Collection Cache",
"duration in milliseconds": 0
"duration in milliseconds": 0,
"children": [
{
"name": "Waiting for Initialization of client to complete",
"duration in milliseconds": 0
}
]
},
{
"name": "Read Collection",
"duration in milliseconds": 0,
"data": {
"Client Side Request Stats": "Redacted To Not Change The Baselines From Run To Run"
}
},
{
"name": "Batch Dispatch Async",
Expand Down
Loading