Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update CosmosDataSinkExtension.cs to enable support for shared throughput in Cosmos containers #108

Merged
merged 3 commits into from
Feb 10, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ public async Task WriteAsync(IAsyncEnumerable<IDataItem> dataItems, IConfigurati
containerProperties.PartitionKeyPath = settings.PartitionKeyPath;
}

ThroughputProperties? throughputProperties = settings.IsServerlessAccount
ThroughputProperties? throughputProperties = settings.IsServerlessAccount || settings.UseSharedThroughput
? null
: settings.UseAutoscaleForCreatedContainer
? ThroughputProperties.CreateAutoscaleThroughput(settings.CreatedContainerMaxThroughput ?? 4000)
Expand Down Expand Up @@ -211,4 +211,4 @@ public record ItemResult(string? Id, HttpStatusCode StatusCode)
public int ItemCount => IsSuccess ? 1 : 0;
}
}
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ public class CosmosSinkSettings : CosmosSettingsBase, IDataExtensionSettings
public int? CreatedContainerMaxThroughput { get; set; }
public bool UseAutoscaleForCreatedContainer { get; set; } = true;
public bool IsServerlessAccount { get; set; } = false;
public bool UseSharedThroughput { get; set; } = false;
public DataWriteMode WriteMode { get; set; } = DataWriteMode.Insert;
public List<string>? PartitionKeyPaths { get; set; }

Expand Down
5 changes: 3 additions & 2 deletions Extensions/Cosmos/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ Or with RBAC:
}
```

Sink requires an additional `PartitionKeyPath` parameter which is used when creating the container if it does not exist. To use hierarchical partition keys, instead use the `PartitionKeyPaths` setting to supply an array of up to 3 paths. It also supports an optional `RecreateContainer` parameter (`false` by default) to delete and then recreate the container to ensure only newly imported data is present. The optional `BatchSize` parameter (100 by default) sets the number of items to accumulate before inserting. `ConnectionMode` can be set to either `Gateway` (default) or `Direct` to control how the client connects to the CosmosDB service. For situations where a container is created as part of the transfer operation `CreatedContainerMaxThroughput` (in RUs) and `UseAutoscaleForCreatedContainer` provide the initial throughput settings which will be in effect when executing the transfer. The optional `WriteMode` parameter specifies the type of data write to use: `InsertStream`, `Insert`, `UpsertStream`, or `Upsert`. The `IsServerlessAccount` parameter specifies whether the target account uses Serverless instead of Provisioned throughput, which affects the way containers are created. Additional parameters allow changing the behavior of the Cosmos client appropriate to your environment.
Sink requires an additional `PartitionKeyPath` parameter which is used when creating the container if it does not exist. To use hierarchical partition keys, instead use the `PartitionKeyPaths` setting to supply an array of up to 3 paths. It also supports an optional `RecreateContainer` parameter (`false` by default) to delete and then recreate the container to ensure only newly imported data is present. The optional `BatchSize` parameter (100 by default) sets the number of items to accumulate before inserting. `ConnectionMode` can be set to either `Gateway` (default) or `Direct` to control how the client connects to the CosmosDB service. For situations where a container is created as part of the transfer operation `CreatedContainerMaxThroughput` (in RUs) and `UseAutoscaleForCreatedContainer` provide the initial throughput settings which will be in effect when executing the transfer. To instead use shared throughput that has been provisioned at the database level, set the `UseSharedThroughput` parameter to `true`. The optional `WriteMode` parameter specifies the type of data write to use: `InsertStream`, `Insert`, `UpsertStream`, or `Upsert`. The `IsServerlessAccount` parameter specifies whether the target account uses Serverless instead of Provisioned throughput, which affects the way containers are created. Additional parameters allow changing the behavior of the Cosmos client appropriate to your environment.

### Sink

Expand All @@ -62,6 +62,7 @@ Sink requires an additional `PartitionKeyPath` parameter which is used when crea
"CreatedContainerMaxThroughput": 1000,
"UseAutoscaleForCreatedContainer": true,
"WriteMode": "InsertStream",
"IsServerlessAccount": false
"IsServerlessAccount": false,
"UseSharedThroughput": false
}
```
Loading