Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

EES-5283 Add missing MetaInsertBatchSize app setting to Public API Data Processor function app #5033

Merged
merged 1 commit into from
Jul 8, 2024

Conversation

benoutram
Copy link
Collaborator

@benoutram benoutram commented Jul 8, 2024

This PR adds a missing app setting AppSettings__MetaInsertBatchSize to the Public API Data Processor function app.

The missing setting is thought to be the cause of data set versions failing to process in Azure environments.

Without this setting we are observing high CPU activity in the ImportMetadata function, thought to be because _appSettingsOptions.MetaInsertBatchSize is resolving to zero in the following block, resulting in the batch iteration to be in an infinite loop.

var batch = options
.Skip(current)
.Take(_appSettingsOptions.MetaInsertBatchSize)
.ToList();

After around 10 minutes (thought to be the function timeout) the exception Npgsql.NpgsqlOperationInProgressException (0x80004005): The connection is already in state 'Executing' was occuring in the following block of code:

var links = await optionTable
.Where(o =>
batchRowKeys.Contains(o.Label + ',' + (o.IsAggregate == true ? "True" : "")))
.OrderBy(o => o.Label)
.Select((option, index) => new FilterOptionMetaLink
{
PublicId = SqidEncoder.Encode(batchStartIndex + index),
MetaId = meta.Id,
OptionId = option.Id
})
.ToListAsync(token: cancellationToken);

This exception can be reproduced locally by running the function app with AppSettings__MetaInsertBatchSize set to zero.

@benoutram benoutram merged commit 22994d0 into dev Jul 8, 2024
2 checks passed
@benoutram benoutram deleted the EES-5283 branch July 8, 2024 10:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants