EES-5283 Add missing MetaInsertBatchSize
app setting to Public API Data Processor function app
#5033
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds a missing app setting
AppSettings__MetaInsertBatchSize
to the Public API Data Processor function app.The missing setting is thought to be the cause of data set versions failing to process in Azure environments.
Without this setting we are observing high CPU activity in the
ImportMetadata
function, thought to be because_appSettingsOptions.MetaInsertBatchSize
is resolving to zero in the following block, resulting in the batch iteration to be in an infinite loop.explore-education-statistics/src/GovUk.Education.ExploreEducationStatistics.Public.Data.Processor/Repository/FilterMetaRepository.cs
Lines 103 to 106 in c52407b
After around 10 minutes (thought to be the function timeout) the exception
Npgsql.NpgsqlOperationInProgressException (0x80004005): The connection is already in state 'Executing'
was occuring in the following block of code:explore-education-statistics/src/GovUk.Education.ExploreEducationStatistics.Public.Data.Processor/Repository/FilterMetaRepository.cs
Lines 115 to 125 in c52407b
This exception can be reproduced locally by running the function app with
AppSettings__MetaInsertBatchSize
set to zero.