Fix a bug in the Archivematica transfer monitor #117
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
When the transfer monitor runs, it's looking for the METS file in the
reporting cluster to see that the transfer completed successfully.
Unfortunately, the name of that file is structured in a way that makes
it slightly difficult to query:
data/objects/submissionDocumentation/transfer-ARTCOOB14-46f0a350-d18d-4123-a162-70589bf9786e/METS.xml
So it has to query the list of storage service files in an approximate
way:
SELECT files
WHERE externalIdentifier={archivematicaIdentifier}
AND suffix=".xml"
which can fail if a bag contains lots of XML files and the METS file
isn't in the first page of results.
We can fix this in the storage service, but we haven't done that work yet.
This ups the result window from 100 to 10,000, so it's more likely we'll
find the METS file in that first page of results. If this still isn't
enough, we can go back to the storage service.
Currently the two stacks are fighting over whether this should be tagged as prod/staging, which creates unnecessary diff churn. Create a new rule per stack to avoid churn.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The transfer monitor provides end-to-end testing of Archivematica – it looks at the files uploaded to the S3 bucket as Archivematica inputs, and checks to see if they've successfully reached the storage service. It does that by looking for the METS file in the reporting cluster. Unfortunately, the name of that file is structured in a way that makes it slightly difficult to query:
So the monitor has to query the list of storage service files in an approximate way:
which can fail if a bag contains lots of XML files and the METS file isn't in the first page of results.
We can fix this in the storage service, but we haven't done that work yet. See wellcomecollection/storage-service#1028
This ups the result window from 100 to 10,000, so it's more likely we'll find the METS file in that first page of results. If this still isn't enough, we can go back to the storage service.