Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

added test for DISABLED digest type to FoxML digest Validation #190

Merged
merged 24 commits into from
Feb 16, 2023
Merged
Show file tree
Hide file tree
Changes from 10 commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
9b7f3ab
added test for DISABLED digest type
LlGC-dof Sep 20, 2022
41e8835
Adding Integration Test and a new exceptin for disabled digests
LlGC-dof Nov 25, 2022
c6be76a
checkstyle fixes
Nov 28, 2022
13f3ce3
changing index to match inline-it et al
Nov 28, 2022
bb13c9e
seperating test index in case of clash
Nov 28, 2022
464ded8
Merge remote-tracking branch 'upstream/main' into FCREPO-3847
Nov 28, 2022
00d36d3
fixing issues with the disabled integration test
Nov 29, 2022
c72a992
removing superflous log message as we throw an exception now
Nov 29, 2022
99d22f4
git wasn't adding an empty dir. used .gitkeep and now the test should…
Surfrdan Nov 29, 2022
d7066ac
.gitkeep broke the indexer. copying a datastream from another test
Surfrdan Nov 29, 2022
888afb2
reverting to logging rather than throwing exception
Surfrdan Jan 12, 2023
8cd2a06
improved DISABLED fixity Integration Test to use fcrepo-storage-ocfl …
Surfrdan Jan 19, 2023
5b3651c
Switch from assembly to shade plugin to correct Jena re-packaging (#189)
whikloj Oct 13, 2022
6c4f373
Alter test assertion to allow for ocfl 1.0 and 1.1 (#192)
whikloj Nov 14, 2022
f1583e9
Bump woodstox-core from 6.2.3 to 6.4.0 (#191)
dependabot[bot] Nov 16, 2022
d391ed4
Increment version to 6.3.1-SNAPSHOT
dbernstein Nov 16, 2022
0753f0b
fixing issues with the disabled integration test
Nov 29, 2022
db3fbd9
removing superflous log message as we throw an exception now
Nov 29, 2022
af793ec
git wasn't adding an empty dir. used .gitkeep and now the test should…
Surfrdan Nov 29, 2022
8f7ddfd
.gitkeep broke the indexer. copying a datastream from another test
Surfrdan Nov 29, 2022
e28c397
reverting to logging rather than throwing exception
Surfrdan Jan 12, 2023
1065504
improved DISABLED fixity Integration Test to use fcrepo-storage-ocfl …
Surfrdan Jan 19, 2023
0116ec4
Merge branch 'FCREPO-3847' of https://github.com/Surfrdan/migration-u…
Surfrdan Jan 19, 2023
0009f78
fixed test to look for a DISABLED digest
Surfrdan Feb 16, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -193,7 +193,13 @@ public void processObject(final StreamingFedoraObjectHandler handler) {
dsInfo = new Foxml11DatastreamInfo(objectInfo, reader);
} else if (reader.getLocalName().equals("datastreamVersion")) {
final var v = new Foxml11DatastreamVersion(dsInfo, reader);
v.validateInlineXml();
try {
v.validateInlineXml();
} catch (RuntimeException e) {
// do we need to do anyting with disabled digests?
LOG.error("Inline Validation failed", e);
throw new RuntimeException(e);
}
handler.processDatastreamVersion(v);
} else {
throw new RuntimeException("Unexpected element! \"" + reader.getLocalName() + "\"!");
Expand Down Expand Up @@ -468,6 +474,11 @@ private String extractInlineXml() throws XMLStreamException {

private void validateInlineXml() {
if (isInlineXml && contentDigest != null && StringUtils.isNotBlank(contentDigest.getDigest())) {

if (StringUtils.equals(contentDigest.getType(), "DISABLED")) {
throw new RuntimeException("DISABLED digest. Skipping digest validation");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm wondering if we want to throw an exception here or not. As I understood the problem initially, it seemed like we wanted to continue migration of the pids with disabled digests because it was a feature of Fedora 3. If so, we could just log this instead.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I went the exception route to assist with building an integration test case but if you think logging would be sufficient I can revert that change.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think logging then return to skip the checksum validation is acceptable. For the integration test we can switch to check if the migration was successful. It looks like the InlineXmlIT does something similar, but for this case we can check if the resources exist in the ocfl object instead of computing checksums.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've now made those changes as suggested @mikejritter

}

final var transformedXml = transformInlineXmlForChecksum();
final var digest = DigestUtils.getDigest(contentDigest.getType());
final var digestBytes = DigestUtils.digest(digest, transformedXml);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -654,10 +654,15 @@ private ResourceHeaders createDatastreamHeaders(final DatastreamVersion dv,
}

if (dv.getContentDigest() != null && !Strings.isNullOrEmpty(dv.getContentDigest().getDigest())) {
final var digest = dv.getContentDigest();
final var digests = new ArrayList<URI>();
digests.add(URI.create("urn:" + digest.getType().toLowerCase() + ":" + digest.getDigest().toLowerCase()));
headers.withDigests(digests);
if (!dv.getContentDigest().getDigest().equals("none")) {
final var digest = dv.getContentDigest();
final var digests = new ArrayList<URI>();
digests.add(URI.create("urn:" + digest.getType().toLowerCase() + ":" +
digest.getDigest().toLowerCase()));
headers.withDigests(digests);
} else {
LOGGER.warn("Digest content 'none' found. Not adding to header");
}
}

headers.withMimeType(mime);
Expand Down
73 changes: 73 additions & 0 deletions src/test/java/org/fcrepo/migration/DisabledDigestIT.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
/**
* Copyright 2015 DuraSpace, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package org.fcrepo.migration;

import org.apache.commons.io.FileUtils;
import org.junit.After;
import org.junit.Test;
import org.springframework.context.ConfigurableApplicationContext;
import org.springframework.context.support.ClassPathXmlApplicationContext;

import java.nio.file.Files;
import java.nio.file.Paths;

import static org.junit.Assert.assertTrue;

/**
* @author Dan Field
*/
public class DisabledDigestIT {

private ConfigurableApplicationContext context;
private Migrator migrator;

@After
public void tearDown() {
if (context != null) {
context.close();
}
}

private void setup(final String name) throws Exception {
final var storage = Paths.get(String.format("target/test/ocfl/%s/storage", name));
final var staging = Paths.get(String.format("target/test/ocfl/%s/staging", name));

if (Files.exists(storage)) {
FileUtils.forceDelete(storage.toFile());
}
if (Files.exists(staging)) {
FileUtils.forceDelete(staging.toFile());
}

Files.createDirectories(storage);
Files.createDirectories(staging);

context = new ClassPathXmlApplicationContext(String.format("spring/%s-setup.xml", name));
migrator = (Migrator) context.getBean("migrator");
}

@Test
public void testMigrateObjectWithExternalDatastreamAndDisabledDigest() throws Exception {
setup("inline-disabled-it");
try {
migrator.run();
} catch (RuntimeException e) {
assertTrue(e.getMessage().contains("DISABLED digest. Skipping digest validation"));
}
}

}
1 change: 1 addition & 0 deletions src/test/java/org/fcrepo/migration/InlineXmlIT.java
Original file line number Diff line number Diff line change
Expand Up @@ -115,4 +115,5 @@ public void failMigrationWhenInlineXmlDoesNotMatchDigest() throws Exception {
}
}


}
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
<?xml version="1.0" encoding="UTF-8"?>
<Policy
xmlns="urn:oasis:names:tc:xacml:3.0:core:schema:wd-17"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:oasis:names:tc:xacml:3.0:core:schema:wd-17 http://digital.library.wisc.edu/1711.dl/XMLSchema-XACML-3.0"
PolicyId="hdl:1711.dl/Access-policy-open-access-UW-Madison-all"
RuleCombiningAlgId="urn:oasis:names:tc:xacml:3.0:rule-combining-algorithm:deny-unless-permit"
Version="1.0">
<Description>
Open Access: any user can view, listen to, and/or download any representation of the object,
as long as they're associated with or present at UW-Madison.
</Description>
<Target />
<Rule Effect="Permit" RuleId="permit-UWMadison">
<Target />
<Condition>
<Apply FunctionId="urn:oasis:names:tc:xacml:1.0:function:string-equal">
<Apply FunctionId="urn:oasis:names:tc:xacml:1.0:function:string-one-and-only">
<AttributeDesignator
AttributeId="http://digital.library.wisc.edu/1711.dl/vocabulary/access/population#"
Category="urn:oasis:names:tc:xacml:1.0:subject-category:access-subject"
DataType="http://www.w3.org/2001/XMLSchema#string"
MustBePresent="true" />
</Apply>
<AttributeValue DataType="http://www.w3.org/2001/XMLSchema#string">http://digital.library.wisc.edu/1711.dl/vocabulary/access/population#uwmadison</AttributeValue>
</Apply>
</Condition>
</Rule>
<Rule Effect="Deny" RuleId="deny-rule"></Rule>
</Policy>
Loading