Tests fail when there are multiple tests with multiple schemas/schematrons

[jordanpadams]

Checked for duplicates

Yes - I've already checked

🐛 Describe the bug

When I try to run back to back tests calling validate, it does not pick up later specified schemas.

🕵️ Expected behavior

I expected the tests to run successfully.

📜 To Reproduce

Playing around with the validate.feature file.

Scenario 1

• Execute as is
• First and last test fail (first test shouldn't pass)

Scenario 2

• Comment out test 1 (github795)
• github797 fails when it should pass. Per the error, validate is not correctly reading the LDDs (Error refers to unrecognized geom:Geometry)

Scenario 3

• Comment out test 1 (github795)
• Swap github795b and github797 in order
• All tests pass

🖥 Environment Info

MacOSx

📚 Version of Software Used

Latest snapshot

🩺 Test Data / Additional context

See repo

🦄 Related requirements

No response

⚙️ Engineering Details

No idea what is happening here. It appears validate is not properly reading/loading the GEOM LDD on the second run.

per comments below, blocked by NASA-PDS/validate#1028

🎉 Integration & Test

No response

[al-niessner]

@jordanpadams

Your expectations are wrong. The first test fails because:
SEED 2.4' is not known to this version of validate.

While you may be using a modern information model, validate must fix NASA-PDS/validate#1028 before this test can pass. There is a block that requires loading it independent of check contents that is causing an internal error.

The second test always processes the same. However, there are tons of SEED errors there as well indicating that constant behavior is probably luck not validate good behavior.

The third, behaves different depending on 2 because one of the schematron errors becomes a warning. I do not know if this would change with SEED no longer causing the internal errors.

It is not too surprising that an internal error would leave all kinds of dodo in validate. I would prioritize fixing NASA-PDS/validate#1028 then retesting here. It also means that both validate and this StepDefs should be updated to always fail on an internal error.

[jordanpadams]

@al-niessner copy that. let's go with that path forward. Thanks for investigating.

[jordanpadams]

@al-niessner unfortunately, the removing the SEED internal error noted above did not fix failures. Removing that file still causes github797 tests to fail, when all 3 tests execute successfully when run independently.