Roll back token datatype to avoid problematic character references

[wendellpiez]

Describe the bug

As reported in the OSCAL repo usnistgov/OSCAL#1127 a regular expression deployed in the schemas on the token datatype definition is breaking in certain tools.

Testing suggests that it is the numeric character escaping (entity syntax) in the expression [\i-[:𐀀-]][\c-[:𐀀-]]* that the tool is not able to handle.

But further testing shows that these character exclusions have no apparent effect on the regex - which suggests we should roll them back.

If the XML-regex syntax here for NCName [\i-[:]][\c-[:]]* is too XMLy, an alternative could be [\w-[:\d]]\w*.

In any case the correction must be made in Metaschema back end, to propagate to into generated schemas.

This is not a backward-compatibility breaking change.

Who is the bug affecting?

Any user of tools that choke on the regex as given.

What is affected by this bug?

Can't validate an OSCAL instance with an appropriate XSD.

When does this occur?

Anytime with the tool in question (C# processor under .NET).

As it happens, the oXygen XML Editor's XML Schema Regular Expression builder also does not support entity syntax, and shows the same error.

Expected behavior (i.e. solution)

The token datatype should be validated appropriately (against NCName constraints).

[aj-stein-nist]

Thanks for adding this @wendellpiez, so are you working on this moving forward or should I learn the dark arts of these regex patterns? :-)

[wendellpiez]

I would like to hear from @david-waltermire-nist what he thinks the right choice is here, on balance, from among the remedies proposed so far, or some other.

Having discussed it I would welcome the help implementing a correction in the XSLT M4 pipeline.

However, it is also unclear to me where this should be done. The bug we are addressing is not actually in the XSD but in a single consuming implemention (we know of); and a workaround is feasible for those cases. @david-waltermire-nist would necessarily have to be involved in any integration since this is the Metaschema support infrastructure. (And it will have a direct impact on tooling he is developing!)

[wendellpiez]

@david-waltermire-nist with AJ's help we have managed to demonstrate:

• The tool in question does appear to have issues dealing with Unicode ranges, or certain Unicode ranges, that other tools are okay with - possible bug there
• Nonetheless, the character range exclusion that causes the regex to break is superfluous, and can be removed without effect, as the excluded characters (despite my earlier research) do not appear to be in the character sets to begin with.

(It turns out a test for this is easy: just try and make an XSD definition for an element named A𐀀.)

So I think the solution here (on the XML side) is to roll back the definition to [\i-[:]][\c-[:]]* (no-colon name without other exclusions).

Issue #182 might give us a reason to do otherwise, but it could also be orthogonal.

[aj-stein-nist]

Yeah I hope that helped. I mothballed the project for now but we can always revive and add more tests. I integrated .NET code with GitHub Actions on a Windows runner so we can further test issues with this particular XML engine in the .NET System.Xml namespace (since that is different from MSXML, we dodged that bullet 😰).

Let me know if I can be of more help.

[wendellpiez]

Now emitting functional XSDs with the new datatype mappings.

Pruning the result XSD so it does not contain unused simpleType definitions is a bit difficult when their definitions are chained, but we are managing some of it at least.

JSON Schema production also looks okay wrt datatypes in this branch.