Correctness of the RFC3339 RegEx in OSCAL Json Schema

[hahsan-ti]

Question

The regular expression used for the datetime fields may not be correct. The pattern is complex as it attempts to cover different date combinations. In the complexity of the pattern, I have noticed the following issues:

1. The timestamp is not recognized if the date falls in January, February, March, May, July, August, October, and December.
2. Two backslashes \\ symbols prefix the period separating the sub-seconds from seconds.
3. The time offset from UTC can range from -99:99 to +99:99, where it should be between -12:00 to +14:00. A timezone offset of more than ±24:00 will put you on a different day.

RegEx from OSCAL POA&M Schema (1.0.4)

^((2000|2400|2800|(19|2[0-9](0[48]|[2468][048]|[13579][26])))-02-29)|(((19|2[0-9])[0-9]{2})-02-(0[1-9]|1[0-9]|2[0-8]))|(((19|2[0-9])[0-9]{2})-(0[13578]|10|12)-(0[1-9]|[12][0-9]|3[01]))|(((19|2[0-9])[0-9]{2})-(0[469]|11)-(0[1-9]|[12][0-9]|30))T(2[0-3]|[01][0-9]):([0-5][0-9]):([0-5][0-9])(\\.[0-9]+)?(Z|[+-][0-9]{2}:[0-9]{2})$

You can see this regex in action here.

I am unsure if any of the above are actual problems or if I need to learn more. However, if I have identified issues, then I propose the following solution.

The first issue is missing parentheses to group the regex for the dates. Adding the parentheses groups the appropriate or together. The second issue is caused by an extra backslash that must be removed. The last issue may be addressed -- if desired -- by expanding the regex to consider all time offsets as they exist at the time of the release.

^(((2000|2400|2800|(19|2[0-9](0[48]|[2468][048]|[13579][26])))-02-29)|(((19|2[0-9])[0-9]{2})-02-(0[1-9]|1[0-9]|2[0-8]))|(((19|2[0-9])[0-9]{2})-(0[13578]|10|12)-(0[1-9]|[12][0-9]|3[01]))|(((19|2[0-9])[0-9]{2})-(0[469]|11)-(0[1-9]|[12][0-9]|30)))T(2[0-3]|[01][0-9]):([0-5][0-9]):([0-5][0-9])(\.[0-9]+)?(Z|(\-((0[0-9]|1[0-2]):00|0[39]:30)|\+((0[0-9]|1[0-4]):00|(0[34569]|10):30|(0[58]|12):45)))$

You can see this regex in action here.

Also, why shouldn't we support a leap second?

[GaryGapinski]

There is an additional interoperability consideration: in an XML Schema dateTime the fractional second string must not end in '0'.

(\.[0-9]+)? should possibly be (\.[0-9]*[1-9])?.

Bonus: I just used an XPath (3.1 in XML Editor 24.1, build 2022062007) statement current-dateTime() to generate a sample dateTime and received 2022-07-30T08:45:27.015550-04:00: go figure.

[michaelhkay]

The rule cited from XML Schema relates to the "canonical representation" - the implication is that the trailing zero carries no information and will be lost when round-tripping, not that it is disallowed. The XPath rules for conversion of a dateTime to a string (which presumably apply to the case you are describing, though you haven't described it in any detail) handle the seconds part using the rules for xs:decimal to string conversion, which refer to the XSD rules for the canonical representation of xs:decimal values, which prohibit the trailing zero in your example: it looks like what you've described is non-conformant as the output of xs:dateTime to string conversion, though it would of course be conformant as the input.

[david-waltermire]

If I am interpreting @michaelhkay correctly, having a more restrictive pattern that enforces trimming the trailing 0s would be ok and we should expect XML tools to produce trimmed time values for fractional seconds. Is this correct?

If correct, I believe the updated pattern should be:

^(((2000|2400|2800|(19|2[0-9](0[48]|[2468][048]|[13579][26])))-02-29)|(((19|2[0-9])[0-9]{2})-02-(0[1-9]|1[0-9]|2[0-8]))|(((19|2[0-9])[0-9]{2})-(0[13578]|10|12)-(0[1-9]|[12][0-9]|3[01]))|(((19|2[0-9])[0-9]{2})-(0[469]|11)-(0[1-9]|[12][0-9]|30)))T(2[0-3]|[01][0-9]):([0-5][0-9]):([0-5][0-9])(\.[0-9]*[1-9])?(Z|(\-((0[0-9]|1[0-2]):00|0[39]:30)|\+((0[0-9]|1[0-4]):00|(0[34569]|10):30|(0[58]|12):45)))$

You can see this regex in action here.

[GaryGapinski]

If I am interpreting @michaelhkay correctly, having a more restrictive pattern that enforces trimming the trailing 0s would be ok and we should expect XML tools to produce trimmed time values for fractional seconds. Is this correct?

After a bit of testing (using the following) I found that an xs:dateTime variable will serialize without a trailing zero.
It also shows that '2022-08-02T12:31:42.339230Z' castable as xs:dateTime returns true.

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:xs="http://www.w3.org/2001/XMLSchema" exclude-result-prefixes="xs"
    expand-text="true" version="3.0">
    <xsl:output method="xml" omit-xml-declaration="true" />
    <xsl:variable name="UTC" as="xs:duration" select="xs:dayTimeDuration('PT0H')" />
    <xsl:variable name="UTC-date" as="xs:date" select="adjust-date-to-timezone(current-date(), $UTC)" />
    <xsl:variable name="UTC-datetime" as="xs:dateTime" select="adjust-dateTime-to-timezone(current-dateTime(), $UTC)" />
    <xsl:template match="/">
        <dateTime test="{'2022-08-02T12:31:42.339230Z' castable as xs:dateTime}">
            <xsl:copy-of select="$UTC-datetime" />
        </dateTime>
    </xsl:template>
</xsl:stylesheet>

When XML, JSON, and YAML are employed, perhaps the canonical form should be enforced. That would provide assurance of equality when (untyped) strings were compared.

[david-waltermire]

We need to first create a PR in develop updating the data types.

[michaelhkay]

Is a regular expression really the most effective specification tool for this task? Michael Kay Saxonica

…

On 16 Aug 2022, at 16:51, Hidayatullah Ahsan ***@***.***> wrote: Question The regular expression used for the datetime fields may not be correct. The pattern is complex as it attempts to cover different date combinations. In the complexity of the pattern, I have noticed the following issues: The timestamp is not recognized if the date falls in January, February, March, May, July, August, October, and December. Two backslashes \\ symbols prefix the period separating the sub-seconds from seconds. The time offset from UTC can range from -99:99 to +99:99, where it should be between -12:00 to +14:00. A timezone offset of more than ±24:00 will put you on a different day. RegEx from OSCAL POA&M Schema (1.0.4) ^((2000|2400|2800|(19|2[0-9](0[48]|[2468][048]|[13579][26])))-02-29)|(((19|2[0-9])[0-9]{2})-02-(0[1-9]|1[0-9]|2[0-8]))|(((19|2[0-9])[0-9]{2})-(0[13578]|10|12)-(0[1-9]|[12][0-9]|3[01]))|(((19|2[0-9])[0-9]{2})-(0[469]|11)-(0[1-9]|[12][0-9]|30))T(2[0-3]|[01][0-9]):([0-5][0-9]):([0-5][0-9])(\\.[0-9]+)?(Z|[+-][0-9]{2}:[0-9]{2})$ You can see this regex in action here <https://regex101.com/r/HIuUPe/1>. I am unsure if any of the above are actual problems or if I need to learn more. However, if I have identified issues, then I propose the following solution. The first issue is missing parentheses to group the regex for the dates. Adding the parentheses groups the appropriate or together. The second issue is caused by an extra backslash that must be removed. The last issue may be addressed -- if desired -- by expanding the regex to consider all time offsets as they exist at the time of the release. ^(((2000|2400|2800|(19|2[0-9](0[48]|[2468][048]|[13579][26])))-02-29)|(((19|2[0-9])[0-9]{2})-02-(0[1-9]|1[0-9]|2[0-8]))|(((19|2[0-9])[0-9]{2})-(0[13578]|10|12)-(0[1-9]|[12][0-9]|3[01]))|(((19|2[0-9])[0-9]{2})-(0[469]|11)-(0[1-9]|[12][0-9]|30)))T(2[0-3]|[01][0-9]):([0-5][0-9]):([0-5][0-9])(\.[0-9]+)?(Z|(\-((0[0-9]|1[0-2]):00|0[39]:30)|\+((0[0-9]|1[0-4]):00|(0[34569]|10):30|(0[58]|12):45)))$ You can see this regex in action here <https://regex101.com/r/HkzaqI/1>. Also, why shouldn't we support a leap second? — Reply to this email directly, view it on GitHub <#224>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AASIQIXDVID5UNDIAJ4CMUTVZO2GRANCNFSM56WLHNAA>. You are receiving this because you were mentioned.

[aj-stein-nist]

Is a regular expression really the most effective specification tool for this task? Michael Kay Saxonica

Is there an implied alternative in the question? :-)

[michaelhkay]

Is a regular expression really the most effective specification tool for this task? Michael Kay Saxonica Is there an implied alternative in the question? :-)

No. I'm just wondering what you're trying to achieve. Sometimes narrative prose is the best way of achieving clarity. Michael Kay Saxonica

[david-waltermire]

Issue #230 identifies follow on work to update the data type documentation. This should include clarifying behavior per @michaelhkay.