Improvements to HTML and Markdown character encoding in XML, JSON, and YAML

[david-waltermire]

User Story:

As an OSCAL content consumer, I need OSCAL XML, JSON, and YAML content that includes properly encoded characters in textual content (e.g., fields). Characters must not be encoded or escaped if the underlying format does not require them to be, as this will require extra processing/parsing to fix. The only exception to this rule are the Markdown formatting characters *`~^ when they are used as literal characters in JSON and YAML strings. This will protect these characters from interactions with Markdown formatting.

Goals:

• All generated OSCAL XML, JSON, and YAML content must use the minimum character encoding needed to correctly represent the character. This should be consistent with the character handling defined in the OSCAL datatypes documentation.

• While & needs to be encoded in HTML as &, this encoding needs to be avoided in JSON and YAML strings.

  The FedRAMP HIGH baseline is an example of where this is occurring incorrectly.

• Confirm that all ' characters in generated OSCAL JSON files are not escaped.

• Confirm that > and < are not encoded as > and < in JSON and YAML. These character should be represented as > and < in JSON and YAML.

• The OSCAL XML HTML content currently uses &#2A; and &#60 to encode * and ` respectively. These characters need to be represented as * and ` in OSCAL XML HTML content.

Dependencies:

None.

Acceptance Criteria

• A Pull Request (PR) is submitted addressing the goals of this User Story. This issue is referenced in the PR.
• The CI-CD build process runs without any reported errors.
• All documentation affected by this issue has been updated.

[anweiss]

@david-waltermire-nist could you provide more specifics as to what might be considered "overly encoded"?

[david-waltermire]

The user story has been updated to remove and clarify what was meant by "overly encoded".

[wendellpiez]

In the XML-to-JSON conversion, handling of characters in strings has been cleaned up, with much better results. Some basic unit testing has also been added for string conversion.

This is ready for inspection. Included in the inspection should be FedRAMP profiles where they contain '&' for example grep "POA&M" or "POA&M" (in XML and JSON respectively).

[wendellpiez]

Also: we need a plan for what to do (if anything) to prevent

XML IN:

<p>Markup can make <em>emphasis</em>, but so can *markdown*</p>

JSON:

"Markup can make *emphasis*, but so can *markdown*"

XML BACK:

<p>Markup can make <em>emphasis</em>, but so can <em>markdown</em></p>

One solution would be to map * and other delimiter characters, in generated Markdown, to Unicode representations such as \u002a. So in this case the JSON would have

"Markup can make *emphasis*, but so can \u002amarkdown\u002a".

This is (much) less legible but more robust.

At a minimum we should document this issue.

[david-waltermire]

According to our docs: https://pages.nist.gov/OSCAL/docs/schemas/datatypes/#specialized-character-mapping XML IN: <p>Markup can make <em>emphasis</em>, but so can *markdown*</p> Would map to JSON: "Markup can make *emphasis*, but so can \*markdown\*" This avoids the roundtrip problem. From: Wendell Piez <[email protected]> Sent: Tuesday, September 17, 2019 11:27 AM To: usnistgov/OSCAL <[email protected]> Cc: Waltermire, David A. (Fed) <[email protected]>; Mention <[email protected]> Subject: Re: [usnistgov/OSCAL] Improvements to HTML and Markdown character encoding in XML, JSON, and YAML (#452) Also: we need a plan for what to do (if anything) to prevent XML IN: <p>Markup can make <em>emphasis</em>, but so can *markdown*</p> JSON: "Markup can make *emphasis*, but so can *markdown*" XML BACK: <p>Markup can make <em>emphasis</em>, but so can <em>markdown</em></p> One solution would be to map * and other delimiter characters, in generated Markdown, to Unicode representations such as \u002a. So in this case the JSON would have "Markup can make *emphasis*, but so can \u002amarkdown\u002a". This is (much) less legible but more robust. Another solution could be to document this issue. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#452?email_source=notifications&email_token=ACLKE2AWLENCDLTZOJXB7YDQKDZLVA5CNFSM4H4HJ2AKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD645KDI#issuecomment-532272397>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ACLKE2FQW5ACTVCRTL53H7DQKDZLVANCNFSM4H4HJ2AA>.

[brian-ruf]

@david-waltermire-nist I think (hope) you are saying that symbols in one format (ie an “*” in XML content) that may be mistaken for formatting in the other format (ie emphasis in MD), should be automatically escaped by the conversion process.

I don’t think we can count on content creators/editors to handle this, nor do I think we want to impose that on them.

In other words, a content creator/editor who works in XML should not have to be concerned with how their data translates to other formats (JSON, YAML) beyond respecting the formatting constraints defined for prose and mixed elements. The conversion process should detect any problematic characters and handle them accordingly. This also future-proofs content if our allowed formatting rules change after the content is created.

[david-waltermire]

Correct. Content authors are only responsible for using the appropriate escaping in the format they are using the create the content. Escaping when converting between formats is handled by the conversion process. Our specification needs to define how this conversion must happen. From: Brian Ruf <[email protected]> Sent: Tuesday, September 17, 2019 11:40 AM To: usnistgov/OSCAL <[email protected]> Cc: Waltermire, David A. (Fed) <[email protected]>; Mention <[email protected]> Subject: Re: [usnistgov/OSCAL] Improvements to HTML and Markdown character encoding in XML, JSON, and YAML (#452) @david-waltermire-nist<https://github.com/david-waltermire-nist> I think (hope) you are saying that symbols in one format (ie an “*” in XML content) that may be mistaken for formatting in the other format (ie emphasis in MD), should be automatically escaped by the conversion process. I don’t think we can count on content creators/editors to handle this, nor do I think we want to impose that on them. In other words, a content creator/editor who works in XML should not have to be concerned with how their data translates to other formats (JSON, YAML) beyond respecting the formatting constraints defined for prose and mixed elements. The conversion process should detect any problematic characters and handle them accordingly. This also future-proofs content if our allowed formatting rules change after the content is created. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#452?email_source=notifications&email_token=ACLKE2FMM3L5YK24ME4C6T3QKD26ZA5CNFSM4H4HJ2AKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD646W6I#issuecomment-532278137>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ACLKE2B5MFMM25SFAQAH6YDQKD26ZANCNFSM4H4HJ2AA>.

[wendellpiez]

@brianrufgsa I think this is what Dave is saying ... in XML, a * is just a star (or asterisk if you prefer). In Markdown, where it might not be just a star, provision must be made for it. Presumably any \* becomes a * again coming back into XML.

(Thanks @david-waltermire-nist for posting that link.)

This means that XML authors won't have to do anything special for *, and JSON/Markdown authors have * if they want it (via \*) and also "tagging" for emphasis when they want it.

[wendellpiez]

@david-waltermire-nist I also noticed that page https://pages.nist.gov/OSCAL/docs/schemas/datatypes/#specialized-character-mapping erroneously references characters “ and ‘ (Unicode 8220 and 8216) rather than the keyboard characters " and ' (Unicode 0034 and 0039), which are undoubtedly what was intended.

Do you want me to correct this or is it being revised in your branch?

[wendellpiez]

Followup - I have managed to produce \* for * -- however it appears no JSON parser will parse this.

\* is the correct representation of * in Markdown, but \\* would be the correct representation of \* in JSON. So I am afraid we have to make * into \\*. In the text delivered by the JSON processor for conversion from Markdown into markup, this will be unescaped to \*.

We could make it into \u002a but that doesn't actually help, as it will look like a * again coming back into XML.

@david-waltermire-nist Advice? Maybe @anweiss also has some thoughts about representing Markdown inside JSON.

[david-waltermire]

Feel free to fix this. You can do so in our integration branch. From: Wendell Piez <[email protected]> Sent: Tuesday, September 17, 2019 12:59 PM To: usnistgov/OSCAL <[email protected]> Cc: Waltermire, David A. (Fed) <[email protected]>; Mention <[email protected]> Subject: Re: [usnistgov/OSCAL] Improvements to HTML and Markdown character encoding in XML, JSON, and YAML (#452) @david-waltermire-nist<https://github.com/david-waltermire-nist> I also noticed that page https://pages.nist.gov/OSCAL/docs/schemas/datatypes/#specialized-character-mapping erroneously references characters “ and ‘ (Unicode 8220 and 8216) rather than the keyboard characters " and ' (Unicode 0034 and 0039), which are undoubtedly what was intended. Do you want me to correct this or is it being revised in your branch? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#452?email_source=notifications&email_token=ACLKE2DP6KXKTPUWU5JLORDQKEEGPA5CNFSM4H4HJ2AKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD65GIBY#issuecomment-532308999>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ACLKE2C4SRSLGJGPEA46DYDQKEEGPANCNFSM4H4HJ2AA>.

[wendellpiez]

For ~ \^ * " we are now getting double-escaped Markdown-in-JSON, e.g. \\*. Amazingly, this is coming back into the XML as the naked character with no escaping. (The JSON-to-XML function converts it to \* and our Markdown handler knows this becomes *.

I have not yet edited the specification in docs\content\documentation\schemas\datatypes.md. This is because it turns out the Liquid parser is responsible for the conversion of characters responsible for the "smart quotes" in display. In the source they are correct.

The Liquid solution to this is to wrap in {% raw %} ... {% endraw %} directives. However, where exactly to do this raises a question of whether there are not other issues with the spec ... as Liquid is also converting our \\ into \. Accordingly I think I should correct this with the help of extra eyes.

[david-waltermire]

Yes. Getting the correct representations through Jekyll is a fresh kind of hell. You can see some examples of this in the current source markdown for the datatypes page. From: Wendell Piez <[email protected]> Sent: Tuesday, September 17, 2019 2:30 PM To: usnistgov/OSCAL <[email protected]> Cc: Waltermire, David A. (Fed) <[email protected]>; Mention <[email protected]> Subject: Re: [usnistgov/OSCAL] Improvements to HTML and Markdown character encoding in XML, JSON, and YAML (#452) For ~ \^ * " we are now getting double-escaped Markdown-in-JSON, e.g. \\*. Amazingly, this is coming back into the XML as the naked character with no escaping. (The JSON-to-XML function converts it to \* and our Markdown handler knows this becomes *. I have not yet edited the specification in docs\content\documentation\schemas\datatypes.md. This is because it turns out the Liquid parser is responsible for the conversion of characters responsible for the "smart quotes" in display. In the source they are correct. The Liquid solution to this is to wrap in {% raw %} ... {% endraw %} directives. However, where exactly to do this raises a question of whether there are not other issues with the spec ... as Liquid is also converting our \\ into \. Accordingly I think I should correct this with the help of extra eyes. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#452?email_source=notifications&email_token=ACLKE2ALSYFXYPCHWQZHBOLQKEOYVA5CNFSM4H4HJ2AKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD65O67I#issuecomment-532344701>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ACLKE2FO24AWLZSBNIXLF63QKEOYVANCNFSM4H4HJ2AA>.

[wendellpiez]

Okay.

Current status:

• The characters &, <, > and ' are now handled as stated in the Issue's requirements - no escaping, no disguising. This should address the issue giving rise to the Issue.
• "Magic" markdown-markup characters including * are being escaped (e.g. \*) when Markdown is produced from XML. For the JSON, the leading reverse solidus \ is then escaped again, so in JSON (not raw Markdown) they come out like \\*. These are able to make it back into XML unescaped.
• These are closer to the spec, but since the spec has issues of its own wrt its display of these characters, we should revisit the spec and test against it before putting this to bed.
• More testing is required around edge cases. As it happens, testing showed a number of small issues in handling Markdown fields vs regular (string) fields, and while these are fixed, there could be more.

[wendellpiez]

The latest commit 4e47cc2 to my branch has changes to the documentation regarding character escapes:

• The table listing required character escapes has been updated
• A sentence has been added at the end regarding escaping the LF character as \n in JSON

[david-waltermire]

This was completed in PR #492.