Regression of processing characters within < >

[ronaldtse]

I have a document that contains text like this: <星>. These were rendered correctly (with asciidoctor 1.5.7.1, metanorma-iso 1.0.6) before.

When I did a bundle update (asciidoctor 1.5.8, metanorma-iso 1.0.11), all of this text disappeared.

This does not seem to be an asciidoctor problem because there was still no output when I manually required the older asciidoctor version (1.5.7.1).

This text <星> is rendered as completely missing in the XML.

[opoudjis]

This may be the smart quote cleaner from https://github.com/pbhogan/sterile (which I've just introduced) being overactive with angle brackets or Chinese text. I'm assuming Chinese text itself renders fine?

[opoudjis]

I've replicated it, will advise.

[ronaldtse]

Thanks!

[opoudjis]

Yes, sterile's smart formatter is converting < back to < (although I can't replicate it in isolation); I'll entity-encode its output, if I have to. I'm at work, so it won't be immediate.

[ronaldtse]

The < character is fine with me, it's just that the text doesn't show up anymore?

[opoudjis]

Yes, it's interpreting the <星> as markup! I will fix it, but this is my last day at work, and ppl are baying for my time. I will fix it tonight.

[ronaldtse]

Holidays always compress stress, right before them 😉

[opoudjis]

This is a problem in Nokogiri: whenever you traverse all nodes in an XML document, the < and > in any text nodes are presented as <, >. &, on the other hand, is left alone. I am ensuring that the four instances where text nodes are manipulated through nokogiri traverse (two here, two in metanorma-gb) convert their <, > back to <, >