Empty result for HTML entities processing in yaml2text

[Intelligent2013]

Source issue: https://github.com/metanorma/pdfa-iso-32000-2/issues/4#issuecomment-1804659727

If the yaml file contains HTML entities like Α, then the Presentation XML doesn't contain any character.

In the sample document document.zip:

• 1st row encoded directly without yaml2text - renders ok
• 2nd row encoded with using of yaml2text, the entities Α and ↔ are using - empty result
• 3rd row encoded with using of yaml2text, the entities Α and ↔ are using - renders ok

| Α | Alpha | 101 | ↔ | arrowboth | 253

{% for row in table.rows %}
{% for character in row %}
| {{ character.entity }} | {{ character.name }} | {{ character.code -}}
{% endfor %}
{% endfor %}

table.yaml:

---
id: D.5
title: Symbol set and encoding
rows:
  - - name: Alpha
      entity: "Α"
      code: 101
    - name: arrowboth
      entity: "↔"
      code: 253

  - - name: Alpha
      entity: "Α"
      code: 101
    - name: arrowboth
      entity: "↔"
      code: 253

The output Presentation XML:

<tbody>
	<tr>
		<td valign="top" align="left">Α</td>
		<td valign="top" align="left">Alpha</td>
		<td valign="top" align="left">101</td>
		<td valign="top" align="left">↔</td>
		<td valign="top" align="left">arrowboth</td>
		<td valign="top" align="left">253</td>
	</tr>
	<tr>
		<td valign="top" align="left"/>
		<td valign="top" align="left">Alpha</td>
		<td valign="top" align="left">101</td>
		<td valign="top" align="left"/>
		<td valign="top" align="left">arrowboth</td>
		<td valign="top" align="left">253</td>
	</tr>
	<tr>
		<td valign="top" align="left">Α</td>
		<td valign="top" align="left">Alpha</td>
		<td valign="top" align="left">101</td>
		<td valign="top" align="left">↔</td>
		<td valign="top" align="left">arrowboth</td>
		<td valign="top" align="left">253</td>
	</tr>
</tbody>

Resulted PDF:

[opoudjis]

Ugh.

The preprocessing that converts HTML entities into XML entities is likely happening before the preprocessing that reads in these YAML files. If so, moving it later will address this.

[opoudjis]

Sure enough:

        preprocessor Metanorma::Standoc::EmbedIncludeProcessor
        preprocessor Metanorma::Standoc::NamedEscapePreprocessor
        preprocessor Metanorma::Standoc::LinkProtectPreprocessor
        preprocessor Metanorma::Standoc::Datamodel::AttributesTablePreprocessor
        preprocessor Metanorma::Standoc::Datamodel::DiagramPreprocessor
        preprocessor Metanorma::Plugin::Datastruct::Json2TextPreprocessor
        preprocessor Metanorma::Plugin::Datastruct::Yaml2TextPreprocessor
        preprocessor Metanorma::Plugin::Glossarist::DatasetPreprocessor

[opoudjis]

I can't test this, so I'm asking you to, @Intelligent2013

[Intelligent2013]

@opoudjis it's working as expected now. Thank you!