InvalidDigest if there is a "xmlns" namespace declaration in the root element

[ralffunk]

I need to sign a XML file.
The signature process works in both cases (with and without the namespace declared in the root element), but the signature verification fails when there is the specific xmlns namespace declared.

Also, when I have less nested levels, then there is no error in both cases, but I need the nesting (the final XML file will have even more nested levels than this example).

The XML files need to be submitted to a government agency (Country Paraguay, SIFEN / EKUATIA, Sistema Nacional de Facturacion Electrónica) and the validation fails if I exclude the namespace from the file.

Here is my code to replicate the error (excluding my key files); I am using Ubuntu 20.04, Python 3.8.10, lxml==4.9.2 and signxml==2.10.1. (I also tried with the newer signxml==3.1.0 and got the same error; but I need to use the older version of signxml because it supports the older Python 3.6 which some of my machines still use).

def test(with_ns):
    print('### parsing XML from string')
    ns = ''
    if with_ns:
        ns = 'xmlns="http://ekuatia.set.gov.py/sifen/xsd"'
    cdc = '0123'*11
    root = etree.fromstring(
        '<rDE {} xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://ekuatia.set.gov.py/sifen/xsd/siRecepDE_v150.xsd">'
            '<dVerFor>1</dVerFor>'
            '<DE Id="{}">'
                '<dDVId>9</dDVId>'
                '<gOpeDE>'
                    '<iTipEmi>1</iTipEmi>'
                '</gOpeDE>'
            '</DE>'
        '</rDE>'.format(ns, cdc))
    output_bytes = etree.tostring(root, pretty_print=True)
    print(output_bytes.decode())

    print()
    print('### signing XML')
    with open(os.path.join(DIR_PATH, 'tb_pub_key.pem')) as f:
        pub_key = f.read()
    with open(os.path.join(DIR_PATH, 'tb_priv_key.pem')) as f:
        priv_key = f.read()
    signer = signxml.XMLSigner(
        c14n_algorithm='http://www.w3.org/TR/2001/REC-xml-c14n-20010315')
    signer.namespaces = {
        None: signxml.namespaces.ds,
    }
    root = signer.sign(
        root,
        key=priv_key,
        cert=pub_key,
        reference_uri=cdc)
    output_bytes = etree.tostring(root, pretty_print=True)
    print(output_bytes.decode())

    print()
    print('### verifying signature')
    result = signxml.XMLVerifier().verify(
        root,
        ca_pem_file=os.path.join(DIR_PATH, 'bundle-doc-mic.pem'))
    output_bytes = etree.tostring(result.signed_xml, pretty_print=True)
    print(output_bytes.decode())

The code runs successfully without the namespace:

>>> test(with_ns=False)
### parsing XML from string
<rDE xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://ekuatia.set.gov.py/sifen/xsd/siRecepDE_v150.xsd">
  <dVerFor>1</dVerFor>
  <DE Id="01230123012301230123012301230123012301230123">
    <dDVId>9</dDVId>
    <gOpeDE>
      <iTipEmi>1</iTipEmi>
    </gOpeDE>
  </DE>
</rDE>
### signing XML
<rDE xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://ekuatia.set.gov.py/sifen/xsd/siRecepDE_v150.xsd">
  <dVerFor>1</dVerFor>
  <DE Id="01230123012301230123012301230123012301230123">
    <dDVId>9</dDVId>
    <gOpeDE>
      <iTipEmi>1</iTipEmi>
    </gOpeDE>
  </DE>
  <Signature xmlns="http://www.w3.org/2000/09/xmldsig#">
    <SignedInfo>
      <CanonicalizationMethod Algorithm="http://www.w3.org/TR/2001/REC-xml-c14n-20010315"/>
      <SignatureMethod Algorithm="http://www.w3.org/2001/04/xmldsig-more#rsa-sha256"/>
      <Reference URI="#01230123012301230123012301230123012301230123">
        <Transforms>
          <Transform Algorithm="http://www.w3.org/2000/09/xmldsig#enveloped-signature"/>
          <Transform Algorithm="http://www.w3.org/TR/2001/REC-xml-c14n-20010315"/>
        </Transforms>
        <DigestMethod Algorithm="http://www.w3.org/2001/04/xmlenc#sha256"/>
        <DigestValue>kKbZeA/kIplpYlTuynW8SsjEovYlkDW4JdiST9qH6s8=</DigestValue>
      </Reference>
    </SignedInfo>
    <SignatureValue>...</SignatureValue>
    <KeyInfo>
      <X509Data>
        <X509Certificate>...</X509Certificate>
      </X509Data>
    </KeyInfo>
  </Signature>
</rDE>
### verifying signature
<DE xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" Id="01230123012301230123012301230123012301230123">
  <dDVId>9</dDVId>
  <gOpeDE>
    <iTipEmi>1</iTipEmi>
  </gOpeDE>
</DE>

But when adding the namespace, I get this error:

>>> test(with_ns=True)
### parsing XML from string
<rDE xmlns="http://ekuatia.set.gov.py/sifen/xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://ekuatia.set.gov.py/sifen/xsd/siRecepDE_v150.xsd">
  <dVerFor>1</dVerFor>
  <DE Id="01230123012301230123012301230123012301230123">
    <dDVId>9</dDVId>
    <gOpeDE>
      <iTipEmi>1</iTipEmi>
    </gOpeDE>
  </DE>
</rDE>
### signing XML
<rDE xmlns="http://ekuatia.set.gov.py/sifen/xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://ekuatia.set.gov.py/sifen/xsd/siRecepDE_v150.xsd">
  <dVerFor>1</dVerFor>
  <DE Id="01230123012301230123012301230123012301230123">
    <dDVId>9</dDVId>
    <gOpeDE>
      <iTipEmi>1</iTipEmi>
    </gOpeDE>
  </DE>
  <Signature xmlns="http://www.w3.org/2000/09/xmldsig#">
    <SignedInfo>
      <CanonicalizationMethod Algorithm="http://www.w3.org/TR/2001/REC-xml-c14n-20010315"/>
      <SignatureMethod Algorithm="http://www.w3.org/2001/04/xmldsig-more#rsa-sha256"/>
      <Reference URI="#01230123012301230123012301230123012301230123">
        <Transforms>
          <Transform Algorithm="http://www.w3.org/2000/09/xmldsig#enveloped-signature"/>
          <Transform Algorithm="http://www.w3.org/TR/2001/REC-xml-c14n-20010315"/>
        </Transforms>
        <DigestMethod Algorithm="http://www.w3.org/2001/04/xmlenc#sha256"/>
        <DigestValue>1PsVs9lH4Yx0i/2ler3MaIUFP4R9yhTm6uksVkh72Sk=</DigestValue>
      </Reference>
    </SignedInfo>
    <SignatureValue>...</SignatureValue>
    <KeyInfo>
      <X509Data>
        <X509Certificate>...</X509Certificate>
      </X509Data>
    </KeyInfo>
  </Signature>
</rDE>
### verifying signature
Traceback (most recent call last):
  File "/snap/pycharm-professional/331/plugins/python/helpers/pydev/pydevconsole.py", line 364, in runcode
    coro = func()
  File "<input>", line 1, in <module>
  File "/home/ralf/PycharmProjects/tb_system_01/base/sifen/base.py", line 111, in test_2
    result = signxml.XMLVerifier().verify(
  File "/home/ralf/PycharmProjects/tb_system_01/.venv/lib/python3.8/site-packages/signxml/__init__.py", line 981, in verify
    raise InvalidDigest("Digest mismatch for reference {}".format(len(verify_results)))
signxml.exceptions.InvalidDigest: Digest mismatch for reference 0

Any ideas why this could be happening? Or how this could be solved?

[ralffunk]

I did ask in Stackoverflow (https://stackoverflow.com/questions/75673679/python-signxml-invaliddigest-if-there-is-a-xmlns-namespace-declaration-in-th), but got no response.

[kislyuk]

Thanks for reporting this, you have identified a bug in how we handle canonicalization. When declaring a default namespace in the root element and then canonicalizing sub-elements of the document, we were detaching the sub-elements before canonicalizing when signing, but not when verifying. This has a subtle effect on the canonicalization algorithm when a default (unprefixed) namespace is used, which manifests as the presence or absence of xmlns="" declarations.

I've opened #226 to research the underlying question but I think what we're doing when signing is correct (and we have now fixed the verifying behavior to trust the signature in this situation).

Released in v3.2.0