Either Moto use of XMLtodict or xmltodict 0.13.0 is broken.

[potiuk]

Either Moto use of XMLtodict or xmltodict 0.13.0 is broken.

We have a sophisticated test harness in Airlfow where we run our tests for latest releases of our dependencies and it caught a strange eror.

I investigated it and traced it to xmltodict 0.13.0 upgrade.

I am not 100% sure whether it's the xmltodict or the way how boto/moto uses it but wanted to give an early warning in case you can investigate it faster.

The problem is that we are starting to ge "ValueError Malformatted input" when 0.13.0 is installed in some of our Amazon EMR tests:

tests/providers/amazon/aws/hooks/test_emr.py::TestEmrHook::test_create_job_flow_extra_args: ValueError: Malformatted input
tests/providers/amazon/aws/hooks/test_emr.py::TestEmrHook::test_create_job_flow_uses_the_emr_config_to_create_a_cluster: ValueError: Malformatted input
tests/providers/amazon/aws/hooks/test_emr.py::TestEmrHook::test_get_cluster_id_by_name: ValueError: Malformatted input

The error is thrown in moto in this part of the code:

         od = OrderedDict()
          for k, v in value.items():
              if k.startswith("@"):
                  continue
      
              if k not in spec:
                  # this can happen when with an older version of
                  # botocore for which the node in XML template is not
                  # defined in service spec.
                  log.warning("Field %s is not defined by the botocore version in use", k)
                  continue
      
              if spec[k]["type"] == "list":
                  if v is None:
                      od[k] = []
                  elif len(spec[k]["member"]) == 1:
                      if isinstance(v["member"], list):
                          od[k] = transform(v["member"], spec[k]["member"])
                      else:
                          od[k] = [transform(v["member"], spec[k]["member"])]
                  elif isinstance(v["member"], list):
                      od[k] = [transform(o, spec[k]["member"]) for o in v["member"]]
                  elif isinstance(v["member"], OrderedDict):
                      od[k] = [transform(v["member"], spec[k]["member"])]
                  else:
  >                   raise ValueError("Malformatted input")
  E                   ValueError: Malformatted input

It's rather easy to reproduce:

1. checkout airflow from https://github.com/apache/airflow
2. install breeze (pipx install -e ./dev/breeze) - airflow development environment
3. run breeze - this will pull and build the docker image that has all necessary deps
4. run pip freeze | grep xmltodict - see that 0.12.0 version is installed
5. once in the container run pytest tests/providers/amazon/aws/hooks/test_emr.py -> they should succed
6. upgrade xmltodoc : pip install xmltodict==0.13.0
7. run pytest tests/providers/amazon/aws/hooks/test_emr.py -> they shoudl fail with the "Malformed input"

Once again 0 I am not sure if the root cause is moto/boto use of xmltodic or xmltodict itself, but I decided to open issues in both - maybe you can investigate and fix the root cause quicker.

Related: apache/airflow#23576
Related: martinblech/xmltodict#297

[bblommers]

Hi @potiuk! Our CI was failing last night because of this as well - I tracked it down to a breaking change in the xmltodict release.
We fixed this on our end in #5111. Moto >= 3.1.9.dev3 should be fully compatible with xmltodict 0.13.0.

[potiuk]

Hi @potiuk! Our CI was failing last night because of this as well - I tracked it down to a breaking change in the xmltodict release. We fixed this on our end in #5111. Moto >= 3.1.9.dev3 should be fully compatible with xmltodict 0.13.0.

Thanks. Cool. In the meantime we tempoarary pinned xmltodict to 0.12.0. Our CI will automatically bump to to 3.1.9 moto when it gets released and when it will., I will remove the pinning. Thanks a lot @bblommers !