Can I use Ellipses in types?

[ricosaurus]

Parameterizations are often expressed as a tuple of keys followed by a sequence of tuples of values in order to express a sequence of mapping of the keys to the values, eg I might write (('foo', 'bar'), ((1, 0.3), (5, 0.9)) to mean ({'foo': 1, 'bar': 0.3}, {'foo': 5, 'bar': 0.9}). I'm trying to do this using jsonargparse, but I am fairly ignorant with python's typing module and argument parsing in general.

This appears to be a valid type: Tuple[Tuple[str, str], Tuple[Tuple[int, float],...]] , as the Ellipsis operate on a 'single' type Tuple[int, float] in the Tuple. However, when I try the code (below) with either Ellipsis (commented out), or repeating two Tuple[int, float] elements, I get the same error (ValueError: not enough values to unpack (expected 3, got 2)). If I only have one nested level it works ok, eg Tuple[Tuple[str, str], Tuple[int, float], Tuple[int, float]] which is often ok, but as I understand it I can't use Ellipsis on a Tuple with elements of more than one type, and this flattened parameterization has two types, the keys and the values.

I really appreciate jsonargparse's ability to map dataclasses to argparser -- that's the functionality I need. I might well be overlooking something regarding argparse or python's typing module, or some of the other dependencies.

from jsonargparse import ArgumentParser, namespace_to_dict
from typing import Dict, Union, List, Tuple
from dataclasses import field, asdict, dataclass

@dataclass
class Schedule:
    #sched: Tuple[Tuple[str, str], Tuple[Tuple[int, float],...]]
    sched: Tuple[Tuple[str, str], Tuple[Tuple[int, float], Tuple[int, float]]]

parser = ArgumentParser()
parser.add_class_arguments(Schedule)

cmds = ["--sched", "[['foo', 'bar'], [[1, 2.02], [3, 3.09]]]"]
cfg = parser.parse_args(cmds)
print(cmds)
print(cfg)

myschedule = Schedule(**namespace_to_dict(cfg))
print(myschedule)

['--sched', "[['foo', 'bar'], [[1, 2.02], [3, 3.09]]]"]

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-112-b905a1e6539a> in <module>
     22 cmds = ["--sched", "[['foo', 'bar'], [[1, 2.02], [3, 3.09]]]"]
     23 print(cmds)
---> 24 cfg = parser.parse_args(cmds)
     25 print(cfg)
     26 

~/anaconda3/envs/py39/lib/python3.9/site-packages/jsonargparse/core.py in parse_args(self, args, namespace, env, defaults, nested, with_meta, _skip_check)
    319         try:
    320             with _suppress_stderr():
--> 321                 cfg, unk = self.parse_known_args(args=args)
    322                 if unk:
    323                     self.error('Unrecognized arguments: %s' % ' '.join(unk))

~/anaconda3/envs/py39/lib/python3.9/site-packages/jsonargparse/core.py in parse_known_args(self, args, namespace)
    201 
    202         try:
--> 203             namespace, args = self._parse_known_args(args, namespace)
    204             if len(args) > 0:
    205                 for action in self._actions:

~/anaconda3/envs/py39/lib/python3.9/argparse.py in _parse_known_args(self, arg_strings, namespace)
   2058 
   2059             # consume the next optional and any arguments for it
-> 2060             start_index = consume_optional(start_index)
   2061 
   2062         # consume any positionals following the last Optional

~/anaconda3/envs/py39/lib/python3.9/argparse.py in consume_optional(start_index)
   1998             assert action_tuples
   1999             for action, args, option_string in action_tuples:
-> 2000                 take_action(action, args, option_string)
   2001             return stop
   2002 

~/anaconda3/envs/py39/lib/python3.9/argparse.py in take_action(action, argument_strings, option_string)
   1926             # (e.g. from a default)
   1927             if argument_values is not SUPPRESS:
-> 1928                 action(self, namespace, argument_values, option_string)
   1929 
   1930         # function to convert arg_strings into an optional action

~/anaconda3/envs/py39/lib/python3.9/site-packages/jsonargparse/jsonschema.py in __call__(self, *args, **kwargs)
    126                 kwargs['help'] = kwargs['help'] % json.dumps(self._validator.schema, sort_keys=True)
    127             return ActionJsonSchema(**kwargs)
--> 128         val = self._check_type(args[2])
    129         if not self._with_meta:
    130             val = strip_meta(val)

~/anaconda3/envs/py39/lib/python3.9/site-packages/jsonargparse/jsonschema.py in _check_type(self, value, cfg)
    141                 if isinstance(val, Namespace):
    142                     val = namespace_to_dict(val)
--> 143                 val = self._adapt_types(val, self._annotation, self._subschemas, reverse=True)
    144                 path_meta = val.pop('__path__') if isinstance(val, dict) and '__path__' in val else None
    145                 self._validator.validate(val)

~/anaconda3/envs/py39/lib/python3.9/site-packages/jsonargparse/jsonschema.py in _adapt_types(val, annotation, subschemas, reverse, instantiate_classes)
    243                 if n < len(subschemas) and subschemas[n] is not None:
    244                     for subschema in subschemas[n]:
--> 245                         val[n] = validate_adapt(v, subschema)
    246             if not reverse:
    247                 val = tuple(val) if annotation.__origin__ in {Tuple, tuple} else set(val)

~/anaconda3/envs/py39/lib/python3.9/site-packages/jsonargparse/jsonschema.py in validate_adapt(v, subschema)
    184         def validate_adapt(v, subschema):
    185             if subschema is not None:
--> 186                 subannotation, subvalidator, subsubschemas = subschema
    187                 if reverse:
    188                     v = ActionJsonSchema._adapt_types(v, subannotation, subsubschemas, reverse, instantiate_classes)

ValueError: not enough values to unpack (expected 3, got 2)

[mauvilsa]

When I was implementing the type hint support I decided to leave ellipses for a future improvement since it wasn't required for me at that moment. From what I see I forgot to mention this in the documentation. But certainly this can be added.

[ricosaurus]

Pretty incredible amount that you did incorporate. Ellipses aren't a huge deal as far as I am concerned, but I am a bit puzzled re: why nesting tuples like this doesn't work for me. Any thoughts?

Tuple[Tuple[str, str], Tuple[Tuple[int, float], Tuple[int, float]]]

I get the ValueError: not enough values to unpack (expected 3, got 2) -- shown above.

[mauvilsa]

I think I have never tried tuples inside tuples so very likely this can be a bug. Need to look at it in more detail.

[mauvilsa]

Indeed nested tuples is not working correctly. And not just because of the error you are getting. For instance Tuple[Tuple[str, str], Tuple[int, float], Tuple[int, float]] does not fail. However, the result is lists inside a tuple (['foo', 'bar'], [1, 2.02], [3, 3.09]) but should be tuples inside a tuple.

[mauvilsa]

Both the support for ellipsis and the correct handling of nested tuples is fixed in the latest release v3.5.0. Please check.

[ricosaurus]

So far so good -- testing nested tuples and with ellipsis. Thanks much!