Namespace gets flattened if under a nested key

[carmocca]

🐛 Bug report

Traceback (most recent call last):
  File "/home/ubuntu/carlos/kk.py", line 18, in <module>
    args = parser.instantiate_classes(args)
  File "/home/ubuntu/.pyenv/versions/env/lib/python3.10/site-packages/jsonargparse/_deprecated.py", line 140, in patched_instantiate_classes
    cfg = self._unpatched_instantiate_classes(cfg, **kwargs)
  File "/home/ubuntu/.pyenv/versions/env/lib/python3.10/site-packages/jsonargparse/_core.py", line 1207, in instantiate_classes
    component.instantiate_class(component, cfg)
  File "/home/ubuntu/.pyenv/versions/env/lib/python3.10/site-packages/jsonargparse/_signatures.py", line 560, in group_instantiate_class
    parent[key] = instantiator_fn(group.group_class, **value)
  File "/home/ubuntu/.pyenv/versions/env/lib/python3.10/site-packages/jsonargparse/_common.py", line 155, in default_class_instantiator
    return class_type(*args, **kwargs)
TypeError: MyConfig.__init__() got an unexpected keyword argument 'optimizer.class_path'

To reproduce

from dataclasses import dataclass
from jsonargparse import ArgumentParser, Namespace
import torch

@dataclass
class Foo:
    f: int

@dataclass
class MyConfig:
    f: Foo
    optimizer: Namespace

parser = ArgumentParser()
parser.add_class_arguments(MyConfig, nested_key="my_config", skip={"optimizer"})
parser.add_subclass_arguments((torch.optim.Optimizer,), nested_key="my_config.optimizer", skip={"params"}, instantiate=False)
args = parser.parse_args(args=["--my_config.f.f=123", "--my_config.optimizer=torch.optim.Adam"])
args = parser.instantiate_classes(args)
print(args)

This works though (there is no nested key):

from dataclasses import dataclass
from jsonargparse import ArgumentParser, Namespace
import torch

@dataclass
class Foo:
    f: int

@dataclass
class MyConfig:
    f: Foo
    optimizer: Namespace

parser = ArgumentParser()
parser.add_class_arguments(MyConfig, skip={"optimizer"})
parser.add_subclass_arguments((torch.optim.Optimizer,), nested_key="optimizer", skip={"params"}, instantiate=False)
args = parser.parse_args(args=["--f.f=123", "--optimizer=torch.optim.Adam"])
args = parser.instantiate_classes(args)
print(args)

Expected behavior

Prints

Namespace(my_config=Namespace(f=Foo(f=123), optimizer=Namespace(class_path='torch.optim.Adam', init_args=Namespace(lr=0.001, betas=(0.9, 0.999), eps=1e-08, weight_decay=0.0, amsgrad=False, foreach=None, maximize=False, capturable=False, differentiable=False, fused=None))))

Environment

• jsonargparse version: 4.35.0
• Python version: 3.10.12
• How jsonargparse was installed: pip
• OS: Linux

Merry christmas

[mauvilsa]

It isn't intended to support Namespace as a type hint. And I wouldn't consider trying to add support for it. Internal logic depends on Namespace instances and trying to support Namespace as a type would restrict how things can be implemented internally and over-complicate things. Actually I think it might be a good idea to simply fail when an argument is added with Namespace as a type.

Why do you want to use Namespace as a type?

[carmocca]

How else should I annotate the type of the optimizer arguments? I dont know beforehand the optimizer type and its arguments. They depend on what the user passes for --my_config.optimizer. I also dont want the CLI to instantiate it for me because their instantiation requires some complex business logic that I want to keep outside the CLI.

Also, it's notable that this works well without a nested key. That's why I thought this was just an edge case

Thanks for your help!

[mauvilsa]

Type Callable[[Iterable], Optimizer] doesn't work? That provides a function to instantiate that you can use whenever. Not sure if it is possible to use that type and disable the conversion to a function. But still, in the parsed namespace you have all the details for instantiation.

[carmocca]

It doesn't. The error is the same.

TypeError: MyConfig.__init__() got an unexpected keyword argument 'optimizer.class_path'

Is this what you suggest?

from dataclasses import dataclass
from jsonargparse import ArgumentParser
import torch
from typing import Callable, Iterable
from torch.optim import Optimizer

@dataclass
class Foo:
    f: int

@dataclass
class MyConfig:
    f: Foo
    optimizer: Callable[[Iterable], Optimizer]

parser = ArgumentParser()
parser.add_class_arguments(MyConfig, nested_key="my_config", skip={"optimizer"})
parser.add_subclass_arguments((torch.optim.Optimizer,), nested_key="my_config.optimizer", skip={"params"}, instantiate=False)
args = parser.parse_args(args=["--my_config.f.f=123", "--my_config.optimizer=torch.optim.Adam"])
args = parser.instantiate_classes(args)
print(args)

I am mainly interested in the resulting namespace. Getting a function to instantiate is convenient but not blocking (I can get that myself). The error when the nested_key="my_config" is added is the main issue

[mauvilsa]

What I meant was

@dataclass
class Foo:
    f: int

@dataclass
class MyConfig:
    f: Foo
    optimizer: Callable[[Iterable], Optimizer]

parser.add_class_arguments(MyConfig, nested_key="my_config")
args = parser.parse_args(args=["--my_config.f.f=123", "--my_config.optimizer=torch.optim.Adam"])
print(args.my_config.optimizer)  # This is what you want
init = parser.instantiate_classes(args)
print(init.my_config.optimizer)  # This you don't want but can ignore

[carmocca]

I think this worked, but I hit another issue. The optimizer works but not my classes: #655

Closing this one. Thanks