RootModel[str] cannot be parsed/requires double quotation.

[user1584]

I am trying to create an extension of standard str class. My idea was to just create a RootModel based on str and then add some custom logic. However, I encountered strange problems when using the resulting class as part of the settings.
My RootModel looks like this (without custom logic):

import pydantic

class CustomStr(pydantic.RootModel[str]):
    root: str

Initializing with CustomStr("hello world") works as expected. The next step was to use it as part of the settings:

import pydantic_settings
import os

os.environ["custom_str"] = 'hello world'

class Settings(pydantic_settings.BaseSettings):
    custom_str: CustomStr

Settings()

This results in 'SettingsError: error parsing value for field "custom_str" from source "EnvSettingsSource"' (see complete traceback below).
If I switch the type of custom_str from CustomStr to standard str, the example works. During lots of debugging, I noticed that it works if I wrap the string in the environment variable in double quotes like os.environ["custom_str"] = '"hello world"' since this can be parsed by json.loads. Note: The order of single and double quotes matters!
So, the question is: Is there any way to let pydantic/pydantic_settings handle the custom model like regular str class?

I am using pydantic version '2.7.1' and pydantic_settings version '2.2.1'.

Here's the complete traceback:

---------------------------------------------------------------------------
JSONDecodeError                           Traceback (most recent call last)
File /home/work/.venv/lib/python3.9/site-packages/pydantic_settings/sources.py:323, in PydanticBaseEnvSettingsSource.__call__(self)
    322 try:
--> 323     field_value = self.prepare_field_value(field_name, field, field_value, value_is_complex)
    324 except ValueError as e:

File /home/work/.venv/lib/python3.9/site-packages/pydantic_settings/sources.py:513, in EnvSettingsSource.prepare_field_value(self, field_name, field, value, value_is_complex)
    512     if not allow_parse_failure:
--> 513         raise e
    515 if isinstance(value, dict):

File /home/work/.venv/lib/python3.9/site-packages/pydantic_settings/sources.py:510, in EnvSettingsSource.prepare_field_value(self, field_name, field, value, value_is_complex)
    509 try:
--> 510     value = self.decode_complex_value(field_name, field, value)
    511 except ValueError as e:

File /home/work/.venv/lib/python3.9/site-packages/pydantic_settings/sources.py:147, in PydanticBaseSettingsSource.decode_complex_value(self, field_name, field, value)
    136 """
    137 Decode the value for a complex field
    138 
   (...)
    145     The decoded value for further preparation
    146 """
--> 147 return json.loads(value)

File /home/work/.venv/lib/python3.9/json/__init__.py:346, in loads(s, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw)
    343 if (cls is None and object_hook is None and
    344         parse_int is None and parse_float is None and
    345         parse_constant is None and object_pairs_hook is None and not kw):
--> 346     return _default_decoder.decode(s)
    347 if cls is None:

File /home/work/.venv/lib/python3.9/json/decoder.py:337, in JSONDecoder.decode(self, s, _w)
    333 """Return the Python representation of ``s`` (a ``str`` instance
    334 containing a JSON document).
    335 
    336 """
--> 337 obj, end = self.raw_decode(s, idx=_w(s, 0).end())
    338 end = _w(s, end).end()

File /home/work/.venv/lib/python3.9/json/decoder.py:355, in JSONDecoder.raw_decode(self, s, idx)
    354 except StopIteration as err:
--> 355     raise JSONDecodeError("Expecting value", s, err.value) from None
    356 return obj, end

JSONDecodeError: Expecting value: line 1 column 1 (char 0)

The above exception was the direct cause of the following exception:

SettingsError                             Traceback (most recent call last)
Cell In[10], line 10
      7     model_config = pydantic_settings.SettingsConfigDict(env_nested_delimiter="__", arbitrary_types_allowed=True)
      8     custom_str: CustomStr
---> 10 Settings()

File /home/work/.venv/lib/python3.9/site-packages/pydantic_settings/main.py:85, in BaseSettings.__init__(__pydantic_self__, _case_sensitive, _env_prefix, _env_file, _env_file_encoding, _env_ignore_empty, _env_nested_delimiter, _env_parse_none_str, _secrets_dir, **values)
     71 def __init__(
     72     __pydantic_self__,
     73     _case_sensitive: bool | None = None,
   (...)
     82 ) -> None:
     83     # Uses something other than `self` the first arg to allow "self" as a settable attribute
     84     super().__init__(
---> 85         **__pydantic_self__._settings_build_values(
     86             values,
     87             _case_sensitive=_case_sensitive,
     88             _env_prefix=_env_prefix,
     89             _env_file=_env_file,
     90             _env_file_encoding=_env_file_encoding,
     91             _env_ignore_empty=_env_ignore_empty,
     92             _env_nested_delimiter=_env_nested_delimiter,
     93             _env_parse_none_str=_env_parse_none_str,
     94             _secrets_dir=_secrets_dir,
     95         )
     96     )

File /home/work/.venv/lib/python3.9/site-packages/pydantic_settings/main.py:187, in BaseSettings._settings_build_values(self, init_kwargs, _case_sensitive, _env_prefix, _env_file, _env_file_encoding, _env_ignore_empty, _env_nested_delimiter, _env_parse_none_str, _secrets_dir)
    179 sources = self.settings_customise_sources(
    180     self.__class__,
    181     init_settings=init_settings,
   (...)
    184     file_secret_settings=file_secret_settings,
    185 )
    186 if sources:
--> 187     return deep_update(*reversed([source() for source in sources]))
    188 else:
    189     # no one should mean to do this, but I think returning an empty dict is marginally preferable
    190     # to an informative error and much better than a confusing error
    191     return {}

File /home/work/.venv/lib/python3.9/site-packages/pydantic_settings/main.py:187, in <listcomp>(.0)
    179 sources = self.settings_customise_sources(
    180     self.__class__,
    181     init_settings=init_settings,
   (...)
    184     file_secret_settings=file_secret_settings,
    185 )
    186 if sources:
--> 187     return deep_update(*reversed([source() for source in sources]))
    188 else:
    189     # no one should mean to do this, but I think returning an empty dict is marginally preferable
    190     # to an informative error and much better than a confusing error
    191     return {}

File /home/work/.venv/lib/python3.9/site-packages/pydantic_settings/sources.py:325, in PydanticBaseEnvSettingsSource.__call__(self)
    323     field_value = self.prepare_field_value(field_name, field, field_value, value_is_complex)
    324 except ValueError as e:
--> 325     raise SettingsError(
    326         f'error parsing value for field "{field_name}" from source "{self.__class__.__name__}"'
    327     ) from e
    329 if field_value is not None:
    330     if self.env_parse_none_str is not None:

SettingsError: error parsing value for field "custom_str" from source "EnvSettingsSource"

[hramezani]

Thanks @user1584 for reporting this issue.

pydantic-settings considers custom_str: CustomStr as a complex field and that's why it parses the value by json.loads.

Right now, it isn't possible to disable the json parsing. probably we need a flag to disable it.

take a look at the similar issue and my comment on it.

[user1584]

Thanks for the incredibly quick answer!
How/where does pydantic-settings decide whether a type is a complex field or simple enough to pass the content of the environment variable directly? Is this something that could be adapted?
A flag might also a good option. Could that be set per field?

[hramezani]

How/where does pydantic-settings decide whether a type is a complex field or simple enough to pass the content of the environment variable directly? Is this something that could be adapted?

It happens here. probably you can create a custom settings source inherited from EnvSettingsSource and change the behaviour for this field.

A flag might also a good option. Could that be set per field?

Not sure, because the Field is imported from Pydantic. agree that is would be good to have control over this behaviour per field

[user1584]

Obviously I am mostly interested to solve my own little problem and I have next to no idea about the inner workings of pydantic/pydantic-settings. With that in mind, would it be viable solution to check whether the RootModels' attribute type is complex?
I monkey-patched the field_is_complex method like this:

from pydantic.fields import FieldInfo
import pydantic_settings

def patched_field_is_complex(self, field: FieldInfo) -> bool:
    """
    Checks whether a field is complex, in which case it will attempt to be parsed as JSON.

    Args:
        field: The field.

    Returns:
        Whether the field is complex.
    """
    if issubclass(field.annotation, pydantic.RootModel):
        return pydantic_settings.sources._annotation_is_complex(field.annotation.__annotations__["root"], field.metadata)
    return pydantic_settings.sources._annotation_is_complex(field.annotation, field.metadata)

pydantic_settings.PydanticBaseSettingsSource.field_is_complex=patched_field_is_complex

Of course, this would only be a solution for models that are based on RootModel.

[hramezani]

seems good. you can open a PR and change the behavior in pydantic-settings if you want. you probably need to change _annotation_is_complex and add some test for it.

[user1584]

I created a PR. However, the python 3.8 & 3.9 tests still fail. Do you have an idea why they fail but all other tests work?

[hramezani]

Fixed in 8fb9abb