Problem deserializing IndexMap where keys are numeric

[phayes]

I have the following type: IndexMap<u8, String>

Serializing it works fine:

{
    "1": "76a04053bda0a88bda5177b86d7881f27eaeb5ce575901bb4",
    "2": "6a15c3b29f559873cb481232fe5417f143d66fda9d32f9807",
    "3": "299cd5743151ac4b2d63ae197f6cd879046f2856f44b9783c"
}

However, deserializing results in the following error:

Err value: Error("invalid type: string "1", expected u8", line: 107, column: 4)'

It looks like there might need to be a bit more logic around deserializing numeric keys.

[cuviper]

Does that work with a regular HashMap? AFAICS we do the same thing, letting MapAccess::next_entry deal with it.

[bluss]

This is a general serde issue, so for workarounds, I'd look around for the people who have already encountered this with other map types in serde.

We already provide serde_seq serialization for indexmap, which corresponds exactly to one of the common workarounds I find (serialize it as a sequence, not a map).

[bluss]

The following test passes for me - it deserializes the integer keys. Code for reproducing the reported issue is needed.

use serde_json;

#[test]
fn test_deser_map() {
    use indexmap::IndexMap;

    let m = serde_json::from_str::<IndexMap<u8, String>>(r#"
    {
        "1": "76a04053bda0a88bda5177b86d7881f27eaeb5ce575901bb4",
        "2": "6a15c3b29f559873cb481232fe5417f143d66fda9d32f9807",
        "3": "299cd5743151ac4b2d63ae197f6cd879046f2856f44b9783c"
    }
    "#);
    println!("{:?}", m);
    let m = m.unwrap();
    assert!(m.contains_key(&1));
    assert!(m.contains_key(&2));
    assert!(m.contains_key(&3));
}

[teohhanhui]

I have a reproducer, though it's not minimal in any way...

https://github.com/rust-malaysia/hyfetch/tree/3c355f54f21e8a7873583da34264f78a28965c65

Steps to reproduce:

1. Run the Python version of hyfetch to generate the config. Make sure to choose a custom / "random" color alignment. Choose to save the config.
2. cargo run -- --debug

You should get an error like this:

Error: Failed to read config

Caused by:
    0: Failed to parse "/home/teohhanhui/.config/hyfetch.json"
    1: color_align: invalid type: string "2", expected u8 at line 13 column 5

[cuviper]

Here's a reproducer from that, even after switching to HashMap:

use serde::Deserialize;
use std::collections::HashMap;

#[derive(Eq, PartialEq, Debug, Deserialize)]
#[serde(rename_all = "lowercase")]
enum ExternalEnum {
    Custom {
        colors: HashMap<u8, usize>,
    },
}

#[derive(Eq, PartialEq, Debug, Deserialize)]
#[serde(rename_all = "lowercase", tag = "mode")]
enum InternalEnum {
    Custom {
        colors: HashMap<u8, usize>,
    },
}

fn main() {
    serde_json::from_str::<ExternalEnum>(
        r#"{
            "custom": {
                "colors": {
                    "1": 123,
                    "2": 234
                }
            }
        }"#,
    )
    .expect("ExternalEnum");

    serde_json::from_str::<InternalEnum>(
        r#"{
            "mode": "custom",
            "colors": {
                "1": 123,
                "2": 234
            }
        }"#,
    )
    .expect("InternalEnum");
}

ExternalEnum passes, but InternalEnum fails:

thread 'main' panicked at src/main.rs:42:6:
InternalEnum: Error("invalid type: string \"1\", expected u8", line: 0, column: 0)

In cargo expand, they look pretty similar in how they call the HashMap deserialization, so I'm not sure what's going on, but I think this is clearly a serde issue.

[cuviper]

It might be serde-rs/json#560 -> serde-rs/serde#1183

[teohhanhui]

For anyone looking for a workaround:

pub(crate) mod index_map_serde {
    use std::fmt;
    use std::fmt::Display;
    use std::hash::Hash;
    use std::marker::PhantomData;
    use std::str::FromStr;

    use indexmap::IndexMap;
    use serde::de::{self, DeserializeSeed, MapAccess, Visitor};
    use serde::{Deserialize, Deserializer};

    pub(crate) fn deserialize<'de, D, K, V>(deserializer: D) -> Result<IndexMap<K, V>, D::Error>
    where
        D: Deserializer<'de>,
        K: Eq + Hash + FromStr,
        K::Err: Display,
        V: Deserialize<'de>,
    {
        struct KeySeed<K> {
            k: PhantomData<K>,
        }

        impl<'de, K> DeserializeSeed<'de> for KeySeed<K>
        where
            K: FromStr,
            K::Err: Display,
        {
            type Value = K;

            fn deserialize<D>(self, deserializer: D) -> Result<Self::Value, D::Error>
            where
                D: Deserializer<'de>,
            {
                deserializer.deserialize_str(self)
            }
        }

        impl<'de, K> Visitor<'de> for KeySeed<K>
        where
            K: FromStr,
            K::Err: Display,
        {
            type Value = K;

            fn expecting(&self, formatter: &mut fmt::Formatter) -> fmt::Result {
                formatter.write_str("a string")
            }

            fn visit_str<E>(self, s: &str) -> Result<Self::Value, E>
            where
                E: de::Error,
            {
                K::from_str(s).map_err(de::Error::custom)
            }
        }

        struct MapVisitor<K, V> {
            k: PhantomData<K>,
            v: PhantomData<V>,
        }

        impl<'de, K, V> Visitor<'de> for MapVisitor<K, V>
        where
            K: Eq + Hash + FromStr,
            K::Err: Display,
            V: Deserialize<'de>,
        {
            type Value = IndexMap<K, V>;

            fn expecting(&self, formatter: &mut fmt::Formatter) -> fmt::Result {
                formatter.write_str("a map")
            }

            fn visit_map<A>(self, mut input: A) -> Result<Self::Value, A::Error>
            where
                A: MapAccess<'de>,
            {
                let mut map = IndexMap::new();
                while let Some((k, v)) =
                    input.next_entry_seed(KeySeed { k: PhantomData }, PhantomData)?
                {
                    map.insert(k, v);
                }
                Ok(map)
            }
        }

        deserializer.deserialize_map(MapVisitor {
            k: PhantomData,
            v: PhantomData,
        })
    }
}

(Lightly adapted from serde-rs/json#560 (comment))

[jonasbb]

serde_with has support for indexmap v1 and v2. You can then annotate the problematic fields with this:

#[serde_as(deserialize_as = "IndexMap<serde_with::DisplayFromStr, _>")]