Add support for MissingMemberHandling to System.Text.Json

[SigmundVik]

We would love to migrate our ASP.NET Core 3 application from using Newtonsoft.Json to System.Text.Json. However, the lack of support/workaround for MissingMemberHandling is a showstopper, because we need a robust way to detect and report deficient [FromBody] model binding. Without this feature, mistakes in the input JSON document can go undetected, and that is not acceptable.

Is adding MissingMemberHandling something that is planned for System.Text.Json?

EDIT @eiriktsarpalis: API proposal & usage examples added

API Proposal

namespace System.Text.Json;

public enum JsonMissingMemberHandling { Ignore = 0, Error = 1 }

public partial class JsonSerializerOptions
{
     // Global configuration
     public JsonMissingMemberHandling MissingMemberHandling { get; set; } = JsonMissingMemberHandling.Ignore;
}

namespace System.Text.Json.Serialization;

// Per-type attribute-level configuration
[AttributeUsageAttribute(AttributeTargets.Class | AttributeTargets.Interface | AttributeTargets.Struct, AllowMultiple=false, Inherited=false)]
public class JsonMissingMemberHandlingAttribute : JsonAttribute
{
       public JsonMissingMemberHandlingAttribute(JsonMissingMemberHandling missingMemberHandling);
       public JsonMissingMemberHandlingAttribute MissingMemberHandling { get; }
}

namespace System.Text.Json.Serialization.Metadata;

public partial class JsonTypeInfo
{
      // Per-type configuration via contract customization.
      public JsonMissingMemberHandling MissingMemberHandling { get; set; } = JsonMissingMemberHandling.Ignore;
}

API Usage

JsonSerializer.Deserialize<MyPoco>("""{ "jsonPropertyNotBindingToPocoProperty" : null}"""); // JsonException: member not found

[JsonMissingMemberHandling(JsonMissingMember.Error)]
public class MyPoco { }

[layomia]

Triage: this can be considered.

Removing api-suggestion as there's no API proposal here.

[eiriktsarpalis]

FWIW the documentation does propose using JsonExtensionDataAttribute as a workaround to achieve this. Granted this would require running validation post-deserialization.

[ghost]

This issue has been marked with the api-needs-work label. This may suggest that the proposal requires further refinement before it can be considered for API review. Please refer to our API review guidelines for a detailed description of the process.

When ready to submit an amended proposal, please ensure that the original post in this issue has been updated, following the API proposal template and examples as provided in the guidelines.

[LetMeSleepAlready]

I just checked the of the json serializer. Maybe it could be as simple as the following (but then again, maybe not)

System.Text.Json.JsonSerializer.LookupProperty

if (jsonPropertyInfo == JsonPropertyInfo.s_missingProperty)
            {
                JsonPropertyInfo? dataExtProperty = state.Current.JsonTypeInfo.DataExtensionProperty;
                if (dataExtProperty != null && dataExtProperty.HasGetter && dataExtProperty.HasSetter)
                {
                    state.Current.JsonPropertyNameAsString = JsonHelpers.Utf8GetString(unescapedPropertyName);
                    if (createExtensionProperty)
                    {
                        CreateDataExtensionProperty(obj, dataExtProperty, options);
                    }
                    jsonPropertyInfo = dataExtProperty;
                    useExtensionProperty = true;
                }
// snippet goes here
            }

add the following snippet:

 else if (options.ThrowOnMissingMember) {
 ThrowHelper.ThrowJsonException_DeserializeUnableToBindMember();
}

Of course this would require an new ThrowHelper, and a new property in JsonSerializerOptions (ThrowOnMissingMember, default false = current way of working), both which are minor additions.

I figured the above might at least get this ticket going in some direction.

[eiriktsarpalis]

Here's a potential workaround using .NET 6 features:

using System.Collections.Generic;
using System.Text.Json;
using System.Text.Json.Serialization;

JsonSerializer.Deserialize<MyPoco>(@"{ ""X"" : 42, ""Y"" : -1 }");
// Unhandled exception. System.Text.Json.JsonException: Contains extra properties: Y.

public class MyPoco : IJsonOnDeserialized
{
    public int X { get; set; }

    [JsonExtensionData]
    public IDictionary<string, object>? ExtraProperties { get; set; }

    void IJsonOnDeserialized.OnDeserialized()
    {
        if (ExtraProperties?.Count > 0)
        {
            throw new JsonException($"Contains extra properties: {string.Join(", ", ExtraProperties.Keys)}.");
        }
    }
}

[ShadyNagy]

All you need to do SetMissingMemberHandling and it will handle every thing for you but you need to install DevBetter.JsonExtensions

var deserializeOptions = new JsonSerializerOptions()
    .SetMissingMemberHandling(MissingMemberHandling.Ignore);

var weatherForecast = JsonSerializer.Deserialize<WeatherForecast>(jsonString, deserializeOptions);

[LetMeSleepAlready]

@eiriktsarpalis that will require us to alter each poco class, which while it would work is the opposite of what we are trying to achieve.. type-agnostic catching of extra members during deserialization.

@ShadyNagy that is exactly the opposite of this ticket.

[ShadyNagy]

@LetMeSleepAlready
Some bad APIs respond with something like this {} on a string or array and the Text.Json throw exceptions and there is not an option to control that turn on and turn off so we made the same as Newtonsoft.Json did
SetMissingMemberHandling(MissingMemberHandling.Ignore)
SetMissingMemberHandling(MissingMemberHandling.Error)

[LetMeSleepAlready]

@ShadyNagy Interesting extension, but I don't see any code that handles the 'MissingMemberHandling.Error' case. So that means it is the default functionality, which is not the intent of this raised ticket.

case:

POCO: class { public string member {get;set;} }
JSON: {"membah","some value"}

The above will not throw an exception, and there is no way to somehow trap this and act on it.

[ShadyNagy]

@LetMeSleepAlready and @SiggyBar
The Devbetter Team Package updated so you can find cover for your problem here
https://github.com/DevBetterCom/DevBetter.JsonExtensions/blob/main/tests/DevBetter.JsonExtensions.Tests/SetMissingMemberErrorHandlingTests.cs
Nuget: DevBetter.JsonExtensions
Source Code: DevBetter.JsonExtensions

@ShadyNagy Interesting extension, but I don't see any code that handles the 'MissingMemberHandling.Error' case. So that means it is the default functionality, which is not the intent of this raised ticket.

case:

POCO: class { public string member {get;set;} }
JSON: {"membah","some value"}

The above will not throw an exception, and there is no way to somehow trap this and act on it.

[SigmundVik]

@ShadyNagy, have you looked at Dahomey.Json? Maybe these efforts could be joined.

[shadow-cs]

I recently stumbled upon this problem and I come to the same conclusion as @LetMeSleepAlready. @eiriktsarpalis could you provide some more insight, whether this feature has any chance of landing and if required is there something the community can do (so maybe pinpoint more areas that might need changing), this would be very helpful to start a PR, thanks 😉

[eiriktsarpalis]

I like the idea of using a per-type handler, but how would the JSON of the missing property be deserialized into an object if we're missing metadata on how to do it?

[layomia]

Action<object, object, string>? SetMissingProperty { get; set; }, // declaring object, value, property name

Is this supposed to be implemented and set by users only (and not the serializer)?

// different name ideas: SetUnhandledProperty, MissingPropertyHandler

HandleMissingProperty seems nice.

[krwq]

Is this supposed to be implemented and set by users only (and not the serializer)?

yes, by default we'd do nothing or use extension data property

@eiriktsarpalis good point, I guess it would need to be Utf8JsonReader (which would require delegate since it would need to be ref) or JsonElement/JsonNode instead (perhaps latter makes more sense from usability point of view but since we're perf oriented we should consider first option as well).

[eiriktsarpalis]

which would require delegate

Given the added complexity, we might want to hold off on adding this unless there is a proven use case. Perhaps focus on exposing enum support for now.

[eiriktsarpalis]

Starting with .NET 7, it is possible enable missing member handling for every type without the added boilerplate using a custom resolver:

var options = new JsonSerializerOptions
{
    TypeInfoResolver = new DefaultJsonTypeInfoResolver
    {
        Modifiers = { AddMissingMemberHandling }
    }
};

string json = """{"Name" : "John", "Surname" : "Doe", "Age" : 99 }""";
// Fails with Unhandled exception. System.Text.Json.JsonException: JSON properties Surname, Age could not bind to any members of type MyPoco
JsonSerializer.Deserialize<MyPoco>(json, options);

static void AddMissingMemberHandling(JsonTypeInfo typeInfo)
{
    if (typeInfo.Kind == JsonTypeInfoKind.Object && 
        typeInfo.Properties.All(prop => !prop.IsExtensionData) &&
        typeInfo.OnDeserialized is null)
    {
        var cwt = new ConditionalWeakTable<object, Dictionary<string, object>>(); // Dynamically attach dictionaries to deserialized objects

        JsonPropertyInfo propertyInfo = typeInfo.CreateJsonPropertyInfo(typeof(Dictionary<string, object>), "__extensionDataAttribute");
        propertyInfo.Get = obj => cwt.TryGetValue(obj, out Dictionary<string, object>? value) ? value : null;
        propertyInfo.Set = (obj, value) => cwt.Add(obj, (Dictionary<string, object>)value!);
        propertyInfo.IsExtensionData = true;
        typeInfo.Properties.Add(propertyInfo);
        typeInfo.OnDeserialized = obj =>
        {
            if (cwt.TryGetValue(obj, out Dictionary<string, object>? dict))
            {
                cwt.Remove(obj);
                throw new JsonException($"JSON properties {String.Join(", ", dict.Keys)} could not bind to any members of type {typeInfo.Type}");
            }
        };
    }
}

public class MyPoco
{
    public string Name { get; set; }
}

[bartonjs]

Video

• Since the proposal involved a two-enum we asked if a boolean would be better, and the conclusion was yes. And then it was no.
• While Newtonsoft.Json calls it MissingMemberHandling, that's actually a debatable/confusing name. The property was specified, but it didn't map to a property/field on the target type (so reflection's MissingMember).
  • So we renamed "MissingMember" to "UnmappedMember". Sorry, future us.

namespace System.Text.Json
{
    public enum JsonUnmappedMemberHandling { Skip, Disallow }

    public partial class JsonSerializerOptions
    {
        // Global configuration
        public JsonUnmappedMemberHandling UnmappedMemberHandling { get; set; } = JsonUnmappedMemberHandling.Skip;
    }
}

namespace System.Text.Json.Serialization
{
    // Per-type attribute-level configuration
    [AttributeUsageAttribute(AttributeTargets.Class | AttributeTargets.Interface | AttributeTargets.Struct, AllowMultiple=false, Inherited=false)]
    public class JsonUnmappedMemberHandlingAttribute : JsonAttribute
    {
        public JsonUnmappedMemberHandlingAttribute(JsonUnmappedMemberHandling unmappedMemberHandling);
        public JsonUnmappedMemberHandling UnmappedMemberHandling { get; }
    }
}

namespace System.Text.Json.Serialization.Metadata
{
    public partial class JsonTypeInfo
    {
        // Per-type configuration via contract customization.
        public JsonUnmappedMemberHandling UnmappedMemberHandling { get; set; }; // defaults to the JsonSerializerOptions value
    }
}

[jeancallisti]

I don't understand. It's 2022. And we're still devising custom solutions in lengthy conversations, in order to be able to deserialize with the absolutely basic option "IF JSON NO MATCH CLASS, DESERIALIZE NO HAPPY" ?

For the last three years I read that Newtonsoft's Json is dying. and I keep trying to adopt System.Text.Json. And yet I'll yet again go back to Newtonsoft because I'm baffled by the absolutely wacky solutions proposed to address basic scenarios that were literally one-liners with Newtonsoft.