System.Text.Json Merge

[tb-mtg]

Using System.Text.Json, is there any way to Merge like Json.Net does?

see Newtonsoft.Json.Linq.JContainer.Merge

public void Merge(
	Object content,
	JsonMergeSettings settings
)

[ahsonkhan]

Since the JsonDocument, JsonElement APIs are read-only, you could workaround this by writing your own Merge method based on the Utf8JsonWriter.

Such an API makes more sense with the writable/modifiable DOM so we should consider adding this merge capability with that feature: https://github.com/dotnet/corefx/issues/39922

If your JSON objects only contain non-null simple/primitive values and the order in which the properties show up isn't particularly concerning, the following, relatively straightforward, code sample should work for you:

public static string SimpleObjectMerge(string originalJson, string newContent)
{
    var outputBuffer = new ArrayBufferWriter<byte>();

    using (JsonDocument jDoc1 = JsonDocument.Parse(originalJson))
    using (JsonDocument jDoc2 = JsonDocument.Parse(newContent))
    using (var jsonWriter = new Utf8JsonWriter(outputBuffer, new JsonWriterOptions { Indented = true }))
    {
        JsonElement root1 = jDoc1.RootElement;
        JsonElement root2 = jDoc2.RootElement;

        // Assuming both JSON strings are single JSON objects (i.e. {...})
        Debug.Assert(root1.ValueKind == JsonValueKind.Object);
        Debug.Assert(root2.ValueKind == JsonValueKind.Object);

        jsonWriter.WriteStartObject();

        // Write all the properties of the first document that don't conflict with the second
        foreach (JsonProperty property in root1.EnumerateObject())
        {
            if (!root2.TryGetProperty(property.Name, out _))
            {
                property.WriteTo(jsonWriter);
            }
        }

        // Write all the properties of the second document (including those that are duplicates which were skipped earlier)
        // The property values of the second document completely override the values of the first
        foreach (JsonProperty property in root2.EnumerateObject())
        {
            property.WriteTo(jsonWriter);
        }

        jsonWriter.WriteEndObject();
    }

    return Encoding.UTF8.GetString(outputBuffer.WrittenSpan);
}

Newtonsoft.Json has different null handling when doing a merge where null doesn't override the value of the non-null property (when there are duplicates). I am not sure if you want that behavior or not. If that's needed, you would need to modify the above method to handle the null cases. Here are the modifications:

public static string SimpleObjectMergeWithNullHandling(string originalJson, string newContent)
{
    var outputBuffer = new ArrayBufferWriter<byte>();

    using (JsonDocument jDoc1 = JsonDocument.Parse(originalJson))
    using (JsonDocument jDoc2 = JsonDocument.Parse(newContent))
    using (var jsonWriter = new Utf8JsonWriter(outputBuffer, new JsonWriterOptions { Indented = true }))
    {
        JsonElement root1 = jDoc1.RootElement;
        JsonElement root2 = jDoc2.RootElement;

        // Assuming both JSON strings are single JSON objects (i.e. {...})
        Debug.Assert(root1.ValueKind == JsonValueKind.Object);
        Debug.Assert(root2.ValueKind == JsonValueKind.Object);

        jsonWriter.WriteStartObject();

        // Write all the properties of the first document that don't conflict with the second
        // Or if the second is overriding it with null, favor the property in the first.
        foreach (JsonProperty property in root1.EnumerateObject())
        {
            if (!root2.TryGetProperty(property.Name, out JsonElement newValue) || newValue.ValueKind == JsonValueKind.Null)
            {
                property.WriteTo(jsonWriter);
            }
        }

        // Write all the properties of the second document (including those that are duplicates which were skipped earlier)
        // The property values of the second document completely override the values of the first, unless they are null in the second.
        foreach (JsonProperty property in root2.EnumerateObject())
        {
            // Don't write null values, unless they are unique to the second document
            if (property.Value.ValueKind != JsonValueKind.Null || !root1.TryGetProperty(property.Name, out _))
            {
                property.WriteTo(jsonWriter);
            }
        }

        jsonWriter.WriteEndObject();
    }

    return Encoding.UTF8.GetString(outputBuffer.WrittenSpan);
}

If your JSON objects can potentially contain nested JSON values including other objects and arrays, you would want to extend the logic to handle that too. Something like this should work:

public static string Merge(string originalJson, string newContent)
{
    var outputBuffer = new ArrayBufferWriter<byte>();

    using (JsonDocument jDoc1 = JsonDocument.Parse(originalJson))
    using (JsonDocument jDoc2 = JsonDocument.Parse(newContent))
    using (var jsonWriter = new Utf8JsonWriter(outputBuffer, new JsonWriterOptions { Indented = true }))
    {
        JsonElement root1 = jDoc1.RootElement;
        JsonElement root2 = jDoc2.RootElement;

        if (root1.ValueKind != JsonValueKind.Array && root1.ValueKind != JsonValueKind.Object)
        {
            throw new InvalidOperationException($"The original JSON document to merge new content into must be a container type. Instead it is {root1.ValueKind}.");
        }

        if (root1.ValueKind != root2.ValueKind)
        {
            return originalJson;
        }

        if (root1.ValueKind == JsonValueKind.Array)
        {
            MergeArrays(jsonWriter, root1, root2);
        }
        else
        {
            MergeObjects(jsonWriter, root1, root2);
        }
    }

    return Encoding.UTF8.GetString(outputBuffer.WrittenSpan);
}

private static void MergeObjects(Utf8JsonWriter jsonWriter, JsonElement root1, JsonElement root2)
{
    Debug.Assert(root1.ValueKind == JsonValueKind.Object);
    Debug.Assert(root2.ValueKind == JsonValueKind.Object);

    jsonWriter.WriteStartObject();

    // Write all the properties of the first document.
    // If a property exists in both documents, either:
    // * Merge them, if the value kinds match (e.g. both are objects or arrays),
    // * Completely override the value of the first with the one from the second, if the value kind mismatches (e.g. one is object, while the other is an array or string),
    // * Or favor the value of the first (regardless of what it may be), if the second one is null (i.e. don't override the first).
    foreach (JsonProperty property in root1.EnumerateObject())
    {
        string propertyName = property.Name;

        JsonValueKind newValueKind;

        if (root2.TryGetProperty(propertyName, out JsonElement newValue) && (newValueKind = newValue.ValueKind) != JsonValueKind.Null)
        {
            jsonWriter.WritePropertyName(propertyName);

            JsonElement originalValue = property.Value;
            JsonValueKind originalValueKind = originalValue.ValueKind;

            if (newValueKind == JsonValueKind.Object && originalValueKind == JsonValueKind.Object)
            {
                MergeObjects(jsonWriter, originalValue, newValue); // Recursive call
            }
            else if (newValueKind == JsonValueKind.Array && originalValueKind == JsonValueKind.Array)
            {
                MergeArrays(jsonWriter, originalValue, newValue);
            }
            else
            {
                newValue.WriteTo(jsonWriter);
            }
        }
        else
        {
            property.WriteTo(jsonWriter);
        }
    }

    // Write all the properties of the second document that are unique to it.
    foreach (JsonProperty property in root2.EnumerateObject())
    {
        if (!root1.TryGetProperty(property.Name, out _))
        {
            property.WriteTo(jsonWriter);
        }
    }

    jsonWriter.WriteEndObject();
}

private static void MergeArrays(Utf8JsonWriter jsonWriter, JsonElement root1, JsonElement root2)
{
    Debug.Assert(root1.ValueKind == JsonValueKind.Array);
    Debug.Assert(root2.ValueKind == JsonValueKind.Array);

    jsonWriter.WriteStartArray();

    // Write all the elements from both JSON arrays
    foreach (JsonElement element in root1.EnumerateArray())
    {
        element.WriteTo(jsonWriter);
    }
    foreach (JsonElement element in root2.EnumerateArray())
    {
        element.WriteTo(jsonWriter);
    }

    jsonWriter.WriteEndArray();
}

This sample was tested with the following:

[Fact]
public static void JsonDocumentMergeTest_ComparedToJContainerMerge()
{
    string jsonString1 = @"{
        ""throw"": null,
        ""duplicate"": null,
        ""id"": 1,
        ""xyz"": null,
        ""nullOverride2"": false,
        ""nullOverride1"": null,
        ""william"": ""shakespeare"",
        ""complex"": {""overwrite"": ""no"", ""type"": ""string"", ""original"": null, ""another"":[]},
        ""nested"": [7, {""another"": true}],
        ""nestedObject"": {""another"": true}
    }";

    string jsonString2 = @"{
        ""william"": ""dafoe"",
        ""duplicate"": null,
        ""foo"": ""bar"",
        ""baz"": {""temp"": 4},
        ""xyz"": [1, 2, 3],
        ""nullOverride1"": true,
        ""nullOverride2"": null,
        ""nested"": [1, 2, 3, null, {""another"": false}],
        ""nestedObject"": [""wow""],
        ""complex"": {""temp"": true, ""overwrite"": ""ok"", ""type"": 14},
        ""temp"": null
    }";

    JObject jObj1 = JObject.Parse(jsonString1);
    JObject jObj2 = JObject.Parse(jsonString2);

    jObj1.Merge(jObj2);
    jObj2.Merge(JObject.Parse(jsonString1));

    Assert.Equal(jObj1.ToString(), Merge(jsonString1, jsonString2));
    Assert.Equal(jObj2.ToString(), Merge(jsonString2, jsonString1));
}

Note: If performance is critical for your scenario, this method (even with writing indented) out-performs the Newtonsoft.Json's Merge method both in terms of runtime and allocations. That said, the implementation could be made faster depending on need (for instance, don't write indented, cache the outputBuffer, don't accept/return strings, etc.).

BenchmarkDotNet=v0.12.0, OS=Windows 10.0.19041
Intel Core i7-6700 CPU 3.40GHz (Skylake), 1 CPU, 8 logical and 4 physical cores
.NET Core SDK=5.0.100-alpha1-015914
  [Host]     : .NET Core 5.0.0 (CoreCLR 5.0.19.56303, CoreFX 5.0.19.56306), X64 RyuJIT
  Job-LACFYV : .NET Core 5.0.0 (CoreCLR 5.0.19.56303, CoreFX 5.0.19.56306), X64 RyuJIT

PowerPlanMode=00000000-0000-0000-0000-000000000000  

Method	Mean	Error	StdDev	Median	Min	Max	Ratio	Gen 0	Gen 1	Gen 2	Allocated
MergeNewtonsoft	29.01 us	0.570 us	0.656 us	28.84 us	28.13 us	30.19 us	1.00	7.0801	0.0610	-	28.98 KB
Merge_New	16.41 us	0.293 us	0.274 us	16.41 us	16.02 us	17.00 us	0.57	1.7090	-	-	6.99 KB

[BenchmarkCategory(Categories.CoreFX, Categories.JSON)]
[Benchmark(Baseline = true)]
public string MergeNewtonsoft()
{
    JObject jObj1 = JObject.Parse(_jsonString1);
    JObject jObj2 = JObject.Parse(_jsonString2);

    jObj1.Merge(jObj2);

    return jObj1.ToString();
}

[BenchmarkCategory(Categories.CoreFX, Categories.JSON)]
[Benchmark]
public string Merge_New()
{
    return Merge(_jsonString1, _jsonString2);
}

[tb-mtg]

Thank you @ahsonkhan, hopefully this feature will be added.

[ahsonkhan]

@tb-mtg, as part of requirements, can you expand on your scenarios and what JsonMergeSettings capabilities are necessary for the Merge APIs (for example MergeArrayHandling, MergeNullValueHandling, PropertyNameComparison).

Are there others that Newtonsoft.Json doesn't support that would be needed? Do you generally merge two JObjects, or is it common to merge any two arbitrary JContainers? What should the behavior be of JArray.Merge(some single JToken)?

Also, what is your particular use case for such an API? Having context around sample usage would help answer some of the requirement questions as well.

[vijer]

Is there a way to do a join? I have two different json files with a common key and I've been looking for a way to filter and join the output.

For example, the controlling file contains categories and the items are in another file. The filter would be where(rc.CategoryID == tt.CategoryID && rc.CategoryID == "metals" && tt.GameVersion == "A" || tt.GameVersion == "2")

All the examples I find are related to merging two files with the same structure.

"ResourceCategories": [
    {
      "CategoryDescription": "Base Upgrades",
      "CategoryID": "baseupgrades",
      "IncludeCategory": true,
      "GameVersion": "A"
    },

and the item file

"TechType": [
    {
      "CategoryID": "crystalline",
      "TechName": "Quartz",
      "SpawnID": "quartz",
      "TechID": 1,
      "GameVersion": "A"
    },
    {
      "CategoryID": "metals",
      "TechName": "Metal Salvage",
      "SpawnID": "scrapmetal",
      "TechID": 2,
      "GameVersion": "A"
    },
    {
      "CategoryID": "outcrop",
      "TechName": "Limestone Outcrop",
      "SpawnID": "limestonechunk",
      "TechID": 4,
      "GameVersion": "A"
    },

[PeterWone]

Is there a way to do a join? I have two different json files with a common key and I've been looking for a way to filter and join the output.

Get the graphs as Json objects and use LINQ.

[eiriktsarpalis]

I believe the new JsonNode type that ships with .NET 6 might be more appropriate to expose this type of functionality. APIs are currently missing, but they are fairly easy to implement as extensions methods:

public static class JsonNodeExtensions
{ 
    public static void AddRange(this JsonArray jsonArray, IEnumerable<JsonNode?> values)
    {
        foreach (var value in values)
        {
            jsonArray.Add(value);
        }
    }

    public static void AddRange(this JsonObject jsonObject, IEnumerable<KeyValuePair<string, JsonNode?>> properties)
    {
        foreach (var kvp in properties)
        {
            jsonObject.Add(kvp);
        }
    }
}

cc @steveharter

[eiriktsarpalis]

Closing in favor of #56592.

[steveharter]

I have concerns about add this feature when the options are not a simple "ignore" or "replace".

The existing semantics of Newtonsoft's MergeNullValueHandling and PropertyNameComparison are fairly intuitive however the MergeArrayHandling option is not. This option applies to both JSON arrays and JSON objects, and works fine if just "ignore" or "replace" is desired, but a "merge" requires an Equals() method which for JSON objects would typically be based on a "key property", and perhaps even a "type identifier property". For "union" that is a separate discussion for objects vs arrays. For ordered JSON arrays, the "key" could be the ordinal, but I think in many cases a true "key" would be desired. For these non-trivial merge cases, I believe Newtonsoft does a DeepEquals() here -- however that is not likely the expected behavior for these cases.

So I believe there are many scenarios where a "merge" done on objects would want a "key property" that typically relates to the primary key in a database. Without such a key, a "merge" would likely combine the properties of two independent objects which will likely not be incorrect. Consider:

Current Json:
[
{"ID":100, "Name":"Steve", "PhoneExtension":111},
{"ID":101, "Name":"Joe", "PhoneExtension":222}
]

JSON to merge:
[
{"ID":101, "PhoneExtension":333}
]

Result? Normally I'd expect this
[
{"ID":100, "Name":"Steve", "PhoneExtension":111},
{"ID":101, "Name":"Joe", "PhoneExtension":333}
]

and not, for example
[
{"ID":100, "Name":"Steve", "PhoneExtension":111},
{"ID":101, "Name":"Joe", "PhoneExtension":222},
{"PhoneExtension":333}
]

Also, since the options [should] apply throughout all nodes in the graph, recursively, (note they don't in Newtonsoft with "concat" and "union") some based on "keys" and some not, I don't see how useful this feature with merge\union would be.

Some options IMO:

• Eirik's extension example above could be used for simple cases that don't have to deal with "keys", and at only one level (not recursive).
• Extend Eirik's extension sample to provide a "key" property (probably needs to be a simple JsonValue type).
• Add a callback pattern where the consumer needs to specify the "replace" or "ignore" semantics and perhaps "merge\union" where custom logic would use the current Path of each node to determine the key and related merge semantics.
• Add the concept of a "key" property to JsonNode by allowing it to be set on each node.

[PeterWone]

@steveharter I think you're right.

The workaround offered to me was very practical and it is not difficult to use it to to handle the issues you mention. Personally I like the simplicity of merge as something you produce from two immutable graphs.

[cajuncoding]

For anyone else finding this thread while looking for a SystemTextJson compatible way to merge Json objects the way that NewtonsoftJson allowed, the new(er) JsonObject, JsonArray, and (their base) JsonNode provide a reasonably decent way to approach this and appear to be a lot more streamlined (less code) than the JsonDocument solution provided above by @ahsonkhan .

Here's a streamlined implementation that has solved my use cases. I haven't had any time yet to benchmark this against the JsonDocument approach above, or Newtonsoft for that matter, but other Json rewriting solutions I've implemented with System.Text.Json all proved to be significantly faster than the exact same logic with Newtonsoft.

The full solution below is also shared out via my gist here . . . but I'm including here for posterity:

EDIT: 07/11/2024 - Updated code to correctly fix missing handling when an object property value is actually a Json Array as described by @IdrissPiard below.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text.Json;
using System.Text.Json.Nodes;

namespace CajunCoding
{
    public static class SystemTextJsonMergeExtensions
    {
        /// <summary>
        /// Merges the specified Json Node into the base JsonNode for which this method is called.
        /// It is null safe and can be easily used with null-check & null coalesce operators for fluent calls.
        /// NOTE: JsonNodes are context aware and track their parent relationships therefore to merge the values both JsonNode objects
        ///         specified are mutated. The Base is mutated with new data while the source is mutated to remove reverences to all
        ///         fields so that they can be added to the base.
        ///
        /// Source taken directly from the open-source Gist here:
        /// https://gist.github.com/cajuncoding/bf78bdcf790782090d231590cbc2438f
        ///
        /// </summary>
        /// <param name="jsonBase"></param>
        /// <param name="jsonMerge"></param>
        /// <returns></returns>
        /// <exception cref="ArgumentException"></exception>
        public static JsonNode Merge(this JsonNode jsonBase, JsonNode jsonMerge)
        {
            if (jsonBase == null || jsonMerge == null)
                return jsonBase;

            switch (jsonBase)
            {
                case JsonObject jsonBaseObj when jsonMerge is JsonObject jsonMergeObj:
                {
                    //NOTE: We must materialize the set (e.g. to an Array), and then clear the merge array so the node can then be 
                    //      re-assigned to the target/base Json; clearing the Object seems to be the most efficient approach...
                    var mergeNodesArray = jsonMergeObj.ToArray();
                    jsonMergeObj.Clear();
                    
                    foreach (var prop in mergeNodesArray)
                    {
                        jsonBaseObj[prop.Key] = jsonBaseObj[prop.Key] switch
                        {
                            JsonObject jsonBaseChildObj when prop.Value is JsonObject jsonMergeChildObj => jsonBaseChildObj.Merge(jsonMergeChildObj),
                            JsonArray jsonBaseChildArray when prop.Value is JsonArray jsonMergeChildArray => jsonBaseChildArray.Merge(jsonMergeChildArray),
                            _ => prop.Value
                        };
                    }
                    break;
                }
                case JsonArray jsonBaseArray when jsonMerge is JsonArray jsonMergeArray:
                {
                    //NOTE: We must materialize the set (e.g. to an Array), and then clear the merge array,
                    //      so they can then be re-assigned to the target/base Json...
                    var mergeNodesArray = jsonMergeArray.ToArray();
                    jsonMergeArray.Clear();
                    foreach(var mergeNode in mergeNodesArray) jsonBaseArray.Add(mergeNode);
                    break;
                }
                default:
                    throw new ArgumentException($"The JsonNode type [{jsonBase.GetType().Name}] is incompatible for merging with the target/base " +
                                                        $"type [{jsonMerge.GetType().Name}]; merge requires the types to be the same.");
                
            }

            return jsonBase;
        }

        /// <summary>
        /// Merges the specified Dictionary of values into the base JsonNode for which this method is called.
        /// </summary>
        /// <typeparam name="TKey"></typeparam>
        /// <typeparam name="TValue"></typeparam>
        /// <param name="jsonBase"></param>
        /// <param name="dictionary"></param>
        /// <param name="options"></param>
        /// <returns></returns>
        public static JsonNode MergeDictionary<TKey, TValue>(this JsonNode jsonBase, IDictionary<TKey, TValue> dictionary, JsonSerializerOptions options = null)
            => jsonBase.Merge(JsonSerializer.SerializeToNode(dictionary, options));
    }
}

[IdrissPiard]

@cajuncoding Hey, I noticed on the members of the jsonBaseObj you only recurse if the member is of type JsonObject. But that exclude JsonArray as the parent type is JsonNode not JsonObject.
That means that you only ever go into the second case if the first call to Merge was done with JsonArray, and it also means it overwrites every arrays with the merge ones, erasing the source array content.
In case that isn't the intended behavior, the code should probably be something like:

/// <summary>
/// Merges the specified Json Node into the base JsonNode for which this method is called.
/// It is null safe and can be easily used with null-check & null coalesce operators for fluent calls.
/// NOTE: JsonNodes are context aware and track their parent relationships therefore to merge the values both JsonNode objects
///         specified are mutated. The Base is mutated with new data while the source is mutated to remove reverences to all
///         fields so that they can be added to the base.
///
/// Source taken directly from the open-source Gist here:
/// https://gist.github.com/cajuncoding/bf78bdcf790782090d231590cbc2438f
///
/// </summary>
/// <param name="jsonBase"></param>
/// <param name="jsonMerge"></param>
/// <returns></returns>
/// <exception cref="ArgumentException"></exception>
public static JsonNode Merge( this JsonNode jsonBase, JsonNode jsonMerge )
{
	if(jsonBase == null || jsonMerge == null)
		return jsonBase;

	switch(jsonBase)
	{
	case JsonObject jsonBaseObj when jsonMerge is JsonObject jsonMergeObj:
		{
			//NOTE: We must materialize the set (e.g. to an Array), and then clear the merge array so the node can then be 
			//      re-assigned to the target/base Json; clearing the Object seems to be the most efficient approach...
			var mergeNodesArray = jsonMergeObj.ToArray();
			jsonMergeObj.Clear();

			foreach(var prop in mergeNodesArray)
			{
				if(jsonBaseObj[ prop.Key ] is JsonNode jsonBaseChildNode && prop.Value is JsonNode jsonMergeChildNode)
					jsonBaseObj[ prop.Key ] = jsonBaseChildNode.Merge( jsonMergeChildNode );
				else
					jsonBaseObj[ prop.Key ] = prop.Value;
			}
			break;
		}
	case JsonArray jsonBaseArray when jsonMerge is JsonArray jsonMergeArray:
		{
			//NOTE: We must materialize the set (e.g. to an Array), and then clear the merge array,
			//      so they can then be re-assigned to the target/base Json...
			var mergeNodesArray = jsonMergeArray.ToArray();
			jsonMergeArray.Clear();
			foreach(var mergeNode in mergeNodesArray) jsonBaseArray.Add( mergeNode );
			break;
		}
	default:
		throw new ArgumentException( $"The JsonNode type [{jsonBase.GetType().Name}] is incompatible for merging with the target/base " +
											$"type [{jsonMerge.GetType().Name}]; merge requires the types to be the same." );

	}

	return jsonBase;
}

/// <summary>
/// Merges the specified Dictionary of values into the base JsonNode for which this method is called.
/// </summary>
/// <typeparam name="TKey"></typeparam>
/// <typeparam name="TValue"></typeparam>
/// <param name="jsonBase"></param>
/// <param name="dictionary"></param>
/// <param name="options"></param>
/// <returns></returns>
public static JsonNode MergeDictionary<TKey, TValue>( this JsonNode jsonBase, IDictionary<TKey, TValue> dictionary, JsonSerializerOptions options = null )
	=> jsonBase.Merge( JsonSerializer.SerializeToNode( dictionary, options ) );

[cajuncoding]

@IdrissPiard Unfortunately your proposed changes won't work and fail my unit test -- because JsonValue is also a JsonNode, and that results in hitting the type mismatch exception thrown by the top level logic of the merge process; which is needed to prevent infinite recursion.

But, you do raise a great point 👍in that I am missing a case whereby a property of a nested object might actually be a JsonArray and that case is missed; this wasn't covered my my initial use cases. The safe solution is actually to add an additional case to handle JsonArray just like the JsonObject....

I have updated my code above, and also updated this in my original Gist. In addition, for the reference of other's I've added the Unit Test case that validates the common cases I've tested to my Gist thread -- and it now includes validation of this case also ✅.

fyi, your code formatter (or something) introduced some odd spacing & indentation changes so I had to normalize all that to see the differences...