proposal: encoding/json: allow defining which fields will be marshalled

[jtorz]

What version of Go are you using (go version)?

$ go version
go version go1.16.3 linux/amd64

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/jtorz/.cache/go-build"
GOENV="/home/jtorz/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/jtorz/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/jtorz/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.16.3"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build3209792062=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Similar issues: #22480, #11939, #45669

The idea of this proposal is to allow defining which columns will be marshal. It would be useful in many situations where we want to reduce the data size like HTTP responses, or marshalling to cache JSON values.

Currently, with the Standard library when we have something like this, the inner Structures are marshal into empty values.
https://play.golang.org/p/UVjAKNTMqW9

package main

import (
	"encoding/json"
	"fmt"
)

type A struct {
	Something1 string `json:"something_1"`
	Something2 string `json:"something_2"`
	Something3 string `json:"something_3"`
	Something4 string `json:"something_4"`
}

type B struct {
	A     A      `json:"a,omitempty"`
	Name  string `json:"name,omitempty"`
	Hello string `json:"hello,omitempty"`
}

func main() {
	b := B{Name: "Jane"}
	var j, _ = json.Marshal(b)
	fmt.Printf("%s\n", j)
}

In the result "a" holds an object with empty values and there are case where thi.

{"a":{"something_1":"","something_2":"","something_3":"","something_4":""},"name":"Joe"}

Solution Proposal

I copied the code from the standard library and did some modifications to allow this. https://github.com/jtorz/jsont

In my module I added the function func MarshalFields(v interface{}, whitelist F) ([]byte, error).

MarshalFields receives a type F map[string]F which works as a whitelist that holds the JSON keys that will be marshal if the key exists in the map, no matter the value, the field is added.

In the next example only the "name" is added to the JSON, all other fields are ignored.

https://play.golang.org/p/ghG6SDyXNLU

In this Example

package main

import (
	"fmt"
	"github.com/jtorz/jsont/v2"
)

type A struct {
	Something1 string `json:"something_1"`
	Something2 string `json:"something_2"`
	Something3 string `json:"something_3"`
	Something4 string `json:"something_4"`
}

type B struct {
	A     A      `json:"a,omitempty"`
	Name  string `json:"name,omitempty"`
	Hello string `json:"hello,omitempty"`
}

func main() {
	b := B{Name: "Joe", Hello: "This will be ignored"}
	j, _ := jsont.MarshalFields(b, jsont.F{
		"name": nil,
	})
	fmt.Printf("%s\n", j)
}

{"name":"Joe"}

Cases

I considered different scenarios like slices, nested structures, or self-referential structures.

The following cases will use the same structs definitions.

type A struct {
	Something1 string `json:"something_1"`
	Something2 string `json:"something_2"`
	Something3 string `json:"something_3"`
	Something4 string `json:"something_4"`
}

type B struct {
	A     A      `json:"a"`
	Name  string `json:"name"`
	Hello string `json:"hello"`
	Bs    []B    `json:"bs,omitempty"`
}

self-referential Slice (Recursive)

b := B{
    Name: "Joe",
    Bs: []B{
        {Name: "Jane"},
        {Name: "John"},
        {Name: "Janah"},
    },
}
j, _ := jsont.MarshalFields(b, jsont.F{
    "name": nil,
    "bs":   jsont.Recursive,
})
fmt.Printf("%s\n", j)

OUTPUT:

{"name":"Joe","bs":[{"name":"Jane"},{"name":"John"},{"name":"Janah"}]}

self-referential Slice (Non Recursive)

b := B{
    Name: "Joe",
    Bs: []B{
        {Hello: "World!"},
    },
}
j, _ := jsont.MarshalFields(b, jsont.F{
    "name": nil,
    "bs": jsont.F{
        "hello": nil,
    },
})
fmt.Printf("%s\n", j)

OUTPUT:

{"name":"Joe","bs":[{"hello":"World!"}]}

Nested Struct (Whole structure)

b := B{
    Hello: "Janah",
    A: A{
        Something1: "value1",
        Something2: "value2",
    },
}
j, _ := jsont.MarshalFields(b, jsont.F{
    "hello": nil,
    "a":     nil,
})
fmt.Printf("%s\n", j)

OUTPUT:

{"hello":"Janah","a":{"something_1":"value1","something_2":"value2","something_3":"","something_4":""}}

Nested Struct (Specific fields)

b := B{
    Hello: "Janah",
    A: A{
        Something1: "value1",
        Something2: "value2",
    },
}
j, _ := jsont.MarshalFields(b, jsont.F{
    "hello": nil,
    "a": jsont.F{
        "something_1": nil,
    },
})
fmt.Printf("%s\n", j)

OUTPUT:

{"hello":"Janah","a":{"something_1":"value1"}}

[ianlancetaylor]

Can you write in this issue exactly what json.MarshalFields does? It's easier to understand a spec than an implementation. Thanks.

[jtorz]

I added more information to the Solution Proposal part

[rsc]

/cc @dsnet

[dsnet]

For prior art, this seems to have overlap with the field_mask feature from protocol buffers.

[seankhliao]

Related #23304

[dsnet]

How does this feature interoperate with the fact that json allows custom marshalers and unmarshalers?

[jtorz]

I added another interface MarshalerFields, if the type implements that interface it uses that encoder. But there is a precedence of Marshalers, if the same type also implements the Marshaler interface it uses that custom Marshaler as the encoder.

type MarshalerFields interface {
	MarshalJSONFields(whitelist F) ([]byte, error)
}

I didn't consider the decoding in my module, but I think the custom Marshal/Unmarshaler interface should be used and ignore the defined values.

[dsnet]

Plumbing down options is a known problem in json and I don't think we should define another interface for another specialized serialization mode.

Even if the problem of plumbing down options was solved, there's the fundamental problem that most implementations will not support some of type of field mask, which makes support for this feature spotty at best.

If the desire is to have semantically correct field mask, then it seems that a better solution is to pre-process JSON input when unmarshaling and post-process JSON output when marshaling to mask out certain fields. It won't be as efficient, but would be guaranteed to be correct. However, if we're willing to accept that this is a pre-process or post-process step, it doesn't seem like this is something that the standard json package needs to support.

[networkimprov]

cc @mvdan