proposal: encoding/json: "nonil" struct tag to marshal nil slices and maps as non-null

[pjebs]

There have been many reports about encoding a nil slice or map as [ ] or { } respectively.

See: #2278

Currently it is encoded as null. This is a source of bugs in many non-theoretical cases (eg consumers of JSON APIs). There are many issues on Google about others falling into this pitfall so it is at least not uncommon.

It would be nice if a struct tag can be used to signify that we intend the nil slice or map to be treated as an empty slice or map respectively for the purposes of marshalling.

Current solutions include: https://github.com/helloeave/json. This is not a nice solution because it requires the assistance of a third-party package.

This is how it's implemented in third party lib: homelight/json@7b6e442

My PR can be found here: #27813 (subject to an agreed struct tag name, which when decided, can be updated in the PR)

[mvdan]

I've retitled this to be a proposal.

I'm not sure if it's a good idea, but it's certainly feasible and wouldn't add much new API. /cc @bradfitz @dsnet

[pjebs]

For slices and maps, it should be noted that the proposed struct tag is mutually exclusive to omitempty. For backwards compatibility, if both omitempty and the new struct tag is present, then omitempty takes precedence.

[pjebs]

Here is my pull request. I hope it gets accepted because I believe it adds value to the "Go" programming language.

#27813

[pjebs]

I called the struct tag: nonil

[bradfitz]

"heed" is a unique word, at least in terms of Go's existing API.

If this is accepted (once it's reviewed together with all the existing JSON proposals), I suspect we'd want to change "heed" to something else.

[pjebs]

I scoured the dictionary and thesaurus for appropriate words that were short and exemplified the intention but couldn't find anything. Hopefully a better word can be found because I was initially hesitant of the word too - although admittedly it is growing on me.

[pjebs]

@bradfitz Any idea of the timeframe to when all the JSON proposals will be examined? Some of them are now 2+ years. The pivotal proposal is 7+ years.

[QtRoS]

@bradfitz @pjebs what about nullsafe or nullfree or nonull?
+1 that heednull is unacceptable.

[pjebs]

For Go2, omitempty should become omitzero. The zero value and what is colloquially called "empty" are different for maps and slices.

[rsc]

Will add this to my JSON scan for this cycle.

[rsc]

Maybe emptynil.

[pjebs]

HEED
verb
1.
pay attention to; take notice of.

noun
1.
careful attention.

[pjebs]

Seems like precisely the right word. It's not part of the Go vocabulary, but it seems like a great addition in various future contexts for programming languages in general.

eg. heed memory, heed security etc

[rsc]

heednull means "pay attention to null" by not respecting that the value is null and instead printing something other than "null"? That seems backward to me.

[pjebs]

If it was called heednil instead, then one can argue that the slice or map was nil, and you are instructing the encoder to "pay attention to it".

I then went from that step to heednull because it flowed off the tongue better. The null in heednull was not really a reference to the eventual json output null.

Perhaps due to the confusion, and ambiguity, heednil is closer to being "Go-like".

[rsc]

On hold for JSON sweep.

[pjebs]

What does that mean?

[dsnet]

The json package already has a bloated API surface. Any additions need to carefully considered together with all the other proposals. I alluded to the need for this: #27813 (comment)

The "JSON sweep" is a sweep through all the JSON proposal trying to come up with a comprehensive yay/nay to each proposal in an effort to balance features desired with not adding too much technical debt.

[tommie]

Current solutions include: https://github.com/helloeave/json

This fork (which sadly I can't use because AppEngine doesn't have sync.Map, it seems) sets an encoder-wide option instead of using a struct tag. It feels to me that either an entire system is fine with having null, or there is an expectation that all slices are non-null. Why is the proposal to do this as a struct tag instead of an encoder option?

[QtRoS]

@tommie in my opinion the problem is that any system doesn't work in isolation. Complex software has complex integrations with different technologies and different expectations. Global flag is not versatile enough.

[tommie]

Complex software has complex integrations with different technologies and different expectations. Global flag is not versatile enough.

That is a non-answer, IMHO (and somewhat belittling). I can see different complex software requiring different options, and AFAIK, this thread has not discussed the already existing solution yet.

• Say you have a frontend that needs non-null results, and a backend that wants null results. It's not per-field, but per encoder invocation.
• Say most fields are "arrays", but one field is an optional array. Then it's per-field.

Defining what the default behavior is could still be an encoder option, while allowing per-field overrides. The two solutions are not mutually exclusive.

The problem I'm trying to solve is interoperability with Javascript. Passing null means my browser-side code needs to handle an empty array in a non-idiomatic way, leading to more fragile code. It seems to me @pjebs is trying to solve a different problem with the struct tag solution, and I'm curious what that is.

[gopherbot]

Change https://golang.org/cl/136761 mentions this issue: encoding/json: added "nonil" option for json encoding (slice/maps)

[as]

I provide an explanation in the similar issue here #37711 why I think a tag is not the right way to address this problem even in a third-party package.

[leaxoy]

Any progress on this, this is a great proposal.

[mvdan]

@leaxoy see #27589 (comment).

We'll update the many json proposals when we have news. Don't hold your breath because it's a big project, but it's in progress. Until then, asking for progress or an ETA won't really help.

[benitogf]

is there any other approach currently for this one?

[VojtechVitek]

I just wrote this little package, which recursively initializes all nil slices in a given object using reflect pkg. This effectively forces json.Marshal() to render empty array [] instead of null values.

https://github.com/golang-cz/nilslice

Might be useful for some, until we have a proper solution merged to encoding/json stdlib pkg.

import "github.com/golang-cz/nilslice"

type Payload struct {
	Items []Item `json:"items"`
}

payload := &Payload{}

b, _ = json.Marshal(nilslice.Initialize(payload))
fmt.Println(string(b))
// {"items": []}

[dsnet]

Hi all, we kicked off a discussion for a possible "encoding/json/v2" package that addresses the spirit of this proposal.
By default v2 proposes that a nil slice or map would marshal as an empty JSON array or object.

For flexibility, it provides the ability to alter this behavior with:

• json.Marshal(v, json.FormatNilSliceAsNull(false)) at the marshal-call level
• struct{ SliceField []T `json:",format:emitnull"` } at the struct-field level

[pkierski]

I just wrote this little package, which recursively initializes all nil slices in a given object using reflect pkg. This effectively forces json.Marshal() to render empty array [] instead of null values.

https://github.com/golang-cz/nilslice

Based on this idea I've prepared more complex solution: The same applies for nil maps (initialized as empty map, rendered as {} instead of null). And also does it recursively for any non-cyclic structures (for cyclic structured crashes as json.Marshal do). https://github.com/pkierski/niltoempty