proposal: embed: allow map[string]string, map[string][]byte, and/or map[string]any types

[myaaaaaaaaa]

Proposal Details

Currently, go:embed allows embedding individual files as strings or []bytes, but directories must go through the embed.FS interface.

Allowing directories to be represented as maps would improve the ergonomics of walking through embedded filesystem trees, since there would no longer be any (nonexistent) errors to handle (or ignore).

Options:

• map[string]string and/or map[string][]byte would represent flattened directory hierarchies (similar to how S3 buckets work)
• map[string]any would represent the full tree, where keys are directory/file names without slashes, and values are map[string]any | []bytes

[earthboundkid]

map[string]string seems like it addresses a common enough need. I don't see the point of the other two types. If you want directories, use an fs.FS. If you want a byte slice, you probably need to mutate it, so having it in a map is unhelpful.

[magical]

You can use this function to convert an embed.FS to a map[string]string.

import (
	"embed"
	"path"
)

func ToDict(fsys embed.FS) map[string]string {
	m := make(map[string]string)
	dirs := []string{"."}
	for len(dirs) > 0 {
		i := len(dirs) - 1
		d := dirs[i]
		dirs = dirs[:i]
		dlist, _ := fsys.ReadDir(d)
		for _, dentry := range dlist {
			p := path.Join(d, dentry.Name())
			if dentry.IsDir() {
				dirs = append(dirs, p)
			} else {
				data, err := fsys.ReadFile(p)
				if err == nil {
					m[p] = string(data)
				}
			}
		}
	}
	return m
}

https://play.golang.org/p/itTm4SbhpEW

---

EDIT: shorter version by mvdan

import (
	"embed"
	"io/fs"
)

func ToDict(fsys embed.FS) map[string]string {
	m := make(map[string]string)
	fs.WalkDir(fsys, ".", func(path string, d fs.DirEntry, err error) error {
		if !d.IsDir() {
			content, _ := fsys.ReadFile(path)
			m[path] = string(content)
		}
		return nil
	})
	return m
}

https://play.golang.org/p/LSE-UbnU9PB

[magical]

If ToDict were implemented in the embed package, it could probably be made much more efficient. embed.FS is currently implemented as a flat list of []file, where each entry contains the full path of a file along with its contents as strings. The strings could be directly reused in the returned map, avoiding any copies. It could also do a straight iteration over the file list instead of a recursive directory traversal.

[mvdan]

That manual walking of directories isn't even needed if you use fs.WalkDir: https://go.dev/play/p/LSE-UbnU9PB

This ignores errors, but I personally think that's fine for embed.FS. I also personally do not think this proposal carries its weight given that you can accomplish the same in under a dozen lines of code, and embed.FS is more versatile than map[string]string.

[myaaaaaaaaa]

ToDict() is enough to effectively replace this proposal (although slightly inelegantly):

//go:embed *
var filesFS embed.FS
var files = filesFS.ToDict()

func main() {
	for fname := range files {
		fmt.Println(fname)
	}
}

[myaaaaaaaaa]

Closing in favor of #69595