cue: Enable builtins to be registered by users

[verdverm]

As a user, I would like to register my own builtins to be used in Cue.

• Expose a RegisterBuiltin function which allows adding builtins to the predefined map
• add the builtins to the runtime (or something else)? This would allow users to have multiple runtimes and different builtins per.
• Enable new stdlib functions to be developed, tested, and used prior to modifying or extending the actual list of builtins
• Collect and document how to decide and register as pure vs tool. Alert to potential issues with non-pure builtins in pure cue, can shoot yourself in the foot.

[shykes]

Doesn't this create the risk of fragmenting the language? If I load a file with the extension .cue, I have certain expectations of available builtins and their behavior. Even more so if we allow builtins to not be pure (no real way to avoid it without extreme sandboxing which is another can of worms).

If custom builtins become widely adopted, it kind of changes the role of Cue from a language to a family of similar but incompatible languages. This in turn could hurt adoption of Cue, and the benefits of using it.

What are your thoughts on this?

[verdverm]

Super good question. @mpvl, would love to hear your thoughts

There are definitely a few that people have been asking for. I have a strong desire for the main functions in structural. I'm about to implement them in Go, along the lines of the update / mutation prototype we got working in slack today.

#348 is the other half of this, to have a place to talk about what builtins Cue should have. There are also the annotations for tooling to work with, and a lot should be possible there. Based on your perspective @shykes, what @mpvl thinks, this PR could go to the chopping block pretty quick.

Another thing that came out of the prototype is the difficulty several of us experienced constructing or mutation Cue values in our Golang based tools. If we improve that, so should the experience around building tools around Cue. At the same time, many of us would like to push as much into Cue as we can. It does seem more tedious to switch back and forth, but then again, maybe we can fix that in the Go API area.

---

Yet another line of thought we can add to the mix is how DSLs mix into this. When we write tools that expect certain values and schemas, we are essentially writing in DSLs. There's a lot more to say about this, but I'm pretty tired and just want to capture this thought when I had it. I'm planning to use attributes in custom tooling in conjunction with these DSL ideas.

@mpvl has some ideas: Cue & DSLs ;]

[ymolists]

@shykes I think its safe to expect that builtins (specially the ones that are meant to not be pure) should only come from plugins that live outside of the main cue source/binary. For example kubectl has an extension framework where people can install plugin binaries (at their own risk of course) that work with the kubectl binary.

To me the real question is:

1. can we define an easier API that make life easier for people who want to use cue as a library instead of a binary ?
2. can we define a plugin api that lives outside of the main binary but people can install plugins to. So the main binary can communicate via htpp or even running the actual external binary as kubectl does today.

Both are i think valid means of extending cue without breaking the main expectations of people who want to use cue as it is now.

[verdverm]

@ymolists if a project adds a new built-in, then there is an unknown token as seen by the CUE implementation. How would it infer that the missing token is a built-in vs other unknown token, like a dev forgot to write something. This would mean that one could not run the CUE cli against this tools "CUE" code.

What do you think CUE should do in the case that an unknown built-in is parsed? Does adding builtins in a project mean that their code is no longer CUE, but rather a dialect of CUE?

@shykes point is that with custom builtins, CUE is not reusable or readable across the ecosystem. That would cause significant support time because people will not be aware of this if everyone calls their dialect "CUE"

[ymolists]

@verdverm if the extention is prefixed with a package name i think the distinction will be clear no ? just as in go everyone knows how to expect a package is from the standard library or from an external package.

[verdverm]

Maybe some code examples of what you are thinking would help the discussion. I wouldn't expect CUE to be able to differentiate regardless of where the built-in comes from, based on my understanding of how a built-in is implemented in the CUE source.

Without the Go source which adds and implements, how would CUE differentiate between attempted use of an unknown built-in and an unknown definition (in example). How would an import statement make this differentiation?

[ymolists]

i am refferring to something like dagger. One could be making a call similar to Line 7 that is backed by a builtin ? at that point the user who is writing the cue file knows they are using an extension and not the standard packages that come with cue.

[verdverm]

CUE attributes were designed for this purpose, dagger is a good example of a tool built around CUE. I built https://github.com/hofstadter-io/hof as a tool for code generation.

The problem is less about who's writing the original code and more with others who come along, see ”cue" code, but the cue cli can't read it. If the tool changes the language, is it still CUE code? Essentially we would end up with numerous CUE dialects that cannot be read by the standard tool.

Having read your other open question, I think you'd want to go the route that dagger and hof use, attributes on schemas the tool understands, yet are still able to be run through the cue reference implementation

[cueckoo]

This issue has been migrated to cue-lang/cue#350.

For more details about CUE's migration to a new home, please see cue-lang/cue#1078.