Debugging slow/huge Cue inputs

[nyarly]

I've come up against the issue of CUE code taking a long time and lots of memory to process a number of times, and I still haven't really gotten a feel for how to fix this. Most recently, after 20 minutes my cue export got OOMkilled. It's unclear to me what exactly causes the time and space of CUE processing to run away like this, or how to find those causes.

For reference, this is about 100kloc of CUE, that results in about 300kloc of Kubernetes YAML. There's a fair amount of literal inclusion there, from importing upstream raw config - I'd guess about 80%.

That said, the stalling out seems to happen as I refactor to better represent intent. My goal right now is to be able to compose the features required by a particular cluster out of struct-based configurations. Eventually I might also want to remove Kustomize from our workflow, but right now it's serving a valuable purpose patching resources and unifying namespaces. I'm making strides towards mergeable feature configurations, only to be held up by extremely lengthy builds.

As new releases of the tool drop, they've made noticeable differences in build speed. Which is greatly appreciated! My temperament wants to solve its own problems, though. Even if that means contributing PRs up to the toolchain.

(I'm on track to get our code into the private unity program, and I'm excited to see how that pans out.)

[verdverm]

I have found manually "unrolling" to improve speeds drastically. This has been largely about inlining definitions, but of course means losing reusable components. There are a number of issues tagged with performance and some known optimizations still to be implemented.

The first thing I might recommend w.r.t. PRs is getting to know the evaluator. There are a lot of comments and todos in the codebase as well as background reading that is helpful. The evaluator is a complicated thing and I'd guess @mpvl is the only one who really understands it's implementation right now. One thing that has helped me is running various simple cases with expanded output. The majority of the interesting code is in the internal directory.

[nyarly]

Unrolling brings up an idea I've been considering for a while: staged CUE compilation. As a way of speeding things up, I'm considering breaking up my project into packages, and then evaluating source packages into "compiled" versions. For instance, I've been using a lot of Kubernetes go imports to get validation at the CUE level - which has been really valuable. But I wonder if I couldn't put all of that into a "schema" package, evaluate it once (or when-changed), and then import the pre-evaluated version. I think I'd need to do something to "hide" the imports I was making, or else they'd be brought into the "DLL" I was using.

[verdverm]

That is an interesting idea, please share any findings. I'd imagine a small example to prove out the workflow would be useful for everyone.

[seh]

I tried to use cue def to accomplish what @nyarly suggested, but it doesn't "consume and compact" the imported packages.

[cueckoo]

This discussion has been migrated to cue-lang/cue#1074.

For more details about CUE's migration to a new home, please see cue-lang/cue#1078.