NewBie: Evaluation of CUE

[cueckoo]

Originally opened by @anjidevarasetty in cuelang/cue#508

Hello everyone,

Newbie Q:

We are working on large number of configuration json files and evaluating CUE to perform operations on that huge data.
Here my requirement is to create & update json files based on requirements.
We are thinking such a way that create/update json files with CUE.

my repo:

  services
    ├── configuration_files
    │    └── 6000 json files
    │── cue
    │   └── cue files to create/modify json files under directory configuration_files

one of my json file

{
  "apiVersion": "v1",
  "kind": "cost_optimization",
  "metadata": {
    "name": "cost_optimization_aug_2020",
    "namespace": "cloud_cost"
  },
  "spec": {
    "cloud": [
      {
        "cloud_name": "GCP",
        "optimization": [
          {
            "pod_id": "-1",
            "module_enable": "0"
          }
        ]
      }
    ]
  }
}

use-case 1:

Based on module_enabled= "1", needs to be updated above json like below

{
  "apiVersion": "v1",
  "kind": "cost_optimization",
  "metadata": {
    "name": "cost_optimization_aug_2020",
    "namespace": "cloud_cost"
  },
  "spec": {
    "cloud": [
      {
        "cloud_name": "GCP",
        "optimization": [
          {
            "pod_id": "-1",
            "module_enable": "1"
          }
        ]
      }
    ]
  }
}

My idea is to provide higher level cue to the users like

cat <<EOF > update_configs.cue
update_configs: {
	module_enabled: "1"
	config_file_list: [
		"cost_optimization_july_2020",
		"cost_optimization_aug_2020"
	]
}
EOF

$cue export update_configs.cue

so when I run cue export <input.cue> , from configurations_list each configuration files needs to be updated with module_enabled value provided in higher level cue.

use-case 2:

Need to create new json cost_optimization_sep_2020 from cost_optimization_aug_2020 and update field value of metadata.name.

For this higher level cue file is

cat <<EOF > clone_configs.cue
clone_configurations: {
	configuration_name:        "cost_optimization_sep_2020"
	configuration_source:      "cost_optimization_aug_2020"
}
EOF

$cue export clone_configs.cue

when I run cue export <input.cue>, new configuration file cost_optimization_sep_2020 needs to be cloned from cost_optimization_aug_2020 and update required fields. it will be like

{
  "apiVersion": "v1",
  "kind": "cost_optimization",
  "metadata": {
    "name": "cost_optimization_sep_2020",
    "namespace": "cloud_cost"
  },
  "spec": {
    "cloud": [
      {
        "cloud_name": "GCP",
        "optimization": [
          {
            "pod_id": "-1",
            "module_enable": "1"
          }
        ]
      }
    ]
  }
}

How can i achieve these kind of operations with CUE ? please provide me some example for my two use-cases

[cueckoo]

Original reply by @cedricgc in cuelang/cue#508 (comment)

If we are looking for the direct answer for those challenges, CUE's upcoming query syntax and code modification capabilities would be a good fit to automate configuration changes (may be only exposed through the Go API right now, unsure). It seems that you control the platform this is used in. Instead of making a tool to make these changes, would you able to make these changes for the user without their intervention?

If you are open to changing your approach, I do have some ideas on how you can use CUE, especially if you are using Kubernetes

[cueckoo]

Original reply by @prakashrj in cuelang/cue#508 (comment)

@cedricgc Thank you very much for your reply. We control the platform and the intention of these higher level abstractions is to hide the complexity from users and provide self-service experience. We use kubernetes and very eager to hear your ideas.

[cueckoo]

Original reply by @cedricgc in cuelang/cue#508 (comment)

Hey, put some thought into this and I wanted clarify. This is less about CUE and more using the platform to help your users. Most of the time users are looking for their platform to take care of things for them, so that they can focus on other things. On the other hand, servicing a broad set of users means that there will be some use cases where there needs to be full control to configure the low level details as well. I think to provide a good experience for everyone a platform needs create higher level abstractions and have a interface that allows for delegating decision-making

Looking at the second example above, the challenge is that cost analysis needs to be run on a schedule, but cost_optimization represents a single run and the onus is on the user to create a new resource every time. Instead, why not create a higher level resource (something like service_cost_analysis) that a user only needs to configure once for their service? Similar to a K8s Deployment that creates as many Pods as needed, we can shift the scheduling of cost analysis to a controller and free up the user from doing so. An added benefit would be reducing the number of files in the repository, since the user is working on higher level policy instead of individual units.

CUE is attractive because it is a better interface for your users. I argue that configuration is usually necessary (as I said in the Configuration Complexity Curse), but we need better tools like a language to deal with it effectively. Now that your users can use libraries with CUE, there is the option for freeing up decision making by setting good defaults and helpers that can be set up and used. This is different from encapsulation where we hide implementation details because users across the platform need different levels of configuration detail. With CUE you do not need to hide details (more important for config) but still create higher level workflows.

[cueckoo]

Original reply by @prakashrj in cuelang/cue#508 (comment)

@cedricgc Thank you very much for sharing your thoughts. Your responses are always insightful. We also have similar goals to what you have described. We use Kubernetes Abstrations to cover all life cycle operations of a service and to reconcile. Many times we might end up with several CRDs or big resource file as we cover all usecases to manage a service. To hide that complexity we are looking upto Cue to create higher level abstractions, as you have explained in detail.

These higher level abstractions are not blackboxes hiding details, but are helpers that set/change default configuration values. To cover all use cases of these higher level abstraction, we should have ability to add, update and delete configuration.

We could cover all known usecases to update configuration. We are currently working on the following to cover other use cases:

1. Adding configuration by selecting predefined validations.
2. Deleting configuration.
3. Deleting/Adding/Replacing values of a list.

We are currently hacking through those items and hoping that Cue can handle them all. Would love to hear your thoughts on how you have handled similar scenarios.