Adapt ytt syntax to leverage K8S CR completion during authoring

[gberche-orange]

Thanks for sharing the great work with ytt for reducing verbosity.

I had worked on similar approach using ytt and kappcontroller for deployment in a gitops maneer without local preprocessing.

My initial goal was not that much reducing verbosity but leveraging IDE built-in code assistance for authoring crossplane XRD and compositions (see crossplane/crossplane#3197 (comment) for screenshot)

I see that in the ytt resources such as the following, the IDE support can't be leveraged

devops-toolkit-crossplane/packages/ytt-k8s/_ytt_lib/google/config.yaml

Lines 59 to 66 in 986cf71

	- name: nodepool
	base:
	apiVersion: container.gcp.crossplane.io/v1beta1
	kind: NodePool
	spec:
	forProvider:
	locations:
	- us-east1-b

Here is code snippet of the yaml ytt templating logic I've used to leverage IDE coding assistance support to author XRD, and Composition

composition.yaml

#@ load("/cloudsql-resource-data.lib.yaml", "resource")
#@ load("/cloudsql-others-data.lib.yaml", "others")
apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
  name: xpostgresqlinstances.gcp.database.orange.com
  labels:
    provider: gcp
spec:
  writeConnectionSecretsToNamespace: 75-crossplane-gcp-cnx-secrets
  compositeTypeRef:
    apiVersion: database.orange.com/v1alpha1
    kind: XPostgreSQLInstance
  #@ resource_yaml_fragment = resource()
  #@ resource = { "base": dict(**resource_yaml_fragment) }
  #@ other_elements = dict(**others()["spec"]["resources"][0])
  #@ other_elements.pop("base")
  #@ resource.update(other_elements)
  #@ resources = [ resource ]
  resources: #@ resources

cloudsql-resource-data.lib.yaml:

#! To be excluded from output, this lib have no yaml document --- header
#@ load("@ytt:data", "data")
#@ def resource():
apiVersion: database.gcp.crossplane.io/v1beta1
kind: CloudSQLInstance
metadata:
  name: cloudsql
spec:
  forProvider:
    databaseVersion: POSTGRES_14
    region: #@ data.values.external_gcp_region
    settings:
      tier: db-f1-micro
      #!              tier: db-custom-1-3840 # 1 CPU, 3840 MB RAM. See https://cloud.google.com/sql/docs/postgres/create-instance#machine-types
      availabilityType: "ZONAL" #! for HA db use REGIONAL
      backupConfiguration: { #! for HA db ie. REGIONAL set both following to true
        binaryLogEnabled: false,
        enabled: false
      }
      
      #!      dataDiskType: PD_SSD
      dataDiskType: PD_HDD #! cheaper
      ipConfiguration:
        ipv4Enabled: false
        privateNetwork: #@ "projects/{}/global/networks/{}".format(data.values.external_gcp_project_id, data.values.external_gcp_poc_openshift_cluster_vpc_network)
        requireSsl: false
      databaseFlags: #! Orange security requirement
        - name: "log_checkpoints"
          value: "on"
        - name: "log_connections"
          value: "on"
        - name: "log_disconnections"
          value: "on"
        - name: "log_lock_waits"
          value: "on"
        - name: "log_temp_files"
          value: "10000" #! 10,000 KB. The security spec isn't providing a value and suggest "0" as default value:  If all temporary files are not logged, it may be more difficult to identify potential performance issues that may be due to either poor application coding or deliberate resource starvation attempts.

  writeConnectionSecretToRef:
    name: default-name-overriden-by-patch
    namespace: 75-crossplane-gcp-cnx-secrets
#@ end

cloudsql-others-data.lib.yaml:

#! To be excluded from output, this lib have no yaml document --- header
#@ def others():
apiVersion: apiextensions.crossplane.io/v1
kind: Composition
spec:
  resources:
    - patches:
        - type: FromCompositeFieldPath
          fromFieldPath: "metadata.uid"
          toFieldPath: "spec.writeConnectionSecretToRef.name"
          transforms:
            - type: string
              string:
                fmt: "%s-postgresql"
        #!        - fromFieldPath: "spec.parameters.storageGB"
        #!          toFieldPath: "spec.forProvider.settings.dataDiskSizeGb"

        #! Try to implement the service binding spec
        #! https://github.com/servicebinding/spec#provisioned-service
        - type: ToCompositeFieldPath
          fromFieldPath: metadata.labels[crossplane.io/claim-name]
          toFieldPath: status.binding.name

        #!  To facilitate discoverability, it is RECOMMENDED that a CustomResourceDefinition exposing a Provisioned Service add servicebinding.io/provisioned-service: "true" as a label.
        - type: ToCompositeFieldPath
          toFieldPath: metadata.labels[servicebinding.io/provisioned-service]
          fromFieldPath: status.atProvider.settingsVersion #! try to wait for a field which is only available when the resource is ready
          transforms:
            #! first convert int to string
            - type: convert
              convert:
                toType: string
            #! then format string with zero length
            - type: string
              string:
                fmt: "true%.0s"
                #! constant string %.0s prints the string with a max zero width. Otherwise gofmt reports
                #! an error about the argument not being used.
                #! EXTRA string
                #! more at https://pkg.go.dev/fmt
                #! https://github.com/golang/go/issues/8151
                #! interactive test https://go.dev/play/
      connectionDetails:
      - type: FromConnectionSecretKey
        name: host
        fromConnectionSecretKey: privateIP
      - type: FromConnectionSecretKey
        name: username
        fromConnectionSecretKey: username
      - type: FromConnectionSecretKey
        name: password
        fromConnectionSecretKey: password
      - type: FromValue
        name: port
        value: "5432"
      - type:  FromValue
        name: type
        value: "postgresql"
      - type:  FromValue
        name: provider
        value: "gcp cloudsql"
      - type:  FromValue
        name: database #! required by spring-clod-bindings when jdbc-url is missing, see https://github.com/spring-cloud/spring-cloud-bindings#postgresql-rdbms
        value: ""



      base: {}

  compositeTypeRef:
    apiVersion: dummy
    kind: dummy

#@ end

I'm still looking for a solution to reduce proportion of imperative syntax, and reduce starlak editions when adding new resources. The overlays is something I need to explore.

I wonder whether this is something you would be interested in exploring ?

[vfarcic]

I'm not sure I would go deeper myself. For a while now, I switched to cdk8s for managing more complex (tedious) cases like Crossplane packages. The ability to download CRDs and generate libraries that, later on, I can use when constructing packages was a winner for me. It makes working in IDEs like VS Code very pleasant.