feat: allow environment variables to configure embedded connect

[agavra]

Description

With this patch, docker users will be able to configure their embedded connect cluster using env variables prefixed with KSQL_CONNECT_.

Testing done

Spun up ksqldb using the following docker compose:

  ksqldb-server:
    image: placeholder/confluentinc/ksql-rest-app:local.build
    hostname: ksqldb-server
    container_name: ksqldb-server
    depends_on:
      - broker
      - schema-registry
    ports:
      - "8088:8088"
    environment:
      KSQL_BOOTSTRAP_SERVERS: "broker:9092"
      KSQL_HOST_NAME: ksqldb-server
      KSQL_LISTENERS: "http://0.0.0.0:8088"
      KSQL_CACHE_MAX_BYTES_BUFFERING: 0
      KSQL_KSQL_SCHEMA_REGISTRY_URL: "http://schema-registry:8081"
      KSQL_KSQL_LOGGING_PROCESSING_STREAM_AUTO_CREATE: "true"
      KSQL_KSQL_LOGGING_PROCESSING_TOPIC_AUTO_CREATE: "true"
      KSQL_LOG4J_OPTS: "-Dlog4j.configuration=file:/etc/ksql/log4j-rolling2.properties -Dlog4j.debug"
      KSQL_CONNECT_GROUP_ID: "ksql-connect-cluster"
      KSQL_CONNECT_BOOTSTRAP_SERVERS: "broker:9092"
      KSQL_CONNECT_KEY_CONVERTER: "io.confluent.connect.avro.AvroConverter"
      KSQL_CONNECT_KEY_CONVERTER_SCHEMA_REGISTRY: "http://schema-registry:8081"
      KSQL_CONNECT_VALUE_CONVERTER: "io.confluent.connect.avro.AvroConverter"
      KSQL_CONNECT_VALUE_CONVERTER_SCHEMA_REGISTRY: "http://schema-registry:8081"
      KSQL_CONNECT_CONFIG_STORAGE_TOPIC: "ksql-connect-configs"
      KSQL_CONNECT_OFFSET_STORAGE_TOPIC: "ksql-connect-offsets"
      KSQL_CONNECT_STATUS_STORAGE_TOPIC: "ksql-connect-statuses"
      KSQL_CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR: 1
      KSQL_CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR: 1
      KSQL_CONNECT_STATUS_STORAGE_REPLICATION_FACTOR: 1
    volumes:
      - ./log4j-rolling.properties:/etc/ksql/log4j-rolling2.properties

Also tested without a required param and it failed with the correct error message.

Reviewer checklist

• Ensure docs are updated if necessary. (eg. if a user visible feature is being added or changed).
• Ensure relevant issues are linked (description should include text like "Fixes #")

[rmoff]

This is great, thanks @agavra !

Two things:

1. I'd suggest is that we should stick to the KSQL prefix of all the other vars to be consistent
   e.g. KSQL_CONNECT_GROUP_ID rather than CONNECT_GROUP_ID
2. We should include the script for checking required values https://github.com/confluentinc/kafka-images/blob/master/kafka-connect-base/include/etc/confluent/docker/configure

[agavra]

I'd suggest is that we should stick to the KSQL prefix of all the other vars to be consistent
e.g. KSQL_CONNECT_GROUP_ID rather than CONNECT_GROUP_ID

@rmoff - I thought about this, but I thought it would make more sense to omit KSQL as they are being passed directly to connect (and for the sake of brevity) but if you feel strongly then I don't (thoughts @derekjn @MichaelDrogalis?)

[MichaelDrogalis]

I liked the reasoning behind @agavra's suggestion, too. But again I don't feel super strongly.

[derekjn]

@rmoff - I thought about this, but I thought it would make more sense to omit KSQL as they are being passed directly to connect (and for the sake of brevity) but if you feel strongly then I don't (thoughts @derekjn @MichaelDrogalis?)

I personally prefer @rmoff's suggestion because it feels more embedded to me. KSQL_CONNECT indicates to me that Connect is subordinate to KSQL, and IMO it draws a sharper distinction between configuring an external Connect cluster versus running in embedded mode. And while these parameters are being passed through to Connect, I see that as an implementation detail that users don't necessarily need to be aware of.

That being said, I am also fine with either option here.

[MichaelDrogalis]

@derekjn That seems like a convincing way to look at it to me. +1

Accepted both comments, will ship after green build

[agavra]

Merging without dev review as this is docker-only build change