Flink on Kubernetes.

Flink achitecture

• One active JobManager(JM)
• Several TaskManagers(TM)

One job per cluster

• Evolving job deployment philosophy
• Tie cluster lifecycle to job lifecycle)
• Ideally, the job is first-class.
  (think about running a job on a platform like Kubernetes, not a running Flink cluster first then submitting the job)

Flink-on-K8s

• JobManager Deployment(jobmanager-deployment.yaml)
  • maintain 1 replica of Flink container running as a JM
  • apply a label like "flink-jobmanager"
  • Jobmanager uses custom Flink Docker image "afilichkin/flink-k8s-with-submit-job" , it start JM and also submit the "app.jar"
• JM service(jobmanager-service.yaml)
  • make JobManager accessiable by a hostname & port
  • select pods with labebl "flink-jobmanager"
• TaskManager Deployment(taskmanager-deployment.yaml)
  • maintain n replica of Flink container running as a TM

custom Flink Docker image "afilichkin/flink-k8s-with-submit-job" has only one change in "docker-entrypoint-custom.sh":

#Customization start
#fix for https://issues.apache.org/jira/browse/FLINK-9937
echo " 127.0.0.1 ${JOB_MANAGER_RPC_ADDRESS}" >> /etc/hosts

echo "Submit flink job in background"
exec   /submit-flink-job.sh &
#Customization end

Main flow

flink-k8s: Flink image that can work with Kubernetes.

Flink image for Kubernetes that fixes Jobmanage connection issue https://issues.apache.org/jira/browse/FLINK-9937

You can use afilichkin/flink-k8s in jobmanager-deployment.yaml file

docker-entrypoint-custom.sh has only one change: echo " 127.0.0.1 ${JOB_MANAGER_RPC_ADDRESS}" >> /etc/hosts