Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Kafka Connect for z/OS support #1323

Open
wants to merge 13 commits into
base: 7.3.x
Choose a base branch
from
Open

Kafka Connect for z/OS support #1323

Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
4e34992
zos support for connect initial
stan-is-hate Feb 1, 2023
843193c
zos support for connect initial
stan-is-hate Feb 1, 2023
e3e6c25
zos support for connect initial
stan-is-hate Feb 1, 2023
37b17de
kerberos zos
stan-is-hate Feb 1, 2023
b533188
force symlink bash
stan-is-hate Feb 1, 2023
d8d759d
processing for ibm mq connectors
stan-is-hate Feb 2, 2023
a86b67b
cleanup
stan-is-hate Feb 2, 2023
2da9f1f
fix env vars
stan-is-hate Feb 3, 2023
a71b033
only copy jar files
stan-is-hate Feb 23, 2023
ce6347e
Copy only MQ jars to ext folder to be loaded by the extension classlo…
pbadani Feb 28, 2023
fa6f6ee
Remove unused var.
pbadani Mar 1, 2023
ac2a417
Merge pull request #1357 from confluentinc/pbadani/ibm-mq
stan-is-hate Mar 1, 2023
ae0ef8a
updated parameters for zos_encode
Aug 21, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/hosts_example.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ all:

#### Setting Proxy Environment variables ####
## To set proxy env vars for the duration of playbook run, uncomment below block and set as necessary
# proxy_env:
# env_vars:
# http_proxy: http://proxy.example.com:8080
# https_proxy: https://proxy.example.com:8080
## Note: You must use Hostnames or IPs to define your no_proxy server addresses, CIDR ranges are not supported.
Expand Down
2 changes: 1 addition & 1 deletion playbooks/all.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
- name: Host Prerequisites
hosts: zookeeper:kafka_broker:schema_registry:kafka_connect:ksql:control_center:kafka_rest:kafka_connect_replicator
gather_facts: false
environment: "{{ proxy_env }}"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a good but breaking change. Can we make an alias for backward compatibility and deprecate proxy_env for couple of releases.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed

environment: "{{ env_vars }}"
tasks:
- import_role:
name: variables
Expand Down
6 changes: 3 additions & 3 deletions playbooks/control_center.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
hosts: control_center
gather_facts: false
tags: control_center
environment: "{{ proxy_env }}"
environment: "{{ env_vars }}"
tasks:
- import_role:
name: variables
Expand All @@ -26,7 +26,7 @@
hosts: control_center_parallel
gather_facts: false
tags: control_center
environment: "{{ proxy_env }}"
environment: "{{ env_vars }}"
tasks:
- import_role:
name: control_center
Expand All @@ -37,7 +37,7 @@
any_errors_fatal: true
gather_facts: false
tags: control_center
environment: "{{ proxy_env }}"
environment: "{{ env_vars }}"
tasks:
- import_role:
name: control_center
Expand Down
10 changes: 5 additions & 5 deletions playbooks/kafka_broker.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
hosts: kafka_broker
gather_facts: false
tags: kafka_broker
environment: "{{ proxy_env }}"
environment: "{{ env_vars }}"
tasks:
- import_role:
name: variables
Expand All @@ -26,7 +26,7 @@
hosts: kafka_broker_parallel
gather_facts: false
tags: kafka_broker
environment: "{{ proxy_env }}"
environment: "{{ env_vars }}"
tasks:
- import_role:
name: kafka_broker
Expand All @@ -35,7 +35,7 @@
hosts: kafka_broker_serial
gather_facts: false
tags: kafka_broker
environment: "{{ proxy_env }}"
environment: "{{ env_vars }}"
tasks:
- import_role:
name: kafka_broker
Expand All @@ -47,7 +47,7 @@
any_errors_fatal: true
gather_facts: false
tags: kafka_broker
environment: "{{ proxy_env }}"
environment: "{{ env_vars }}"
tasks:
- import_role:
name: kafka_broker
Expand All @@ -62,7 +62,7 @@
any_errors_fatal: true
gather_facts: false
tags: kafka_broker
environment: "{{ proxy_env }}"
environment: "{{ env_vars }}"
tasks:
- import_role:
name: kafka_broker
Expand Down
6 changes: 3 additions & 3 deletions playbooks/kafka_connect.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
hosts: kafka_connect
gather_facts: false
tags: kafka_connect
environment: "{{ proxy_env }}"
environment: "{{ env_vars }}"
tasks:
- import_role:
name: variables
Expand All @@ -26,7 +26,7 @@
hosts: kafka_connect_parallel
gather_facts: false
tags: kafka_connect
environment: "{{ proxy_env }}"
environment: "{{ env_vars }}"
tasks:
- import_role:
name: kafka_connect
Expand All @@ -37,7 +37,7 @@
any_errors_fatal: true
gather_facts: false
tags: kafka_connect
environment: "{{ proxy_env }}"
environment: "{{ env_vars }}"
tasks:
- import_role:
name: kafka_connect
Expand Down
6 changes: 3 additions & 3 deletions playbooks/kafka_connect_replicator.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
hosts: kafka_connect_replicator
gather_facts: false
tags: kafka_connect_replicator
environment: "{{ proxy_env }}"
environment: "{{ env_vars }}"
tasks:
- import_role:
name: variables
Expand All @@ -26,7 +26,7 @@
hosts: kafka_connect_replicator_parallel
gather_facts: false
tags: kafka_connect_replicator
environment: "{{ proxy_env }}"
environment: "{{ env_vars }}"
tasks:
- import_role:
name: kafka_connect_replicator
Expand All @@ -37,7 +37,7 @@
any_errors_fatal: true
gather_facts: false
tags: kafka_connect_replicator
environment: "{{ proxy_env }}"
environment: "{{ env_vars }}"
tasks:
- import_role:
name: kafka_connect_replicator
Expand Down
6 changes: 3 additions & 3 deletions playbooks/kafka_rest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
hosts: kafka_rest
gather_facts: false
tags: kafka_rest
environment: "{{ proxy_env }}"
environment: "{{ env_vars }}"
tasks:
- import_role:
name: variables
Expand All @@ -26,7 +26,7 @@
hosts: kafka_rest_parallel
gather_facts: false
tags: kafka_rest
environment: "{{ proxy_env }}"
environment: "{{ env_vars }}"
tasks:
- import_role:
name: kafka_rest
Expand All @@ -37,7 +37,7 @@
any_errors_fatal: true
gather_facts: false
tags: kafka_rest
environment: "{{ proxy_env }}"
environment: "{{ env_vars }}"
tasks:
- import_role:
name: kafka_rest
Expand Down
6 changes: 3 additions & 3 deletions playbooks/ksql.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
hosts: ksql
gather_facts: false
tags: ksql
environment: "{{ proxy_env }}"
environment: "{{ env_vars }}"
tasks:
- import_role:
name: variables
Expand All @@ -26,7 +26,7 @@
hosts: ksql_parallel
gather_facts: false
tags: ksql
environment: "{{ proxy_env }}"
environment: "{{ env_vars }}"
tasks:
- import_role:
name: ksql
Expand All @@ -36,7 +36,7 @@
serial: 1
gather_facts: false
tags: ksql
environment: "{{ proxy_env }}"
environment: "{{ env_vars }}"
tasks:
- import_role:
name: ksql
Expand Down
2 changes: 1 addition & 1 deletion playbooks/schema_registry.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
any_errors_fatal: true
gather_facts: false
tags: schema_registry
environment: "{{ proxy_env }}"
environment: "{{ env_vars }}"
tasks:
- import_role:
name: schema_registry
Expand Down
10 changes: 5 additions & 5 deletions playbooks/zookeeper.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
hosts: zookeeper
gather_facts: false
tags: zookeeper
environment: "{{ proxy_env }}"
environment: "{{ env_vars }}"
tasks:
- import_role:
name: variables
Expand All @@ -26,7 +26,7 @@
hosts: zookeeper_parallel
gather_facts: false
tags: zookeeper
environment: "{{ proxy_env }}"
environment: "{{ env_vars }}"
tasks:
- import_role:
name: zookeeper
Expand All @@ -35,7 +35,7 @@
hosts: zookeeper_serial
gather_facts: false
tags: zookeeper
environment: "{{ proxy_env }}"
environment: "{{ env_vars }}"
tasks:
- import_role:
name: zookeeper
Expand All @@ -47,7 +47,7 @@
any_errors_fatal: true
gather_facts: false
tags: zookeeper
environment: "{{ proxy_env }}"
environment: "{{ env_vars }}"
tasks:
- import_role:
name: zookeeper
Expand All @@ -62,7 +62,7 @@
any_errors_fatal: true
gather_facts: false
tags: zookeeper
environment: "{{ proxy_env }}"
environment: "{{ env_vars }}"
tasks:
- import_role:
name: zookeeper
Expand Down
7 changes: 7 additions & 0 deletions plugins/filter/filters.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ def filters(self):
'get_hostnames': self.get_hostnames,
'get_roles': self.get_roles,
'resolve_hostname': self.resolve_hostname,
'health_check_hostname': self.health_check_hostname,
'resolve_hostnames': self.resolve_hostnames,
'cert_extension': self.cert_extension,
'ssl_required': self.ssl_required,
Expand Down Expand Up @@ -91,6 +92,12 @@ def resolve_hostname(self, hosts_hostvars_dict):
else:
return hosts_hostvars_dict.get('inventory_hostname')

def health_check_hostname(self, hosts_hostvars_dict):
# if health_check_hostname var is provided, use it for healthcheck; if not, revert to the usual algorithm
hc_hostname = hosts_hostvars_dict.get('health_check_hostname')
return hc_hostname if hc_hostname else self.resolve_hostname(hosts_hostvars_dict)


def resolve_hostnames(self, hosts, hostvars_dict):
# Given a collection of hosts, usually from a group, will resolve the correct hostname to use for each.
hostnames = []
Expand Down
16 changes: 16 additions & 0 deletions roles/common/tasks/copy_files.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,10 +9,26 @@
loop: "{{ copy_files }}"

- name: Copy Files
when: ansible_os_family != "OS/390"
copy:
src: "{{ item.source_path }}"
dest: "{{ item.destination_path }}"
mode: "{{ item.file_mode | default('0640') }}"
owner: "{{user}}"
group: "{{group}}"
loop: "{{ copy_files }}"

- name: Copy Files (z/OS)
when: ansible_os_family == "OS/390"
ibm.ibm_zos_core.zos_copy:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this part of ansible core? If not, we need to add this dependency in our documentation.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

src: "{{ item.source_path }}"
dest: "{{ item.destination_path }}"
mode: "{{ item.file_mode | default('0640') }}"
owner: "{{ user }}"
group: "{{ group }}"
validate: yes
backup: yes
force: yes
is_binary: "{{ item.is_binary | default(False) }}"
remote_src: "{{ item.remote_src | default(False) }}"
loop: "{{ copy_files }}"
Loading