Skip to content

Commit

Permalink
k8s_cp - a new module for copying files to/from a Pod (#127)
Browse files Browse the repository at this point in the history
* k8s_cp module

* add documentation for k8s_cp module

* add doc for the new module

* pods should be running

* support for binary, archive and zip file

* sanity

* Delete file.txt

* remove unused

* set back

* Update collection.txt

* Update test_copy_errors.yml

* Update plugins/modules/k8s_cp.py

Co-authored-by: Abhijeet Kasurde <[email protected]>

* Update k8s_cp.py

* Update k8s_cp.py

* tar binary requirements

* Update common.py

* Update k8s_cp.py

* Update k8s_cp.py

* replace kind with binary file

* Update test_copy_large_file.yml

* Update plugins/action/k8s_info.py

Co-authored-by: Mike Graves <[email protected]>

* Update k8s_info.py

* Update k8s_info.py

* Update k8s_cp.py

Co-authored-by: Abhijeet Kasurde <[email protected]>
Co-authored-by: Mike Graves <[email protected]>
  • Loading branch information
3 people authored Jul 26, 2021
1 parent 2f59c3d commit c330c7e
Show file tree
Hide file tree
Showing 31 changed files with 2,007 additions and 2 deletions.
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,7 @@ Name | Description
[kubernetes.core.helm_template](https://github.com/ansible-collections/kubernetes.core/blob/main/docs/kubernetes.core.helm_template_module.rst)|Render chart templates
[kubernetes.core.k8s](https://github.com/ansible-collections/kubernetes.core/blob/main/docs/kubernetes.core.k8s_module.rst)|Manage Kubernetes (K8s) objects
[kubernetes.core.k8s_cluster_info](https://github.com/ansible-collections/kubernetes.core/blob/main/docs/kubernetes.core.k8s_cluster_info_module.rst)|Describe Kubernetes (K8s) cluster, APIs available and their respective versions
[kubernetes.core.k8s_cp](https://github.com/ansible-collections/kubernetes.core/blob/main/docs/kubernetes.core.k8s_cp_module.rst)|Copy files and directories to and from pod.
[kubernetes.core.k8s_exec](https://github.com/ansible-collections/kubernetes.core/blob/main/docs/kubernetes.core.k8s_exec_module.rst)|Execute command in Pod
[kubernetes.core.k8s_info](https://github.com/ansible-collections/kubernetes.core/blob/main/docs/kubernetes.core.k8s_info_module.rst)|Describe Kubernetes (K8s) objects
[kubernetes.core.k8s_json_patch](https://github.com/ansible-collections/kubernetes.core/blob/main/docs/kubernetes.core.k8s_json_patch_module.rst)|Apply JSON patch operations to existing objects
Expand Down
549 changes: 549 additions & 0 deletions docs/kubernetes.core.k8s_cp_module.rst

Large diffs are not rendered by default.

3 changes: 3 additions & 0 deletions meta/runtime.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ action_groups:
- k8s_log
- k8s_scale
- k8s_service
- k8s_cp

plugin_routing:
action:
Expand Down Expand Up @@ -43,6 +44,8 @@ plugin_routing:
redirect: kubernetes.core.k8s_info
k8s_service:
redirect: kubernetes.core.k8s_info
k8s_cp:
redirect: kubernetes.core.k8s_info
inventory:
openshift:
redirect: community.okd.openshift
Expand Down
5 changes: 5 additions & 0 deletions molecule/default/converge.yml
Original file line number Diff line number Diff line change
Expand Up @@ -177,6 +177,11 @@
tags:
- helm

- role: k8scopy
tags:
- copy
- k8s

post_tasks:
- name: Ensure namespace exists
k8s:
Expand Down
15 changes: 15 additions & 0 deletions molecule/default/roles/k8scopy/defaults/main.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
---
# defaults file for k8copy
copy_namespace: copy

pod_with_one_container:
name: pod-copy-0
container: container-00

pod_with_two_container:
name: pod-copy-1
container:
- container-10
- container-11

kubectl_path: /tmp/kubectl
Binary file not shown.
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
kubernetes.core
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
k8s_cp
1 change: 1 addition & 0 deletions molecule/default/roles/k8scopy/files/data/file.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
This is a simple file used to test k8s_cp module on ansible.
2 changes: 2 additions & 0 deletions molecule/default/roles/k8scopy/files/data/teams/ansible.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
cloud team
content team
Binary file added molecule/default/roles/k8scopy/files/hello
Binary file not shown.
1 change: 1 addition & 0 deletions molecule/default/roles/k8scopy/files/simple_file.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
This content will be copied into remote Pod.
Binary file not shown.
91 changes: 91 additions & 0 deletions molecule/default/roles/k8scopy/library/k8s_create_file.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
#!/usr/bin/python
# -*- coding: utf-8 -*-
# Copyright: (c) 2021, Aubin Bikouo <@abikouo>
# GNU General Public License v3.0+ (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt)

from __future__ import absolute_import, division, print_function

__metaclass__ = type


DOCUMENTATION = r'''
module: k8s_diff
short_description: Create large file with a defined size.
author:
- Aubin Bikouo (@abikouo)
description:
- This module is used to validate k8s_cp module.
options:
path:
description:
- The destination path for the file to create.
type: path
required: yes
size:
description:
- The size of the output file in MB.
type: int
default: 400
binary:
description:
- If this flag is set to yes, the generated file content binary data.
type: bool
default: False
'''

EXAMPLES = r'''
- name: create 150MB file
k8s_diff:
path: large_file.txt
size: 150
'''


RETURN = r'''
'''

import os

from ansible.module_utils.basic import AnsibleModule
from ansible.module_utils._text import to_native


def execute_module(module):
try:
size = module.params.get('size') * 1024 * 1024
path = module.params.get('path')
write_mode = "w"
if module.params.get('binary'):
content = os.urandom(size)
write_mode = "wb"
else:
content = ""
count = 0
while len(content) < size:
content += f"This file has been generated using ansible: {count}\n"
count += 1

with open(path, write_mode) as f:
f.write(content)
module.exit_json(changed=True, size=len(content))
except Exception as e:
module.fail_json(msg="failed to create file due to: {0}".format(to_native(e)))


def main():
argument_spec = {}
argument_spec['size'] = {'type': 'int', 'default': 400}
argument_spec['path'] = {'type': 'path', 'required': True}
argument_spec['binary'] = {'type': 'bool', 'default': False}
module = AnsibleModule(argument_spec=argument_spec)

execute_module(module)


if __name__ == '__main__':
main()
215 changes: 215 additions & 0 deletions molecule/default/roles/k8scopy/library/kubectl_file_compare.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,215 @@
#!/usr/bin/python
# -*- coding: utf-8 -*-
# Copyright: (c) 2021, Aubin Bikouo <@abikouo>
# GNU General Public License v3.0+ (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt)

from __future__ import absolute_import, division, print_function

__metaclass__ = type


DOCUMENTATION = r'''
module: kubectl_file_compare
short_description: Compare file and directory using kubectl
author:
- Aubin Bikouo (@abikouo)
description:
- This module is used to validate k8s_cp module.
- Compare the local file/directory with the remote pod version
notes:
- This module authenticates on kubernetes cluster using default kubeconfig only.
options:
namespace:
description:
- The pod namespace name
type: str
required: yes
pod:
description:
- The pod name
type: str
required: yes
container:
description:
- The container to retrieve files from.
type: str
remote_path:
description:
- Path of the file or directory on Pod.
type: path
required: yes
local_path:
description:
- Path of the local file or directory.
type: path
content:
description:
- local content to compare with remote file from pod.
- mutually exclusive with option I(local_path).
type: path
required: yes
args:
description:
- The file is considered to be an executable.
- The tool will be run locally and on pod and compare result from output and stderr.
type: list
kubectl_path:
description:
- Path to the kubectl executable, if not specified it will be download.
type: path
'''

EXAMPLES = r'''
- name: compare local /tmp/foo with /tmp/bar in a remote pod
kubectl_file_compare:
namespace: some-namespace
pod: some-pod
remote_path: /tmp/bar
local_path: /tmp/foo
kubectl_path: /tmp/test/kubectl
- name: Compare executable running help command
kubectl_file_compare:
namespace: some-namespace
pod: some-pod
remote_path: /tmp/test/kubectl
local_path: kubectl
kubectl_path: /tmp/test/kubectl
args:
- "--help"
'''


RETURN = r'''
'''

import os
import filecmp

from tempfile import NamedTemporaryFile, TemporaryDirectory
from ansible.module_utils.basic import AnsibleModule


def kubectl_get_content(module, dest_dir):
kubectl_path = module.params.get('kubectl_path')
if kubectl_path is None:
kubectl_path = module.get_bin_path('kubectl', required=True)

namespace = module.params.get('namespace')
pod = module.params.get('pod')
file = module.params.get('remote_path')

cmd = [
kubectl_path,
'cp',
"{0}/{1}:{2}".format(namespace, pod, file)
]
container = module.params.get('container')
if container:
cmd += ['-c', container]
local_file = os.path.join(dest_dir, os.path.basename(module.params.get('remote_path')))
cmd.append(local_file)
rc, out, err = module.run_command(cmd)
return local_file, err, rc, out


def kubectl_run_from_pod(module):
kubectl_path = module.params.get('kubectl_path')
if kubectl_path is None:
kubectl_path = module.get_bin_path('kubectl', required=True)

cmd = [
kubectl_path,
'exec',
module.params.get('pod'),
'-n',
module.params.get('namespace')
]
container = module.params.get('container')
if container:
cmd += ['-c', container]
cmd += ['--', module.params.get('remote_path')]
cmd += module.params.get('args')
return module.run_command(cmd)


def compare_directories(dir1, dir2):
test = filecmp.dircmp(dir1, dir2)
if any([len(test.left_only) > 0, len(test.right_only) > 0, len(test.funny_files) > 0]):
return False
(t, mismatch, errors) = filecmp.cmpfiles(dir1, dir2, test.common_files, shallow=False)
if len(mismatch) > 0 or len(errors) > 0:
return False
for common_dir in test.common_dirs:
new_dir1 = os.path.join(dir1, common_dir)
new_dir2 = os.path.join(dir2, common_dir)
if not compare_directories(new_dir1, new_dir2):
return False
return True


def execute_module(module):

args = module.params.get('args')
local_path = module.params.get('local_path')
namespace = module.params.get('namespace')
pod = module.params.get('pod')
file = module.params.get('remote_path')
content = module.params.get('content')
if args:
pod_rc, pod_out, pod_err = kubectl_run_from_pod(module)
rc, out, err = module.run_command([module.params.get('local_path')] + args)
if rc == pod_rc and out == pod_out:
module.exit_json(msg=f"{local_path} and {namespace}/{pod}:{file} are same.", rc=rc, stderr=err, stdout=out)
result = dict(local=dict(rc=rc, out=out, err=err), remote=dict(rc=pod_rc, out=pod_out, err=pod_err))
module.fail_json(msg=f"{local_path} and {namespace}/{pod}:{file} are same.", **result)
else:
with TemporaryDirectory() as tmpdirname:
file_from_pod, err, rc, out = kubectl_get_content(module=module, dest_dir=tmpdirname)
if not os.path.exists(file_from_pod):
module.fail_json(msg="failed to copy content from pod", error=err, output=out)

if content is not None:
with NamedTemporaryFile(mode="w") as tmp_file:
tmp_file.write(content)
tmp_file.flush()
if filecmp.cmp(file_from_pod, tmp_file.name):
module.exit_json(msg=f"defined content and {namespace}/{pod}:{file} are same.")
module.fail_json(msg=f"defined content and {namespace}/{pod}:{file} are same.")

if os.path.isfile(local_path):
if filecmp.cmp(file_from_pod, local_path):
module.exit_json(msg=f"{local_path} and {namespace}/{pod}:{file} are same.")
module.fail_json(msg=f"{local_path} and {namespace}/{pod}:{file} are same.")

if os.path.isdir(local_path):
if compare_directories(file_from_pod, local_path):
module.exit_json(msg=f"{local_path} and {namespace}/{pod}:{file} are same.")
module.fail_json(msg=f"{local_path} and {namespace}/{pod}:{file} are same.")


def main():
argument_spec = {}
argument_spec['namespace'] = {'type': 'str', 'required': True}
argument_spec['pod'] = {'type': 'str', 'required': True}
argument_spec['container'] = {}
argument_spec['remote_path'] = {'type': 'path', 'required': True}
argument_spec['local_path'] = {'type': 'path'}
argument_spec['content'] = {'type': 'str'}
argument_spec['kubectl_path'] = {'type': 'path'}
argument_spec['args'] = {'type': 'list'}
module = AnsibleModule(argument_spec=argument_spec,
mutually_exclusive=[('local_path', 'content')],
required_one_of=[['local_path', 'content']])

execute_module(module)


if __name__ == '__main__':
main()
3 changes: 3 additions & 0 deletions molecule/default/roles/k8scopy/meta/main.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
---
collections:
- kubernetes.core
Loading

0 comments on commit c330c7e

Please sign in to comment.