theforeman.foreman.content_upload fails with error "non_field_errors: Chunk size does not match content range."

[pescobar]

SUMMARY

theforeman.foreman.content_upload fails with error "non_field_errors: Chunk size does not match content range." when uploading this binary file

The same tasks works ok when uploading a small txt file instead of a binary file

The file triggering the error can be uploaded using hammer doing hammer repository upload-content --organization-id 1 --id 1548 --content-type file --path /tmp/docker-compose-Linux-x86_64 . Hammer upload test is done from the same host where katello is running. Ansible task triggering the error is executed from a different host.

ISSUE TYPE

• Bug Report

ANSIBLE VERSION

ansible 2.10.2
  config file = /home/escobar/ansible/playbooks/ansible-playbook-foreman-katello/ansible.cfg
  configured module search path = ['/home/escobar/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/escobar/tmp/venv_ansible_2.10.2/lib/python3.6/site-packages/ansible
  executable location = /home/escobar/tmp/venv_ansible_2.10.2/bin/ansible
  python version = 3.6.8 (default, Nov 21 2019, 19:31:34) [GCC 8.3.1 20190507 (Red Hat 8.3.1-4)]

COLLECTION VERSION

# /home/escobar/ansible/playbooks/ansible-playbook-foreman-katello/collections/ansible_collections
Collection         Version
------------------ -------
theforeman.foreman 1.4.0

KATELLO/FOREMAN VERSION

tfm-rubygem-katello-3.16.0-1.el7.noarch
foreman-2.1.2-1.el7.noarch

STEPS TO REPRODUCE

wget -P /tmp/ https://github.com/docker/compose/releases/download/1.27.4/docker-compose-Linux-x86_64

- name: "Create product 'docker-compose'"
  theforeman.foreman.product:
    username: "{{ foreman_username }}"
    password: "{{ foreman_password }}"
    server_url: "{{ foreman_url }}"
    validate_certs: "{{ foreman_validate_certs }}"
    organization: "{{ foreman_initial_organization }}"
    name: "docker-compose"
    state: present

- name: "Create files repository 'docker-compose-linux-x86_64' for product 'docker-compose'"
  theforeman.foreman.repository:
    username: "{{ foreman_username }}"
    password: "{{ foreman_password }}"
    server_url: "{{ foreman_url }}"
    validate_certs: "{{ foreman_validate_certs }}"
    organization: "{{ foreman_initial_organization }}"
    name: "docker-compose-linux-x86_64"
    state: present
    content_type: "file"
    product: "docker-compose"
  register: _repo_info

- name: "Upload a test file"
  theforeman.foreman.content_upload:
    username: "{{ foreman_username }}"
    password: "{{ foreman_password }}"
    server_url: "{{ foreman_url }}"
    validate_certs: "{{ foreman_validate_certs }}"
    organization: "{{ foreman_initial_organization }}"
    src: "/tmp/docker-compose-Linux-x86_64"
    #src: "/tmp/lele.txt"
    product: "docker-compose"
    repository: "docker-compose-linux-x86_64"

EXPECTED RESULTS

File should be uploaded to the repository but it fails. The same task works OK when uploading a small txt file. e.g

echo "test" > /tmp/lelele.txt

- name: "Upload a test file"
  theforeman.foreman.content_upload:
    username: "{{ foreman_username }}"
    password: "{{ foreman_password }}"
    server_url: "{{ foreman_url }}"
    validate_certs: "{{ foreman_validate_certs }}"
    organization: "{{ foreman_initial_organization }}"
    src: "/tmp/lelele.txt"
    product: "docker-compose"
    repository: "docker-compose-linux-x86_64"

[evgeni]

Thanks for the report!

Do you know from the top of your head whether you're using pilp2 or pulp3 for file contents? If that's a fresh Katello 3.16 it should be 3, but on upgraded installs it might be either.

I'll look what hammer does.

[evgeni]

Interestingly, our chunk size is even smaller than hammers...

foreman-ansible-modules/plugins/modules/content_upload.py

Line 94 in 909e920

CONTENT_CHUNK_SIZE = 2 * 1024 * 1024

https://github.com/Katello/hammer-cli-katello/blob/ccc1b9d541b66778cce4855e87246b83dc7b8683/lib/hammer_cli_katello/repository.rb#L347

[pescobar]

Do you know from the top of your head whether you're using pilp2 or pulp3 for file contents? If that's a fresh Katello 3.16 it should be 3, but on upgraded installs it might be either.

I got this from administer >> about >> smart proxies >> services

Pulp Version 2.1.0
Pulp server version 2.21.3 

I think this katello instance was upgraded from 3.15

[evgeni]

Thanks, I finally managed to find some time and repro this on a fresh 3.16. Will dig further.

[evgeni]

I think we need to port Katello/hammer-cli-katello@9276ae5

[pescobar]

Glad to hear you could reproduce the issue and found a possible fix. Let me know if you want me to do some testing.

[evgeni]

Weird, we have the size parameter implemented since #659

[evgeni]

The only other commit in hammer that touches this is Katello/hammer-cli-katello@0d0f877#diff-fe176ff6849ed0578304d0d8c0aa8b54fefa5acb8ba48579cb94748b3d14829f

[evgeni]

Okay, so the following patch should fix it for you:

--- plugins/modules/content_upload.py
+++ plugins/modules/content_upload.py
@@ -174,7 +174,7 @@ def main():
 
                 with open(b_src, 'rb') as contentfile:
                     for chunk in iter(lambda: contentfile.read(CONTENT_CHUNK_SIZE), b""):
-                        data = {'content': chunk, 'offset': offset, 'size': len(chunk)}
+                        data = {'content': chunk, 'offset': offset, 'size': size}
                         module.resource_action('content_uploads', 'update', params=content_upload_scope, data=data)
 
                         offset += len(chunk)

Now I just need to find out why it worked before XD

[mdellweg]

Now I just need to find out why it worked before XD

Maybe no one ever chunked a file before...

[evgeni]

Maybe no one ever chunked a file before...

I do almost daily, just not against Pulp3, so it seems its ignored on Pulp3 installs.

[mdellweg]

Hmm, looking at pulp code, there is no evidence that the size field is used anywhere. Not sure, how katello translates that though. Pulp expects the content range as a header specifying start and end byte.

https://github.com/pulp/pulpcore/blob/master/pulpcore/app/serializers/upload.py#L27

[evgeni]

Yeah, before the patch, the Katello generated header was "range bytes 2097152-2097152/2097152", which is so wrong ;)

[mdellweg]

Sure but still only the first two numbers of that content range are processed by pulp. Katello is capping finish with size, which when given the chunk size, will produce exactly that in the second chunk.

https://github.com/Katello/katello/blob/540e695c5e108df99d84acae620ab641b6c898f0/app/services/katello/pulp3/content.rb#L54

[pescobar]

I tested the patch and I could upload the file. Thanks @evgeni !