Intermittent issue while trying to sync community devsec.hardening collection

[newswangerd]

Original issue: https://issues.redhat.com/browse/AAH-1202

[bmbouter]

What versions of pulpcore and pulp_ansible should we test against?

[newswangerd]

Pulpcore 3.15.0 and pulp ansible 0.10.1

[bmbouter]

We should attempt to reproduce this, and then determine (if we can) if this pulpcore change resolves it.

[mdellweg]

We should attempt to reproduce this, and then determine (if we can) if this pulpcore change resolves it.

Actually i think this fix needs amendments:
pulp/pulpcore#1851
pulp/pulpcore#2122

[bmbouter]

@newswangerd which pulpcore versions do these fixes need to go into? Just 3.15.z?

[newswangerd]

@bmbouter just 3.15.z

[bmbouter]

Here's a script I used to try to sync galaxy, which includes the rate limiting values galaxy_ng uses.

#!/bin/bash

set -ev

repo_name="repo$RANDOM"
remote_name="remote$RANDOM"
url="https://galaxy.ansible.com"
distribution_name="distribution$RANDOM"
base_path="$RANDOM"

pulp ansible remote -t collection create --name $remote_name --url $url --download-concurrency 10 --rate-limit 8
pulp ansible repository create --name $repo_name --remote $remote_name
pulp ansible repository sync --name $repo_name

echo "Created and synced repo $repo_name"

This is on a system where I pip install pulp_ansible==0.10.1.

[bmclaughlin]

The default rate limiting options we use in galaxy_ng are:
download concurrency = 10
rate limit = 8

[bmbouter]

thanks @bmclaughlin I've revised the reproducer to use those values and I'm rerunning the full mirror now.

[bmbouter]

Well I didn't get the exception of interest, but I did get a failure. I got this:

pulp [42527f144b314854a96714c904237add]: pulpcore.tasking.pulpcore_worker:INFO: Task c1c4a3a2-0883-4371-bb19-302b6768d5f6 failed ('NoneType' object has no attribute 'get')
pulp [42527f144b314854a96714c904237add]: pulpcore.tasking.pulpcore_worker:INFO:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/pulpcore/tasking/pulpcore_worker.py", line 370, in _perform_task
    result = func(*args, **kwargs)
  File "/usr/local/lib/pulp/lib64/python3.9/site-packages/pulp_ansible/app/tasks/collections.py", line 145, in sync
    repo_version = d_version.create()
  File "/usr/local/lib/pulp/lib64/python3.9/site-packages/pulpcore/plugin/stages/declarative_version.py", line 161, in create
    loop.run_until_complete(pipeline)
  File "/usr/lib64/python3.9/asyncio/base_events.py", line 642, in run_until_complete 
    return future.result()
  File "/usr/local/lib/pulp/lib64/python3.9/site-packages/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
    await asyncio.gather(*futures)
  File "/usr/local/lib/pulp/lib64/python3.9/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__
    await self.run()
  File "/usr/local/lib/pulp/lib64/python3.9/site-packages/pulpcore/plugin/stages/content_stages.py", line 168, in run
    await sync_to_async(process_batch)()
  File "/usr/local/lib/pulp/lib64/python3.9/site-packages/asgiref/sync.py", line 414, in __call__
    ret = await asyncio.wait_for(future, timeout=None)
  File "/usr/lib64/python3.9/asyncio/tasks.py", line 442, in wait_for
    return await fut
  File "/usr/lib64/python3.9/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/pulp/lib64/python3.9/site-packages/asgiref/sync.py", line 455, in thread_handler
    return func(*args, **kwargs)
  File "/usr/local/lib/pulp/lib64/python3.9/site-packages/pulpcore/plugin/stages/content_stages.py", line 166, in process_batch
    self._post_save(batch)
  File "/usr/local/lib/pulp/lib64/python3.9/site-packages/pulp_ansible/app/tasks/collections.py", line 912, in _post_save
    collection_version.requires_ansible = runtime_yaml.get("requires_ansible")

[bmbouter]

I received the same exception again. I believe there is a data quality issue on galaxy.ansible.com that our code is not adequately handling.

[bmbouter]

I moved that issue to it's own issue here to track it's fix separately. It's not the same root cause as the claim in this bug. #806 I'm going to fix 806 and then resume with this.

[mdellweg]

Pulpcore 3.15.0 and pulp ansible 0.10.1

Trying to install from the git tags:

ERROR: Cannot install pulp-ansible==0.10.1 and pulpcore 3.15.0 (from /home/vagrant/devel/pulpcore) because these package versions have conflicting dependencies.

The conflict is caused by:
    The user requested pulpcore 3.15.0 (from /home/vagrant/devel/pulpcore)
    pulp-ansible 0.10.1 depends on pulpcore<3.17 and >=3.15.1

How sure are you about those numbers?

[bmbouter]

Yeah when I installed pulp_ansible==0.10.1 I actually received pulpcore==3.16.2

[mdellweg]

I will try to reproduce with this snippet now:

#!/bin/bash

set -ev

for i in $(seq 100)
do
    repo_name="repo$RANDOM"
    remote_name="remote$RANDOM"
    url="https://galaxy.ansible.com"
    distribution_name="distribution$RANDOM"
    base_path="$RANDOM"
    requirements='collections: [devsec.hardening]'

    pulp ansible remote -t collection create --name $remote_name --url $url --download-concurrency 10 --rate-limit 8 --requirements "$requirements"
    pulp ansible repository create --name $repo_name --remote $remote_name
    pulp ansible repository sync --name $repo_name

    echo "Created and synced repo $repo_name"

    pulp ansible content --type collection-version list | jq 'length'

    pulp ansible repository destroy --name "$repo_name"
    pulp ansible remote destroy --name "$remote_name"
    pulp orphan cleanup --protection-time 0
done

Versions:

{
      "component": "core",
      "version": "3.16.0"
    },
    {
      "component": "certguard",
      "version": "1.6.0.dev"
    },
    {
      "component": "herminig",
      "version": "0.1.0a1.dev"
    },
    {
      "component": "file",
      "version": "1.10.1"
    },
    {
      "component": "deb",
      "version": "2.15.1"
    },
    {
      "component": "container",
      "version": "2.9.1"
    },
    {
      "component": "ansible",
      "version": "0.10.1"
    }

edit: No luck here. Seems like i ran into rate limiting. But no trace of the reported bug.

[bmbouter]

I was able to sync all collections on galaxy.ansible.com (13488 collectionVersions) in 1 hour and 24 minutes with no errors. I have not been able to reproduce the issue. I'm going to close this issue while awaiting more info on a reproducer.