Cannot mirror galaxy

[bmbouter]

When using pulp_ansible 0.10.1 at least if you use the reproducer below, you'll receive this exception:

pulp [42527f144b314854a96714c904237add]: pulpcore.tasking.pulpcore_worker:INFO: Task c1c4a3a2-0883-4371-bb19-302b6768d5f6 failed ('NoneType' object has no attribute 'get')
pulp [42527f144b314854a96714c904237add]: pulpcore.tasking.pulpcore_worker:INFO:   File "/usr/local/lib/pulp/lib64/python3.9/site-packages/pulpcore/tasking/pulpcore_worker.py", line 370, in _perform_task
    result = func(*args, **kwargs)
  File "/usr/local/lib/pulp/lib64/python3.9/site-packages/pulp_ansible/app/tasks/collections.py", line 145, in sync
    repo_version = d_version.create()
  File "/usr/local/lib/pulp/lib64/python3.9/site-packages/pulpcore/plugin/stages/declarative_version.py", line 161, in create
    loop.run_until_complete(pipeline)
  File "/usr/lib64/python3.9/asyncio/base_events.py", line 642, in run_until_complete 
    return future.result()
  File "/usr/local/lib/pulp/lib64/python3.9/site-packages/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
    await asyncio.gather(*futures)
  File "/usr/local/lib/pulp/lib64/python3.9/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__
    await self.run()
  File "/usr/local/lib/pulp/lib64/python3.9/site-packages/pulpcore/plugin/stages/content_stages.py", line 168, in run
    await sync_to_async(process_batch)()
  File "/usr/local/lib/pulp/lib64/python3.9/site-packages/asgiref/sync.py", line 414, in __call__
    ret = await asyncio.wait_for(future, timeout=None)
  File "/usr/lib64/python3.9/asyncio/tasks.py", line 442, in wait_for
    return await fut
  File "/usr/lib64/python3.9/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/pulp/lib64/python3.9/site-packages/asgiref/sync.py", line 455, in thread_handler
    return func(*args, **kwargs)
  File "/usr/local/lib/pulp/lib64/python3.9/site-packages/pulpcore/plugin/stages/content_stages.py", line 166, in process_batch
    self._post_save(batch)
  File "/usr/local/lib/pulp/lib64/python3.9/site-packages/pulp_ansible/app/tasks/collections.py", line 912, in _post_save
    collection_version.requires_ansible = runtime_yaml.get("requires_ansible")

This happens repeatedly. The reproducer is:

#!/bin/bash

set -ev

repo_name="repo$RANDOM"
remote_name="remote$RANDOM"
url="https://galaxy.ansible.com"
distribution_name="distribution$RANDOM"
base_path="$RANDOM"

pulp ansible remote -t collection create --name $remote_name --url $url --download-concurrency 10 --rate-limit 8
pulp ansible repository create --name $repo_name --remote $remote_name
pulp ansible repository sync --name $repo_name

echo "Created and synced repo $repo_name"

[bmbouter]

The issue I'm seeing is for the collection manifest below. When the code reads from meta/runtime.yml there is something in there, but it's not valid yaml because runtime_yaml is None which causes pulp_ansible to fail syncing it.

{
 "collection_info": {
  "namespace": "inexio",
  "name": "thola",
  "version": "1.0.0",
  "authors": [
   "Thola team [[email protected]]"
  ],
  "readme": "README.md",
  "tags": [
   "collection"
  ],
  "description": "Integrates the functionalities of Thola into Ansible",
  "license": [],
  "license_file": "LICENSE",
  "dependencies": {},
  "repository": "https://github.com/inexio/thola-ansible",
  "documentation": null,
  "homepage": "https://thola.io",
  "issues": null
 },
 "file_manifest_file": {
  "name": "FILES.json",
  "ftype": "file",
  "chksum_type": "sha256",
  "chksum_sha256": "8b1c818c21f4a3d1ad2327c60269485d0534c0719a1c3f43f4ac70b2df05aa45",
  "format": 1
 },
 "format": 1
}

[awcrosby]

When collections come through the galaxy-importer for the galaxy_ng repo, we require that requires_ansible exists in meta/runtime.yml. But for the galaxy.ansible.com galaxy repo we do not require that, so a workaround to handle that is probably best.