Dock sources like Git or storage systems need additional processing

[saheljalal]

go-getter seems to only be able to download entire folders or repositories/buckets likely and not a single file. So if you try to reference a single file to a private git repo like this:

nostromo dock git::https://github.com/org/private-repo/blob/main/manifest.yaml?sshkey=<key>

it'll fail. We'll need to detect these folders first and keep track of the root to download the source first into the /tmp folder, and then extract the actual file from there into the downloads/ folder for processing the merge.

Also, have not successfully been able to clone private repo either yet, so needs investigation and fix.

[gedw99]

I am using go-getter to grab single files on public repos and it works.

Under the hood it downloads the whole folder and caches it

Maybe it’s a big on private repos ?

[saheljalal]

Ah maybe, will need to try out on public repo. I assumed it should work but couldn't get it since I was trying to get a file from a private repo. Thanks! Will update ticket to reflect after trying.

[saheljalal]

Finally got this working in latest release. A couple of issues I was running into:

• The download location was being created first and I think it was causing issues for me so made unique folder references now
• The URLs for git needed either explicit git:: prefixes or should match the exact HTTPS clone link from GitHub for me

Now dock locations should be able to read multiple manifests from repos or single files and sync.