Merge fetches full repo?

[arvidj]

I recently had the following time out with marge bot:

2021-07-15 08:30:47,908 INFO Unassigning from MR !3210
self.git('fetch', '--prune', remote_name)
raise TimeoutExpired(
File "/marge-bot/marge/git.py", line 201, in _run
return _run(*command, env=env, check=True, timeout=timeout_seconds)
File "/marge-bot/marge/git.py", line 185, in git
subprocess.TimeoutExpired: Command '[b'git', b'-C', b'/tmp/tmp2tl8pzw3/tmphm0rwq7u', b'fetch', b'--prune', b'source']' timed out after 120.0 seconds
File "/marge-bot/marge/git.py", line 52, in fetch
self._repo.fetch(
File "/marge-bot/marge/job.py", line 262, in fetch_source_project
source_project, source_repo_url, _ = self.fetch_source_project(merge_request)
File "/marge-bot/marge/single_merge_job.py", line 50, in update_merge_request_and_accept
subprocess.TimeoutExpired: Command '[b'git', b'-C', b'/tmp/tmp2tl8pzw3/tmphm0rwq7u', b'fetch', b'--prune', b'source']' timed out after 120.0 seconds
File "/marge-bot/marge/single_merge_job.py", line 24, in execute
Traceback (most recent call last):
The above exception was the direct cause of the following exception:
self.update_merge_request_and_accept(approvals)
raise TimeoutExpired(
File "/usr/lib/python3.8/subprocess.py", line 1072, in _check_timeout
self._check_timeout(endtime, orig_timeout, stdout, stderr)
File "/usr/lib/python3.8/subprocess.py", line 1869, in _communicate
stdout, stderr = self._communicate(input, endtime, timeout)
File "/usr/lib/python3.8/subprocess.py", line 1028, in communicate
stdout, stderr = process.communicate(input, timeout=timeout)
File "/marge-bot/marge/git.py", line 197, in _run
Traceback (most recent call last):
2021-07-15 08:30:47,405 ERROR Unexpected Exception
2021-07-15 08:28:37,374 INFO Running git -C /tmp/tmp2tl8pzw3/tmphm0rwq7u fetch --prune source
2021-07-15 08:28:37,370 INFO Running git -C /tmp/tmp2tl8pzw3/tmphm0rwq7u remote add source [email protected]:nomadic-labs/tezos.git
2021-07-15 08:28:37,370 WARNING stderr: b"fatal: No such remote: 'source'\n"
2021-07-15 08:28:37,370 WARNING stdout: b''
2021-07-15 08:28:37,370 WARNING git returned 128
2021-07-15 08:28:37,366 INFO Running git -C /tmp/tmp2tl8pzw3/tmphm0rwq7u remote rm source
2021-07-15 08:28:35,401 INFO Ensuring MR !3210 is mergeable
2021-07-15 08:28:34,057 INFO Processing !3210 - 'Snoop: inline latex library'
2021-07-15 08:28:34,057 INFO Attempting to merge the oldest MR...
2021-07-15 08:28:34,057 INFO Got 1 requests to merge;

If I'm reading correctly, Marge is attempting to fetch the full source repo. However, our repo is a bit large, it's normal that it takes some time. Is it really necessary to fetch the full repo? Couldn't it just fetch the source branch?

[snim2]

Good question, perhaps it's worth thinking about shallow cloning too?

[xtermi2]

I would also vote to support shallow cloning, maybe via a configflag:

--git-shallow-clone GIT_SHALLOW_CLONE
                        The number of changes to fetch from GitLab when cloning a repository. Lower values can speed up marge, especially on large repositories. Set to 0 or blank to fetch all branches and tags.
                           [env var: MARGE_GIT_SHALLOW_CLONE] (default: 0)

We have a lage repo an cloning takes about 2 minutes and shallow cloning would speed-up marge a lot.