feat: improve clone performance and reduce memory usage.

[orisano]

I tried to clone github.com/knqyf263/vuln-list, but PlainClone took a too long time.
I improve clone performance and reduce memory usage!

benchstat results

name          old time/op    new time/op    delta
PlainClone-4      605s ± 4%      173s ± 5%  -71.42%  (p=0.008 n=5+5)

name          old alloc/op   new alloc/op   delta
PlainClone-4    64.9GB ± 0%    29.8GB ± 0%  -54.12%  (p=0.008 n=5+5)

name          old allocs/op  new allocs/op  delta
PlainClone-4      117M ± 0%      114M ± 0%   -2.50%  (p=0.008 n=5+5)

thanks!

[jfontan]

Have you found substantial difference using a buffered writer to save the index?

[orisano]

local benchmark result. reverted bufio.Writer.

name          old time/op    new time/op    delta
PlainClone-4      173s ± 5%      251s ± 9%  +44.84%  (p=0.008 n=5+5)

name          old alloc/op   new alloc/op   delta
PlainClone-4    29.8GB ± 0%    29.4GB ± 0%   -1.12%  (p=0.008 n=5+5)

name          old allocs/op  new allocs/op  delta
PlainClone-4      114M ± 0%      114M ± 0%   -0.04%  (p=0.008 n=5+5)

[jfontan]

I'm doing more tests with our repository downloader and these changes makes it a little bit slower and consumes more memory. I'll continue digging why this could be.

[jfontan]

It seems that the biggest improvement are the changes in worktree.go and storage/filesystem/index.go. Can yo split this PR in two? One with changes for those files and another with changes for packfile code.

I've been reviewing the changes for packfile and make sense but in some cases they make the code slower. I'm pretty sure this can be improved to get good gains in both memory and speed when it's generating the index.

@mcuadros what do you think?

[orisano]

Ok, I split PR.

[orisano]

Shall I close this PR?

[jfontan]

@orisano I would love to take a look to the changes for packfile you did. Even if they are not increasing speed now they look very promising. Can you open a new PR with only the changes from the files in plumbing/format/packfile/*. We can work from there to improve them.

Thanks!

[orisano]

ok

[jfontan]

The changes from this PR are now in #1180