Support static pages

[SoniEx2]

It'd be nice if static pages were supported because we want to rewrite our thing in Rust and for security reasons we recommend not running complex scripts and programs on the HTTP server, which means we recommend configuring all git repos with the so-called "dumb" protocol.

Our current thing currently just uses git directly and clones/fetches everything into a single cache repo, which means we can't parallelize it, so we're looking into alternatives for that, whether by using something like gitoxide or having multiple temporary caches and joining them together after fetching. However, having static pages support is a must.

[Byron]

Thanks for letting me know! The current stance of gitoxide is to not implement the 'dumb' protocol on the client side. This is due to it being seemingly underspecified.

From what's stated here I gather that you are interested in the server side. There I also don't know how it would work at all to generate these caches and what the problems are specifically that you are running into.

I think further comments here could try to fill in the gaps to understand the requirements and allow an implementation to happen.

Thank you.

[SoniEx2]

Nah, we only need it on the "client" (the ability to clone).

Take a look at https://github.com/ganarchy/GAnarchy

On the "server", or more specifically on the git host, plain git works fine - git is good enough at generating static pages when pushing over SSH, so there's no point trying to replace it there.

[Byron]

I took a cursory look at GAnarchy but it didn't jump at me how gitoxide could help there.

Since the latter isn't able yet to do an entire clone itself and lots of features are missing, I recommend checking back when there is more of the feature set that I think you would need.

In the meantime, I am closing this issue as it lacks the detail necessary to aid any implementation.

[SoniEx2]

These would be replaced with a Gitoxide-based equivalent: https://github.com/ganarchy/GAnarchy/blob/ganarchy/ganarchy/git.py

Many of the repos here do not support the v1/v2 protocol: https://github.com/ganarchy/ganarchy.github.io/blob/master/index.toml

[SoniEx2]

The "dumb" protocol is simply ".git dir over HTTP"

Since HTTP doesn't provide an index, git has to provide that index, but otherwise you just have a .git like you would on a filesystem. This comes with all the stuff a normal .git comes with including packfiles and the like.

We'd like to ask for this again, as our plan is to have self-embedding git repos for our HTTP-based projects, and it'd be nice if gitoxide could handle them so we don't need to require system-wide git.

[Byron]

Welcome back!

I took a cursory look at git.py and thought that a lot of that should be doable in gitoxide already. When it comes to maintaining an index to support the 'dumb' protocol, I would think that something like this exists or can be written by the ganarchy project as a git hook maybe.

By the end of this year gitoxide will be production ready for cloning/fetching repositories and checking out a worktree correctly and completely, if that helps.

[SoniEx2]

Can gitoxide fetch all of the repos listed here? https://ganarchy.autistic.space/index.toml

This is the main blocker for us being able to use gitoxide.

(Also, that new git.py (it is updated from when it was originally linked) was written specifically to parallelize a large amount of git fetch. And it does an amazing job at that.)

[Byron]

Here is a listing of the supported transport protocols - dump http is not among them so I presume gitoxide cannot fetch these repos. It looks like git2 also doesn't support this protocol, so the chances of ever supporting it on the client side in gitoxide aren't high unless the implementation is contributed.

GitHub doesn't support line-links in markdowns unfortunately, so here is what the link above should have displayed.

[SoniEx2]

how hard would it be to use the on-disk format code ("the on-disk format must remain compatible, and we will never contend with it.") in the HTTP transport?

(it'd be nice to reuse it so it can be kept in sync... vs maintaining a separate copy)

[Byron]

how hard would it be to use the on-disk format code ("the on-disk format must remain compatible, and we will never contend with it.") in the HTTP transport?

I don't know how these two related and would assume one is entirely independent of another. So one would probably have to find out.

[SoniEx2]

the static pages protocol is just the on-disk format over HTTP. so they're very related.

decoupling them allows better transfer optimizations but then you kinda have to maintain the same thing twice.

[Byron]

Your work will be valuable as it will pave the way for eventually merging such capability back into mainline or be the basis for a backport. It's not on my agenda at all to add static http support, but I am happy to assist with questions should they arise on your own journey on getting it done.

[SoniEx2]

hmm, thoughts on optimizing gitoxide for network-attached storage and the like? (gitoxide over google drive maybe?) such work would likely benefit attempts to add static pages protocol support (at least, using shared code).

basically, can you make the on-disk format driver backend-agnostic and optimized for latency?

[Byron]

hmm, thoughts on optimizing gitoxide for network-attached storage and the like?

No. But I think a first step would be to get anything to work in this regard.

[SoniEx2]

actually hmm does gitoxide have support for VFS and local filesystem clone?

[Byron]

gitoxide can perform a git-aware clone from a location specified by path. Since I don't know what VFS means exactly in this context, I'd say 'no' to this one.

[SoniEx2]

hmm... does that mean this is unsupported?

       -l, --local
           When the repository to clone from is on a local machine, this flag
           bypasses the normal "Git aware" transport mechanism and clones the
           repository by making a copy of HEAD and everything under objects
           and refs directories. The files under .git/objects/ directory are
           hardlinked to save space when possible.

           If the repository is specified as a local path (e.g.,
           /path/to/repo), this is the default, and --local is essentially a
           no-op. If the repository is specified as a URL, then this flag is
           ignored (and we never use the local optimizations). Specifying
           --no-local will override the default when /path/to/repo is given,
           using the regular Git transport instead.

           NOTE: this operation can race with concurrent modification to the
           source repository, similar to running cp -r src dst while modifying
           src.

[Byron]

That is unsupported indeed - gitoxide currently uses 'git-aware' mechanisms only. Local clone optimizations can happen at some point though, it's just not a priority yet.

[SoniEx2]

ahh. hmm...

"VFS" just means virtual filesystem. e.g. sshfs. however we now realize that wouldn't work for implementing this. we think we just have to do it properly... does gitoxide support http/2 (especially multiplexing)? that would help us a lot. where would a proper implementation of this go?

[Byron]

http is supported in a blocking fashion via curl and the implementation of that is here: https://github.com/Byron/gitoxide/blob/9509ce4faeca8b4e1527bac625370403495bb03c/git-transport/src/client/blocking_io/http/mod.rs .

Since you are interested in static http support, I recommend adding support for it into the curl-based http implementation. I think right now it specifically checks for the 'smart' transport, and with the right test or two it should be possible to support the 'dumb' one as well without breakage.