Support static pages

[SoniEx2]

It'd be nice if static pages were supported because we want to rewrite our thing in Rust and for security reasons we recommend not running complex scripts and programs on the HTTP server, which means we recommend configuring all git repos with the so-called "dumb" protocol.

Our current thing currently just uses git directly and clones/fetches everything into a single cache repo, which means we can't parallelize it, so we're looking into alternatives for that, whether by using something like gitoxide or having multiple temporary caches and joining them together after fetching. However, having static pages support is a must.

[Byron]

Thanks for letting me know! The current stance of gitoxide is to not implement the 'dumb' protocol on the client side. This is due to it being seemingly underspecified.

From what's stated here I gather that you are interested in the server side. There I also don't know how it would work at all to generate these caches and what the problems are specifically that you are running into.

I think further comments here could try to fill in the gaps to understand the requirements and allow an implementation to happen.

Thank you.

[SoniEx2]

Nah, we only need it on the "client" (the ability to clone).

Take a look at https://github.com/ganarchy/GAnarchy

On the "server", or more specifically on the git host, plain git works fine - git is good enough at generating static pages when pushing over SSH, so there's no point trying to replace it there.

[Byron]

I took a cursory look at GAnarchy but it didn't jump at me how gitoxide could help there.

Since the latter isn't able yet to do an entire clone itself and lots of features are missing, I recommend checking back when there is more of the feature set that I think you would need.

In the meantime, I am closing this issue as it lacks the detail necessary to aid any implementation.

[SoniEx2]

These would be replaced with a Gitoxide-based equivalent: https://github.com/ganarchy/GAnarchy/blob/ganarchy/ganarchy/git.py

Many of the repos here do not support the v1/v2 protocol: https://github.com/ganarchy/ganarchy.github.io/blob/master/index.toml

[SoniEx2]

The "dumb" protocol is simply ".git dir over HTTP"

Since HTTP doesn't provide an index, git has to provide that index, but otherwise you just have a .git like you would on a filesystem. This comes with all the stuff a normal .git comes with including packfiles and the like.

We'd like to ask for this again, as our plan is to have self-embedding git repos for our HTTP-based projects, and it'd be nice if gitoxide could handle them so we don't need to require system-wide git.

[Byron]

Welcome back!

I took a cursory look at git.py and thought that a lot of that should be doable in gitoxide already. When it comes to maintaining an index to support the 'dumb' protocol, I would think that something like this exists or can be written by the ganarchy project as a git hook maybe.

By the end of this year gitoxide will be production ready for cloning/fetching repositories and checking out a worktree correctly and completely, if that helps.

[SoniEx2]

Can gitoxide fetch all of the repos listed here? https://ganarchy.autistic.space/index.toml

This is the main blocker for us being able to use gitoxide.

(Also, that new git.py (it is updated from when it was originally linked) was written specifically to parallelize a large amount of git fetch. And it does an amazing job at that.)

[Byron]

Here is a listing of the supported transport protocols - dump http is not among them so I presume gitoxide cannot fetch these repos. It looks like git2 also doesn't support this protocol, so the chances of ever supporting it on the client side in gitoxide aren't high unless the implementation is contributed.

GitHub doesn't support line-links in markdowns unfortunately, so here is what the link above should have displayed.

[SoniEx2]

how hard would it be to use the on-disk format code ("the on-disk format must remain compatible, and we will never contend with it.") in the HTTP transport?

(it'd be nice to reuse it so it can be kept in sync... vs maintaining a separate copy)

[Byron]

how hard would it be to use the on-disk format code ("the on-disk format must remain compatible, and we will never contend with it.") in the HTTP transport?

I don't know how these two related and would assume one is entirely independent of another. So one would probably have to find out.

[SoniEx2]

the static pages protocol is just the on-disk format over HTTP. so they're very related.

decoupling them allows better transfer optimizations but then you kinda have to maintain the same thing twice.

[Byron]

Your work will be valuable as it will pave the way for eventually merging such capability back into mainline or be the basis for a backport. It's not on my agenda at all to add static http support, but I am happy to assist with questions should they arise on your own journey on getting it done.

[SoniEx2]

hmm, thoughts on optimizing gitoxide for network-attached storage and the like? (gitoxide over google drive maybe?) such work would likely benefit attempts to add static pages protocol support (at least, using shared code).

basically, can you make the on-disk format driver backend-agnostic and optimized for latency?