git:// is not used -- let's remove it #111

Open
opened 2025-10-13 17:21:25 +00:00 by fkobi · 8 comments
Owner

Current state

Right now we store just the URL without the protocol & git suffix:

def __sanitize_url(url: str) -> str:
"""Remove schema and extension from an URL"""
if url:
if (scheme := url.find("://")) != -1:
url = url[scheme + 3 :] # noqa: E203
if url.lower().endswith(".git"):
url = url[:-4]
return url

Removing the suffix does not matter:

$ git ls-remote git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git HEAD
3a8660878839faadb4f1a6dd72c3179c1df56787	HEAD
$ git ls-remote git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux HEAD
3a8660878839faadb4f1a6dd72c3179c1df56787	HEAD

Later we try to retrieve information from that URL with guessing the protocols in that order:

for protocol in ("https", "git", "http"):
try:
latest_hash(protocol, url)
return protocol
except NotARepository:

The problem is that every entry in our database works with https:

for url in $(psql -U fsfe -d reuse -c "SELECT url FROM repository" | awk 'NR >2 {print}'); do
    git ls-remote -q https://$url HEAD > /dev/null || echo $url
done # no extensive testing done but I am 90% sure

So the git protocol is not even used.

I assume that we want https for encryption and if that is impossible we fallback to http.


Why drop it?

  1. We don't need it it's efficiency
    git:// is designed of efficient transfers of large volumes of data and we literally ask it for 46 bytes:
    $ git ls-remote https://git.fsfe.org/reuse/api HEAD | wc -c
    46
    
    and it's efficiency is it's only strength over http (that I know of)
  2. It would allow us to simplify things
## Current state Right now we store just the URL without the protocol & git suffix: https://git.fsfe.org/reuse/api/src/commit/2ea966aefff2f8f0d4130bea4fbf4dc51de27cbc/reuse_api/views.py#L36-L44 Removing the suffix does not matter: ```console $ git ls-remote git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git HEAD 3a8660878839faadb4f1a6dd72c3179c1df56787 HEAD $ git ls-remote git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux HEAD 3a8660878839faadb4f1a6dd72c3179c1df56787 HEAD ``` Later we try to retrieve information from that URL with guessing the protocols in that order: https://git.fsfe.org/reuse/api/src/commit/2ea966aefff2f8f0d4130bea4fbf4dc51de27cbc/reuse_api/scheduler.py#L94-L98 The problem is that every entry in our database works with https: ```sh for url in $(psql -U fsfe -d reuse -c "SELECT url FROM repository" | awk 'NR >2 {print}'); do git ls-remote -q https://$url HEAD > /dev/null || echo $url done # no extensive testing done but I am 90% sure ``` So the git protocol is not even used. I assume that we want https for encryption and if that is impossible we fallback to http. --- **Why drop it?** 1. We don't need it it's efficiency `git://` is designed of efficient transfers of large volumes of data and we literally ask it for 46 bytes: ```console $ git ls-remote https://git.fsfe.org/reuse/api HEAD | wc -c 46 ``` and it's efficiency is it's only strength over http (that I know of) 2. It would allow us to simplify things
Author
Owner

This would also require changing the website:
https://api.reuse.software/register

This would also require changing the website: https://api.reuse.software/register
Author
Owner

@carmenbianca @floriansnow do I have your ACK for this?

@carmenbianca @floriansnow do I have your ACK for this?
Owner

I guess that is biased by market share. There are solution for hosting git repositories that to not provide a Web UI for the repositories. Removing the git protocol would mean that the service could not be used by projects using these solutions. I vote against removing it.

I guess that is biased by market share. There are solution for hosting git repositories that to not provide a Web UI for the repositories. Removing the `git` protocol would mean that the service could not be used by projects using these solutions. I vote against removing it.
Author
Owner

I guess that is biased by market share.

I do not understand what you are referring to.


I agree with you on principle, that we should not exclude simpler hosting software.

However my suggestion is not principle but data-driven: git:// is unused even though it is supported.
There are also repositories using Mercurial, which we do not support at all.
We support the git protocol but do not use it.

We have too little manhours to spend them on maintaining the code that people not use (see git log from before I started contributing).
We should measure our strengths for our intentions -- I think it is way better to have a smaller project that works well rather than a messy project that barely works (and our code smells)

We can add the git protocol back when people ask for it and up until that point refactoring the code will be much easier!

> I guess that is biased by market share. I do not understand what you are referring to. --- I agree with you on principle, that we should not exclude simpler hosting software. However my suggestion is not principle but data-driven: **`git://` is unused** even though it is supported. There are also repositories using Mercurial, which we do not support at all. We support the git protocol but do not use it. We have too little manhours to spend them on maintaining the code that people not use (see git log from before I started contributing). We should [measure our strengths for our intentions](https://en.wiktionary.org/wiki/mierzy%C4%87_si%C5%82y_na_zamiary) -- I think it is way better to have a smaller project that works well rather than a messy project that barely works (and our code **smells**) We can add the git protocol back when people ask for it and up until that point refactoring the code will be much easier!
Owner

is unused even though it is supported

And this is, in my opinion, biased by the marked share of the forges that provide both, git:// and https://.

> is unused even though it is supported And this is, in my opinion, biased by the marked share of the forges that provide both, `git://` and `https://`.
Author
Owner

I would not call that bias but preference; our software checks for https first and git second.
That preference is our choice: d709385097

That preference is also irrelevant to the fact that currently we do not have any repositories in our database that are not accessible via https but are via git.

I would not call that bias but preference; our software checks for https first and git second. That preference is our choice: https://git.fsfe.org/reuse/api/commit/d709385097173c8753649b6d10111d797eaf1594 That preference is also irrelevant to the fact that currently we do not have any repositories in our database that are not accessible via https but are via git.
fkobi added the
component
backend
label 2025-11-14 09:42:41 +00:00
fkobi added the
prio
mid
label 2025-11-14 09:49:56 +00:00
Author
Owner

In my 2.0 rewrite of this git:// support will not be present.
It can be added later.

In my 2.0 rewrite of this `git://` support will not be present. It can be added later.
fkobi added this to the 2.0 -- A thought-out backend redesign milestone 2025-11-14 10:21:36 +00:00
fkobi self-assigned this 2025-11-14 10:21:40 +00:00
fkobi added the enhancement label 2025-12-17 11:37:01 +00:00
fkobi added the bug label 2025-12-17 11:41:56 +00:00
Author
Owner

I guess later is now, 5 months later.
If I can readd it quickly then it will be added before the deployment

I guess later is now, 5 months later. If I can readd it quickly then it will be added before the deployment
Sign in to join this conversation.