feat/versions (#5512)

# Summary
Add a stage to the build process that sparse clones deps and puts them in the required places.

This should make it much easier to keep track of vendored deps, and similar.

Solves #5426 and helps makes progress towards #4825

This pr moves almost all thirdparty deps from being vendored to being added by the build process.

This pr should be deployed on test and some cgi scripts tested before merge, there is some possibility of issues. But it should be minimal.

One thing: we manually mention the versions of things in https://fsfe.org/about/js-licences

I think we could make this automatic perhaps?

This pr atm does make us dependent on our sources being online for full builds. This is common, but perhaps not acceptable for us?

We could instead do one of the following:
- cache downloaded repos outside of the website dir, so they survive a full rebuild
- make this script an updater script, so all deps are still vendored, but we have a nice script to update them. Not a big fan of this.
- Use nix to fetch all the deps. Now reliant on nixos org instead of github, significant barrier for new contributors.
- have a mirror of all the repos we use for builds on this git server. I kinda like this option, it leaves the most sensible and "normal" build process. Does not seem hard: https://docs.gitea.com/usage/repo-mirror

# State of deps
## Moved
- PHPmailer
- Bootstrap ( we were using different versions for the less and js. this is almost certainly A mistake, but I have kept it the same in this issue to minimize disruption, and will try and fix it later). Also we had some bootstrap js we do not use anywhere, we just have a small bit of custom bootstrap js, not the whole package.
- Lunr (could not get a minimized version from upstream repo, so we have lost minimization. Not very important as it is only used on search page)
- Jquery ( this can be dropped once we upgrade bootstrap, only used as a bootstrap dep)

## Could not move
- moderniser (I hope to drop this down the line, but we use a custom build of an old version that I could find no way to recreate/pull from their git repo. So I could not fetch it as a dep)
- the various xml files. These are incredibly static compared to everything else, so I did not bother.
- Fonts: they are often only available as release artifacts from some site, and so downloading them would massively complicate the dep script. I do not want to maintain a package manager, and so if we think that necessary I would advocate for nix. Also, some fonts are only available from google fonts which does not seem to expose versions. This would again be a problem to be solved with nix.

# Checklist
- [x] site looks correct locally
- [x] search works locally
- [x] no missing scripts using inspector locally
- [x] Deployed on test

Co-authored-by: Darragh Elliott <me@delliott.net>
Reviewed-on: #5512
Reviewed-by: tobiasd <tobiasd@fsfe.org>
Co-authored-by: delliott <delliott@fsfe.org>
Co-committed-by: delliott <delliott@fsfe.org>

build/fsfe_website_build/build.py

@@ -11,6 +11,7 @@ from pathlib import Path
 from textwrap import dedent
 
 from .lib.misc import lang_from_filename
+from .phase0.clean_cache import clean_cache
 from .phase0.full import full
 from .phase0.global_symlinks import global_symlinks
 from .phase0.prepare_early_subdirectories import prepare_early_subdirectories
@@ -32,6 +33,11 @@ def _parse_arguments() -> argparse.Namespace:
         help="Force a full rebuild of all webpages.",
         action="store_true",
     )
+    parser.add_argument(
+        "--clean-cache",
+        help="Clean the global cache stored in the platform cache directory.",
+        action="store_true",
+    )
     parser.add_argument(
         "--languages",
         help="Languages to build website in.",
@@ -107,6 +113,8 @@ def build(args: argparse.Namespace) -> None:
     with multiprocessing.Pool(args.processes) as pool:
         logger.info("Starting phase 0 - Global Conditional Setup")
 
+        if args.clean_cache:
+            clean_cache()
         # TODO Should also be triggered whenever any build python file is changed
         if args.full:
             full(args.source)

build/fsfe_website_build/globals.py

@@ -0,0 +1,10 @@
+# SPDX-FileCopyrightText: Free Software Foundation Europe e.V. <https://fsfe.org>
+#
+# SPDX-License-Identifier: GPL-3.0-or-later
+"""Global variables for build process."""
+
+from pathlib import Path
+
+from platformdirs import user_cache_dir
+
+CACHE_DIR = Path(user_cache_dir("fsfe-website-build", "fsfe"))

build/fsfe_website_build/phase0/clean_cache.py

@@ -0,0 +1,23 @@
+# SPDX-FileCopyrightText: Free Software Foundation Europe e.V. <https://fsfe.org>
+#
+# SPDX-License-Identifier: GPL-3.0-or-later
+
+"""Clean global cache."""
+
+import logging
+import shutil
+
+from fsfe_website_build.globals import CACHE_DIR
+
+logger = logging.getLogger(__name__)
+
+
+def clean_cache() -> None:
+    """Remove the global cache folder."""
+    logger.info("Cleaning global cache")
+    # Slightly more complex logic to handle the cache dir being a mount in docker
+    for item in CACHE_DIR.iterdir():
+        if item.is_file() or item.is_symlink():
+            item.unlink()
+        elif item.is_dir():
+            shutil.rmtree(item)

build/fsfe_website_build/phase0/full.py

@@ -25,12 +25,10 @@ def full(source: Path) -> None:
         [
             "git",
             "--git-dir",
-            str(source) + "/.git",
+            str(source / ".git"),
             "clean",
-            "-fdx",
+            "-ffdx",
             "--exclude",
             "/.venv",
-            "--exclude",
-            "/.nltk_data",
         ],
     )

build/fsfe_website_build/phase1/get_dependencies.py

@@ -0,0 +1,101 @@
+# SPDX-FileCopyrightText: Free Software Foundation Europe e.V. <https://fsfe.org>
+#
+# SPDX-License-Identifier: GPL-3.0-or-later
+"""Download the dependencies of a site, should it be necessary."""
+
+import logging
+import tomllib
+from collections import defaultdict
+from pathlib import Path
+
+from fsfe_website_build.globals import CACHE_DIR
+from fsfe_website_build.lib.misc import run_command
+
+logger = logging.getLogger(__name__)
+
+
+def fetch_sparse(
+    cache: Path,
+    repo: str,
+    rev: str,
+    file_mappings: list[dict[str, str]],
+) -> None:
+    """Clone source, and move necessary files into place with source/dest pairs."""
+    clone_dir = cache / f"{Path(repo).name}_{rev}"
+
+    if not (clone_dir / ".git").exists():
+        clone_dir.mkdir(exist_ok=True)
+        run_command(
+            ["git", "-C", str(clone_dir), "init", "--quiet", "--no-initial-branch"]
+        )
+        run_command(["git", "-C", str(clone_dir), "remote", "add", "origin", repo])
+        run_command(
+            ["git", "-C", str(clone_dir), "config", "core.sparseCheckout", "true"]
+        )
+        # Extract all paths for sparse checkout
+        paths = [mapping["source"] for mapping in file_mappings]
+        sparse_checkout_path = clone_dir / ".git" / "info" / "sparse-checkout"
+        # if path is ".", checkout the whole repo
+        if any(path == "." for path in paths):
+            sparse_checkout_path.write_text("/*\n")
+        else:
+            sparse_checkout_path.write_text("\n".join(paths) + "\n")
+
+        # Fetch the required revision
+        run_command(["git", "-C", str(clone_dir), "fetch", "--depth=1", "origin", rev])
+        # Checkout the fetched revision
+        run_command(["git", "-C", str(clone_dir), "checkout", "FETCH_HEAD"])
+    # Copy each file to its destination
+    for mapping in file_mappings:
+        # create our source path
+        # the source syntax is that of .gitignore, and so may have a leading /
+        # to say to interpret it only relative to the root
+        # and so we remove that so joining gives us a proper path
+        src = clone_dir / mapping["source"].lstrip("/")
+        dest_path = Path(mapping["dest"])
+        dest_path.parent.mkdir(parents=True, exist_ok=True)
+
+        run_command(
+            [
+                "rsync",
+                "-avz",
+                "--del",
+                "--exclude=.git",
+                str(src) if not src.is_dir() else str(src) + "/",
+                str(dest_path),
+            ]
+        )
+
+
+def get_dependencies(
+    source_dir: Path,
+) -> None:
+    """Download and put in place all website dependencies."""
+    logger.info("Getting Dependencies")
+    cache = CACHE_DIR / "repos"
+    cache.mkdir(parents=True, exist_ok=True)
+    deps_file = source_dir / "dependencies.toml"
+    if deps_file.exists():
+        with deps_file.open("rb") as file:
+            cfg = tomllib.load(file)
+
+        # Group file mappings by repository and revision
+        repo_tasks: defaultdict[tuple[str, str], list[dict[str, str]]] = defaultdict(
+            list
+        )
+
+        for data in cfg.values():
+            repo = str(data["repo"])
+            rev = str(data["rev"])
+            key = (repo, rev)
+
+            for file_set in data["file_sets"]:
+                # Make path relative to source dir
+                dest = source_dir / file_set["target"]
+                repo_tasks[key].append(
+                    {"source": str(file_set["source"]), "dest": str(dest)}
+                )
+
+        # Process each repository/revision only once
+        for (repo, rev), file_mappings in repo_tasks.items():
+            fetch_sparse(cache, repo, rev, file_mappings)

build/fsfe_website_build/phase1/run.py

@@ -12,6 +12,7 @@ directory tree and does not touch the target directory tree at all.
 import logging
 from typing import TYPE_CHECKING
 
+from .get_dependencies import get_dependencies
 from .prepare_subdirectories import prepare_subdirectories
 from .update_css import update_css
 from .update_defaultxsls import update_defaultxsls
@@ -35,6 +36,7 @@ def phase1_run(
 ) -> None:
     """Run all the necessary sub functions for phase1."""
     logger.info("Starting Phase 1 - Setup")
+    get_dependencies(source_site)
     update_css(source_site)
     update_stylesheets(source_site, pool)
     prepare_subdirectories(source, source_site, languages, processes)

build/fsfe_website_build_tests_ci/caching_test.py

@@ -15,13 +15,12 @@ def no_rebuild_twice_test(mocker: MockFixture) -> None:
     # first, run a full build
     args = Namespace(
         full=True,
+        clean_cache=False,
         languages=[
             "en",
-            "nl",
-            "de",
         ],
         log_level="CRITICAL",  # by only logging critical messages
-        # the build should be faster, as evaluating less trhings to strings
+        # the build should be faster, as evaluating less things to strings
         processes=8,
         source=Path(),
         serve=False,