FSFE/fsfe-website
27
32
Fork 93
Code Issues 63 Pull Requests 18 Wiki Activity

feat/versions #5512

Merged
tobiasd merged 19 commits from feat/versions into master 2025-12-24 13:00:47 +00:00
Conversation 10 Commits 19 Files Changed 110 +960 -15404
delliott commented 2025-11-21 14:47:08 +00:00
Member
Copy Link

Summary

Add a stage to the build process that sparse clones deps and puts them in the required places.

This should make it much easier to keep track of vendored deps, and similar.

Solves #5426 and helps makes progress towards #4825

This pr moves almost all thirdparty deps from being vendored to being added by the build process.

This pr should be deployed on test and some cgi scripts tested before merge, there is some possibility of issues. But it should be minimal.

One thing: we manually mention the versions of things in https://fsfe.org/about/js-licences

I think we could make this automatic perhaps?

This pr atm does make us dependent on our sources being online for full builds. This is common, but perhaps not acceptable for us?

We could instead do one of the following:

  • cache downloaded repos outside of the website dir, so they survive a full rebuild
  • make this script an updater script, so all deps are still vendored, but we have a nice script to update them. Not a big fan of this.
  • Use nix to fetch all the deps. Now reliant on nixos org instead of github, significant barrier for new contributors.
  • have a mirror of all the repos we use for builds on this git server. I kinda like this option, it leaves the most sensible and "normal" build process. Does not seem hard: https://docs.gitea.com/usage/repo-mirror

State of deps

Moved

  • PHPmailer
  • Bootstrap ( we were using different versions for the less and js. this is almost certainly A mistake, but I have kept it the same in this issue to minimize disruption, and will try and fix it later). Also we had some bootstrap js we do not use anywhere, we just have a small bit of custom bootstrap js, not the whole package.
  • Lunr (could not get a minimized version from upstream repo, so we have lost minimization. Not very important as it is only used on search page)
  • Jquery ( this can be dropped once we upgrade bootstrap, only used as a bootstrap dep)

Could not move

  • moderniser (I hope to drop this down the line, but we use a custom build of an old version that I could find no way to recreate/pull from their git repo. So I could not fetch it as a dep)
  • the various xml files. These are incredibly static compared to everything else, so I did not bother.
  • Fonts: they are often only available as release artifacts from some site, and so downloading them would massively complicate the dep script. I do not want to maintain a package manager, and so if we think that necessary I would advocate for nix. Also, some fonts are only available from google fonts which does not seem to expose versions. This would again be a problem to be solved with nix.

Checklist

  • site looks correct locally
  • search works locally
  • no missing scripts using inspector locally
  • Deployed on test
# Summary Add a stage to the build process that sparse clones deps and puts them in the required places. This should make it much easier to keep track of vendored deps, and similar. Solves https://git.fsfe.org/FSFE/fsfe-website/issues/5426 and helps makes progress towards https://git.fsfe.org/FSFE/fsfe-website/issues/4825 This pr moves almost all thirdparty deps from being vendored to being added by the build process. This pr should be deployed on test and some cgi scripts tested before merge, there is some possibility of issues. But it should be minimal. One thing: we manually mention the versions of things in https://fsfe.org/about/js-licences I think we could make this automatic perhaps? This pr atm does make us dependent on our sources being online for full builds. This is common, but perhaps not acceptable for us? We could instead do one of the following: - cache downloaded repos outside of the website dir, so they survive a full rebuild - make this script an updater script, so all deps are still vendored, but we have a nice script to update them. Not a big fan of this. - Use nix to fetch all the deps. Now reliant on nixos org instead of github, significant barrier for new contributors. - have a mirror of all the repos we use for builds on this git server. I kinda like this option, it leaves the most sensible and "normal" build process. Does not seem hard: https://docs.gitea.com/usage/repo-mirror # State of deps ## Moved - PHPmailer - Bootstrap ( we were using different versions for the less and js. this is almost certainly A mistake, but I have kept it the same in this issue to minimize disruption, and will try and fix it later). Also we had some bootstrap js we do not use anywhere, we just have a small bit of custom bootstrap js, not the whole package. - Lunr (could not get a minimized version from upstream repo, so we have lost minimization. Not very important as it is only used on search page) - Jquery ( this can be dropped once we upgrade bootstrap, only used as a bootstrap dep) ## Could not move - moderniser (I hope to drop this down the line, but we use a custom build of an old version that I could find no way to recreate/pull from their git repo. So I could not fetch it as a dep) - the various xml files. These are incredibly static compared to everything else, so I did not bother. - Fonts: they are often only available as release artifacts from some site, and so downloading them would massively complicate the dep script. I do not want to maintain a package manager, and so if we think that necessary I would advocate for nix. Also, some fonts are only available from google fonts which does not seem to expose versions. This would again be a problem to be solved with nix. # Checklist - [x] site looks correct locally - [x] search works locally - [x] no missing scripts using inspector locally - [x] Deployed on test
delliott requested review from tobiasd 2025-11-21 16:21:39 +00:00
delliott requested review from fkobi 2025-11-21 16:21:47 +00:00
delliott force-pushed feat/versions from 72f6766f74 to ea1c800083 2025-11-21 16:23:44 +00:00 Compare
delliott commented 2025-11-22 15:09:23 +00:00
Author
Member
Copy Link

It might make sense to merge #5513 , which removes some unused dependencies, before this PR.

It might make sense to merge https://git.fsfe.org/FSFE/fsfe-website/pulls/5513 , which removes some unused dependencies, before this PR.
fkobi commented 2025-11-24 08:22:27 +00:00
Member
Copy Link

I have not contributed to fsfe-website nor do I know webdev well so I do not think I am the best suited to review this, especially with the diff being so large.

I have not contributed to fsfe-website nor do I know webdev well so I do not think I am the best suited to review this, especially with the diff being so large.
fkobi declined to review 2025-11-24 08:22:37 +00:00
delliott commented 2025-11-24 08:35:18 +00:00
Author
Member
Copy Link

I have not contributed to fsfe-website nor do I know webdev well so I do not think I am the best suited to review this, especially with the diff being so large.

Cool, I pinged you as you created #5426

> I have not contributed to fsfe-website nor do I know webdev well so I do not think I am the best suited to review this, especially with the diff being so large. Cool, I pinged you as you created https://git.fsfe.org/FSFE/fsfe-website/issues/5426
👍 1
delliott force-pushed feat/versions from 4c324aa9ee to 735d63004c 2025-11-24 09:05:34 +00:00 Compare
tobiasd commented 2025-11-24 09:41:24 +00:00
Owner
Copy Link

Use nix to fetch all the deps.

Do we need to? And what need people who work on the website need to install for building?

> Use nix to fetch all the deps. Do we need to? And what need people who work on the website need to install for building?
delliott commented 2025-11-24 09:50:09 +00:00
Author
Member
Copy Link

Use nix to fetch all the deps.

Do we need to? And what need people who work on the website need to install for building?

Sorry, that list of bullet points are all different options. So using nix is one option, but there are several others.

> > Use nix to fetch all the deps. > > Do we need to? And what need people who work on the website need to install for building? Sorry, that list of bullet points are all different options. So using nix is one option, but there are several others.
delliott force-pushed feat/versions from 735d63004c to 18330ab184 2025-11-24 11:38:34 +00:00 Compare
delliott force-pushed feat/versions from 18330ab184 to 4561ac40c0 2025-11-24 13:43:10 +00:00 Compare
delliott force-pushed feat/versions from 4561ac40c0 to 323cd7fb97 2025-11-24 13:43:33 +00:00 Compare
delliott force-pushed feat/versions from 323cd7fb97 to b2e370ff14 2025-11-28 20:12:23 +00:00 Compare
delliott commented 2025-11-28 20:53:38 +00:00
Author
Member
Copy Link

Decided to move ahead with a POC implementation of using gitea mirrors of dependency repos.

Have to say, its very easy, seems like a great plan to me.

This pr now uses a bootstrap mirror at https://git.fsfe.org/delliott/bootstrap , just temporarily as a poc.

As we dont need to update often automatically (if ever) we could have a very low poll rate so as not to bother github, say once a week or even longer.

Main question I would have is where we put the mirrors? Probably system-hackers, we already have some mirrors there: https://git.fsfe.org/fsfe-system-hackers/?q=&mirror=1

Also might make sense to have a new namespace for mirrors of thirdparty software, keep it clear what is ours and what is external?

Any thoughts @tobiasd ?

Decided to move ahead with a POC implementation of using gitea mirrors of dependency repos. Have to say, its very easy, seems like a great plan to me. This pr now uses a bootstrap mirror at https://git.fsfe.org/delliott/bootstrap , just temporarily as a poc. As we dont need to update often automatically (if ever) we could have a very low poll rate so as not to bother github, say once a week or even longer. Main question I would have is where we put the mirrors? Probably system-hackers, we already have some mirrors there: https://git.fsfe.org/fsfe-system-hackers/?q=&mirror=1 Also might make sense to have a new namespace for mirrors of thirdparty software, keep it clear what is ours and what is external? Any thoughts @tobiasd ?
delliott force-pushed feat/versions from de3baafaaa to 403a19731a 2025-11-28 20:55:08 +00:00 Compare
delliott force-pushed feat/versions from 403a19731a to 64a66b1a24 2025-12-02 12:42:16 +00:00 Compare
delliott force-pushed feat/versions from d4fba95851 to 054087c7f0 2025-12-02 13:24:10 +00:00 Compare
delliott force-pushed feat/versions from 054087c7f0 to f9a1176a81 2025-12-02 15:28:34 +00:00 Compare
delliott force-pushed feat/versions from 413f8577b1 to 76fe7bfd56 2025-12-02 18:51:15 +00:00 Compare
delliott commented 2025-12-02 20:14:32 +00:00
Author
Member
Copy Link

Discussed in the system hackers jourfix today.

The plan

  • Cache repos in global cache, so they survive a full rebuild, with a new flag to clean them too (--clean-cache). Should persist cache across ci runs.
  • @tobiasd to make new namespace for mirrors of stuff
  • Mirror all deps, using that namespace
Discussed in the system hackers jourfix today. The plan - [x] Cache repos in global cache, so they survive a full rebuild, with a new flag to clean them too (`--clean-cache`). Should persist cache across ci runs. - [x] @tobiasd to make new namespace for mirrors of stuff - [x] Mirror all deps, using that namespace
delliott force-pushed feat/versions from 76fe7bfd56 to e63a276694 2025-12-03 10:32:15 +00:00 Compare
delliott force-pushed feat/versions from e81943688f to d4b3d98f28 2025-12-03 11:48:41 +00:00 Compare
delliott force-pushed feat/versions from ced2213c70 to 2d4a99804c 2025-12-04 09:17:12 +00:00 Compare
delliott force-pushed feat/versions from 2d4a99804c to e340a3878e 2025-12-04 10:29:28 +00:00 Compare
delliott force-pushed feat/versions from e340a3878e to ecf2715792 2025-12-04 12:16:50 +00:00 Compare
delliott force-pushed feat/versions from ecf2715792 to 4b77db2b96 2025-12-04 15:30:01 +00:00 Compare
delliott force-pushed feat/versions from 4b77db2b96 to 382d8cf9e8 2025-12-05 12:20:48 +00:00 Compare
delliott force-pushed feat/versions from 028300bd2d to 7a73dbdd02 2025-12-05 12:54:00 +00:00 Compare
delliott force-pushed feat/versions from e724947a15 to 4291d035f9 2025-12-05 14:19:00 +00:00 Compare
delliott force-pushed feat/versions from 4291d035f9 to b0547295f8 2025-12-05 14:29:09 +00:00 Compare
delliott force-pushed feat/versions from b0547295f8 to 357ab1e552 2025-12-05 14:35:30 +00:00 Compare
delliott force-pushed feat/versions from 357ab1e552 to 957d7c7bc0 2025-12-05 14:38:21 +00:00 Compare
delliott force-pushed feat/versions from 957d7c7bc0 to 6767b898c5 2025-12-05 15:28:13 +00:00 Compare
delliott force-pushed feat/versions from 6767b898c5 to 38ee28e359 2025-12-05 15:31:04 +00:00 Compare
delliott force-pushed feat/versions from 38ee28e359 to 6cc799f7c7 2025-12-05 15:39:51 +00:00 Compare
delliott force-pushed feat/versions from 6cc799f7c7 to d487680fe1 2025-12-05 15:48:02 +00:00 Compare
delliott force-pushed feat/versions from d487680fe1 to 2896c78846 2025-12-05 15:54:33 +00:00 Compare
delliott changed title from WIP: feat/versions to feat/versions 2025-12-05 15:59:58 +00:00
delliott commented 2025-12-05 16:01:16 +00:00
Author
Member
Copy Link

Right, this is now deployed on test to play with, and seems like it should be good to merge to me.

Might be no harm to test perdiem.php and reimbursement.php on test before merge.

They are internal tooling for employees, which I am not, so I cannot test them.

Right, this is now deployed on test to play with, and seems like it should be good to merge to me. Might be no harm to test `perdiem.php` and `reimbursement.php` on test before merge. They are internal tooling for employees, which I am not, so I cannot test them.
delliott force-pushed feat/versions from 2896c78846 to 5e2ad0f4ab 2025-12-14 17:47:47 +00:00 Compare
delliott added the buildenhancementfeature-request labels 2025-12-19 13:07:33 +00:00
delliott force-pushed feat/versions from 5e2ad0f4ab to d0b45d9476 2025-12-21 17:03:17 +00:00 Compare
tobiasd referenced this issue from a commit 2025-12-22 14:59:04 +00:00
fix/php-cs-fixer (#5538)
delliott force-pushed feat/versions from d0b45d9476 to dc704799c2 2025-12-23 18:20:57 +00:00 Compare
delliott force-pushed feat/versions from dc704799c2 to 26eee13d52 2025-12-24 11:10:17 +00:00 Compare
tobiasd requested changes 2025-12-24 11:49:57 +00:00
Dismissed
build/fsfe_website_build/globals.py Outdated
@@ -0,0 +1,10 @@
# SPDX-FileCopyrightText: Free Software Foundation Europe e.V. <https://fsfe.org>
tobiasd commented 2025-12-24 10:51:37 +00:00
Owner
Copy Link

Does this file need to be executable?

Does this file need to be executable?
delliott commented 2025-12-24 12:02:48 +00:00
Author
Member
Copy Link

No

No
delliott commented 2025-12-24 12:07:04 +00:00
Author
Member
Copy Link

Fixed

Fixed
👍 1
delliott marked this conversation as resolved
delliott added 1 commit 2025-12-24 12:06:52 +00:00
fix: remove executable bits from globals.py
All checks were successful
continuous-integration/drone/pr Build is passing
Details
239069b4e4
tobiasd approved these changes 2025-12-24 13:00:34 +00:00
tobiasd merged commit 7f1263b7d3 into master 2025-12-24 13:00:47 +00:00
tobiasd deleted branch feat/versions 2025-12-24 13:00:48 +00:00
tobiasd referenced this issue from a commit 2025-12-24 13:00:49 +00:00
feat/versions (#5512)
Sign in to join this conversation.
Reviewers
No Reviewers
tobiasd
Labels
Clear labels
bug
build
cgi Scripting
design
disruptive
Requires careful coordination and documentation changes
 documentation
duplicate
easy
enhancement
Improving what is already there
 feature-request
help wanted
javascript
priority/low
question
system-hackers
tagging
text
translations
wait/bugfix
wait/inprogress
wait/misc
wait/proofread
wontfix
xsl
No Label build enhancement feature-request
Milestone
No items
No Milestone
Assignees
Clear assignees
albert alex.busch alex.sander anaghz ao bcludts bonnie cryptie dario delliott dfajfer dmaphy doczkal eal egnun eventregbot fi fkobi floriansnow gabriel.ku guido hl hugo ineiev jn jzarl linus lucabon lucas.lasota max.mehl mk monochromec mweimann nico.rikken patrick reinhard renovate-bot repentinus schiessle sofiaritz tobiasd vincent
No Assignees
3 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: FSFE/fsfe-website#5512
Delete Branch "feat/versions"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?