- use nargs instead of splitting strings. Nicer logic, more standard
- Show default values of args in help message
- each site can now have a `config.toml` with deps and required files for deployment.
- cleaner argument handling and custom value generation
- use dataclass to hold args, or cleaner typing
- pass config types to run commands instead of individual args
- clean up the whole build.py file
Overall, this is a nice internal refactor that also offers the feature of per site config.
It does have a breaking change of how multiple args are handled, but the arg help page should explain it just fine.
Co-authored-by: Darragh Elliott <me@delliott.net>
Reviewed-on: #5590
Co-authored-by: delliott <delliott@fsfe.org>
Co-committed-by: delliott <delliott@fsfe.org>
# Summary
Add a stage to the build process that sparse clones deps and puts them in the required places.
This should make it much easier to keep track of vendored deps, and similar.
Solves #5426 and helps makes progress towards #4825
This pr moves almost all thirdparty deps from being vendored to being added by the build process.
This pr should be deployed on test and some cgi scripts tested before merge, there is some possibility of issues. But it should be minimal.
One thing: we manually mention the versions of things in https://fsfe.org/about/js-licences
I think we could make this automatic perhaps?
This pr atm does make us dependent on our sources being online for full builds. This is common, but perhaps not acceptable for us?
We could instead do one of the following:
- cache downloaded repos outside of the website dir, so they survive a full rebuild
- make this script an updater script, so all deps are still vendored, but we have a nice script to update them. Not a big fan of this.
- Use nix to fetch all the deps. Now reliant on nixos org instead of github, significant barrier for new contributors.
- have a mirror of all the repos we use for builds on this git server. I kinda like this option, it leaves the most sensible and "normal" build process. Does not seem hard: https://docs.gitea.com/usage/repo-mirror
# State of deps
## Moved
- PHPmailer
- Bootstrap ( we were using different versions for the less and js. this is almost certainly A mistake, but I have kept it the same in this issue to minimize disruption, and will try and fix it later). Also we had some bootstrap js we do not use anywhere, we just have a small bit of custom bootstrap js, not the whole package.
- Lunr (could not get a minimized version from upstream repo, so we have lost minimization. Not very important as it is only used on search page)
- Jquery ( this can be dropped once we upgrade bootstrap, only used as a bootstrap dep)
## Could not move
- moderniser (I hope to drop this down the line, but we use a custom build of an old version that I could find no way to recreate/pull from their git repo. So I could not fetch it as a dep)
- the various xml files. These are incredibly static compared to everything else, so I did not bother.
- Fonts: they are often only available as release artifacts from some site, and so downloading them would massively complicate the dep script. I do not want to maintain a package manager, and so if we think that necessary I would advocate for nix. Also, some fonts are only available from google fonts which does not seem to expose versions. This would again be a problem to be solved with nix.
# Checklist
- [x] site looks correct locally
- [x] search works locally
- [x] no missing scripts using inspector locally
- [x] Deployed on test
Co-authored-by: Darragh Elliott <me@delliott.net>
Reviewed-on: #5512
Reviewed-by: tobiasd <tobiasd@fsfe.org>
Co-authored-by: delliott <delliott@fsfe.org>
Co-committed-by: delliott <delliott@fsfe.org>
Renames some files to ignored file extensions, and ignores a few filetypes/names that should be ignored.
Also adds file extensions to pretty much every file in the repo, makes sure they get caught by pre-commit and such properly. And makes it easier to check what types we have around.
Removes some old files that should have been deleted, but were instead renamed to `.disabled` or similar.
adds a way to manually specify files to copy from each site, regardless of if they would have been ignored by default filters.
Co-authored-by: Darragh Elliott <me@delliott.net>
Reviewed-on: #5528
Reviewed-by: tobiasd <tobiasd@fsfe.org>
Co-authored-by: delliott <delliott@fsfe.org>
Co-committed-by: delliott <delliott@fsfe.org>
add function to get localised files
add tests for localised file func
Code is quite a lot cleaner, and easier to understand, should be less errors in future.
Fixes localization of links in menus, which is apparently broken atm
Co-authored-by: Darragh Elliott <me@delliott.net>
Reviewed-on: #5564
Co-authored-by: delliott <delliott@fsfe.org>
Co-committed-by: delliott <delliott@fsfe.org>
Very small tweak, in tested code.
Should be a little more performant and handle edge cases better.
Co-authored-by: Darragh Elliott <me@delliott.net>
Reviewed-on: #5567
fix for #5557
I have checked that the news, thankgnus, events and activities localmenus are working correctly.
Co-authored-by: Darragh Elliott <me@delliott.net>
Reviewed-on: #5558
Co-authored-by: delliott <delliott@fsfe.org>
Co-committed-by: delliott <delliott@fsfe.org>
prevents issues with processing files when there are empty xmllists
issue only shows when building with very few languages
Co-authored-by: Darragh Elliott <me@delliott.net>
Reviewed-on: #5514
Co-authored-by: delliott <delliott@fsfe.org>
Co-committed-by: delliott <delliott@fsfe.org>
Whitelist some common attributes/elements that should be localized.
This was done by making the whitelist a list of xpaths.
Co-authored-by: Darragh Elliott <me@delliott.net>
Reviewed-on: #5500
Co-authored-by: delliott <delliott@fsfe.org>
Co-committed-by: delliott <delliott@fsfe.org>
fix some stuff not being cached, and add test to catch caching regressions.
Test takes quite a while to run, not sure it is a good idea?
Co-authored-by: Darragh Elliott <me@delliott.net>
Reviewed-on: #5494
- enable pyright strict mode
- fix the issues raised
This should make the build process cleaner and more resilient.
It should make it much easier to catch design flaws.
Co-authored-by: Darragh Elliott <me@delliott.net>
Reviewed-on: #5451
Reviewed-by: tobiasd <tobiasd@fsfe.org>
Co-authored-by: delliott <delliott@fsfe.org>
Co-committed-by: delliott <delliott@fsfe.org>
Adds reuse compliance checks for python files only, using `reuse lint-file`.
It is managed as a pre-commit hook, and hence ran in ci.
Over time we may add more files to be covered, but I suspect we would use something like `reuse.toml` for setting the licensing for `xml/xhtml/xsl` files, as there are just too many for adding the same text snippet to each to be justifiable.
Co-authored-by: Darragh Elliott <me@delliott.net>
Reviewed-on: #5440
Co-authored-by: delliott <delliott@fsfe.org>
Co-committed-by: delliott <delliott@fsfe.org>
- fixes issues with xml structure pre commit hook
- Moves comprea-files to lib
- adds a n overview page on status.fsfe.org to see what files have different structures
Co-authored-by: Darragh Elliott <me@delliott.net>
Reviewed-on: #5435
Co-authored-by: delliott <delliott@fsfe.org>
Co-committed-by: delliott <delliott@fsfe.org>
I do not expect the flag to be used much, but it enforces better code design, and prevents implicitly relying on cwd.
Just a general code style improvement really.
Co-authored-by: Darragh Elliott <me@delliott.net>
Co-authored-by: tobiasd <tobiasd@fsfe.org>
Reviewed-on: #5334
Co-authored-by: delliott <delliott@fsfe.org>
Co-committed-by: delliott <delliott@fsfe.org>
# About
## Summary
Before we iterated over xhtml files by directory, and rss/ics ones by processor (xsl stylesheet).
This worked, but meant we had to pass the processor filepath to the processing functions, which meant we were reprocessing the xsl several times, probably hundred for some xsl's, like the default one for a site.
We now instead iterate by processor, pre parsing it before passing it to the processing code.
This gets us a significant speed increase (~30%), and makes the code a little cleaner.
# Benchmarks
## Before
```
Benchmark 1: uv run build --full
Time (mean ± σ): 269.798 s ± 12.501 s [User: 1566.724 s, System: 75.804 s]
Range (min … max): 262.183 s … 284.226 s 3 runs
```
## After
```
Benchmark 1: uv run build --full
Time (mean ± σ): 206.330 s ± 5.088 s [User: 498.556 s, System: 47.233 s]
Range (min … max): 202.760 s … 212.155 s 3 runs
```
Co-authored-by: Darragh Elliott <me@delliott.net>
Co-authored-by: tobiasd <tobiasd@fsfe.org>
Reviewed-on: #5397
Co-authored-by: delliott <delliott@fsfe.org>
Co-committed-by: delliott <delliott@fsfe.org>
This properly separtes out the python build process (theoretically portable and useable for different sites) and the xslt (custom to the fsfe sites, not really portable as a whole)
It also moves some thirdparty tooling around to make it clearer its thirdparty and unify xslt sources.
Co-authored-by: Darragh Elliott <me@delliott.net>
Reviewed-on: #5299
Reviewed-by: tobiasd <tobiasd@fsfe.org>
Co-authored-by: delliott <delliott@fsfe.org>
Co-committed-by: delliott <delliott@fsfe.org>
if has some stuff pretty specific to fsfe.org in there, and hence should not be in global build
Co-authored-by: Darragh Elliott <me@delliott.net>
Reviewed-on: #5298
Co-authored-by: delliott <delliott@fsfe.org>
Co-committed-by: delliott <delliott@fsfe.org>
Allow specifying a subject in our email element.
Eg the xhtml content
```xml
<email mailto="yes" subject="Ada and Zangemann">contact@fsfe.org</email>
```
Will now generate
```xml
<a href="mailto:contact@fsfe.org?subject=Ada and Zangemann">contact@fsfe.org</a>
```
Which when clicked will open an email client, and correctly populate the subject.
I also added the subject to all ada zangemann pages, and have documented it on the web features page.
Resolves #2742
Co-authored-by: Darragh Elliott <me@delliott.net>
Reviewed-on: #5281
Co-authored-by: delliott <delliott@fsfe.org>
Co-committed-by: delliott <delliott@fsfe.org>
it is highly specific to fsfe.org, and so should probably be a subdir instead of a build step
Co-authored-by: Darragh Elliott <me@delliott.net>
Reviewed-on: #5273
Co-authored-by: delliott <delliott@fsfe.org>
Co-committed-by: delliott <delliott@fsfe.org>
- add pytest
- add tests for all exposed lib functions
- Add a whole bunch more ruff checks and apply them
Benchmarking:
Before:
```
❯ hyperfine --runs 3 "uv run build --full"
Benchmark 1: uv run build --full
Time (mean ± σ): 287.463 s ± 15.161 s [User: 1573.732 s, System: 72.548 s]
Range (min … max): 272.953 s … 303.201 s 3 runs
```
After
```
❯ hyperfine --runs 3 "uv run build --full"
Benchmark 1: uv run build --full
Time (mean ± σ): 280.184 s ± 9.605 s [User: 1557.788 s, System: 72.693 s]
Range (min … max): 269.878 s … 288.886 s 3 runs
```
So we gain a few seconds, but that is all.
The main benefit is that we have better testing and more linting.
Co-authored-by: Darragh Elliott <me@delliott.net>
Reviewed-on: #5268
update docs
update nix-shell to auto install
rework custom pre-commit hooks to integrate better
add dockerfile for lefthook and integrate into drone
add some more hooks for formatting and linting some files
Co-authored-by: Darragh Elliott <me@delliott.net>
Reviewed-on: #5248
Co-authored-by: delliott <delliott@fsfe.org>
Co-committed-by: delliott <delliott@fsfe.org>
#5146 attempt 2, texts and topbanner issue resolved.
After the issues I have deployed this to https://test.fsfe.org, so we can see if there are further issues there.
It seems not though.
Co-authored-by: Darragh Elliott <me@delliott.xyz>
Reviewed-on: #5181
Co-authored-by: delliott <delliott@fsfe.org>
Co-committed-by: delliott <delliott@fsfe.org>
Code looks much cleaner and easier to read, as well as being more performant.
Also prevents the construction of invalid xml.
Some benchmarks:
```
❯ hyperfine --runs 3 "uv run build --full "; git checkout master; hyperfine --runs 3 "uv run build --full "
Benchmark 1: uv run build --full
Time (mean ± σ): 287.612 s ± 4.557 s [User: 1613.681 s, System: 151.556 s]
Range (min … max): 283.550 s … 292.539 s 3 runs
Switched to branch 'master'
Your branch is up to date with 'origin/master'.
Benchmark 1: uv run build --full
Time (mean ± σ): 357.248 s ± 5.578 s [User: 2052.506 s, System: 163.057 s]
Range (min … max): 350.810 s … 360.609 s 3 runs
```
So this reduces build times by about a minute, getting us down to just under 5 minutes for a full build on my machine.
Co-authored-by: Darragh Elliott <me@delliott.xyz>
Reviewed-on: #5146
Co-authored-by: delliott <delliott@fsfe.org>
Co-committed-by: delliott <delliott@fsfe.org>
