open serve urls in a crossplatform way
open urls after they have started to be served
---------
Co-authored-by: Darragh Elliott <me@delliott.net>
Reviewed-on: #6013
Co-authored-by: delliott <420+delliott@fsfe.org>
Co-committed-by: delliott <420+delliott@fsfe.org>
move similar logic to main nix shell, and document.
misc doc fixes while passing through
---------
Co-authored-by: Darragh Elliott <me@delliott.net>
Reviewed-on: #6014
Co-authored-by: delliott <420+delliott@fsfe.org>
Co-committed-by: delliott <420+delliott@fsfe.org>
This reverts commit ed19e107f8.
Now we have a timeline that looks somewhat nice, we can plan restoring it to prominence. This pr should wait till we have added new content, maybe?
---------
Co-authored-by: Darragh Elliott <me@delliott.net>
Reviewed-on: #5720
Co-authored-by: delliott <420+delliott@fsfe.org>
Co-committed-by: delliott <420+delliott@fsfe.org>
Solves #5621
I feel the images should be moved to pics.fsfe.org too, but not sure what folder to place them in? Also I do not seem to have access, so perhaps I should not be the one to do that?
---------
Co-authored-by: Darragh Elliott <me@delliott.net>
Reviewed-on: #5921
Co-authored-by: delliott <420+delliott@fsfe.org>
Co-committed-by: delliott <420+delliott@fsfe.org>
Require 100% line coverage for lib functions. At present we only test lib functions.
Add the tests needed to achieve this coverage.
---------
Co-authored-by: Darragh Elliott <me@delliott.net>
Reviewed-on: #5839
Co-authored-by: delliott <420+delliott@fsfe.org>
Co-committed-by: delliott <420+delliott@fsfe.org>
Based on feedback in today's tech session, this pr makes some changes to the xml structure ci check
- The main one is that the check no longer applies to files matching the regex `r"^fsfe\.org/news/([0-9]{4}|nl)/.*"`.
- This is essentially all news items and all newsletters.
- This or something similar requested by @anaghz and proposed by @mk .
- Pr also adds some more items to the whitelist and patches updates the list for the status page.
Co-authored-by: Darragh Elliott <me@delliott.net>
Reviewed-on: #5809
Reviewed-by: tobiasd <tobiasd@fsfe.org>
Co-authored-by: delliott <delliott@fsfe.org>
Co-committed-by: delliott <delliott@fsfe.org>
Just a simple bump of dependencies.
Ran a simple build, no issues.
Co-authored-by: Darragh Elliott <me@delliott.net>
Reviewed-on: #5794
Co-authored-by: delliott <delliott@fsfe.org>
Co-committed-by: delliott <delliott@fsfe.org>
So, this is a basic attempt at implementing the fedi redirects as described on the webpage.
Things I do not understand:
- In what format should we expect the address request to come?
- how one should determine the "location" param to be set in the header?
- What actually are the `.well-known` locations we need to redirect to the script?
Note that the php script is completely untested and almost certainly broken: this is a very rough POC
Co-authored-by: Darragh Elliott <me@delliott.net>
Reviewed-on: #5682
Co-authored-by: delliott <delliott@fsfe.org>
Co-committed-by: delliott <delliott@fsfe.org>
some improvements to the subdir script
I think this is what we wanted to have news show up to be translated?
Co-authored-by: Darragh Elliott <me@delliott.net>
Reviewed-on: #5732
Co-authored-by: delliott <delliott@fsfe.org>
Co-committed-by: delliott <delliott@fsfe.org>
port the bash precommit hooks to python in a nice extensible way.
It should have feature parity and be much easier to read and maintain. It also allows for reducing the number of dependencies in the pre commit docker image.
I have tested that the checks can both pass and fail correctly, but it is somewhat possible there might be some edge cases and bugs I have missed. So for a short while after merging if failing ci look at it a little more carefully, in case the checks need some tuning.
Co-authored-by: Darragh Elliott <me@delliott.net>
Reviewed-on: #5651
Reviewed-by: tobiasd <tobiasd@fsfe.org>
Co-authored-by: delliott <delliott@fsfe.org>
Co-committed-by: delliott <delliott@fsfe.org>
looks like a ci script was broken and introduced them a while back, and we never noticed
Co-authored-by: Darragh Elliott <me@delliott.net>
Reviewed-on: #5629
Co-authored-by: delliott <delliott@fsfe.org>
Co-committed-by: delliott <delliott@fsfe.org>
- use nargs instead of splitting strings. Nicer logic, more standard
- Show default values of args in help message
- each site can now have a `config.toml` with deps and required files for deployment.
- cleaner argument handling and custom value generation
- use dataclass to hold args, or cleaner typing
- pass config types to run commands instead of individual args
- clean up the whole build.py file
Overall, this is a nice internal refactor that also offers the feature of per site config.
It does have a breaking change of how multiple args are handled, but the arg help page should explain it just fine.
Co-authored-by: Darragh Elliott <me@delliott.net>
Reviewed-on: #5590
Co-authored-by: delliott <delliott@fsfe.org>
Co-committed-by: delliott <delliott@fsfe.org>
apparently someties it could decide to decode to a stdClass, as seen in #5606
this should fix the above issue
Co-authored-by: Darragh Elliott <me@delliott.net>
Reviewed-on: #5608
Co-authored-by: delliott <delliott@fsfe.org>
Co-committed-by: delliott <delliott@fsfe.org>
Its a better toml formatter that has better musl support, for when we move the images to alpine.
Co-authored-by: Darragh Elliott <me@delliott.net>
Reviewed-on: #5591
Reviewed-by: tobiasd <tobiasd@fsfe.org>
Co-authored-by: delliott <delliott@fsfe.org>
Co-committed-by: delliott <delliott@fsfe.org>
- save search index as json instead of js
- optimise the search index generation script
Co-authored-by: Darragh Elliott <me@delliott.net>
Reviewed-on: #5588
Co-authored-by: delliott <delliott@fsfe.org>
Co-committed-by: delliott <delliott@fsfe.org>
# Summary
Add a stage to the build process that sparse clones deps and puts them in the required places.
This should make it much easier to keep track of vendored deps, and similar.
Solves #5426 and helps makes progress towards #4825
This pr moves almost all thirdparty deps from being vendored to being added by the build process.
This pr should be deployed on test and some cgi scripts tested before merge, there is some possibility of issues. But it should be minimal.
One thing: we manually mention the versions of things in https://fsfe.org/about/js-licences
I think we could make this automatic perhaps?
This pr atm does make us dependent on our sources being online for full builds. This is common, but perhaps not acceptable for us?
We could instead do one of the following:
- cache downloaded repos outside of the website dir, so they survive a full rebuild
- make this script an updater script, so all deps are still vendored, but we have a nice script to update them. Not a big fan of this.
- Use nix to fetch all the deps. Now reliant on nixos org instead of github, significant barrier for new contributors.
- have a mirror of all the repos we use for builds on this git server. I kinda like this option, it leaves the most sensible and "normal" build process. Does not seem hard: https://docs.gitea.com/usage/repo-mirror
# State of deps
## Moved
- PHPmailer
- Bootstrap ( we were using different versions for the less and js. this is almost certainly A mistake, but I have kept it the same in this issue to minimize disruption, and will try and fix it later). Also we had some bootstrap js we do not use anywhere, we just have a small bit of custom bootstrap js, not the whole package.
- Lunr (could not get a minimized version from upstream repo, so we have lost minimization. Not very important as it is only used on search page)
- Jquery ( this can be dropped once we upgrade bootstrap, only used as a bootstrap dep)
## Could not move
- moderniser (I hope to drop this down the line, but we use a custom build of an old version that I could find no way to recreate/pull from their git repo. So I could not fetch it as a dep)
- the various xml files. These are incredibly static compared to everything else, so I did not bother.
- Fonts: they are often only available as release artifacts from some site, and so downloading them would massively complicate the dep script. I do not want to maintain a package manager, and so if we think that necessary I would advocate for nix. Also, some fonts are only available from google fonts which does not seem to expose versions. This would again be a problem to be solved with nix.
# Checklist
- [x] site looks correct locally
- [x] search works locally
- [x] no missing scripts using inspector locally
- [x] Deployed on test
Co-authored-by: Darragh Elliott <me@delliott.net>
Reviewed-on: #5512
Reviewed-by: tobiasd <tobiasd@fsfe.org>
Co-authored-by: delliott <delliott@fsfe.org>
Co-committed-by: delliott <delliott@fsfe.org>
Renames some files to ignored file extensions, and ignores a few filetypes/names that should be ignored.
Also adds file extensions to pretty much every file in the repo, makes sure they get caught by pre-commit and such properly. And makes it easier to check what types we have around.
Removes some old files that should have been deleted, but were instead renamed to `.disabled` or similar.
adds a way to manually specify files to copy from each site, regardless of if they would have been ignored by default filters.
Co-authored-by: Darragh Elliott <me@delliott.net>
Reviewed-on: #5528
Reviewed-by: tobiasd <tobiasd@fsfe.org>
Co-authored-by: delliott <delliott@fsfe.org>
Co-committed-by: delliott <delliott@fsfe.org>
fixes some issues with the latest version of php-cs-fixer and runs it on all files.
solved the ci hangs we are seeing in #5528 and #5512
Co-authored-by: Darragh Elliott <me@delliott.net>
Reviewed-on: #5538
Co-authored-by: delliott <delliott@fsfe.org>
Co-committed-by: delliott <delliott@fsfe.org>
add function to get localised files
add tests for localised file func
Code is quite a lot cleaner, and easier to understand, should be less errors in future.
Fixes localization of links in menus, which is apparently broken atm
Co-authored-by: Darragh Elliott <me@delliott.net>
Reviewed-on: #5564
Co-authored-by: delliott <delliott@fsfe.org>
Co-committed-by: delliott <delliott@fsfe.org>
Draw incidents from an xml source file, order them nicely, use details and some xsl to get nice time ranges and such.
Uses details to get a pretty list. Looks like image below, where clicking on a detail shows a list of updates.
Co-authored-by: Darragh Elliott <me@delliott.net>
Reviewed-on: #5568
Co-authored-by: delliott <delliott@fsfe.org>
Co-committed-by: delliott <delliott@fsfe.org>
Very small tweak, in tested code.
Should be a little more performant and handle edge cases better.
Co-authored-by: Darragh Elliott <me@delliott.net>
Reviewed-on: #5567
fix for #5557
I have checked that the news, thankgnus, events and activities localmenus are working correctly.
Co-authored-by: Darragh Elliott <me@delliott.net>
Reviewed-on: #5558
Co-authored-by: delliott <delliott@fsfe.org>
Co-committed-by: delliott <delliott@fsfe.org>
- Fix regex in ci check to make sure we actually check the files we should be checking
- Remove check that has been superseded by compare-xml-structure
- See stdout of successful ci checks.
Co-authored-by: Darragh Elliott <me@delliott.net>
Reviewed-on: #5522
Co-authored-by: delliott <delliott@fsfe.org>
Co-committed-by: delliott <delliott@fsfe.org>
Remove (hopefully) unused js files.
Did a bunch of grepping for their use, and looked through the history as best I could to determine if they are needed.
Co-authored-by: Darragh Elliott <me@delliott.net>
Reviewed-on: #5513
Reviewed-by: tobiasd <tobiasd@fsfe.org>
Co-authored-by: delliott <delliott@fsfe.org>
Co-committed-by: delliott <delliott@fsfe.org>
prevents issues with processing files when there are empty xmllists
issue only shows when building with very few languages
Co-authored-by: Darragh Elliott <me@delliott.net>
Reviewed-on: #5514
Co-authored-by: delliott <delliott@fsfe.org>
Co-committed-by: delliott <delliott@fsfe.org>
Should close #2471
Move podcasts to a per year structure. This allows for them to be displayed in the archive properly and all that jazz.
This required some enhancements of the subdir script to properly handle more nesting levels, but that is all acceptable.
Also move the transcripts, to a year structure, for consistency.
I have added rewrite rules for the old locations of the podcast episodes, breakage should be minimal.
I am going to keep this as WIP until @bonnie approves it, as it is possible there are breaking issues with this I have failed to notice.
It is also deployed to [test.fsfe.org](https://test.fsfe.org), so feel free to take a look there.
Co-authored-by: Darragh Elliott <me@delliott.net>
Reviewed-on: #5400
Reviewed-by: bonnie <bonnie@fsfe.org>
Reviewed-by: tobiasd <tobiasd@fsfe.org>
Co-authored-by: delliott <delliott@fsfe.org>
Co-committed-by: delliott <delliott@fsfe.org>
Whitelist some common attributes/elements that should be localized.
This was done by making the whitelist a list of xpaths.
Co-authored-by: Darragh Elliott <me@delliott.net>
Reviewed-on: #5500
Co-authored-by: delliott <delliott@fsfe.org>
Co-committed-by: delliott <delliott@fsfe.org>
Bump the uv lockfile, and upgrade python versions.
This opens up some fun possibilities for further improvements using the new features in python 3.14.
Co-authored-by: Darragh Elliott <me@delliott.net>
Reviewed-on: #5503
Co-authored-by: delliott <delliott@fsfe.org>
Co-committed-by: delliott <delliott@fsfe.org>
fix some stuff not being cached, and add test to catch caching regressions.
Test takes quite a while to run, not sure it is a good idea?
Co-authored-by: Darragh Elliott <me@delliott.net>
Reviewed-on: #5494
- enable pyright strict mode
- fix the issues raised
This should make the build process cleaner and more resilient.
It should make it much easier to catch design flaws.
Co-authored-by: Darragh Elliott <me@delliott.net>
Reviewed-on: #5451
Reviewed-by: tobiasd <tobiasd@fsfe.org>
Co-authored-by: delliott <delliott@fsfe.org>
Co-committed-by: delliott <delliott@fsfe.org>
Currently implements both a format hook to remove white-space paragraphs, and a check to make their existence a critical failure.
In the end may implement only one or none of these, subject to major design changes.
Solves #5409
Co-authored-by: Darragh Elliott <me@delliott.net>
Reviewed-on: #5431
Co-authored-by: delliott <delliott@fsfe.org>
Co-committed-by: delliott <delliott@fsfe.org>
Adds reuse compliance checks for python files only, using `reuse lint-file`.
It is managed as a pre-commit hook, and hence ran in ci.
Over time we may add more files to be covered, but I suspect we would use something like `reuse.toml` for setting the licensing for `xml/xhtml/xsl` files, as there are just too many for adding the same text snippet to each to be justifiable.
Co-authored-by: Darragh Elliott <me@delliott.net>
Reviewed-on: #5440
Co-authored-by: delliott <delliott@fsfe.org>
Co-committed-by: delliott <delliott@fsfe.org>
Adds an overview page to see what and where are we still using style attributes.
Should help coordinate efforts for #2736
Co-authored-by: Darragh Elliott <me@delliott.net>
Reviewed-on: #5437
Co-authored-by: delliott <delliott@fsfe.org>
Co-committed-by: delliott <delliott@fsfe.org>
- Cleanup lefthook (pre-commit hook program) config file
- Make some types of formatting fail CI if it changes the files
Co-authored-by: Darragh Elliott <me@delliott.net>
Reviewed-on: #5436
Co-authored-by: delliott <delliott@fsfe.org>
Co-committed-by: delliott <delliott@fsfe.org>
- fixes issues with xml structure pre commit hook
- Moves comprea-files to lib
- adds a n overview page on status.fsfe.org to see what files have different structures
Co-authored-by: Darragh Elliott <me@delliott.net>
Reviewed-on: #5435
Co-authored-by: delliott <delliott@fsfe.org>
Co-committed-by: delliott <delliott@fsfe.org>
Adds a pre-commit hook that checks that `xml` and `xhtml` files of the same version have matched elements and attributes.
The value of elements may differ, as that is translated text.
Some attributes are allowed to have different values, for example image alt text.
This is a continuation of @max.mehl 's work in #2556, and should hopefully enable closing that issue eventually.
Co-authored-by: Darragh Elliott <me@delliott.net>
Reviewed-on: #5432
Reviewed-by: tobiasd <tobiasd@fsfe.org>
Co-authored-by: delliott <delliott@fsfe.org>
Co-committed-by: delliott <delliott@fsfe.org>
I do not expect the flag to be used much, but it enforces better code design, and prevents implicitly relying on cwd.
Just a general code style improvement really.
Co-authored-by: Darragh Elliott <me@delliott.net>
Co-authored-by: tobiasd <tobiasd@fsfe.org>
Reviewed-on: #5334
Co-authored-by: delliott <delliott@fsfe.org>
Co-committed-by: delliott <delliott@fsfe.org>
it is all minimized in bootstrap.min.js and bootstrap.custom.js
I am largely basing my belief that it is unused on the fact that
1. It looks like it is
2. I can find no mention or linking to it using `rg bootstrap/`. All matched entries are for other things
3. When I load the site and look in the console/debugger I see no errors associated with it.
Co-authored-by: Darragh Elliott <me@delliott.net>
Reviewed-on: #5398
Co-authored-by: delliott <delliott@fsfe.org>
Co-committed-by: delliott <delliott@fsfe.org>
# About
## Summary
Before we iterated over xhtml files by directory, and rss/ics ones by processor (xsl stylesheet).
This worked, but meant we had to pass the processor filepath to the processing functions, which meant we were reprocessing the xsl several times, probably hundred for some xsl's, like the default one for a site.
We now instead iterate by processor, pre parsing it before passing it to the processing code.
This gets us a significant speed increase (~30%), and makes the code a little cleaner.
# Benchmarks
## Before
```
Benchmark 1: uv run build --full
Time (mean ± σ): 269.798 s ± 12.501 s [User: 1566.724 s, System: 75.804 s]
Range (min … max): 262.183 s … 284.226 s 3 runs
```
## After
```
Benchmark 1: uv run build --full
Time (mean ± σ): 206.330 s ± 5.088 s [User: 498.556 s, System: 47.233 s]
Range (min … max): 202.760 s … 212.155 s 3 runs
```
Co-authored-by: Darragh Elliott <me@delliott.net>
Co-authored-by: tobiasd <tobiasd@fsfe.org>
Reviewed-on: #5397
Co-authored-by: delliott <delliott@fsfe.org>
Co-committed-by: delliott <delliott@fsfe.org>
prevent the fsfe logo floating over the menu stuff when pages are in an rtl language
eg arabic
Should fix #4255
Co-authored-by: Darragh Elliott <me@delliott.net>
Reviewed-on: #5399
Reviewed-by: tobiasd <tobiasd@fsfe.org>
Co-authored-by: delliott <delliott@fsfe.org>
Co-committed-by: delliott <delliott@fsfe.org>
In #5333 I added docker image caching using Gitea artifact Registry.
This turned out to be a waste of time, that actually slowed things down.
This is because the docker instance on the drone runner host actually caches the images internally anyway, so the time cost of downloading an image, rebuilding it and re pushing it was being paid for no reason or benefit.
This pr removes the caching, but does translate a step to ensure that the build docker container can be built in a pr, which is a good check to have pre merge to master.
Co-authored-by: Darragh Elliott <me@delliott.net>
Reviewed-on: #5392
Co-authored-by: delliott <delliott@fsfe.org>
Co-committed-by: delliott <delliott@fsfe.org>
Enable the building and pushing of docker containers on master, and then using them as a cache base in prs.
This should massively reduce the build times, and the load we are placing on other projects servers.
Depends on #5330
Co-authored-by: Darragh Elliott <me@delliott.net>
Reviewed-on: #5333
Co-authored-by: delliott <delliott@fsfe.org>
Co-committed-by: delliott <delliott@fsfe.org>
This should mean we can used cached docker images properly in future.
Co-authored-by: Darragh Elliott <me@delliott.net>
Reviewed-on: #5330
Co-authored-by: delliott <delliott@fsfe.org>
Co-committed-by: delliott <delliott@fsfe.org>
This seems like something we should be doing.
I have some vague memory of trying this before and there being issues as we are coming from the `fsfe` namespace rather than the `fsfe-system-hackers` namespace, but we should be able to get it working eventually regardless.
I will let it run on prs for testing, but before merge we should have it only run on commits to master
Co-authored-by: Darragh Elliott <me@delliott.net>
Reviewed-on: #5274
This properly separtes out the python build process (theoretically portable and useable for different sites) and the xslt (custom to the fsfe sites, not really portable as a whole)
It also moves some thirdparty tooling around to make it clearer its thirdparty and unify xslt sources.
Co-authored-by: Darragh Elliott <me@delliott.net>
Reviewed-on: #5299
Reviewed-by: tobiasd <tobiasd@fsfe.org>
Co-authored-by: delliott <delliott@fsfe.org>
Co-committed-by: delliott <delliott@fsfe.org>
if has some stuff pretty specific to fsfe.org in there, and hence should not be in global build
Co-authored-by: Darragh Elliott <me@delliott.net>
Reviewed-on: #5298
Co-authored-by: delliott <delliott@fsfe.org>
Co-committed-by: delliott <delliott@fsfe.org>
Allow specifying a subject in our email element.
Eg the xhtml content
```xml
<email mailto="yes" subject="Ada and Zangemann">contact@fsfe.org</email>
```
Will now generate
```xml
<a href="mailto:contact@fsfe.org?subject=Ada and Zangemann">contact@fsfe.org</a>
```
Which when clicked will open an email client, and correctly populate the subject.
I also added the subject to all ada zangemann pages, and have documented it on the web features page.
Resolves #2742
Co-authored-by: Darragh Elliott <me@delliott.net>
Reviewed-on: #5281
Co-authored-by: delliott <delliott@fsfe.org>
Co-committed-by: delliott <delliott@fsfe.org>
it is highly specific to fsfe.org, and so should probably be a subdir instead of a build step
Co-authored-by: Darragh Elliott <me@delliott.net>
Reviewed-on: #5273
Co-authored-by: delliott <delliott@fsfe.org>
Co-committed-by: delliott <delliott@fsfe.org>
- add pytest
- add tests for all exposed lib functions
- Add a whole bunch more ruff checks and apply them
Benchmarking:
Before:
```
❯ hyperfine --runs 3 "uv run build --full"
Benchmark 1: uv run build --full
Time (mean ± σ): 287.463 s ± 15.161 s [User: 1573.732 s, System: 72.548 s]
Range (min … max): 272.953 s … 303.201 s 3 runs
```
After
```
❯ hyperfine --runs 3 "uv run build --full"
Benchmark 1: uv run build --full
Time (mean ± σ): 280.184 s ± 9.605 s [User: 1557.788 s, System: 72.693 s]
Range (min … max): 269.878 s … 288.886 s 3 runs
```
So we gain a few seconds, but that is all.
The main benefit is that we have better testing and more linting.
Co-authored-by: Darragh Elliott <me@delliott.net>
Reviewed-on: #5268
- Update deps to latest versions
- stop using tight constraints and instead just use loose ones and commit the uv lockfile
- Greater reproducibility, and much easier to bump deps in future.
Co-authored-by: Darragh Elliott <me@delliott.net>
Reviewed-on: #5264
Before if none of the files submitted to the check contained the tag, and there were hence no matches the script would fail incorrectly.
This is no longer the case
Co-authored-by: Darragh Elliott <me@delliott.net>
Reviewed-on: #5265
Co-authored-by: delliott <delliott@fsfe.org>
Co-committed-by: delliott <delliott@fsfe.org>
