feat/python-rewrite #4762

Merged
tobiasd merged 79 commits from :feat/python-rewrite into master 2025-04-24 12:56:33 +00:00

79 Commits

Author SHA1 Message Date
0bcd0e69fa Merge branch 'master' into feat/python-rewrite 2025-04-24 12:53:21 +00:00
16fd6b9cc4 Merge branch 'feat/python-rewrite' of https://git.fsfe.org/delliott/fsfe-website into feat/python-rewrite 2025-04-24 11:37:39 +02:00
0b66e625e5 Merge branch 'master' into feat/python-rewrite 2025-04-24 09:36:50 +00:00
971877faf0 Merge branch 'master' into feat/python-rewrite 2025-04-24 11:19:40 +02:00
12a3ef746d add target for master 2025-04-23 14:44:14 +02:00
6c1da900c4 Fix environ 2 2025-04-23 14:39:21 +02:00
76f27c6895 Fix environ 2025-04-23 14:39:10 +02:00
24ee1225fe Create activities XML file 2025-04-23 14:38:44 +02:00
16bb89445e Merge branch 'master' into feat/python-rewrite 2025-04-15 08:45:52 +00:00
f5a8cb3614 Copy .htaccess and .htuser files 2025-04-11 15:49:52 +02:00
Darragh Elliott
a7e3b10a87 chore: bump python deps 2025-04-10 08:31:25 +00:00
Darragh Elliott
a52ccbd85c fix: single file hrefs 2025-04-10 08:31:25 +00:00
Darragh Elliott
e7d8a2d6cb feat: add spdx headers for reuse compliance 2025-04-10 08:31:25 +00:00
Darragh Elliott
b3f67a8cd5 chore: add todos and fix comments 2025-04-10 08:31:25 +00:00
Darragh Elliott
e227dfab6e feat: use norris cont2 instead of cont1 2025-04-10 08:31:25 +00:00
Darragh Elliott
d2725e877a fix: localmenu urls 2025-04-10 08:31:25 +00:00
Darragh Elliott
3c5aef8139 feat: do install recommended packages 2025-04-10 08:31:25 +00:00
Darragh Elliott
4566e8dcce fix: do not sync times 2025-04-10 08:31:25 +00:00
Darragh Elliott
2733349409 feat: use noris instead of plutex 2025-04-10 08:31:25 +00:00
Darragh Elliott
4ce40634f6 feat: remove now unneeded enabling of ipv6 2025-04-10 08:31:25 +00:00
Darragh Elliott
3c8381ceae feat: add support for speciying port per target, and use ipv4 proxies for deployment 2025-04-10 08:31:25 +00:00
Darragh Elliott
18c35ca6bb feat: ipv6 networking 2025-04-10 08:31:25 +00:00
Darragh Elliott
c37879e997 feat: sign drone 2025-04-10 08:31:25 +00:00
Darragh Elliott
b97e07a2a1 fix: load sshkey properly 2025-04-10 08:31:25 +00:00
Darragh Elliott
05eab28088 feat: simplify drone 2025-04-10 08:31:25 +00:00
Darragh Elliott
2b27a1a905 fix: ssh key loading 2025-04-10 08:31:25 +00:00
Darragh Elliott
b861e6bf79 feat: improve dockerfile, port to debian, fix some compat issues with older pythons 2025-04-10 08:31:25 +00:00
Darragh Elliott
84a882712e feat: deploy test branch only from test, on pushes to test. 2025-04-10 08:31:25 +00:00
Darragh Elliott
cb732c5fbf feat: use sshkey with password 2025-04-10 08:31:25 +00:00
Darragh Elliott
3f2cc872eb feat: change to only one required key 2025-04-10 08:31:25 +00:00
Darragh Elliott
468a4585e6 feat: stopy copyting python files, and add special case for order xml 2025-04-10 08:31:25 +00:00
Darragh Elliott
6b8f0233cf fix: some small code tweaks and optimisations 2025-04-10 08:31:25 +00:00
Darragh Elliott
6f2e8f0c98 feat: multithread the rsync to stage 2025-04-10 08:31:25 +00:00
Darragh Elliott
d9fc3139b9 feat: remove the less script file
Not needed anymore, and has not been for many commits
2025-04-10 08:31:25 +00:00
Darragh Elliott
8a90514402 fix: some small docker tweaks 2025-04-10 08:31:25 +00:00
Darragh Elliott
34358bb151 feat: better tagging
fix some bugs in generating tag pages
use lxml instead of string templating
actually use translation of pages for generating tagged-*.??.xhtml (this was broken on master too)
previusly it just always used the en version, even if a translation was present
2025-04-10 08:31:25 +00:00
Darragh Elliott
8a78711c23 fix: use only one pool in translations subdir 2025-04-10 08:31:25 +00:00
Darragh Elliott
3790e3c729 fix: do not add another language to .thml urls that already have a language
mainly a fix for the translation bar
2025-04-10 08:31:25 +00:00
Darragh Elliott
c31ca09cfc feat: use lxml instead of string templating for generating localemnus
some other tweaks
2025-04-10 08:31:25 +00:00
Darragh Elliott
707333b438 fix: use xpath to find on root level instead of a an innefficient recursive iter 2025-04-10 08:31:25 +00:00
Darragh Elliott
6841166a64 feat: add proper targets for test branch
sign drone
2025-04-10 08:31:25 +00:00
Darragh Elliott
86ae2035b5 feat: better docker caching 2025-04-10 08:31:25 +00:00
Darragh Elliott
384c24ce40 chore: update python deps, and silence pip update warnings 2025-04-10 08:31:25 +00:00
Darragh Elliott
0e63b0900c chore: remove perl from build deps 2025-04-10 08:31:25 +00:00
Darragh Elliott
44447347cf chore: add some packages minifiers needs if cannot use chached builds 2025-04-10 08:31:25 +00:00
Darragh Elliott
cf36add2f6 feat: add python formatters to nix shell 2025-04-10 08:31:25 +00:00
Darragh Elliott
97df49b0a1 feat: use internal lib function for subprocesses 2025-04-10 08:31:25 +00:00
Darragh Elliott
54b2e078a3 feat: use maps and filters instead of list comphrehension in many places 2025-04-10 08:31:25 +00:00
Darragh Elliott
c9e1a625f2 feat: use one pool for all multiprocessing
Cannot use the same pool for subdirs due to them being dynamically imported
need to use a pool with as many processes as necessary for website serve
2025-04-10 08:31:25 +00:00
Darragh Elliott
50ae1453ca feat: reorganise python a little 2025-04-10 08:31:25 +00:00
Darragh Elliott
8af65ab0d4 feat: black formatter 2025-04-10 08:31:25 +00:00
Darragh Elliott
d073acd8d7 feat: add processes argument
Useful to reduce number of processes than can run at a a time, keeping parallellism down
2025-04-10 08:31:25 +00:00
Darragh Elliott
120a4b7ad0 feat: add support for sshkeys for authenticating into targets
not very well tested
2025-04-10 08:31:25 +00:00
Darragh Elliott
7ddfc5b804 fix: formatting tweaks, typo fix, replace folder with dir 2025-04-10 08:31:25 +00:00
Darragh Elliott
1e2df847e8 feat: move translation status stuff to a subdir, as that makes way more sense.
also fis one or two small issues with it
2025-04-10 08:31:25 +00:00
Darragh Elliott
683d8c03ce fix: disable the local testing alteration to the drone pipeline 2025-04-10 08:31:25 +00:00
Darragh Elliott
dea7d02f6b feat: use unicode rather than html escapes for arabic 2025-04-10 08:31:25 +00:00
Darragh Elliott
158566295d feat: add better drone pipeline support 2025-04-10 08:31:25 +00:00
Darragh Elliott
26afbf9a95 feat: implement translation status logic in python
made some sense changes to the xsl sheet while at it,
as part of this noticed some issues with process_file and fixed them
Also removed the old tooling
2025-04-10 08:31:25 +00:00
Darragh Elliott
98de6e834f feat: remove the status script and instead point people towards drone 2025-04-10 08:31:25 +00:00
Darragh Elliott
31aa7131ee fix: fix urls to source code 2025-04-10 08:31:25 +00:00
Darragh Elliott
ed3bfd0023 feat: use lib functions for getting file paths and names wihout the last two suffixes 2025-04-10 08:31:25 +00:00
Darragh Elliott
10e84c4134 fix: use a lib function for getting lang code from filenames 2025-04-10 08:31:25 +00:00
Darragh Elliott
553a0a1556 feat: add proper stage support 2025-04-10 08:31:25 +00:00
Darragh Elliott
1a0215c6f6 feat: move process file to general use folder,
rename build_rss* to process_rss*
remove now unneeded commented out code from xhtml|rss processing
Multithread xhtml processing again
2025-04-10 08:31:25 +00:00
Darragh Elliott
d14d1a804d fix: build docs 2025-04-10 08:31:25 +00:00
Darragh Elliott
bce0a2da87 feat: rmeove unneeded update stub, implement full rebuild logic 2025-04-10 08:31:25 +00:00
Darragh Elliott
e2e0b28937 feat: enhance dockerfile, nix-shell
drop uv dep and just use python and pip

Docker file now chaches builds correctly, so we get the benefits of incremental builds in docker
No longer uses mount ow working dir, so user direcotries are not polluted
2025-04-10 08:31:25 +00:00
Darragh Elliott
15744f7c50 feat: implement processing in python
As fo this commit the build process is fully functional with no bash deps.
But could still use some optimisation, and implementation of full builds and update, maybe
2025-04-10 08:31:25 +00:00
Darragh Elliott
9bef3e7784 fix: building of rsss and ics files 2025-04-10 08:31:25 +00:00
Darragh Elliott
0c3cf40010 feat: fix subdir path 2025-04-10 08:31:25 +00:00
Darragh Elliott
83888b5bb2 feat: use latest python3 2025-04-10 08:31:25 +00:00
Darragh Elliott
4c3bb738d4 feat: replace subdir makefiles with python modules 2025-04-10 08:31:25 +00:00
Darragh Elliott
be8cf2edb4 feat: better docs 2025-04-10 08:31:25 +00:00
Darragh Elliott
129d6a3a4b feat: implement makerules in.sh in python 2025-04-10 08:31:25 +00:00
Darragh Elliott
0484e78817 feat: phase2 in python
run pre phase1 and phase 1 directly from build.py
use uv for dep management
2025-04-10 08:31:25 +00:00
Darragh Elliott
c43b6e958f feat: cleaner stage dir logic
Makes stage dirs an internal thing, largely decided by if there are multiple targets or not.
2025-04-10 08:31:25 +00:00
Darragh Elliott
a1abc9c019 feat: remove build-env stuff 2025-04-10 08:31:25 +00:00
Darragh Elliott
bdb44640fc feat: Phase1 in python
Install python deps using venv during the build process
Convert the entirety of phase1 to python
2025-04-10 08:31:25 +00:00