FSFE/fsfe-website
27
32
Fork 93
Code Issues 63 Pull Requests 19 Wiki Activity

feat/python-rewrite #4762

Merged
tobiasd merged 79 commits from :feat/python-rewrite into master 2025-04-24 12:56:33 +00:00
Conversation 20 Commits 79 Files Changed 83 +2643 -2558
delliott commented 2025-01-20 19:20:38 +00:00
Member
Copy Link

Rewrite the whole build process in python, for superior speed and maintenance

Rewrite the whole build process in python, for superior speed and maintenance
delliott force-pushed feat/python-rewrite from f8b17154a1 to 5d00068988 2025-01-20 23:29:41 +00:00 Compare
delliott force-pushed feat/python-rewrite from b05a51078e to af81f98f68 2025-01-26 16:15:32 +00:00 Compare
delliott commented 2025-01-26 22:06:16 +00:00
Author
Member
Copy Link

So, as of now this is, as far as I can tell, a fully functional website build replacement that correctly builds all pages, menus, etc. There may be some small issues somewhere.

The phase1 is significantly faster than before (7s worst case vs 120s worst case).

I have not benchmarked phase 2 yet. I believe there are some optimisations I can make to phase 2, mainly regarding that at the moment the same xhtml|xml files are parsed multiple times with lxml. I should be able to reduce this, but passing lxml objects between funcs can be tricky.

Additionally, I have enhanced the Dockerfile, Docker compose, and introduced entrypoint.sh such that the docker build process no longer interferes with the users repo. It also means that the docker builds properly use caching, so I believe it will be viable to use drone to deploy the website, as we use drone for all other deployments.

Unresolved issues as of now:

How to get files from docker volume to the several serves they need to be on? I think it would be unwise/impossible to allow the container access to the host sshkeys for rsync? easiest thing is probably a cron job on build-server that rsyncs the result from the volume to the targets. I will try and discuss with other syshackers.

While there are some features I want to add I think its best to first move to this enhanced but very similar in behavior to original builder, then add features/ enhancements. This is a significant change, so probably wisest to deploy on the test branch, then on master.

So, as of now this is, as far as I can tell, a fully functional website build replacement that correctly builds all pages, menus, etc. There may be some small issues somewhere. The phase1 is significantly faster than before (7s worst case vs 120s worst case). I have not benchmarked phase 2 yet. I believe there are some optimisations I can make to phase 2, mainly regarding that at the moment the same xhtml|xml files are parsed multiple times with lxml. I should be able to reduce this, but passing lxml objects between funcs can be tricky. Additionally, I have enhanced the Dockerfile, Docker compose, and introduced entrypoint.sh such that the docker build process no longer interferes with the users repo. It also means that the docker builds properly use caching, so I believe it will be viable to use drone to deploy the website, as we use drone for all other deployments. Unresolved issues as of now: How to get files from docker volume to the several serves they need to be on? I think it would be unwise/impossible to allow the container access to the host sshkeys for rsync? easiest thing is probably a cron job on build-server that rsyncs the result from the volume to the targets. I will try and discuss with other syshackers. While there are some features I want to add I think its best to first move to this enhanced but very similar in behavior to original builder, then add features/ enhancements. This is a significant change, so probably wisest to deploy on the test branch, then on master.
🎉 1
delliott force-pushed feat/python-rewrite from d507cff1a2 to d140dafa39 2025-01-26 22:09:46 +00:00 Compare
sofiaritz reviewed 2025-01-27 08:03:15 +00:00
Dockerfile Outdated
@@ -1,23 +1,22 @@
FROM debian:bookworm-slim
RUN apt update
FROM fedora:latest
sofiaritz commented 2025-01-27 08:03:15 +00:00
Member
Copy Link

Great work! I've never seen Fedora used in a Dockerfile :)
Why Fedora and not something like Alpine?

Great work! I've never seen Fedora used in a Dockerfile :) Why Fedora and not something like [Alpine](https://hub.docker.com/_/alpine)?
delliott commented 2025-01-27 09:13:40 +00:00
Author
Member
Copy Link

Under alpine the compilation of tdewolff-minify fails with error __vfprintf_chk: symbol not found. It would seem that this can be solved by building the go binary with different compile flags, E.G here, but I did not feel I knew enough about go to investigate it.

And the go version in the latest debian is too old to compile tdewolff-minify.

I get that it seems the problem here is tdewolff-minify, but it does seem to be the best minifier around for speed and supporting multiple filetypes.

At the moment it is only used for the css generated from less. But in the future, probably in a followup PR, I would like to use it to minify all css, js and html files.

Under alpine the compilation of `tdewolff-minify` fails with error `__vfprintf_chk: symbol not found`. It would seem that this can be solved by building the go binary with different compile flags, E.G [here](https://github.com/influxdata/influxdb/issues/5554), but I did not feel I knew enough about go to investigate it. And the go version in the latest debian is too old to compile `tdewolff-minify`. I get that it seems the problem here is `tdewolff-minify`, but it does seem to be the best minifier around for speed and supporting multiple filetypes. At the moment it is only used for the css generated from less. But in the future, probably in a followup PR, I would like to use it to minify all `css`, `js` and `html` files.
👍 1
delliott commented 2025-03-08 11:40:59 +00:00
Author
Member
Copy Link

Some issues came up with fedora (broken node) and I managed to port it do debian successfully.

Some issues came up with fedora (broken node) and I managed to port it do debian successfully.
delliott marked this conversation as resolved
delliott commented 2025-01-27 09:17:40 +00:00
Author
Member
Copy Link

Things this branch does not do that the original script does

  • I need to fix the staging bit. At the moment it will not actually copy the result from the stage dir to the target dir.

  • It does not do the generated manifest and then the removal of files from the target. So only a full rebuild using --stage will actually remove unused files from the target. I would kinda prefer to avoid implementing this removal logic, as I don't really want to treat the generated website dir as something that is carefully managed, I would prefer that if there are any questions we just full rebuild. But the manifest and removal may be necessary.

Things this branch does not do that the original script does - I need to fix the staging bit. At the moment it will not actually copy the result from the stage dir to the target dir. - It does not do the generated manifest and then the removal of files from the target. So only a full rebuild using `--stage` will actually remove unused files from the target. I would kinda prefer to avoid implementing this removal logic, as I don't really want to treat the generated website dir as something that is carefully managed, I would prefer that if there are any questions we just full rebuild. But the manifest and removal may be necessary.
delliott commented 2025-01-27 09:23:47 +00:00
Author
Member
Copy Link

Some code style to work on:

There are a bunch of pretty common pathlib operation I do, like chaining `with_suffix("") and some stuff that should probably become lib functions.

Some code style to work on: There are a bunch of pretty common pathlib operation I do, like chaining `with_suffix("") and some stuff that should probably become lib functions.
delliott force-pushed feat/python-rewrite from d140dafa39 to 96e24da3d5 2025-01-27 09:34:23 +00:00 Compare
delliott commented 2025-01-27 15:43:36 +00:00
Author
Member
Copy Link

Unresolved issues as of now:

How to get files from docker volume to the several serves they need to be on? I think it would be unwise/impossible to allow the container access to the host sshkeys for rsync? easiest thing is probably a cron job on build-server that rsyncs the result from the volume to the targets. I will try and discuss with other syshackers.

Oh, of course. We can use drone secrets to pass things to the container. So we can pass the secrets keys as env vars, add them to files, load them and deploy.

Or the solution described in this comment seems quite elegant, if less neatly encapsulated in drone. https://stackoverflow.com/a/36648428

> Unresolved issues as of now: > > How to get files from docker volume to the several serves they need to be on? I think it would be unwise/impossible to allow the container access to the host sshkeys for rsync? easiest thing is probably a cron job on build-server that rsyncs the result from the volume to the targets. I will try and discuss with other syshackers. > Oh, of course. We can use drone secrets to pass things to the container. So we can pass the secrets keys as env vars, add them to files, load them and deploy. Or the solution described in this comment seems quite elegant, if less neatly encapsulated in drone. https://stackoverflow.com/a/36648428
delliott commented 2025-01-27 15:50:05 +00:00
Author
Member
Copy Link

Things this branch does not do that the original script does

  • I need to fix the staging bit. At the moment it will not actually copy the result from the stage dir to the target dir.

Fixed in latest commit, it now stages correctly.

> Things this branch does not do that the original script does > > - I need to fix the staging bit. At the moment it will not actually copy the result from the stage dir to the target dir. > Fixed in latest commit, it now stages correctly.
delliott commented 2025-01-27 15:52:42 +00:00
Author
Member
Copy Link

Oh, and another issue: the fsfe status cgi script is no longer works meangingfully.

I can patch the build process and the scig script to show the new logs, but if we are using drone that is kinda just unnecessary? Probably better to remove it.

I also plan to add a buildtime flag from generating the translation status pages, so we should get to replace that stuff.

Oh, and another issue: the fsfe status cgi script is no longer works meangingfully. I can patch the build process and the scig script to show the new logs, but if we are using drone that is kinda just unnecessary? Probably better to remove it. I also plan to add a buildtime flag from generating the translation status pages, so we should get to replace that stuff.
delliott commented 2025-01-27 17:32:47 +00:00
Author
Member
Copy Link

Did some basic benchmarking on my laptop, should not be taken as solid proof of anything, but shows general trends. Both for full rebuilds with no caching. Current bash build script was ran first, so as not to make improvements seem better than real.

Old bash build: 24 mins

New build: 8 mins

Did some basic benchmarking on my laptop, should not be taken as solid proof of anything, but shows general trends. Both for full rebuilds with no caching. Current bash build script was ran first, so as not to make improvements seem better than real. Old bash build: 24 mins New build: 8 mins
delliott commented 2025-01-28 16:07:09 +00:00
Author
Member
Copy Link

Some code style to work on:

There are a bunch of pretty common pathlib operation I do, like chaining `with_suffix("") and some stuff that should probably become lib functions.

The worst offenders for this have now been functionised.

> Some code style to work on: > > There are a bunch of pretty common pathlib operation I do, like chaining `with_suffix("") and some stuff that should probably become lib functions. The worst offenders for this have now been functionised.
delliott force-pushed feat/python-rewrite from 5105374e10 to 89623e5407 2025-01-30 16:39:54 +00:00 Compare
delliott changed title from WIP: feat/python-rewrite to feat/python-rewrite 2025-02-06 13:27:42 +00:00
delliott changed title from feat/python-rewrite to WIP: feat/python-rewrite 2025-02-06 13:28:57 +00:00
delliott force-pushed feat/python-rewrite from 89623e5407 to 4ea6d88597 2025-02-06 13:29:44 +00:00 Compare
delliott force-pushed feat/python-rewrite from 4ea6d88597 to 6e978b576d 2025-02-08 19:41:54 +00:00 Compare
delliott commented 2025-02-08 19:43:25 +00:00
Author
Member
Copy Link

Have now rewritten the translation-status script ad added it to the build process.

New one takes ~30s for all languages, old took ~33 mins for all languages.

So thats a nice improvement.

Have now rewritten the translation-status script ad added it to the build process. New one takes ~30s for all languages, old took ~33 mins for all languages. So thats a nice improvement.
delliott commented 2025-02-08 21:44:42 +00:00
Author
Member
Copy Link

As far as I can tell at this point, the build process works properly using the script locally, using docker and using drone exec to run the pipeline locally.

The next step is to disable the pull based ci for the test branch. I have a pr to do this fsfe-system-hackers/build-server#15. After its merged someone with access will have to redeploy the webserver.

Then merge this pr to the test branch. Alter the config so it aims at the right dirs on the webservers.

This will need sshkeys to access the webservers to deploy the test branch builds too. Load them into ssh as per https://stackoverflow.com/a/72654766

And to sign the drone file, rerun it, etc I need more drone permissions for this repo. I have commit access for the repo, but for some reason Tobias and I cant figure that does not give me drone access.

This branch is stuck waiting on the above requirements.

Pinging @tobiasd as maintainer.

As far as I can tell at this point, the build process works properly using the script locally, using docker and using drone exec to run the pipeline locally. The next step is to disable the pull based ci for the test branch. I have a pr to do this https://git.fsfe.org/fsfe-system-hackers/build-server/pulls/15. After its merged someone with access will have to redeploy the webserver. Then merge this pr to the test branch. Alter the config so it aims at the right dirs on the webservers. This will need sshkeys to access the webservers to deploy the test branch builds too. Load them into ssh as per https://stackoverflow.com/a/72654766 And to sign the drone file, rerun it, etc I need more drone permissions for this repo. I have commit access for the repo, but for some reason Tobias and I cant figure that does not give me drone access. This branch is stuck waiting on the above requirements. Pinging @tobiasd as maintainer.
delliott force-pushed feat/python-rewrite from 24391c7ef5 to 8769c12efb 2025-02-10 10:34:36 +00:00 Compare
delliott force-pushed feat/python-rewrite from 797de26bfd to e6bd3ada8f 2025-02-12 14:19:39 +00:00 Compare
delliott force-pushed feat/python-rewrite from 4115fedf9d to 30d5eaa8b4 2025-02-13 13:20:44 +00:00 Compare
delliott force-pushed feat/python-rewrite from 30d5eaa8b4 to 329e09084d 2025-02-13 13:34:34 +00:00 Compare
delliott force-pushed feat/python-rewrite from 3b6745aa93 to 17bfcf6476 2025-02-14 14:15:12 +00:00 Compare
delliott force-pushed feat/python-rewrite from c1bcf03375 to aa8d373e67 2025-02-15 13:53:48 +00:00 Compare
delliott commented 2025-02-15 13:55:52 +00:00
Author
Member
Copy Link

So, I appreciate that it seems like this is still under heavy development, but frankly I am just grabbing some small optimizations and code style improvements.

So, I appreciate that it seems like this is still under heavy development, but frankly I am just grabbing some small optimizations and code style improvements.
delliott force-pushed feat/python-rewrite from 2d9514e976 to aac039d57d 2025-03-02 19:40:26 +00:00 Compare
delliott force-pushed feat/python-rewrite from aac039d57d to c22c26453c 2025-03-04 16:19:37 +00:00 Compare
delliott force-pushed feat/python-rewrite from a593c6daf9 to ccb4d0e5d9 2025-03-07 16:24:55 +00:00 Compare
delliott commented 2025-03-08 11:40:20 +00:00
Author
Member
Copy Link

This is now deployed on the test branch.

This is now deployed on the test branch.
🎉 1
delliott force-pushed feat/python-rewrite from 8d75825b82 to 1d3edd02f6 2025-03-10 18:47:06 +00:00 Compare
delliott commented 2025-03-10 18:49:25 +00:00
Author
Member
Copy Link

So, progress report:
Loading of secrets works (I think), and the docker compose file enables ipv6 properly. Ran it locally, so a build from this branch is currently live on test.fsfe.org
But it does not work when ran in drone, I think because the parent docker container, the official docker image, does not enable ipv6.

So we can enable ipv6 by default for all images, in docker daemon conf, but thats not ideal.

So, progress report: Loading of secrets works (I think), and the docker compose file enables ipv6 properly. Ran it locally, so a build from this branch is currently live on test.fsfe.org But it does not work when ran in drone, I think because the parent docker container, the official docker image, does not enable ipv6. So we can enable ipv6 by default for all images, in docker daemon conf, but thats not ideal.
delliott commented 2025-03-10 18:52:27 +00:00
Author
Member
Copy Link

Other problem we have is that the ephemeral docker image used by drone means that layers dont get cached between runs, massively increasing the build time.

I think we maybe use a drone exec pipeline to run with one less layer of docker, I think would fix our ipv6+caching issues.

But less secure, and different to how most of our services work.

Other problem we have is that the ephemeral docker image used by drone means that layers dont get cached between runs, massively increasing the build time. I think we maybe use a drone exec pipeline to run with one less layer of docker, I think would fix our ipv6+caching issues. But less secure, and different to how most of our services work.
delliott force-pushed feat/python-rewrite from 1d3edd02f6 to 88721f94b9 2025-03-12 11:58:08 +00:00 Compare
delliott commented 2025-03-12 11:58:49 +00:00
Author
Member
Copy Link

Ipv6 now not needed, as using ipv4 proxies to the ipv6 servers.

Caching still an issue, final blocker I am aware of.

Ipv6 now not needed, as using ipv4 proxies to the ipv6 servers. Caching still an issue, final blocker I am aware of.
delliott force-pushed feat/python-rewrite from 88721f94b9 to edc241c185 2025-03-12 14:19:40 +00:00 Compare
delliott commented 2025-03-12 14:21:32 +00:00
Author
Member
Copy Link

Caching is now fixed.

The docker in docker container is pruned nightly, so the first build every day will do the whole dep reinstall. But I have optimized that a little, so it's not too slow.

And after that cached docker layers will be used till the next day.

And fixed the build incremental caching, so now it works. There are a few bits that dont seem to cache properly, but I think that hunting them down is a future problem.

Caching is now fixed. The docker in docker container is pruned nightly, so the first build every day will do the whole dep reinstall. But I have optimized that a little, so it's not too slow. And after that cached docker layers will be used till the next day. And fixed the build incremental caching, so now it works. There are a few bits that dont seem to cache properly, but I think that hunting them down is a future problem.
delliott force-pushed feat/python-rewrite from 35e5515feb to 99a93470b0 2025-03-13 15:16:49 +00:00 Compare
delliott commented 2025-03-13 15:20:17 +00:00
Author
Member
Copy Link

So.

This now largely works, and is deployed on test.fsfe.org. There are a few things to iron out.

I will be very busy the next while, so there will likely be no further development for a bit.

So. This now largely works, and is deployed on `test.fsfe.org`. There are a few things to iron out. I will be very busy the next while, so there will likely be no further development for a bit.
delliott force-pushed feat/python-rewrite from 99a93470b0 to edd80f302c 2025-04-05 14:34:42 +00:00 Compare
delliott force-pushed feat/python-rewrite from 7468236a03 to 7b0e3bf3b4 2025-04-05 15:46:39 +00:00 Compare
delliott commented 2025-04-05 15:49:52 +00:00
Author
Member
Copy Link

So,
What is left for this PR?

Done

It would seem that the site builds correctly, all links working and etc. The CI setup for deploying it to test.fsfe.org works correctly.

The build itself is much faster than before, and seems to cache properly.

Todo

In python performance

There are several places where some speed in the python itself can be gained, all marked with TODOs. These would give slightly better performance, but I think its best to leave that to a follow up pr. This one is already huge, and reworking some of the stuff has significant breakage risks.

TLDR: Some optimization possible leave to future pr.

CI

The check step uses a different image, and takes a while to install and run its tests. Before builds did not actually need this to pass of event start to start building and deployment, so they did not have the delay it adds. Probably just a price to pay for not building and deployed commits that fail ci. Nothing to do really.

We delete our drone runner image/clean our caches every night. This means that the first run the following day takes another 2 minutes or so. This should be okay as we will presumably just set a nighty cron to build the webpages anyway. Just an FYI really.

TLDR: Some small delays that are slightly annoying, but no real mitigation available. And only small delays.

Deployment

If one inspects the drone yml file they will see a step for sending to test.fsfe.org. A similar one would need to be introduced to deploy it to fsfe.org proper pre merge.

Additionally, before this could be merged, one must disable the existing build process for the website, as was done in fsfe-system-hackers/build-server#15 for the test branch.

So, What is left for this PR? ## Done It would seem that the site builds correctly, all links working and etc. The CI setup for deploying it to `test.fsfe.org` works correctly. The build itself is much faster than before, and seems to cache properly. ## Todo ### In python performance There are several places where some speed in the python itself can be gained, all marked with TODOs. These would give slightly better performance, but I think its best to leave that to a follow up pr. This one is already huge, and reworking some of the stuff has significant breakage risks. TLDR: Some optimization possible leave to future pr. ### CI The check step uses a different image, and takes a while to install and run its tests. Before builds did not actually need this to pass of event start to start building and deployment, so they did not have the delay it adds. Probably just a price to pay for not building and deployed commits that fail ci. Nothing to do really. We delete our drone runner image/clean our caches every night. This means that the first run the following day takes another 2 minutes or so. This should be okay as we will presumably just set a nighty cron to build the webpages anyway. Just an FYI really. TLDR: Some small delays that are slightly annoying, but no real mitigation available. And only small delays. ### Deployment If one inspects the drone yml file they will see a step for sending to test.fsfe.org. A similar one would need to be introduced to deploy it to `fsfe.org` proper pre merge. Additionally, before this could be merged, one must disable the existing build process for the website, as was done in https://git.fsfe.org/fsfe-system-hackers/build-server/pulls/15 for the test branch.
delliott commented 2025-04-08 11:43:51 +00:00
Author
Member
Copy Link

Oh, and we should probably be using a pyproject.toml, but again, that can wait for a future pr.

Oh, and we should probably be using a `pyproject.toml`, but again, that can wait for a future pr.
delliott force-pushed feat/python-rewrite from 7b0e3bf3b4 to 9d3daed622 2025-04-08 13:40:21 +00:00 Compare
sofiaritz force-pushed feat/python-rewrite from 0e4850c286 to a7e3b10a87 2025-04-10 08:31:41 +00:00 Compare
sofiaritz added 1 commit 2025-04-11 13:50:18 +00:00
Copy .htaccess and .htuser files f5a8cb3614
tobiasd added 1 commit 2025-04-15 08:45:56 +00:00
Merge branch 'master' into feat/python-rewrite 16bb89445e
tobiasd added 1 commit 2025-04-24 09:36:56 +00:00
Merge branch 'master' into feat/python-rewrite 0b66e625e5
tobiasd added 6 commits 2025-04-24 11:36:45 +00:00
Create activities XML file 24ee1225fe
Fix environ 76f27c6895
Fix environ 2 6c1da900c4
add target for master 12a3ef746d
Merge branch 'master' into feat/python-rewrite 971877faf0
Merge branch 'feat/python-rewrite' of https://git.fsfe.org/delliott/fsfe-website into feat/python-rewrite 16fd6b9cc4
tobiasd changed title from WIP: feat/python-rewrite to feat/python-rewrite 2025-04-24 12:53:11 +00:00
tobiasd added 1 commit 2025-04-24 12:53:25 +00:00
Merge branch 'master' into feat/python-rewrite 0bcd0e69fa
tobiasd merged commit 4468cf2337 into master 2025-04-24 12:56:33 +00:00
tobiasd referenced this issue from a commit 2025-04-24 12:56:34 +00:00
feat/python-rewrite (#4762)
Sign in to join this conversation.
Reviewers
No Reviewers
sofiaritz
Labels
Clear labels
bug
build
cgi Scripting
design
disruptive
Requires careful coordination and documentation changes
 documentation
duplicate
easy
enhancement
Improving what is already there
 feature-request
help wanted
javascript
priority/low
question
system-hackers
tagging
text
translations
wait/bugfix
wait/inprogress
wait/misc
wait/proofread
wontfix
xsl
No Label
Milestone
No items
No Milestone
Assignees
Clear assignees
albert alex.busch alex.sander anaghz ao bcludts bonnie cryptie dario delliott dfajfer dmaphy doczkal eal egnun eventregbot fi fkobi floriansnow gabriel.ku guido hl hugo ineiev jn jzarl linus lucabon lucas.lasota max.mehl mk monochromec mweimann nico.rikken patrick reinhard renovate-bot repentinus schiessle sofiaritz tobiasd vincent
No Assignees
3 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: FSFE/fsfe-website#4762
Delete Branch ":feat/python-rewrite"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?