FSFE/fsfe-website
25
30
Fork 89
Code Issues 63 Pull Requests 9 Wiki Activity

WIP: feat: python phase 1 #4609

Draft
delliott wants to merge 1 commits from delliott/fsfe-website:feat/python-tools into master
pull from: delliott/fsfe-website:feat/python-tools
merge into: FSFE:master
Conversation 5 Commits 1 Files Changed 38 +1220 -944
delliott commented 2024-11-16 13:26:26 +00:00
Contributor
Copy Link

Install python deps using venv during the build process

Rewrite the entirety of Phase1 in python.
This brings significant speed ups. At time of writing, pre some further optimization, I am seeing ~6secs vs ~113secs for worst case scenario (Full rebuild)

This will unfortunately require some additionally dependencies on the build server. The upside is that any future python deps can be added without touching the build server.
The additional deps are:

python3-venv
python3-pip

I believe this worthwhile for the speed improvements, superior portability and more reproducible development environments.

Pinging @tobiasd as maintainer.

Install python deps using venv during the build process Rewrite the entirety of Phase1 in python. This brings significant speed ups. At time of writing, pre some further optimization, I am seeing ~6secs vs ~113secs for worst case scenario (Full rebuild) This will unfortunately require some additionally dependencies on the build server. The upside is that any future python deps can be added without touching the build server. The additional deps are: ``` python3-venv python3-pip ``` I believe this worthwhile for the speed improvements, superior portability and more reproducible development environments. Pinging @tobiasd as maintainer.
delliott force-pushed feat/python-tools from 910ed37845 to 8e510f094b 2024-11-18 11:40:35 +00:00 Compare
delliott commented 2024-11-18 13:03:54 +00:00
Author
Contributor
Copy Link

Pr for updating build server as required found here: fsfe-system-hackers/build-server#14

Pr for updating build server as required found here: https://git.fsfe.org/fsfe-system-hackers/build-server/pulls/14
tobiasd reviewed 2024-11-26 10:05:32 +00:00
README.md Outdated
@ -86,3 +86,3 @@
The package names for Debian, are
```
bash bash-completion coreutils diffutils findutils inotify-tools libxml2-utils libxslt make procps python3 rsync
bash bash-completion coreutils diffutils findutils inotify-tools libxml2-utils libxslt make procps python3 pip rsync
tobiasd commented 2024-11-26 10:05:32 +00:00
Owner
Copy Link

The Debian package for pip is python3-pip I think

The Debian package for `pip` is `python3-pip` I think
delliott commented 2024-11-26 14:46:10 +00:00
Author
Contributor
Copy Link

Yes, I think you are right

Yes, I think you are right
delliott marked this conversation as resolved
tobiasd reviewed 2024-11-26 10:06:17 +00:00
build/build_main.sh Outdated
@ -23,3 +23,3 @@
# Check dependencies for all kinds of build envs (e.g. development, fsfe.org)
check_dependencies realpath rsync xsltproc xmllint sed find egrep grep wc make tee date iconv wget shuf python3
check_dependencies realpath rsync xsltproc xmllint sed find egrep grep wc make tee date iconv wget shuf python3 pip
tobiasd commented 2024-11-26 10:06:17 +00:00
Owner
Copy Link

Here as well python3-pip right?

Here as well `python3-pip` right?
delliott marked this conversation as resolved
delliott force-pushed feat/python-tools from 8e510f094b to c5defbf983 2024-11-26 20:56:48 +00:00 Compare
delliott force-pushed feat/python-tools from c5defbf983 to d15c2c5a76 2024-11-27 23:19:00 +00:00 Compare
delliott force-pushed feat/python-tools from d15c2c5a76 to 279100dd71 2024-12-03 12:20:32 +00:00 Compare
delliott commented 2024-12-03 12:24:35 +00:00
Author
Contributor
Copy Link

Pr now includes a python build.py wrapper over the main build scripts. It enjoys far more sensible arguments than build_main.sh, and is easier to write and maintain than build.sh.

My goal is that all builds of the website will be done using this script, and then the build process itself can be slowly rewritten with build.py as its stable interface.

Pr now includes a python `build.py` wrapper over the main build scripts. It enjoys far more sensible arguments than `build_main.sh`, and is easier to write and maintain than `build.sh`. My goal is that all builds of the website will be done using this script, and then the build process itself can be slowly rewritten with `build.py` as its stable interface.
delliott commented 2024-12-14 14:57:19 +00:00
Author
Contributor
Copy Link

As of now this pr is designed such that it requires #4660 and #4564

As of now this pr is designed such that it requires https://git.fsfe.org/FSFE/fsfe-website/pulls/4660 and https://git.fsfe.org/FSFE/fsfe-website/pulls/4564
delliott force-pushed feat/python-tools from 279100dd71 to 2302fbddbe 2024-12-14 20:33:33 +00:00 Compare
delliott force-pushed feat/python-tools from 2302fbddbe to 930acdc344 2024-12-15 17:56:00 +00:00 Compare
delliott commented 2024-12-15 18:03:05 +00:00
Author
Contributor
Copy Link

The scope of this pr has increased a little from when it started, and I have updated the main comment to reflect this.
This pr does not require altering the cron job on the server, although it does need the deps mentioned above.

After this has been merged for a bit I would like to alter the cronjobs to use the interface provided by build.py. Its far easier to make sense of, and I intend build.py so serve as a stable frontend throughout the rest of the rewrite process.

The scope of this pr has increased a little from when it started, and I have updated the main comment to reflect this. This pr does not require altering the cron job on the server, although it does need the deps mentioned above. After this has been merged for a bit I would like to alter the cronjobs to use the interface provided by `build.py`. Its far easier to make sense of, and I intend `build.py` so serve as a stable frontend throughout the rest of the rewrite process.
delliott requested review from tobiasd 2024-12-15 18:06:53 +00:00
delliott force-pushed feat/python-tools from 930acdc344 to 8914820be7 2024-12-15 23:10:03 +00:00 Compare
delliott changed title from WIP: feat: pythonisation to WIP: feat: python phase 1 2024-12-15 23:10:04 +00:00
delliott force-pushed feat/python-tools from 8914820be7 to 6ac91ca7bc 2024-12-18 13:46:22 +00:00 Compare
delliott commented 2024-12-18 13:48:03 +00:00
Author
Contributor
Copy Link

Is currently fully functional and seems to correctly handle all pages, tags etc.

Will keep in wip until the two prerequisites are merged, but can be tested in the meantime.

Is currently fully functional and seems to correctly handle all pages, tags etc. Will keep in wip until the two prerequisites are merged, but can be tested in the meantime.
delliott force-pushed feat/python-tools from 6ac91ca7bc to 881b24d83f 2024-12-29 23:17:28 +00:00 Compare
delliott force-pushed feat/python-tools from 881b24d83f to 3393a77836 2025-01-01 22:02:33 +00:00 Compare
delliott removed review request for tobiasd 2025-01-08 10:13:48 +00:00
delliott force-pushed feat/python-tools from 3393a77836 to ccc40ab9a5 2025-01-08 11:37:26 +00:00 Compare
All checks were successful
continuous-integration/drone/pr Build is passing
Required
Details
This pull request is marked as a work in progress.
This branch is out-of-date with the base branch
View command line instructions.

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u feat/python-tools:delliott-feat/python-tools
git checkout delliott-feat/python-tools
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
bug

   
build

   
cgi Scripting

   
design

   
disruptive

Requires careful coordination and documentation changes

  
documentation

   
duplicate

   
easy

   
feature-request

   
help wanted

   
javascript

   
priority/low

   
question

   
system-hackers

   
tagging

   
text

   
translations

   
wait/bugfix

   
wait/inprogress

   
wait/misc

   
wait/proofread

   
wontfix

   
xsl
No Label
bug
build
cgi Scripting
design
disruptive
documentation
duplicate
easy
feature-request
help wanted
javascript
priority/low
question
system-hackers
tagging
text
translations
wait/bugfix
wait/inprogress
wait/misc
wait/proofread
wontfix
xsl
Milestone
Clear milestone
No items
No Milestone
Assignees
Clear assignees
No Assignees
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: FSFE/fsfe-website#4609
No description provided.
Delete Branch "delliott/fsfe-website:feat/python-tools"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?