FSFE/fsfe-website
28
32
Fork 93
Code Issues 64 Pull Requests 13 Wiki Activity

WIP: feat: python phase 1 #4609

Closed
delliott wants to merge 1 commits from (deleted):feat/python-tools into master
pull from: (deleted):feat/python-tools
merge into: FSFE:master
Conversation 8 Commits 1 Files Changed 38 +1220 -944
delliott commented 2024-11-16 13:26:26 +00:00
Member
Copy Link

Install python deps using venv during the build process

Rewrite the entirety of Phase1 in python.
This brings significant speed ups. At time of writing, pre some further optimization, I am seeing ~6secs vs ~113secs for worst case scenario (Full rebuild)

This will unfortunately require some additionally dependencies on the build server. The upside is that any future python deps can be added without touching the build server.
The additional deps are:

python3-venv
python3-pip

I believe this worthwhile for the speed improvements, superior portability and more reproducible development environments.

Pinging @tobiasd as maintainer.

Install python deps using venv during the build process Rewrite the entirety of Phase1 in python. This brings significant speed ups. At time of writing, pre some further optimization, I am seeing ~6secs vs ~113secs for worst case scenario (Full rebuild) This will unfortunately require some additionally dependencies on the build server. The upside is that any future python deps can be added without touching the build server. The additional deps are: ``` python3-venv python3-pip ``` I believe this worthwhile for the speed improvements, superior portability and more reproducible development environments. Pinging @tobiasd as maintainer.
delliott force-pushed feat/python-tools from 910ed37845 to 8e510f094b 2024-11-18 11:40:35 +00:00 Compare
delliott commented 2024-11-18 13:03:54 +00:00
Author
Member
Copy Link

Pr for updating build server as required found here: fsfe-system-hackers/build-server#14

Pr for updating build server as required found here: https://git.fsfe.org/fsfe-system-hackers/build-server/pulls/14
tobiasd reviewed 2024-11-26 10:05:32 +00:00
README.md Outdated
@@ -86,3 +86,3 @@
The package names for Debian, are
```
bash bash-completion coreutils diffutils findutils inotify-tools libxml2-utils libxslt make procps python3 rsync
bash bash-completion coreutils diffutils findutils inotify-tools libxml2-utils libxslt make procps python3 pip rsync
tobiasd commented 2024-11-26 10:05:32 +00:00
Owner
Copy Link

The Debian package for pip is python3-pip I think

The Debian package for `pip` is `python3-pip` I think
delliott commented 2024-11-26 14:46:10 +00:00
Author
Member
Copy Link

Yes, I think you are right

Yes, I think you are right
delliott marked this conversation as resolved
tobiasd reviewed 2024-11-26 10:06:17 +00:00
build/build_main.sh Outdated
@@ -23,3 +23,3 @@
# Check dependencies for all kinds of build envs (e.g. development, fsfe.org)
check_dependencies realpath rsync xsltproc xmllint sed find egrep grep wc make tee date iconv wget shuf python3
check_dependencies realpath rsync xsltproc xmllint sed find egrep grep wc make tee date iconv wget shuf python3 pip
tobiasd commented 2024-11-26 10:06:17 +00:00
Owner
Copy Link

Here as well python3-pip right?

Here as well `python3-pip` right?
delliott marked this conversation as resolved
delliott force-pushed feat/python-tools from 8e510f094b to c5defbf983 2024-11-26 20:56:48 +00:00 Compare
delliott force-pushed feat/python-tools from c5defbf983 to d15c2c5a76 2024-11-27 23:19:00 +00:00 Compare
delliott force-pushed feat/python-tools from d15c2c5a76 to 279100dd71 2024-12-03 12:20:32 +00:00 Compare
delliott commented 2024-12-03 12:24:35 +00:00
Author
Member
Copy Link

Pr now includes a python build.py wrapper over the main build scripts. It enjoys far more sensible arguments than build_main.sh, and is easier to write and maintain than build.sh.

My goal is that all builds of the website will be done using this script, and then the build process itself can be slowly rewritten with build.py as its stable interface.

Pr now includes a python `build.py` wrapper over the main build scripts. It enjoys far more sensible arguments than `build_main.sh`, and is easier to write and maintain than `build.sh`. My goal is that all builds of the website will be done using this script, and then the build process itself can be slowly rewritten with `build.py` as its stable interface.
delliott commented 2024-12-14 14:57:19 +00:00
Author
Member
Copy Link

As of now this pr is designed such that it requires #4660 and #4564

As of now this pr is designed such that it requires https://git.fsfe.org/FSFE/fsfe-website/pulls/4660 and https://git.fsfe.org/FSFE/fsfe-website/pulls/4564
delliott force-pushed feat/python-tools from 279100dd71 to 2302fbddbe 2024-12-14 20:33:33 +00:00 Compare
delliott force-pushed feat/python-tools from 2302fbddbe to 930acdc344 2024-12-15 17:56:00 +00:00 Compare
delliott commented 2024-12-15 18:03:05 +00:00
Author
Member
Copy Link

The scope of this pr has increased a little from when it started, and I have updated the main comment to reflect this.
This pr does not require altering the cron job on the server, although it does need the deps mentioned above.

After this has been merged for a bit I would like to alter the cronjobs to use the interface provided by build.py. Its far easier to make sense of, and I intend build.py so serve as a stable frontend throughout the rest of the rewrite process.

The scope of this pr has increased a little from when it started, and I have updated the main comment to reflect this. This pr does not require altering the cron job on the server, although it does need the deps mentioned above. After this has been merged for a bit I would like to alter the cronjobs to use the interface provided by `build.py`. Its far easier to make sense of, and I intend `build.py` so serve as a stable frontend throughout the rest of the rewrite process.
delliott requested review from tobiasd 2024-12-15 18:06:53 +00:00
delliott force-pushed feat/python-tools from 930acdc344 to 8914820be7 2024-12-15 23:10:03 +00:00 Compare
delliott changed title from WIP: feat: pythonisation to WIP: feat: python phase 1 2024-12-15 23:10:04 +00:00
delliott force-pushed feat/python-tools from 8914820be7 to 6ac91ca7bc 2024-12-18 13:46:22 +00:00 Compare
delliott commented 2024-12-18 13:48:03 +00:00
Author
Member
Copy Link

Is currently fully functional and seems to correctly handle all pages, tags etc.

Will keep in wip until the two prerequisites are merged, but can be tested in the meantime.

Is currently fully functional and seems to correctly handle all pages, tags etc. Will keep in wip until the two prerequisites are merged, but can be tested in the meantime.
delliott force-pushed feat/python-tools from 6ac91ca7bc to 881b24d83f 2024-12-29 23:17:28 +00:00 Compare
delliott force-pushed feat/python-tools from 881b24d83f to 3393a77836 2025-01-01 22:02:33 +00:00 Compare
delliott removed review request for tobiasd 2025-01-08 10:13:48 +00:00
delliott force-pushed feat/python-tools from 3393a77836 to ccc40ab9a5 2025-01-08 11:37:26 +00:00 Compare
delliott commented 2025-01-20 10:48:20 +00:00
Author
Member
Copy Link

Will conglomerate these changes in a full python rewrite

Will conglomerate these changes in a full python rewrite
delliott closed this pull request 2025-01-20 10:48:21 +00:00
All checks were successful
continuous-integration/drone/pr Build is passing
Required
Details

Pull request closed

This pull request cannot be reopened because the branch was deleted.
Sign in to join this conversation.
Reviewers
No Reviewers
tobiasd
Labels
Clear labels
bug
build
cgi Scripting
design
disruptive
Requires careful coordination and documentation changes
 documentation
duplicate
easy
enhancement
Improving what is already there
 feature-request
help wanted
javascript
priority/low
question
system-hackers
tagging
text
translations
wait/bugfix
wait/inprogress
wait/misc
wait/proofread
wontfix
xsl
No Label
Milestone
No items
No Milestone
Assignees
Clear assignees
albert alex.busch alex.sander anaghz ao bcludts bonnie cryptie dario delliott dfajfer dmaphy doczkal eal egnun eventregbot fi fkobi floriansnow gabriel.ku guido hl hugo ineiev jn jzarl linus lucabon lucas.lasota max.mehl mk monochromec mweimann nico.rikken patrick reinhard renovate-bot repentinus schiessle sofiaritz tobiasd vincent
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: FSFE/fsfe-website#4609
Delete Branch "(deleted):feat/python-tools"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?