lucabon/fsfe-website
1
0
Fork 0
forked from FSFE/fsfe-website
Code Issues Pull Requests Releases Wiki Activity

feat: build by site (#5031)

Browse Source 
phase 1 and 2 now by site folder

This allows for us building each site in a different set of languages, wheres before we built every site in every language any site was in.

At the moment this will build each site in every language that there is at least one file of that language in the site.

So if a site has on file in lang AA, it will be built in lang AA. But if it has no files, it will not be built in that lang.

This is a performance enhancement, will do benchmarks later.

Some more complex heuristics for when we do/do not use a language for a file are being discussed in FSFE/fsfe-website#4601 .

Co-authored-by: Darragh Elliott <me@delliott.xyz>
Reviewed-on: FSFE/fsfe-website#5031
Reviewed-by: tobiasd <tobiasd@fsfe.org>
Reviewed-by: Sofía Aritz <sofiaritz@fsfe.org>
Co-authored-by: delliott <delliott@fsfe.org>
Co-committed-by: delliott <delliott@fsfe.org>
This commit is contained in:
delliott
2025-05-26 12:18:43 +00:00
committed by tobiasd
parent d3c66e8918
commit 9c15f9400a
28 changed files with 321 additions and 160 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
README.md
View File

@@ -76,7 +76,7 @@ You can see the current status of translation progress of fsfe.org at [status.fs
## Build
There are two ways to build and develop the directory locally. Initial builds of the webpages may take ~12 minutes, but subsequent builds should be much faster. Using the `--languages` flag to avoid building all supported languages can make this much faster. Run `./build.py --help` for more information.
There are two ways to build and develop the directory locally. Initial builds of the webpages may take ~12 minutes, but subsequent builds should be much faster. Using the `--languages` flag to avoid building all supported languages can make this much faster. The `--sites` flag allows for building only some of the sites in this repo, which can also provide a speed boost to the developer experience. Run `./build.py --help` for more information.
Alterations to build scripts or the files used site-wide will result in near full rebuilds.

109
build.py
View File

@@ -8,11 +8,18 @@ import argparse
import logging
import multiprocessing
import os
import sys
from pathlib import Path
from build.lib.misc import lang_from_filename
from build.phase0.full import full
from build.phase0.global_symlinks import global_symlinks
from build.phase0.prepare_early_subdirectories import prepare_early_subdirectories
from build.phase1.run import phase1_run
from build.phase2.run import phase2_run
from build.phase3.serve_websites import serve_websites
from build.phase3.stage_to_target import stage_to_target
@@ -28,53 +35,50 @@ def parse_arguments() -> argparse.Namespace:
description="Python script to handle building of the fsfe webpage"
)
parser.add_argument(
"--target",
dest="target",
help="Final dirs for websites to be build to. Can be a single path, or a comma separated list of valid rsync targets. Supports custom rsynx extension for specifying ports for ssh targets, name@host:path?port.",
type=str,
default="./output/final",
"--full",
help="Force a full rebuild of all webpages.",
action="store_true",
)
parser.add_argument(
"--languages",
help="Languages to build website in.",
default=[],
type=lambda input: input.split(","),
)
parser.add_argument(
"--log-level",
dest="log_level",
type=str,
default="INFO",
choices=["DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL"],
help="Set the logging level (default: INFO)",
)
parser.add_argument(
"--full",
dest="full",
help="Force a full rebuild of all webpages.",
action="store_true",
)
parser.add_argument(
"--processes",
dest="processes",
help="Number of processes to use when building the website",
type=int,
default=multiprocessing.cpu_count(),
)
parser.add_argument(
"--languages",
dest="languages",
help="Languages to build website in.",
default=list(
map(lambda path: path.name, Path(".").glob("global/languages/??"))
),
type=lambda input: input.split(","),
"--serve",
help="Serve the webpages after rebuild",
action="store_true",
)
parser.add_argument(
"--sites",
help="What site directories to build",
default=list(filter(lambda path: path.is_dir(), Path().glob("?*.??*"))),
type=lambda input: list(map(lambda site: Path(site), input.split(","))),
)
parser.add_argument(
"--stage",
dest="stage",
help="Force the use of an internal staging directory.",
action="store_true",
)
parser.add_argument(
"--serve",
dest="serve",
help="Serve the webpages after rebuild",
action="store_true",
"--target",
help="Final dirs for websites to be build to. Can be a single path, or a comma separated list of valid rsync targets. Supports custom rsynx extension for specifying ports for ssh targets, name@host:path?port.",
type=str,
default="./output/final",
)
args = parser.parse_args()
return args
@@ -89,22 +93,65 @@ def main(args: argparse.Namespace):
logger.debug(args)
with multiprocessing.Pool(args.processes) as pool:
logger.info("Starting phase 0 - Conditional Setup")
logger.info("Starting phase 0 - Global Conditional Setup")
# TODO Should also be triggered whenever any build python file is changed
if args.full:
full()
# -----------------------------------------------------------------------------
# Create XML symlinks
# -----------------------------------------------------------------------------
# After this step, the following symlinks will exist:
# * global/data/texts/.texts.<lang>.xml for each language
# * global/data/topbanner/.topbanner.<lang>.xml for each language
# Each of these symlinks will point to the corresponding file without a dot at
# the beginning of the filename, if present, and to the English version
# otherwise. This symlinks make sure that phase 2 can easily use the right file
# for each language
global_symlinks(
args.languages
if args.languages
else list(
map(lambda path: path.name, Path(".").glob("global/languages/??"))
),
pool,
)
# Early subdirs
# for subdir actions that need to be performed very early in the build process. Do not get access to languages to be built in, and other benefits of being ran later.
prepare_early_subdirectories(
Path(),
args.processes,
)
stage_required = any(
[args.stage, "@" in args.target, ":" in args.target, "," in args.target]
)
working_target = Path("./output/stage" if stage_required else args.target)
# the two middle phases are unconditional, and run on a per site basis
for site in args.sites:
logger.info(f"Processing {site}")
if not site.exists():
logger.critical(f"Site {site} does not exist, exiting")
sys.exit(1)
languages = (
args.languages
if args.languages
else list(
set(
map(
lambda path: lang_from_filename(path),
site.glob("**/*.*.xhtml"),
)
)
)
)
# Processes needed only for subdir stuff
phase1_run(site, languages, args.processes, pool)
phase2_run(site, languages, pool, working_target.joinpath(site))
# Processes needed only for subdir stuff
phase1_run(args.languages, args.processes, pool)
phase2_run(args.languages, pool, working_target)
logger.info("Starting Phase 3 - Conditional Finishing")
logger.info("Starting Phase 3 - Global Conditional Finishing")
if stage_required:
stage_to_target(working_target, args.target, pool)

2
build/lib/process_file.py
View File

@@ -68,8 +68,6 @@ def _list_langs(file: Path) -> str:
Path(f"global/languages/{lang_from_filename(path)}")
.read_text()
.strip()
if Path(f"global/languages/{lang_from_filename(path)}").exists()
else lang_from_filename(path)
)
+ "</tr>"
),

0
build/phase1/global_symlinks.py → build/phase0/global_symlinks.py
View File

30
build/phase0/prepare_early_subdirectories.py Normal file
View File

@@ -0,0 +1,30 @@
# SPDX-FileCopyrightText: Free Software Foundation Europe e.V. <https://fsfe.org>
#
# SPDX-License-Identifier: GPL-3.0-or-later
import logging
import sys
from pathlib import Path
logger = logging.getLogger(__name__)
def prepare_early_subdirectories(source_dir: Path, processes: int) -> None:
"""
Find any early subdir scripts in subdirectories and run them
"""
logger.info("Preparing Early Subdirectories")
for subdir_path in map(
lambda path: path.parent, source_dir.glob("**/early_subdir.py")
):
logger.info(f"Preparing early subdirectory {subdir_path}")
sys.path.append(str(subdir_path.resolve()))
import early_subdir
early_subdir.run(processes, subdir_path)
# Remove its path from where things can be imported
sys.path.remove(str(subdir_path.resolve()))
# Remove it from loaded modules
sys.modules.pop("early_subdir")
# prevent us from accessing it again
del early_subdir

15
build/phase1/index_website.py
View File

@@ -68,7 +68,9 @@ def _process_file(file: Path, stopwords: set[str]) -> dict:
}
def index_websites(languages: list[str], pool: multiprocessing.Pool) -> None:
def index_websites(
source_dir: Path, languages: list[str], pool: multiprocessing.Pool
) -> None:
"""
Generate a search index for all sites that have a search/search.js file
"""
@@ -78,11 +80,8 @@ def index_websites(languages: list[str], pool: multiprocessing.Pool) -> None:
nltk.data.path = [nltkdir] + nltk.data.path
nltk.download("stopwords", download_dir=nltkdir, quiet=True)
# Iterate over sites
for site in filter(
lambda path: path.joinpath("search/search.js").exists(),
Path(".").glob("?*.??*"),
):
logger.debug(f"Indexing {site}")
if source_dir.joinpath("search/search.js").exists():
logger.debug(f"Indexing {source_dir}")
# Get all xhtml files in languages to be processed
# Create a list of tuples
@@ -109,13 +108,13 @@ def index_websites(languages: list[str], pool: multiprocessing.Pool) -> None:
),
filter(
lambda file: file.suffixes[0].removeprefix(".") in languages,
Path(site).glob("**/*.??.xhtml"),
source_dir.glob("**/*.??.xhtml"),
),
)
articles = pool.starmap(_process_file, files_with_stopwords)
update_if_changed(
Path(f"{site}/search/index.js"),
source_dir.joinpath("search/index.js"),
"var pages = " + json.dumps(articles, ensure_ascii=False),
)

10
build/phase1/prepare_subdirectories.py
View File

@@ -9,14 +9,14 @@ from pathlib import Path
logger = logging.getLogger(__name__)
def prepare_subdirectories(languages: list[str], processes: int) -> None:
def prepare_subdirectories(
source_dir: Path, languages: list[str], processes: int
) -> None:
"""
Find any makefiles in subdirectories and run them
Find any subdir scripts in subdirectories and run them
"""
logger.info("Preparing Subdirectories")
for subdir_path in map(
lambda path: path.parent, Path("").glob("?*.?*/**/subdir.py")
):
for subdir_path in map(lambda path: path.parent, source_dir.glob("**/subdir.py")):
logger.info(f"Preparing subdirectory {subdir_path}")
sys.path.append(str(subdir_path.resolve()))
import subdir

40
build/phase1/run.py
View File

@@ -12,8 +12,8 @@
# -----------------------------------------------------------------------------
import logging
import multiprocessing
from pathlib import Path
from .global_symlinks import global_symlinks
from .index_website import index_websites
from .prepare_subdirectories import prepare_subdirectories
from .update_css import update_css
@@ -26,7 +26,12 @@ from .update_xmllists import update_xmllists
logger = logging.getLogger(__name__)
def phase1_run(languages: list[str], processes: int, pool: multiprocessing.Pool):
def phase1_run(
source_dir: Path,
languages: list[str] or None,
processes: int,
pool: multiprocessing.Pool,
):
"""
Run all the necessary sub functions for phase1.
"""
@@ -39,14 +44,16 @@ def phase1_run(languages: list[str], processes: int, pool: multiprocessing.Pool)
# This step runs a Python tool that creates an index of all news and
# articles. It extracts titles, teaser, tags, dates and potentially more.
# The result will be fed into a JS file.
index_websites(languages, pool)
index_websites(source_dir, languages, pool)
# -----------------------------------------------------------------------------
# Update CSS files
# -----------------------------------------------------------------------------
# This step recompiles the less files into the final CSS files to be
# distributed to the web server.
update_css()
update_css(
source_dir,
)
# -----------------------------------------------------------------------------
# Update XSL stylesheets
# -----------------------------------------------------------------------------
@@ -60,25 +67,12 @@ def phase1_run(languages: list[str], processes: int, pool: multiprocessing.Pool)
# and events directories, the XSL files, if updated, will be copied for the
# per-year archives.
update_stylesheets(pool)
update_stylesheets(source_dir, pool)
# -----------------------------------------------------------------------------
# Dive into subdirectories
# -----------------------------------------------------------------------------
# Find any makefiles in subdirectories and run them
prepare_subdirectories(languages, processes)
# -----------------------------------------------------------------------------
# Create XML symlinks
# -----------------------------------------------------------------------------
# After this step, the following symlinks will exist:
# * global/data/texts/.texts.<lang>.xml for each language
# * global/data/topbanner/.topbanner.<lang>.xml for each language
# Each of these symlinks will point to the corresponding file without a dot at
# the beginning of the filename, if present, and to the English version
# otherwise. This symlinks make sure that phase 2 can easily use the right file
# for each language, also as a prerequisite in the Makefile.
global_symlinks(languages, pool)
prepare_subdirectories(source_dir, languages, processes)
# -----------------------------------------------------------------------------
# Create XSL symlinks
@@ -90,14 +84,14 @@ def phase1_run(languages: list[str], processes: int, pool: multiprocessing.Pool)
# determine which XSL script should be used to build a HTML page from a source
# file.
update_defaultxsls(pool)
update_defaultxsls(source_dir, pool)
# -----------------------------------------------------------------------------
# Update local menus
# -----------------------------------------------------------------------------
# After this step, all .localmenu.??.xml files will be up to date.
update_localmenus(languages, pool)
update_localmenus(source_dir, languages, pool)
# -----------------------------------------------------------------------------
# Update tags
# -----------------------------------------------------------------------------
@@ -108,7 +102,7 @@ def phase1_run(languages: list[str], processes: int, pool: multiprocessing.Pool)
# in phase 2 are built into pages listing all news items and events for a
# tag.
# * tags/.tags.??.xml with a list of the tags used.
update_tags(languages, pool)
update_tags(source_dir, languages, pool)
# -----------------------------------------------------------------------------
# Update XML filelists
# -----------------------------------------------------------------------------
@@ -119,4 +113,4 @@ def phase1_run(languages: list[str], processes: int, pool: multiprocessing.Pool)
# correct XML files when generating the HTML pages. It is taken care that
# these files are only updated whenever their content actually changes, so
# they can serve as a prerequisite in the phase 2 Makefile.
update_xmllists(languages, pool)
update_xmllists(source_dir, languages, pool)

7
build/phase1/update_css.py
View File

@@ -12,14 +12,17 @@ from build.lib.misc import run_command, update_if_changed
logger = logging.getLogger(__name__)
def update_css() -> None:
def update_css(
source_dir: Path,
) -> None:
"""
If any less files have been changed, update the css.
Compile less found at website/look/(fsfe.less|valentine.less)
Then minify it, and place it in the expected location for the build process.
"""
logger.info("Updating css")
for dir in Path("").glob("?*.?*/look"):
dir = source_dir.joinpath("look")
if dir.exists():
for name in ["fsfe", "valentine"]:
if dir.joinpath(name + ".less").exists() and (
not dir.joinpath(name + ".min.css").exists()

6
build/phase1/update_defaultxsls.py
View File

@@ -22,7 +22,7 @@ def _do_symlinking(directory: Path) -> None:
)
def update_defaultxsls(pool: multiprocessing.Pool) -> None:
def update_defaultxsls(source_dir: Path, pool: multiprocessing.Pool) -> None:
"""
Place a .default.xsl into each directory containing source files for
HTML pages (*.xhtml). These .default.xsl are symlinks to the first
@@ -33,9 +33,7 @@ def update_defaultxsls(pool: multiprocessing.Pool) -> None:
logger.info("Updating default xsl's")
# Get a set of all directories containing .xhtml source files
directories = set(
map(lambda path: path.parent, Path(".").glob("*?.?*/**/*.*.xhtml"))
)
directories = set(map(lambda path: path.parent, source_dir.glob("**/*.*.xhtml")))
# Do all directories asynchronously
pool.map(_do_symlinking, directories)

6
build/phase1/update_localmenus.py
View File

@@ -77,7 +77,9 @@ def _write_localmenus(
)
def update_localmenus(languages: list[str], pool: multiprocessing.Pool) -> None:
def update_localmenus(
source_dir: Path, languages: list[str], pool: multiprocessing.Pool
) -> None:
"""
Update all the .localmenu.*.xml files containing the local menus.
"""
@@ -86,7 +88,7 @@ def update_localmenus(languages: list[str], pool: multiprocessing.Pool) -> None:
files_by_dir = {}
for file in filter(
lambda path: "-template" not in str(path),
Path(".").glob("*?.?*/**/*.??.xhtml"),
source_dir.glob("**/*.??.xhtml"),
):
xslt_root = etree.parse(file)
if xslt_root.xpath("//localmenu"):

4
build/phase1/update_stylesheets.py
View File

@@ -30,7 +30,7 @@ def _update_sheet(file: Path) -> None:
touch_if_newer_dep(file, imports)
def update_stylesheets(pool: multiprocessing.Pool) -> None:
def update_stylesheets(source_dir: Path, pool: multiprocessing.Pool) -> None:
"""
This script is called from the phase 1 Makefile and touches all XSL files
which depend on another XSL file that has changed since the last build run.
@@ -44,6 +44,6 @@ def update_stylesheets(pool: multiprocessing.Pool) -> None:
_update_sheet,
filter(
lambda file: re.match(banned, str(file)) is None,
Path(".").glob("**/*.xsl"),
source_dir.glob("**/*.xsl"),
),
)

29
build/phase1/update_tags.py
View File

@@ -25,9 +25,9 @@ def _update_tag_pages(site: Path, tag: str, languages: list[str]) -> None:
Update the xhtml pages and xmllists for a given tag
"""
for lang in languages:
tagfile_source = Path(f"{site}/tags/tagged.{lang}.xhtml")
tagfile_source = site.joinpath(f"tags/tagged.{lang}.xhtml")
if tagfile_source.exists():
taggedfile = Path(f"{site}/tags/tagged-{tag}.{lang}.xhtml")
taggedfile = site.joinpath(f"tags/tagged-{tag}.{lang}.xhtml")
content = tagfile_source.read_text().replace("XXX_TAGNAME_XXX", tag)
update_if_changed(taggedfile, content)
@@ -65,12 +65,14 @@ def _update_tag_sets(
page, "tag", section=section, key=tag, count=str(count)
).text = label
update_if_changed(
Path(f"{site}/tags/.tags.{lang}.xml"),
site.joinpath(f"tags/.tags.{lang}.xml"),
etree.tostring(page, encoding="utf-8").decode("utf-8"),
)
def update_tags(languages: list[str], pool: multiprocessing.Pool) -> None:
def update_tags(
source_dir: Path, languages: list[str], pool: multiprocessing.Pool
) -> None:
"""
Update Tag pages, xmllists and xmls
@@ -89,20 +91,17 @@ def update_tags(languages: list[str], pool: multiprocessing.Pool) -> None:
When a tag has been removed from the last XML file where it has been used,
the tagged-* are correctly deleted.
"""
for site in filter(
lambda path: path.joinpath("tags").exists(),
Path(".").glob("?*.??*"),
):
logger.info(f"Updating tags for {site}")
if source_dir.joinpath("tags").exists():
logger.info(f"Updating tags for {source_dir}")
# Create a complete and current map of which tag is used in which files
files_by_tag = {}
tags_by_lang = {}
# Fill out files_by_tag and tags_by_lang
for file in filter(
lambda file:
# Not in tags dir of a site
site.joinpath("tags") not in file.parents,
site.glob("**/*.xml"),
# Not in tags dir of a source_dir
source_dir.joinpath("tags") not in file.parents,
source_dir.glob("**/*.xml"),
):
for tag in etree.parse(file).xpath("//tag"):
# Get the key attribute, and filter out some invalid chars
@@ -141,7 +140,7 @@ def update_tags(languages: list[str], pool: multiprocessing.Pool) -> None:
logger.debug("Updating tag pages")
pool.starmap(
_update_tag_pages,
map(lambda tag: (site, tag, languages), files_by_tag.keys()),
map(lambda tag: (source_dir, tag, languages), files_by_tag.keys()),
)
logger.debug("Updating tag lists")
@@ -149,7 +148,7 @@ def update_tags(languages: list[str], pool: multiprocessing.Pool) -> None:
update_if_changed,
map(
lambda tag: (
Path(f"{site}/tags/.tagged-{tag}.xmllist"),
Path(f"{source_dir}/tags/.tagged-{tag}.xmllist"),
("\n".join(map(lambda file: str(file), files_by_tag[tag])) + "\n"),
),
files_by_tag.keys(),
@@ -173,7 +172,7 @@ def update_tags(languages: list[str], pool: multiprocessing.Pool) -> None:
pool.starmap(
_update_tag_sets,
map(
lambda lang: (site, lang, filecount, files_by_tag, tags_by_lang),
lambda lang: (source_dir, lang, filecount, files_by_tag, tags_by_lang),
filter(lambda lang: lang in languages, tags_by_lang.keys()),
),
)

83
build/phase1/update_xmllists.py
View File

@@ -76,49 +76,48 @@ def _update_for_base(
)
def _update_module_xmllists(languages: list[str], pool: multiprocessing.Pool) -> None:
def _update_module_xmllists(
source_dir: Path, languages: list[str], pool: multiprocessing.Pool
) -> None:
"""
Update .xmllist files for .sources and .xhtml containing <module>s
"""
logger.info("Updating XML lists")
# Store current dir
for site in filter(lambda path: path.is_dir(), Path(".").glob("?*.??*")):
logger.info(f"Updating xmllists for {site}")
# Get all the bases and stuff before multithreading the update bit
all_xml = set(
map(
lambda path: get_basepath(path),
filter(
lambda path: lang_from_filename(path) in languages,
list(site.glob("**/*.*.xml"))
+ list(Path("global/").glob("**/*.*.xml")),
),
)
# Get all the bases and stuff before multithreading the update bit
all_xml = set(
map(
lambda path: get_basepath(path),
filter(
lambda path: lang_from_filename(path) in languages,
list(source_dir.glob("**/*.*.xml"))
+ list(Path("global/").glob("**/*.*.xml")),
),
)
source_bases = set(
map(
lambda path: path.with_suffix(""),
site.glob("**/*.sources"),
)
)
source_bases = set(
map(
lambda path: path.with_suffix(""),
source_dir.glob("**/*.sources"),
)
module_bases = set(
map(
lambda path: get_basepath(path),
filter(
lambda path: lang_from_filename(path) in languages
and etree.parse(path).xpath("//module"),
site.glob("**/*.*.xhtml"),
),
)
)
all_bases = source_bases | module_bases
nextyear = str(datetime.datetime.today().year + 1)
thisyear = str(datetime.datetime.today().year)
lastyear = str(datetime.datetime.today().year - 1)
pool.starmap(
_update_for_base,
map(lambda base: (base, all_xml, nextyear, thisyear, lastyear), all_bases),
)
module_bases = set(
map(
lambda path: get_basepath(path),
filter(
lambda path: lang_from_filename(path) in languages
and etree.parse(path).xpath("//module"),
source_dir.glob("**/*.*.xhtml"),
),
)
)
all_bases = source_bases | module_bases
nextyear = str(datetime.datetime.today().year + 1)
thisyear = str(datetime.datetime.today().year)
lastyear = str(datetime.datetime.today().year - 1)
pool.starmap(
_update_for_base,
map(lambda base: (base, all_xml, nextyear, thisyear, lastyear), all_bases),
)
def _check_xmllist_deps(file: Path) -> None:
@@ -134,16 +133,18 @@ def _check_xmllist_deps(file: Path) -> None:
def _touch_xmllists_with_updated_deps(
languages: list[str], pool: multiprocessing.Pool
source_dir: Path, languages: list[str], pool: multiprocessing.Pool
) -> None:
"""
Touch all .xmllist files where one of the contained files has changed
"""
logger.info("Checking contents of XML lists")
pool.map(_check_xmllist_deps, Path("").glob("./**/.*.xmllist"))
pool.map(_check_xmllist_deps, source_dir.glob("**/.*.xmllist"))
def update_xmllists(languages: list[str], pool: multiprocessing.Pool) -> None:
def update_xmllists(
source_dir: Path, languages: list[str], pool: multiprocessing.Pool
) -> None:
"""
Update XML filelists (*.xmllist)
@@ -161,5 +162,5 @@ def update_xmllists(languages: list[str], pool: multiprocessing.Pool) -> None:
When a tag has been removed from the last XML file where it has been used,
the tagged-* are correctly deleted.
"""
_update_module_xmllists(languages, pool)
_touch_xmllists_with_updated_deps(languages, pool)
_update_module_xmllists(source_dir, languages, pool)
_touch_xmllists_with_updated_deps(source_dir, languages, pool)

12
build/phase2/copy_files.py
View File

@@ -10,8 +10,8 @@ from pathlib import Path
logger = logging.getLogger(__name__)
def _copy_file(target: Path, source_file: Path) -> None:
target_file = target.joinpath(source_file)
def _copy_file(target: Path, source_dir: Path, source_file: Path) -> None:
target_file = target.joinpath(source_file.relative_to(source_dir))
if (
not target_file.exists()
or source_file.stat().st_mtime > target_file.stat().st_mtime
@@ -23,7 +23,7 @@ def _copy_file(target: Path, source_file: Path) -> None:
shutil.copymode(source_file, target_file)
def copy_files(pool: multiprocessing.Pool, target: Path) -> None:
def copy_files(source_dir: Path, pool: multiprocessing.Pool, target: Path) -> None:
"""
Copy images, docments etc
"""
@@ -31,7 +31,7 @@ def copy_files(pool: multiprocessing.Pool, target: Path) -> None:
pool.starmap(
_copy_file,
map(
lambda file: (target, file),
lambda file: (target, source_dir, file),
list(
filter(
lambda path: path.is_file()
@@ -50,10 +50,10 @@ def copy_files(pool: multiprocessing.Pool, target: Path) -> None:
".pyc",
]
and path.name not in ["Makefile"],
Path("").glob("*?.?*/**/*"),
source_dir.glob("**/*"),
)
)
# Special case hard code pass over orde items xml required by cgi script
+ list(Path("").glob("*?.?*/order/data/items.en.xml")),
+ list(source_dir.glob("order/data/items.en.xml")),
),
)

4
build/phase2/create_index_symlinks.py
View File

@@ -19,7 +19,9 @@ def _do_symlinking(target: Path) -> None:
source.symlink_to(target.relative_to(source.parent))
def create_index_symlinks(pool: multiprocessing.Pool, target: Path) -> None:
def create_index_symlinks(
source_dir: Path, pool: multiprocessing.Pool, target: Path
) -> None:
"""
Create index.* symlinks
"""

4
build/phase2/create_language_symlinks.py
View File

@@ -15,7 +15,9 @@ def _do_symlinking(target: Path) -> None:
source.symlink_to(target.relative_to(source.parent))
def create_language_symlinks(pool: multiprocessing.Pool, target: Path) -> None:
def create_language_symlinks(
source_dir: Path, pool: multiprocessing.Pool, target: Path
) -> None:
"""
Create symlinks from file.<lang>.html to file.html.<lang>
"""

28
build/phase2/process_files.py
View File

@@ -48,11 +48,15 @@ def _run_process(
target_file.write_text(result)
def _process_dir(languages: list[str], target: Path, dir: Path) -> None:
def _process_dir(
source_dir: Path, languages: list[str], target: Path, dir: Path
) -> None:
for basename in set(map(lambda path: path.with_suffix(""), dir.glob("*.??.xhtml"))):
for lang in languages:
source_file = basename.with_suffix(f".{lang}.xhtml")
target_file = target.joinpath(source_file).with_suffix(".html")
target_file = target.joinpath(
source_file.relative_to(source_dir)
).with_suffix(".html")
processor = (
basename.with_suffix(".xsl")
if basename.with_suffix(".xsl").exists()
@@ -61,9 +65,11 @@ def _process_dir(languages: list[str], target: Path, dir: Path) -> None:
_run_process(target_file, processor, source_file, basename, lang)
def _process_stylesheet(languages: list[str], target: Path, processor: Path) -> None:
def _process_stylesheet(
source_dir: Path, languages: list[str], target: Path, processor: Path
) -> None:
basename = get_basepath(processor)
destination_base = target.joinpath(basename)
destination_base = target.joinpath(basename.relative_to(source_dir))
for lang in languages:
target_file = destination_base.with_suffix(
f".{lang}{processor.with_suffix('').suffix}"
@@ -73,7 +79,7 @@ def _process_stylesheet(languages: list[str], target: Path, processor: Path) ->
def process_files(
languages: list[str], pool: multiprocessing.Pool, target: Path
source_dir: Path, languages: list[str], pool: multiprocessing.Pool, target: Path
) -> None:
"""
Build .html, .rss and .ics files from .xhtml sources
@@ -84,23 +90,23 @@ def process_files(
pool.starmap(
_process_dir,
map(
lambda dir: (languages, target, dir),
set(map(lambda path: path.parent, Path("").glob("*?.?*/**/*.*.xhtml"))),
lambda dir: (source_dir, languages, target, dir),
set(map(lambda path: path.parent, source_dir.glob("**/*.*.xhtml"))),
),
)
logger.info("Processing rss files")
pool.starmap(
_process_stylesheet,
map(
lambda processor: (languages, target, processor),
Path("").glob("*?.?*/**/*.rss.xsl"),
lambda processor: (source_dir, languages, target, processor),
source_dir.glob("**/*.rss.xsl"),
),
)
logger.info("Processing ics files")
pool.starmap(
_process_stylesheet,
map(
lambda processor: (languages, target, processor),
Path("").glob("*?.?*/**/*.ics.xsl"),
lambda processor: (source_dir, languages, target, processor),
source_dir.glob("**/*.ics.xsl"),
),
)

12
build/phase2/run.py
View File

@@ -17,12 +17,14 @@ from .process_files import process_files
logger = logging.getLogger(__name__)
def phase2_run(languages: list[str], pool: multiprocessing.Pool, target: Path):
def phase2_run(
source_dir: Path, languages: list[str], pool: multiprocessing.Pool, target: Path
):
"""
Run all the necessary sub functions for phase2.
"""
logger.info("Starting Phase 2 - Generating output")
process_files(languages, pool, target)
create_index_symlinks(pool, target)
create_language_symlinks(pool, target)
copy_files(pool, target)
process_files(source_dir, languages, pool, target)
create_index_symlinks(source_dir, pool, target)
create_language_symlinks(source_dir, pool, target)
copy_files(source_dir, pool, target)

4
docs/overview.md
View File

@@ -29,6 +29,10 @@ For details on the phases and exact implementation please examine the codebase
The build process can be conceptually divided into four phases: Phases 0-3
Phases 0 and 3 contain steps that may or not be performed based on passed arguments. They also act over all sites at once.
Phases 1 and 2 always run all steps inside them, and are run on a per-site basis.
### [phase0](./phase0.md)
### [phase1](./phase1.md)

1
global/languages/eo Normal file
View File

@@ -0,0 +1 @@
Esperanto

1
global/languages/gl Normal file
View File

@@ -0,0 +1 @@
Galego

1
global/languages/he Normal file
View File

@@ -0,0 +1 @@
עברית

1
global/languages/ku Normal file
View File

@@ -0,0 +1 @@
کوردی

1
status.fsfe.org/.gitignore vendored
View File

@@ -1,2 +1,3 @@
## Status dir stuff
*/data*/*
filler/*.xhtml

9
status.fsfe.org/filler/README.md Normal file
View File

@@ -0,0 +1,9 @@
# README
The languages to build a site in for the fsfe webpages are determined by what languages are available for that site, along with some heuristics about percentage of files translated, etc.
The status.fsfe.org page is a little different, as it needs to be built in languages that it has no proper files in.
It was decided that the best way to resolve this quandary was to just add some near empty files to the folder that would then successfully convince the build process to build it in all relevant languages.
This is accomplised programatically by means of the early_subdir.py script

6
status.fsfe.org/filler/__init__.py Normal file
View File

@@ -0,0 +1,6 @@
# SPDX-FileCopyrightText: Free Software Foundation Europe e.V. <https://fsfe.org>
#
# SPDX-License-Identifier: GPL-3.0-or-later
# __init__.py is a special Python file that allows a directory to become
# a Python package so it can be accessed using the 'import' statement.

54
status.fsfe.org/filler/early_subdir.py Normal file
View File

@@ -0,0 +1,54 @@
# SPDX-FileCopyrightText: Free Software Foundation Europe e.V. <https://fsfe.org>
#
# SPDX-License-Identifier: GPL-3.0-or-later
import logging
import multiprocessing
from pathlib import Path
import lxml.etree as etree
from build.lib.misc import (
update_if_changed,
)
logger = logging.getLogger(__name__)
def _create_index(
target_file: Path,
):
# Create the root element
page = etree.Element("html")
# Add the subelements
version = etree.SubElement(page, "version")
version.text = "1"
head = etree.SubElement(page, "head")
title = etree.SubElement(head, "title")
title.text = "filler"
head = etree.SubElement(page, "body")
result_str = etree.tostring(page, xml_declaration=True, encoding="utf-8").decode(
"utf-8"
)
update_if_changed(target_file, result_str)
def run(processes: int, working_dir: Path) -> None:
"""
Place filler indices to encourgae the site to ensure that status pages for all langs are build.
"""
with multiprocessing.Pool(processes) as pool:
pool.map(
_create_index,
map(
lambda path: (
working_dir.joinpath(
f"index.{path.name}.xhtml",
)
),
Path().glob("global/languages/*"),
),
)