Source files of fsfe.org, pdfreaders.org, freeyourandroid.org, ilovefs.org, drm.info, and test.fsfe.org. Contribute: https://fsfe.org/contribute/web/
https://fsfe.org
You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
373 lines
17 KiB
373 lines
17 KiB
<?xml version="1.0" encoding="UTF-8"?> |
|
<html newsdate="2021-12-03"> |
|
<version>1</version> |
|
|
|
<head> |
|
<title>Infrastructure living the ideals of software freedom</title> |
|
</head> |
|
|
|
<body> |
|
|
|
<h1>Infrastructure living the ideals of software freedom</h1> |
|
|
|
<p> |
|
Can organisations with limited resources be digitally sovereign and still |
|
provide modern services? It is not trivial, but the FSFE proves it's possible. |
|
Take a deep dive with us into our infrastructure to learn how we run all the |
|
different services within the FSFE and cope with numerous challenges. A story |
|
non only for techies. |
|
</p> |
|
|
|
<p> |
|
Charity, non-profit organisations run into limits every day: personnel, |
|
budget, time, and the pressing question how to use donations most efficiently. |
|
When it comes to technical infrastructure, many organisations unfortunately |
|
decide to outsource and use proprietary, non-free services. By this, they give |
|
up software freedom and thereby digital sovereignty and independence. |
|
</p> |
|
|
|
<p> |
|
Since its founding more than 20 years ago, the FSFE has been pursuing the |
|
opposite way. Right from the start, we have relied on Free Software although |
|
it sometimes meant not being able to use and offer trendy services. Also, |
|
given the limited resources, we constantly have to choose between useful |
|
features and maintainability. |
|
</p> |
|
|
|
<figure> |
|
<img |
|
src="https://pics.fsfe.org/uploads/medium/a9669e241527f1769e2fe67418a77f0d.jpg" |
|
alt="The four freedoms and Free Software services on top" /> |
|
</figure> |
|
|
|
<p> |
|
And still, neither is our infrastructure perfect nor is it 1:1 transferable to |
|
other organisations. But we think it's important that organisations exchange |
|
their experiences and learning, especially when it's about something as |
|
important as software freedom. |
|
</p> |
|
|
|
<p> |
|
Therefore, let us take you on a journey through our infrastructure and its |
|
principles, from shiny user interfaces of our <a |
|
href="#services">services</a>, crossing the <a |
|
href="#virtualisation">virtualisation methods</a> and <a |
|
href="#monitoring">monitoring</a>, down to the <a href="#servers">bare metal |
|
servers</a> they are running on. This is a story not only for techies, but for |
|
everyone interested in making or keeping an NGO independent from proprietary |
|
service providers. |
|
</p> |
|
|
|
<h2 id="team">One team and its principles</h2> |
|
|
|
<p> |
|
All of the FSFE's infrastructure is managed by the <a |
|
href="https://wiki.fsfe.org/Teams/System-Hackers">System Hackers</a>. Only a |
|
few years ago, in a time of larger technical debt and restructuring, the team |
|
was only three people. Fortunately, since then we have experienced a steady |
|
growth of membership. Today, it is a healthy team consisting of 9 active |
|
volunteers, complemented by two staff members who dedicate some of their |
|
working time to the team's tasks. |
|
</p> |
|
|
|
<figure> |
|
<img |
|
src="https://pics.fsfe.org/uploads/medium/04ff9b7b037924fa522e2a1e47ac3a94.JPG" |
|
alt="A group picture of the System Hackers in 2020" /> |
|
<figcaption> |
|
Picture from the 2020 meeting of the System Hackers in Lyon |
|
</figcaption> |
|
</figure> |
|
|
|
<p> |
|
With a team this size it was crucial to define the <a |
|
href="https://wiki.fsfe.org/Teams/System-Hackers/Principles">most basic |
|
principles of this team</a>. They form the basis of the System Hackers' goals: |
|
as much control as possible over services and servers by using 100% Free |
|
Software, internal and external transparency, and bearable complexity, and at |
|
the same time providing useful features for the various FSFE teams and the |
|
whole community. |
|
</p> |
|
|
|
<p> |
|
Aside from purely asynchronous work, emails, and chat, the System Hackers |
|
also met at least once per year in pre-pandemic times, and continue doing that |
|
in virtual form. During these meetings, we were able to tackle more complex |
|
decisions and technical changes efficiently, but also just enjoyed |
|
non-technical conversations and fun activities. The team is coordinated by us, |
|
the authors of this text, Albert and Max, who also maintain a large number of |
|
services and systems. |
|
</p> |
|
|
|
<h2 id="services">Services, services everywhere</h2> |
|
|
|
<p> |
|
The FSFE's infrastructure is very service-oriented. Volunteers and staff rely |
|
on basic functionality like sending and receiving email or exchanging files, |
|
but also on a website fed by a version control system, a wiki, or video chat |
|
systems. To give an example for the complex interconnection of different |
|
components, just drafting and releasing this news item involved at least 12 |
|
services that have to seamlessly work together. |
|
</p> |
|
|
|
<module id="banner-become-supporter" /> |
|
|
|
<p> |
|
The currently most crucial and used services contain Mailman for mailing |
|
lists, or Gitea as our Git version control system. To allow sharing and saving |
|
knowledge our <a href="https://wiki.fsfe.org/Teams/WikiCaretakers">Wiki |
|
Caretakers</a> maintain MoinMoin, while Björn takes care of Nextcloud to share |
|
files, coordinate tasks, and collaboratively edit documents. We run our own |
|
BigBlueButton and Jitsi instances for video conferencing, and XMPP/Jabber for |
|
text-based chats. Very recent service additions are Matrix as another chat |
|
service, initiated and maintained by Michael, and Peertube for hosting and |
|
sharing our videos on own infrastructure, made possible by Alvar. |
|
</p> |
|
|
|
<p> |
|
The list of services is <a href="https://wiki.fsfe.org/TechDocs/Services">much |
|
larger</a> and also contains fundamental systems like the Account Management |
|
System and Community Database developed by <a |
|
href="/news/2021/news-20210305-01.html">Reinhard</a>, our own DNS servers, or |
|
Drone as our CI/CD system that processes data from Gitea, checks them, and |
|
deploys them on other servers eventually. |
|
</p> |
|
|
|
<p> |
|
Needless to say, the team regularly receives requests to provide |
|
additional services. Here, the challenge is to make a careful selection |
|
based on available resources (computing, space, volunteer time) and |
|
estimated use for the organisation, and evaluating whether a solution is |
|
well-maintainable, not largely overlapping with existing services, and |
|
generally leaving a good impression. |
|
</p> |
|
|
|
<p> |
|
Sometimes that also means we have to test solutions in practice over a |
|
longer period of time. Real-time editing of documents is a good example |
|
for that, for which we currently have multiple possibilities available. |
|
Longer documents are often edited as ODF files via Collabora Online |
|
attached to Nextcloud, but some editors prefer Etherpad or use Git directly. |
|
All solutions have advantages and downsides, and finding a good path |
|
between having diverse options on one side, and tool overload on the other |
|
is anything but trivial. |
|
</p> |
|
|
|
<h2 id="virtualisation">Virtualisation and deployment</h2> |
|
|
|
<p> |
|
While the FSFE owns dedicated servers in actual racks (we will come to |
|
that later), all services run in some sort of virtualisation. In total |
|
there are 43 virtual machines distributed over different data centres |
|
at the time of writing this article. Some have a purely internal |
|
role, for instance being a gateway for other virtual machines or |
|
assisting web services with obtaining TLS certificates. |
|
</p> |
|
|
|
<figure class="no-border"> |
|
<a href="https://pics.fsfe.org/uploads/big/0b41e57a4d5163ef911cfd913b9af640.png"> |
|
<img src="https://pics.fsfe.org/uploads/medium/0b41e57a4d5163ef911cfd913b9af640.png" alt="All virtual servers of the FSFE"/> |
|
</a> |
|
<figcaption> |
|
An overview of the currently running virtual servers of the FSFE, and their |
|
distribution over the different clusters. |
|
</figcaption> |
|
</figure> |
|
|
|
<p> |
|
Some others, in turn, themselves host a number of diverse services. Since 2017 |
|
we have been using <a href="https://git.fsfe.org/explore/repos?q=docker&topic=1">Docker</a> as a |
|
container engine. That has not been an easy choice since the technology adds |
|
a lot of complexity and sometimes requires stunning workarounds to be operated |
|
in a secure fashion. On the other hand, especially for smaller services or |
|
larger self-coded tools, it is great to spin them up quickly, test and deploy |
|
them via our Continuous Integration system, provide a more or less uniform |
|
local development environment, and maintain an (admittedly limited) |
|
reproducibility of configurations. |
|
</p> |
|
|
|
<p> |
|
We are regularly evaluating alternative engines that provide a more |
|
seamless rootless mode, are still compatible with our CI/CD system |
|
(Drone) and reverse proxy, and ideally do not require a major rewrite of |
|
existing deployment code. Also, here again, it is important for the |
|
System Hackers that at least two members understand a high-priority |
|
technology in depth. |
|
</p> |
|
|
|
<p> |
|
To bootstrap virtual machines and deploy non-container services in a |
|
reproducible way, we <a href="https://git.fsfe.org/explore/repos?q=ansible&topic=1">rely a lot on |
|
Ansible</a>, a provisioning and configuration management tool. While Ansible |
|
deployments may also have their shortcomings, we appreciate the |
|
infrastructure-as-code approach that can be executed from any computer and |
|
does not require a central server. Meanwhile, only the minority of services |
|
are deployed via neither Ansible nor Docker which makes understanding existing |
|
infrastructure, onboarding new volunteers, and documenting changes much |
|
easier. |
|
</p> |
|
|
|
<h2 id="monitoring">The all-seeing eye and uniformity</h2> |
|
|
|
<p> |
|
Dozens of servers and services: how do you know if there is a problem with one |
|
of them? We have to admit that up until two years ago we sometimes only learnt |
|
about a crashed server via an email from a random friendly community member. |
|
Now a <a href="https://git.fsfe.org/fsfe-system-hackers/monitoring">monitoring system |
|
based on Icinga2</a> takes care of this. Currently 50 hosts and 690 services |
|
are continuously checked, for example for pending upgrades of the operating |
|
system, systemd services, failed backups, disk space, or TLS certificate |
|
validity. |
|
</p> |
|
|
|
<p> |
|
This is eased by other parts of our strategy: except 4 virtual machines, all |
|
run on Debian GNU/Linux. After initial creation, a new machine will experience |
|
a <a href="https://git.fsfe.org/fsfe-system-hackers/baseline">baseline</a> |
|
setup taking care of fundamental security settings, monitoring, backup, and |
|
automatic upgrades. Thanks to this, maintenance of a server is mostly limited |
|
to the service it is running, and other improvements can be applied to a large |
|
number of hosts automatically within a few minutes. |
|
</p> |
|
|
|
<p> |
|
To further ease management and maintenance, we sometimes write our own |
|
tools. For instance |
|
<a href="https://git.fsfe.org/fsfe-system-hackers/ssh-key-distributor">ssh-key-distributor</a> |
|
provides a simple interface and deployment method to manage and |
|
document access via SSH on our servers. Or |
|
<a href="https://git.fsfe.org/fsfe-system-hackers/docker-utils">docker-utils</a> |
|
which is tailored to our Docker infrastructure and takes care of analysing and |
|
upgrading Docker images and containers. Both tools have been created by our |
|
working student Linus. You can find more tools and generally most |
|
Ansible/Docker deployment code in the System Hackers' |
|
<a href="https://git.fsfe.org/fsfe-system-hackers">Git organisation</a>. |
|
</p> |
|
|
|
<h2 id="servers">Bare metal servers</h2> |
|
|
|
<p> |
|
Unlike the current trend of the IT industry, we are proud to run the |
|
vast majority of services on our own physical servers. This guarantees |
|
the most sovereignty, data security by full disk encryption, and |
|
technological independence for us. To increase resilience, we bundle |
|
three servers each into a Proxmox cluster with Ceph storage, so if one |
|
physical server crashes a virtual machine is just moved to one of the |
|
two other servers in the cluster. |
|
</p> |
|
|
|
<p> |
|
As an additional safety net, the three clusters and a solo machine are spread |
|
over four different data centres which <a href="/donate/hardware.html">kindly |
|
donate the colocation</a> to us. |
|
</p> |
|
|
|
<p> |
|
However, this is only possible thanks to a fortunate combination of |
|
conditions. First of all, we are lucky to have Albert with us who has a lot of |
|
experience with, among various other areas, Proxmox, Ceph, and the depths of |
|
networking. Then, we have the kind support of the data centres providing |
|
colocation as well as hardware donors, and the <fsfe-cd-donate-link>FSFE |
|
supporters</fsfe-cd-donate-link> that contribute financially. And we also |
|
profit from a more or less identical setup on all four sites which makes |
|
maintenance a bit easier. But still, we are considering giving up the cluster |
|
containing the oldest servers as well as the solo physical machine to reduce |
|
work and complexity. |
|
</p> |
|
|
|
<h2 id="challenges">Challenges behind and ahead</h2> |
|
|
|
<p> |
|
Over the course of the years, we managed to overcome many challenges: |
|
technical debt, antique software that blocked operating system upgrades, |
|
lack of hardware resources, fatal crashes of single points of failures, |
|
and hard technical decisions about how to develop the infrastructure |
|
further. In one of these hard times, our back-then intern and since then System Hacker |
|
<a href="/about/people/interviews/lequertier.html">Vincent</a> |
|
played an important role and helped set the foundation for the good state we |
|
are in today. Despite all preparation and evaluation, we have made many |
|
mistakes, but most importantly we have learnt a lot by them. |
|
</p> |
|
|
|
<module id="banner-subscribe" /> |
|
|
|
<p> |
|
Facing us right now are new hurdles. For example, with the high amount of |
|
servers, we can no longer give every virtual machine a dedicated IPv4 address. |
|
Unfortunately, many technologies and internet services, even large proprietary |
|
and supposedly professional companies, still do not support the more modern, |
|
future-proof, and already 20 years old IPv6 protocol. This requires us to |
|
fiddle around with a few hacks like <a |
|
href="https://git.fsfe.org/fsfe-system-hackers/ip-proxy/">reverse proxies</a>, |
|
<a href="https://git.fsfe.org/fsfe-system-hackers/docker2caddy/">container |
|
discovery services</a>, |
|
<a href="https://git.fsfe.org/fsfe-system-hackers/fsfe-gw">NAT</a>, and <a |
|
href="https://git.fsfe.org/fsfe-system-hackers/innernet-playbook">VPNs</a>. |
|
</p> |
|
|
|
<p> |
|
Another interesting decision ahead is the one of communication channels. While |
|
plain-text email and, since 2004, XMPP/Jabber |
|
have formed the de-facto standard within the FSFE, many people meanwhile |
|
prefer <a href="https://wiki.fsfe.org/TechDocs/Matrix">Matrix</a>, |
|
<a href="https://community.fsfe.org">Discourse</a>, or still the traditional |
|
IRC. While we see advantages and disadvantages for all of them, we also want |
|
to avoid fragmentation of our community. This is not a pure technical |
|
question, but a great example of the need for good inter-team communication |
|
and decision-making. |
|
</p> |
|
|
|
<p> |
|
Last but not least, we have a few technologies that are head scratchers |
|
for us. Let's take Mailman 2 as an example that has powered our 116 |
|
mailing lists for years. Unfortunately, the project is no longer |
|
actively developed, and the successor and its alternatives all have |
|
serious downsides. Here, we need to conduct a careful evaluation and many |
|
tests, and eventually find the best solution in the mass of imperfect |
|
options. |
|
</p> |
|
|
|
<p> |
|
With all that said, we would like to express our thanks to the many Free |
|
Software projects and their developers out there. They grant us the ability to |
|
choose from different solutions, they form the basis of our infrastructure, |
|
and they provide astonishing solutions that make our lives easier every day. |
|
It is a pleasure to be part of this large community. |
|
</p> |
|
|
|
<h2>Why all of this?</h2> |
|
|
|
<p> |
|
As you can see, the technical infrastructure of our organisation is anything |
|
but boring or simple. This is not only due to the size of the FSFE and its |
|
community, but also due to our own high standards: living software freedom in |
|
practice and maintaining as much technical independence and sovereignty as |
|
possible. At the same time, we care for technical accessibility to also allow |
|
non-technical contributors to interact with our services. That sometimes |
|
requires extra work and tools, but we are convinced that it is worth it. |
|
</p> |
|
|
|
<p> |
|
All of this depends on the dedication and long-time commitment of many people, |
|
and is fed by its productive use and the feedback of the whole FSFE community. |
|
And it is only possible thanks to the FSFE's supporters who enable us to |
|
invest resources in a fully Free Software infrastructure. If you share this |
|
ideal, <fsfe-cd-donate-link>please consider a donation</fsfe-cd-donate-link>. |
|
</p> |
|
|
|
</body> |
|
|
|
<tags> |
|
<tag key="front-page"/> |
|
<tag key="internal">FSFE Internal</tag> |
|
<tag key="tech-teams">Technical Teams</tag> |
|
</tags> |
|
|
|
<author id="dengg" /> |
|
<author id="mehl" /> |
|
|
|
<discussion href="https://community.fsfe.org/t/775" /> |
|
|
|
<image url="https://pics.fsfe.org/uploads/medium/a9669e241527f1769e2fe67418a77f0d.jpg" alt="The four freedoms and Free Software services on top"/> |
|
|
|
</html>
|
|
|