Analysis of periodic stuck issue #167

New Issue

2026-05-12T11:11:45Z

max.mehl commented

2026-05-12 11:11:45 +00:00

Once again, especially newly registered repos were stuck in the init phase and never got checked. IMHO this is a serious issue, affecting the trustworthiness of the FSFE's service.

I spent some tokens on analyzing the issue in the hope that it will help you pinning down the issue. If you wish, I can also dive into fixing them, but that would require some synchronisation of efforts.

Root cause analysis — ranked by severity × probability

1. CRITICAL / HIGH PROBABILITY — Uninitialised repos are never re-enqueued on subsequent requests

Where: scheduler.py, specifically the schedule() method's decision tree.

The bug: When a repo already exists in the DB (repository is not None) but has never been successfully checked (last_access is None, hash is None), the only branch that would re-enqueue it is:

elif repository.hash != latest:

But repository.hash is None (never set) and latest is a real hash string. In Python, None != "abc123..." is True, so this does re-enqueue. That part is OK by accident.

However, if the first check failed (SSH error 255, timeout, empty output, or JSONDecodeError), the DB row already exists with hash=None, the task is completed and removed from the in-memory queue via done() at scheduler.py, but the DB is never updated. On the next request, repository.hash (None) != latest evaluates to True, so it should re-enqueue. This path does work — but only if someone actually visits /info/ or /status/ again.

The real problem: if nobody visits those pages again after the first failure, the repo stays uninitialised forever. There is no retry mechanism, no background sweep, no watchdog. This is the most likely root cause for repos stuck "after weeks" — the initial check failed silently, and nobody re-triggered it.

Fix direction: Add a lightweight periodic sweep (e.g. a background thread or timer) that queries the DB for rows where last_access IS NULL and re-enqueues them. This is the single highest-impact change.

2. HIGH / HIGH PROBABILITY — `TaskQueue` is a process-wide singleton with class-level mutable state shared across gunicorn workers via `fork()`

Where: task.py

_instance = None
__urls: set[str] = set()
__urls_lock: Lock = Lock()

These are class variables. TaskQueue uses a singleton pattern via __new__. With gunicorn --workers=4, gunicorn forks after create_app() is called. At fork time:

The _instance, __urls set, and __urls_lock are copied into each worker.
Each worker gets its own independent TaskQueue with its own __urls set.
Each worker also gets its own Scheduler with its own Runner threads.

This means:

A task enqueued in Worker A is invisible to Workers B, C, D.
If Worker A creates a DB row and enqueues a task, but the next request from the same user lands on Worker B, Worker B sees the DB row (repository is not None) but the task is not in Worker B's queue, so it falls through to the hash != latest check. This can cause duplicate concurrent checks of the same repo across workers.
If a Runner thread dies (see item 3), only that worker loses capacity — no other worker compensates.

But more critically for the bug: after a gunicorn worker is recycled (which gunicorn does periodically via max_requests or on crash), the in-memory queue is lost. Any tasks that were in-flight or queued are silently dropped. The DB row exists but was never updated, so last_access remains None.

Fix direction: Either run with --workers=1 --threads=N (trading parallelism), use --preload and ensure the scheduler is started post-fork, or move to a durable queue (Redis, database table) shared across workers.

3. HIGH / MEDIUM PROBABILITY — Unhandled exceptions in `Runner.run()` silently kill worker threads

Where: scheduler.py

The try/except in Runner.run() only catches:

subprocess.TimeoutExpired (line ~112)
JSONDecodeError (line ~139)

Any other exception — e.g. AttributeError if Repository.find(self.url) returns None in task.py, a database connection error, an OSError from SSH, or an encoding error — will propagate up, exit the while self.__running loop, and kill the thread permanently. There is no respawn logic.

With NB_RUNNER=6 threads per worker process, losing threads over weeks of uptime is very plausible. Once all 6 are dead, no more tasks are ever processed in that worker, even though they're enqueued. The queue grows silently. No monitoring or health check exists.

Fix direction: Wrap the entire loop body in a broad except Exception with logging, so a single bad task can't kill a thread. Optionally add a health-check endpoint that reports the number of alive runner threads.

4. MEDIUM / HIGH PROBABILITY — SSH failure (exit code 255) leaves repo permanently uninitialised with no retry

Where: scheduler.py

When SSH returns 255, the code logs a warning and explicitly skips the DB update. The task is then marked done() and removed from the queue. The repo stays with hash=None and last_access=None.

If the SSH failure is transient (network blip, worker restart, connection limit), this is fine — the next request re-enqueues. But if it's persistent (misconfigured key, firewall rule, DNS issue), every attempt will fail with 255 and be silently discarded, with no escalation, no retry backoff, and no alert beyond a log warning.

Over weeks, if the API worker is temporarily unreachable (maintenance, restart), every repo checked during that window gets silently dropped and must wait for another user request to retry.

Fix direction: Re-enqueue tasks that fail with SSH 255 after a delay, with a retry counter. Or at minimum, don't call done() so the task stays in the queue (though this risks infinite retry storms without backoff).

5. MEDIUM / MEDIUM PROBABILITY — Empty output from linting silently leaves repo uninitialised

Where: scheduler.py

When output is empty, the code logs a warning but still passes the empty string to task.update_db(output). json_loads("") raises JSONDecodeError, which is caught at line 139 and logged — but the DB is not updated. The task is marked done. Same outcome as SSH 255: repo stays stuck.

Fix direction: Skip update_db() when output is empty, and consider re-enqueuing.

6. MEDIUM / LOW-MEDIUM PROBABILITY — Case-sensitivity mismatch between `create()` and `find()`

Where: models.py vs models.py

Repository.create() stores the URL as-is (case-preserving). Repository.find() uses db.func.lower() for case-insensitive lookup. But is_initialised() at models.py uses filter_by(url=url), which is case-sensitive.

If a user registers git.example.com/Org/Repo but later visits /info/git.example.com/org/repo, then:

find() returns the row (case-insensitive match) → repository is not None
schedule() sees hash is None, re-enqueues with the lowercase URL
update_db() calls find(lowercase_url) → finds the row → updates it with the lowercase URL
But is_initialised(original_case_url) uses filter_by(url=...) (case-sensitive) → returns False because the stored URL was changed to lowercase by the update

This can cause a repo to flip between "initialised" and "uninitialised" depending on the case of the URL in the request, and trigger redundant re-checks.

Fix direction: Make is_initialised() and is_compliant() use the same case-insensitive lookup as find().

7. LOW / MEDIUM PROBABILITY — `is_registered()` reads `FORMS_FILE` from disk on every request with no caching

Where: models.py

Every call to is_registered() opens and parses the entire JSON file. This is called from status() (for badges), info(), schedule() (via create()), etc. If the file is large or the filesystem is slow (network mount in Docker), this adds latency. More importantly, if the file is being written to by the forms app at the same moment (no locking on the read side), the read could get a partial/corrupt JSON and raise an unhandled exception that propagates up to the request handler — or worse, into a Runner thread if called from create() inside schedule().

Fix direction: Cache the file contents with a short TTL, or use FileLock on reads (matching the write side in json_store.py).

Summary table

#	Issue	Severity	Probability	Key file	Fix effort
1	No retry/sweep for failed first checks	Critical	High	scheduler.py	Medium — add background sweep
2	Per-worker in-memory queue lost on fork/recycle	High	High	task.py, Dockerfile	Medium-High — shared queue or single worker
3	Unhandled exceptions kill runner threads	High	Medium	scheduler.py	Low — add broad `except` in loop
4	SSH 255 silently drops task with no retry	Medium	High	scheduler.py	Low — re-enqueue with backoff
5	Empty output still calls `update_db`, fails silently	Medium	Medium	scheduler.py	Low — guard + re-enqueue
6	Case mismatch between `find()` and `is_initialised()`	Medium	Low-Med	models.py	Low — use `find()` in both
7	`FORMS_FILE` read with no locking or caching	Low	Medium	models.py	Low — add cache or lock

Most likely root cause for the observed symptom ("newly registered projects stuck after weeks"): a combination of # 1 and # 3/# 4. The initial check fails (SSH issue, timeout, empty output, or unhandled exception), the runner thread either dies or discards the result, and no mechanism ever retries the check unless a user manually visits /info/ or /status/ again. Over weeks, runner threads may gradually die from unhandled exceptions (# 3), reducing throughput to zero in affected workers.

Once again, especially newly registered repos were stuck in the init phase and never got checked. IMHO this is a serious issue, affecting the trustworthiness of the FSFE's service. I spent some tokens on analyzing the issue in the hope that it will help you pinning down the issue. If you wish, I can also dive into fixing them, but that would require some synchronisation of efforts. --- ## Root cause analysis — ranked by severity × probability ### 1. **CRITICAL / HIGH PROBABILITY — Uninitialised repos are never re-enqueued on subsequent requests** **Where:** scheduler.py, specifically the `schedule()` method's decision tree. **The bug:** When a repo already exists in the DB (`repository is not None`) but has never been successfully checked (`last_access` is `None`, `hash` is `None`), the only branch that would re-enqueue it is: ```python elif repository.hash != latest: ``` But `repository.hash` is `None` (never set) and `latest` is a real hash string. In Python, `None != "abc123..."` is `True`, so this **does** re-enqueue. That part is OK by accident. **However**, if the first check failed (SSH error 255, timeout, empty output, or `JSONDecodeError`), the DB row already exists with `hash=None`, the task is completed and removed from the in-memory queue via `done()` at scheduler.py, but the DB is never updated. On the *next* request, `repository.hash` (`None`) `!= latest` evaluates to `True`, so it should re-enqueue. **This path does work** — but only if someone actually visits `/info/` or `/status/` again. **The real problem:** if nobody visits those pages again after the first failure, the repo stays uninitialised forever. There is no retry mechanism, no background sweep, no watchdog. This is the most likely root cause for repos stuck "after weeks" — the initial check failed silently, and nobody re-triggered it. **Fix direction:** Add a lightweight periodic sweep (e.g. a background thread or timer) that queries the DB for rows where `last_access IS NULL` and re-enqueues them. This is the single highest-impact change. --- ### 2. **HIGH / HIGH PROBABILITY — `TaskQueue` is a process-wide singleton with class-level mutable state shared across gunicorn workers via `fork()`** **Where:** task.py ```python _instance = None __urls: set[str] = set() __urls_lock: Lock = Lock() ``` These are **class variables**. `TaskQueue` uses a singleton pattern via `__new__`. With `gunicorn --workers=4`, gunicorn forks after `create_app()` is called. At fork time: - The `_instance`, `__urls` set, and `__urls_lock` are **copied** into each worker. - Each worker gets its own independent `TaskQueue` with its own `__urls` set. - Each worker also gets its own `Scheduler` with its own `Runner` threads. This means: - A task enqueued in Worker A is invisible to Workers B, C, D. - If Worker A creates a DB row and enqueues a task, but the next request from the same user lands on Worker B, Worker B sees the DB row (`repository is not None`) but the task is not in Worker B's queue, so it falls through to the `hash != latest` check. This can cause **duplicate concurrent checks** of the same repo across workers. - If a `Runner` thread dies (see item 3), only that worker loses capacity — no other worker compensates. **But more critically for the bug:** after a `gunicorn` worker is recycled (which gunicorn does periodically via `max_requests` or on crash), the in-memory queue is lost. Any tasks that were in-flight or queued are silently dropped. The DB row exists but was never updated, so `last_access` remains `None`. **Fix direction:** Either run with `--workers=1 --threads=N` (trading parallelism), use `--preload` and ensure the scheduler is started post-fork, or move to a durable queue (Redis, database table) shared across workers. --- ### 3. **HIGH / MEDIUM PROBABILITY — Unhandled exceptions in `Runner.run()` silently kill worker threads** **Where:** scheduler.py The `try/except` in `Runner.run()` only catches: - `subprocess.TimeoutExpired` (line ~112) - `JSONDecodeError` (line ~139) Any other exception — e.g. `AttributeError` if `Repository.find(self.url)` returns `None` in task.py, a database connection error, an `OSError` from SSH, or an encoding error — will propagate up, exit the `while self.__running` loop, and **kill the thread permanently**. There is no respawn logic. With `NB_RUNNER=6` threads per worker process, losing threads over weeks of uptime is very plausible. Once all 6 are dead, no more tasks are ever processed in that worker, even though they're enqueued. The queue grows silently. No monitoring or health check exists. **Fix direction:** Wrap the entire loop body in a broad `except Exception` with logging, so a single bad task can't kill a thread. Optionally add a health-check endpoint that reports the number of alive runner threads. --- ### 4. **MEDIUM / HIGH PROBABILITY — SSH failure (exit code 255) leaves repo permanently uninitialised with no retry** **Where:** scheduler.py When SSH returns 255, the code logs a warning and explicitly skips the DB update. The task is then marked `done()` and removed from the queue. The repo stays with `hash=None` and `last_access=None`. If the SSH failure is transient (network blip, worker restart, connection limit), this is fine — the next request re-enqueues. But if it's persistent (misconfigured key, firewall rule, DNS issue), every attempt will fail with 255 and be silently discarded, with no escalation, no retry backoff, and no alert beyond a log warning. Over weeks, if the API worker is temporarily unreachable (maintenance, restart), every repo checked during that window gets silently dropped and must wait for another user request to retry. **Fix direction:** Re-enqueue tasks that fail with SSH 255 after a delay, with a retry counter. Or at minimum, don't call `done()` so the task stays in the queue (though this risks infinite retry storms without backoff). --- ### 5. **MEDIUM / MEDIUM PROBABILITY — Empty output from linting silently leaves repo uninitialised** **Where:** scheduler.py When `output` is empty, the code logs a warning but **still passes the empty string to `task.update_db(output)`**. `json_loads("")` raises `JSONDecodeError`, which is caught at line 139 and logged — but the DB is not updated. The task is marked done. Same outcome as SSH 255: repo stays stuck. **Fix direction:** Skip `update_db()` when output is empty, and consider re-enqueuing. --- ### 6. **MEDIUM / LOW-MEDIUM PROBABILITY — Case-sensitivity mismatch between `create()` and `find()`** **Where:** models.py vs models.py `Repository.create()` stores the URL as-is (case-preserving). `Repository.find()` uses `db.func.lower()` for case-insensitive lookup. But `is_initialised()` at models.py uses `filter_by(url=url)`, which is **case-sensitive**. If a user registers `git.example.com/Org/Repo` but later visits `/info/git.example.com/org/repo`, then: - `find()` returns the row (case-insensitive match) → `repository is not None` - `schedule()` sees hash is `None`, re-enqueues with the lowercase URL - `update_db()` calls `find(lowercase_url)` → finds the row → updates it with the lowercase URL - But `is_initialised(original_case_url)` uses `filter_by(url=...)` (case-sensitive) → **returns `False`** because the stored URL was changed to lowercase by the update This can cause a repo to flip between "initialised" and "uninitialised" depending on the case of the URL in the request, and trigger redundant re-checks. **Fix direction:** Make `is_initialised()` and `is_compliant()` use the same case-insensitive lookup as `find()`. --- ### 7. **LOW / MEDIUM PROBABILITY — `is_registered()` reads `FORMS_FILE` from disk on every request with no caching** **Where:** models.py Every call to `is_registered()` opens and parses the entire JSON file. This is called from `status()` (for badges), `info()`, `schedule()` (via `create()`), etc. If the file is large or the filesystem is slow (network mount in Docker), this adds latency. More importantly, if the file is being written to by the forms app at the same moment (no locking on the read side), the read could get a partial/corrupt JSON and raise an unhandled exception that propagates up to the request handler — or worse, into a `Runner` thread if called from `create()` inside `schedule()`. **Fix direction:** Cache the file contents with a short TTL, or use `FileLock` on reads (matching the write side in json_store.py). --- ### Summary table | # | Issue | Severity | Probability | Key file | Fix effort | |---|-------|----------|-------------|----------|------------| | 1 | No retry/sweep for failed first checks | Critical | High | scheduler.py | Medium — add background sweep | | 2 | Per-worker in-memory queue lost on fork/recycle | High | High | task.py, Dockerfile | Medium-High — shared queue or single worker | | 3 | Unhandled exceptions kill runner threads | High | Medium | scheduler.py | Low — add broad `except` in loop | | 4 | SSH 255 silently drops task with no retry | Medium | High | scheduler.py | Low — re-enqueue with backoff | | 5 | Empty output still calls `update_db`, fails silently | Medium | Medium | scheduler.py | Low — guard + re-enqueue | | 6 | Case mismatch between `find()` and `is_initialised()` | Medium | Low-Med | models.py | Low — use `find()` in both | | 7 | `FORMS_FILE` read with no locking or caching | Low | Medium | models.py | Low — add cache or lock | **Most likely root cause for the observed symptom** ("newly registered projects stuck after weeks"): a combination of **# 1** and **# 3**/**# 4**. The initial check fails (SSH issue, timeout, empty output, or unhandled exception), the runner thread either dies or discards the result, and no mechanism ever retries the check unless a user manually visits `/info/` or `/status/` again. Over weeks, runner threads may gradually die from unhandled exceptions (# 3), reducing throughput to zero in affected workers.

fkobi added the

component

backend

bug labels 2026-07-17 15:08:42 +00:00

fkobi commented

2026-07-17 15:09:36 +00:00

Is this not caused by #152?

Is this not caused by https://git.fsfe.org/reuse/api/issues/152?

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: reuse/api#167

Analysis of periodic stuck issue #167

Root cause analysis — ranked by severity × probability

1. CRITICAL / HIGH PROBABILITY — Uninitialised repos are never re-enqueued on subsequent requests

2. HIGH / HIGH PROBABILITY — TaskQueue is a process-wide singleton with class-level mutable state shared across gunicorn workers via fork()

3. HIGH / MEDIUM PROBABILITY — Unhandled exceptions in Runner.run() silently kill worker threads

4. MEDIUM / HIGH PROBABILITY — SSH failure (exit code 255) leaves repo permanently uninitialised with no retry

5. MEDIUM / MEDIUM PROBABILITY — Empty output from linting silently leaves repo uninitialised

6. MEDIUM / LOW-MEDIUM PROBABILITY — Case-sensitivity mismatch between create() and find()

7. LOW / MEDIUM PROBABILITY — is_registered() reads FORMS_FILE from disk on every request with no caching

Summary table

2. HIGH / HIGH PROBABILITY — `TaskQueue` is a process-wide singleton with class-level mutable state shared across gunicorn workers via `fork()`

3. HIGH / MEDIUM PROBABILITY — Unhandled exceptions in `Runner.run()` silently kill worker threads

6. MEDIUM / LOW-MEDIUM PROBABILITY — Case-sensitivity mismatch between `create()` and `find()`

7. LOW / MEDIUM PROBABILITY — `is_registered()` reads `FORMS_FILE` from disk on every request with no caching