fix(updates,timelapse): a hung update step or slow timelapse stitch no longer gets stuck or loses its work

- In-app update: git steps time out after 5 min, pip/npm after 15; a timed-out
  step gets SIGTERM, then SIGKILL (pip/npm with their children), and the update
  ends with an error instead of blocking every retry. A timed-out frontend build
  keeps the prebuilt files; after a timed-out pip or git reset the status says
  to run the update again before restarting.
- Layer timelapse: the stitch deadline grows with the frame count (0.5 s/frame,
  5 min to 1 h). If ffmpeg fails or Bambuddy stops mid-stitch, the frames are
  kept for 24 h with the ffmpeg command to stitch them by hand; no partial mp4
  is left. The frames cleanup also runs hourly and protects every stitch in
  progress.

.basedpyright/baseline.json

@@ -3002,16 +3002,6 @@
                 }
             }
         ],
-        "./backend/app/services/layer_timelapse.py": [
-            {
-                "code": "reportPossiblyUnboundVariable",
-                "range": {
-                    "startColumn": 15,
-                    "endColumn": 22,
-                    "lineCount": 1
-                }
-            }
-        ],
         "./backend/app/services/library_folder_access.py": [
             {
                 "code": "reportArgumentType",

CHANGELOG.md

@@ -164,6 +164,8 @@ All notable changes to Bambuddy will be documented in this file.
 - **The frontend build no longer warns about `path` and `crypto` being externalized for the STEP previewer (#2976)** — `occt-import-js`, the Emscripten build behind STEP previews, requires both modules, but only inside its `ENVIRONMENT_IS_NODE` branches; in the browser it loads its `.wasm` from the URL the preview worker passes and draws randomness from `crypto.getRandomValues`. Vite still externalized both and printed two warnings on every build. `vite.config.ts` now drops exactly those two warnings for that one package through `build.rolldownOptions.onLog`, so an externalization anywhere else, or of any other module, still shows.
 
 ### Fixed
+- **Long prints keep their layer timelapse** — Stitching a layer timelapse was stopped after 5 minutes and its frames were then deleted, so a print with more than a few hundred layers on a Raspberry Pi lost its timelapse for good. Stitching now gets half a second per layer, from 5 minutes up to an hour. If ffmpeg fails anyway, or Bambuddy stops while it runs, the frames are kept for about 24 hours with an ffmpeg command to make the video by hand; when ffmpeg fails, the log names the folder. Without ffmpeg the frames are still deleted.
+- **An in-app update that hangs no longer blocks further updates** — A stalled `git fetch`, `pip install` or `npm install` left the update showing "downloading" or "installing" forever, and every new attempt was refused with "Update already in progress" until Bambuddy was restarted. Each step now has a time limit (5 minutes for git, 15 for pip and npm); a git or pip step that runs out of time ends the update with an error, and checking for updates again lets you retry. When pip or `git reset` is the one that ran out of time the new code may already be in place, so run the update again before restarting. A frontend build that runs out of time keeps the files shipped with the release.
 - **The plate picker downloads a printer's 3MF once** — Opening a 3MF in the printer's file browser downloaded the whole file again for the plate list and for every plate thumbnail, all at once: a ten-plate 60 MB file meant eleven full downloads, slow on any printer and slower still on an A1. The plate list, plate thumbnails and G-code preview now share one download and keep it for a minute (the 3D view still fetches its own copy).
 - **Printer file downloads tell a missing file from a failed one, and big files get the time they need** — Downloading a file from the printer, and the file browser's previews (G-code, plates, plate thumbnails), treated every failure as "File not found", including a slow transfer of a file just listed. Bambuddy now answers "not found" only when the printer says the file is gone, and otherwise that the download failed or took too long; the download's message and the timelapse messages show which. A file download was also held to 10 minutes whatever its size, so a 250 MB camera recording on a slow link always failed, and the previews to 5 minutes; both may now take as long as the file's size calls for, up to the 30-minute limit. A timelapse that could not be downloaded is reported the same way instead of as a server error.
 - **A timed-out printer download no longer runs on beside its retry** — When a timelapse download ran out of time, it kept going in the background while the next attempt opened a second connection to the printer. The transfer is now stopped first, and a download that already had the time its size allows is not retried.

backend/app/api/routes/updates.py

@@ -5,6 +5,7 @@ import logging
 import os
 import re
 import shutil
+import signal
 import sys
 import time
 
@@ -25,6 +26,53 @@ logger = logging.getLogger(__name__)
 router = APIRouter(prefix="/updates", tags=["updates"])
 
 # Global state for update progress
+# How long one update step may run. Generous, for slow links and a Raspberry
+# Pi, but finite: a step that hung (a stalled git fetch or npm install) left the
+# status at "downloading" or "installing" for good, and every retry answered
+# "Update already in progress" until Bambuddy was restarted.
+GIT_STEP_TIMEOUT_SECONDS = 300
+INSTALL_STEP_TIMEOUT_SECONDS = 900
+TERMINATE_GRACE_SECONDS = 5
+
+
+class _StepTimedOut(Exception):
+    def __init__(self, step: str, seconds: float) -> None:
+        super().__init__(f"{step} timed out after {seconds:.0f} s")
+        self.step = step
+
+
+async def _communicate(process, timeout: float, step: str) -> tuple[bytes, bytes]:
+    """``process.communicate()`` with a deadline; at it the step is stopped."""
+    try:
+        return await asyncio.wait_for(process.communicate(), timeout)
+    except TimeoutError:
+        # Asked first: git removes its lock files on SIGTERM, and a lock left
+        # by SIGKILL would make every retry fail. Killed if it does not go.
+        for sig, grace in ((signal.SIGTERM, TERMINATE_GRACE_SECONDS), (signal.SIGKILL, 10)):
+            _signal_step(process, sig)
+            try:
+                await asyncio.wait_for(process.wait(), grace)
+                break
+            except (TimeoutError, ProcessLookupError):
+                if sig == signal.SIGKILL:
+                    logger.warning("%s did not exit after being killed", step)
+        raise _StepTimedOut(step, timeout) from None
+
+
+def _signal_step(process, sig) -> None:
+    """The whole process group where the step was started in its own session
+    (npm and pip start children of their own); otherwise the process itself."""
+    try:
+        if hasattr(os, "killpg") and isinstance(process.pid, int) and os.getpgid(process.pid) == process.pid:
+            os.killpg(process.pid, sig)
+        elif sig == signal.SIGTERM:
+            process.terminate()
+        else:
+            process.kill()
+    except (ProcessLookupError, PermissionError, OSError):
+        pass
+
+
 _update_status = {
     "status": "idle",  # idle, checking, downloading, installing, complete, error
     "progress": 0,
@@ -324,8 +372,8 @@ async def _origin_points_at_repo(git_path: str, git_config: list[str], app_dir,
             stdout=asyncio.subprocess.PIPE,
             stderr=asyncio.subprocess.PIPE,
         )
-        stdout, _ = await process.communicate()
-    except (OSError, asyncio.CancelledError):
+        stdout, _ = await _communicate(process, GIT_STEP_TIMEOUT_SECONDS, "git remote get-url")
+    except (OSError, asyncio.CancelledError, _StepTimedOut):
         # Fail closed: let the caller go through the rewrite branch if we
         # can't even invoke git. The unconditional set-url is the safer
         # fallback, only mildly destructive.
@@ -753,7 +801,7 @@ async def _perform_update(target_ref: str):
                 stdout=asyncio.subprocess.PIPE,
                 stderr=asyncio.subprocess.PIPE,
             )
-            await process.communicate()
+            await _communicate(process, GIT_STEP_TIMEOUT_SECONDS, "git remote set-url")
 
         _update_status = {
             "status": "downloading",
@@ -787,7 +835,7 @@ async def _perform_update(target_ref: str):
             stdout=asyncio.subprocess.PIPE,
             stderr=asyncio.subprocess.PIPE,
         )
-        stdout, stderr = await process.communicate()
+        stdout, stderr = await _communicate(process, GIT_STEP_TIMEOUT_SECONDS, "git fetch")
 
         if process.returncode != 0:
             error_msg = stderr.decode() if stderr else "Git fetch failed"
@@ -823,7 +871,7 @@ async def _perform_update(target_ref: str):
             stdout=asyncio.subprocess.PIPE,
             stderr=asyncio.subprocess.PIPE,
         )
-        stdout, stderr = await process.communicate()
+        stdout, stderr = await _communicate(process, GIT_STEP_TIMEOUT_SECONDS, "git reset")
 
         if process.returncode != 0:
             error_msg = stderr.decode() if stderr else "Git reset failed"
@@ -858,8 +906,9 @@ async def _perform_update(target_ref: str):
             cwd=str(app_dir),
             stdout=asyncio.subprocess.PIPE,
             stderr=asyncio.subprocess.PIPE,
+            start_new_session=True,
         )
-        stdout, stderr = await process.communicate()
+        stdout, stderr = await _communicate(process, INSTALL_STEP_TIMEOUT_SECONDS, "pip install")
 
         if process.returncode != 0:
             logger.warning("pip install warning: %s", stderr.decode() if stderr else "unknown")
@@ -876,29 +925,36 @@ async def _perform_update(target_ref: str):
                 "error": None,
             }
 
-            # npm install
-            process = await asyncio.create_subprocess_exec(
-                npm_path,
-                "install",
-                cwd=str(frontend_dir),
-                stdout=asyncio.subprocess.PIPE,
-                stderr=asyncio.subprocess.PIPE,
-            )
-            await process.communicate()
-
-            # npm run build
-            process = await asyncio.create_subprocess_exec(
-                npm_path,
-                "run",
-                "build",
-                cwd=str(frontend_dir),
-                stdout=asyncio.subprocess.PIPE,
-                stderr=asyncio.subprocess.PIPE,
-            )
-            stdout, stderr = await process.communicate()
-
-            if process.returncode != 0:
-                logger.warning("Frontend build warning: %s", stderr.decode() if stderr else "unknown")
+            # Optional: the release ships pre-built static files, so a build
+            # that fails or runs out of time leaves those in place.
+            try:
+                # npm install
+                process = await asyncio.create_subprocess_exec(
+                    npm_path,
+                    "install",
+                    cwd=str(frontend_dir),
+                    stdout=asyncio.subprocess.PIPE,
+                    stderr=asyncio.subprocess.PIPE,
+                    start_new_session=True,
+                )
+                await _communicate(process, INSTALL_STEP_TIMEOUT_SECONDS, "npm install")
+
+                # npm run build
+                process = await asyncio.create_subprocess_exec(
+                    npm_path,
+                    "run",
+                    "build",
+                    cwd=str(frontend_dir),
+                    stdout=asyncio.subprocess.PIPE,
+                    stderr=asyncio.subprocess.PIPE,
+                    start_new_session=True,
+                )
+                stdout, stderr = await _communicate(process, INSTALL_STEP_TIMEOUT_SECONDS, "npm run build")
+
+                if process.returncode != 0:
+                    logger.warning("Frontend build warning: %s", stderr.decode() if stderr else "unknown")
+            except _StepTimedOut as exc:
+                logger.warning("%s; keeping the pre-built frontend", exc)
         else:
             logger.info("npm not found or frontend dir missing - using pre-built static files")
 
@@ -911,6 +967,29 @@ async def _perform_update(target_ref: str):
 
         logger.info("Update completed successfully")
 
+    except _StepTimedOut as exc:
+        logger.error("Update failed: %s", exc)
+        if exc.step == "pip install":
+            # git reset has already moved the code: it must not be started on
+            # dependencies that were never installed.
+            advice = (
+                "The new code is in place but its dependencies were not installed. "
+                "Check for updates and run the update again before restarting."
+            )
+        elif exc.step == "git reset":
+            # Stopped while the files were being replaced: some may be new.
+            advice = (
+                "The new code may be only partly in place. "
+                "Check for updates and run the update again before restarting."
+            )
+        else:
+            advice = "Nothing was installed. Check for updates and run the update again."
+        _update_status = {
+            "status": "error",
+            "progress": 0,
+            "message": f"Update failed: {exc.step} took too long",
+            "error": f"{exc}. {advice}",
+        }
     except Exception as e:
         logger.error("Update failed: %s", e)
         _update_status = {

backend/app/main.py

@@ -9198,11 +9198,14 @@ async def on_print_complete(printer_id: int, data: dict):
                 )
         except Exception as e:
             logger.warning("[LAYER-TL] Failed: %s", e)
-            # Try to cancel session on error
-            try:
-                cancel_session(printer_id)
-            except Exception:
-                pass  # Best-effort timelapse session cancellation on error
+            # A completed print's session left _active_sessions when its stitch
+            # began: the one there now belongs to the next print, which may have
+            # started during the stitch, and must not lose its frames.
+            if print_status != "completed":
+                try:
+                    cancel_session(printer_id)
+                except Exception:
+                    pass  # Best-effort timelapse session cancellation on error
 
     spawn_background_task(_background_layer_timelapse(), name="background-layer-timelapse")
 
@@ -10105,17 +10108,32 @@ _camera_cleanup_task: asyncio.Task | None = None
 CAMERA_CLEANUP_INTERVAL = 60
 
 
+# Every this many camera cleanups (about an hour), the timelapse janitor too:
+# kept frames of a failed stitch go after a day, not at the next restart.
+TIMELAPSE_CLEANUP_EVERY = 60
+
+
 async def _camera_cleanup_loop():
-    """Periodically clean up orphaned ffmpeg processes."""
+    """Periodically clean up orphaned ffmpeg processes and timelapse leftovers."""
     from backend.app.api.routes.camera import cleanup_orphaned_streams
+    from backend.app.services.layer_timelapse import cleanup_orphaned_timelapse_sessions
 
+    rounds = 0
     while True:
         try:
+            rounds += 1
             await cleanup_orphaned_streams()
         except asyncio.CancelledError:
             break
         except Exception as e:
             logging.getLogger(__name__).warning("Camera stream cleanup failed: %s", e)
+        try:
+            if rounds % TIMELAPSE_CLEANUP_EVERY == 0:
+                await asyncio.to_thread(cleanup_orphaned_timelapse_sessions)
+        except asyncio.CancelledError:
+            break
+        except Exception as e:
+            logging.getLogger(__name__).warning("Timelapse frames cleanup failed: %s", e)
         await asyncio.sleep(CAMERA_CLEANUP_INTERVAL)
 
 

backend/app/services/layer_timelapse.py

@@ -4,6 +4,7 @@ Captures a frame on each layer change and stitches them into a video on print co
 """
 
 import asyncio
+import contextlib
 import logging
 import shutil
 import time
@@ -23,12 +24,30 @@ _active_sessions: dict[int, "TimelapseSession"] = {}
 
 # Sessions whose frames are being stitched right now: {printer_id: session_id}.
 # on_print_complete removes the session from _active_sessions *before* handing
-# frames_dir to ffmpeg, so for the length of a stitch (up to 300s) nothing in
-# _active_sessions marks that directory as in use. Without this second registry
-# the only thing standing between an in-progress stitch and
-# cleanup_orphaned_timelapse_sessions() is the age margin — whose default is
-# exactly the stitch timeout, so there is no headroom at all.
-_finalizing_sessions: dict[int, str] = {}
+# frames_dir to ffmpeg, so for the length of a stitch (up to an hour, see
+# _stitch_timeout) nothing in _active_sessions marks that directory as in use.
+# Without this second registry the only thing standing between an in-progress
+# stitch and cleanup_orphaned_timelapse_sessions() is the age margin, which is
+# far shorter than a long stitch.
+# A set per printer: a short print can finish while a long one still stitches.
+_finalizing_sessions: dict[int, set[str]] = {}
+
+# Stitch deadline: a libx264 encode of every frame takes about this long per
+# frame on a Raspberry Pi, never less than 5 minutes and never more than an
+# hour. A flat 300 s lost the timelapse of any print with more than a few
+# hundred layers on slow hardware.
+_STITCH_SECONDS_PER_FRAME = 0.5
+_STITCH_MIN_SECONDS = 300.0
+_STITCH_MAX_SECONDS = 3600.0
+
+# A stitch that fails keeps its frames, marked with this file, so the video can
+# still be made from them; the orphan janitor leaves them for this long.
+KEPT_FRAMES_MARKER = "STITCH_FAILED"
+KEEP_FAILED_FRAMES_SECONDS = 24 * 60 * 60
+
+
+def _stitch_timeout(frame_count: int) -> float:
+    return min(max(_STITCH_MIN_SECONDS, frame_count * _STITCH_SECONDS_PER_FRAME), _STITCH_MAX_SECONDS)
 
 
 def get_ffmpeg_path() -> str | None:
@@ -58,6 +77,10 @@ class TimelapseSession:
     frame_count: int = 0
     session_id: str = field(default_factory=lambda: datetime.now().strftime("%Y%m%d_%H%M%S"))
     frames_dir: Path = field(init=False)
+    # Set once ffmpeg has been started on the frames: only a stitch that got
+    # that far and failed keeps them. Without ffmpeg, or without frames, there
+    # is nothing a later attempt could do differently.
+    ffmpeg_ran: bool = field(default=False, init=False)
 
     def __post_init__(self):
         self.frames_dir = settings.base_dir / "timelapse_frames" / str(self.printer_id) / self.session_id
@@ -179,6 +202,11 @@ class TimelapseSession:
             str(output_path),
         ]
 
+        process = None
+        timeout = _STITCH_MIN_SECONDS
+        # Marked before ffmpeg starts, so a restart in the middle of a stitch
+        # (which can now run an hour) keeps the frames too.
+        self.mark_kept("The timelapse was being stitched when Bambuddy stopped.")
         try:
             process = await asyncio.create_subprocess_exec(
                 *cmd,
@@ -186,25 +214,58 @@ class TimelapseSession:
                 stderr=asyncio.subprocess.PIPE,
                 cwd=str(self.frames_dir),  # Run in frames dir so relative paths work
             )
+            self.ffmpeg_ran = True
 
-            stdout, stderr = await asyncio.wait_for(process.communicate(), timeout=300)
+            timeout = _stitch_timeout(len(frame_files))
+            stdout, stderr = await asyncio.wait_for(process.communicate(), timeout=timeout)
 
             if process.returncode != 0:
                 logger.error("ffmpeg timelapse stitch failed: %s", summarize_ffmpeg_stderr(stderr) or NO_FFMPEG_OUTPUT)
+                output_path.unlink(missing_ok=True)
                 return False
 
             logger.info("Created timelapse video: %s (%s frames)", output_path, self.frame_count)
             return True
 
         except TimeoutError:
-            logger.error("Timelapse stitching timed out")
+            logger.error("Timelapse stitching timed out after %.0f s (%s frames)", timeout, len(frame_files))
             if process:
-                process.kill()
+                with contextlib.suppress(ProcessLookupError):
+                    process.kill()
+                try:
+                    await asyncio.wait_for(process.wait(), 10)
+                except TimeoutError:
+                    logger.warning("ffmpeg did not exit after the timelapse stitch was killed")
+            output_path.unlink(missing_ok=True)
             return False
         except Exception as e:
             logger.error("Timelapse stitch failed: %s", e)
             return False
 
+    def mark_kept(self, reason: str) -> bool:
+        """Mark the frames to be kept for KEEP_FAILED_FRAMES_SECONDS."""
+        try:
+            marker = self.frames_dir / KEPT_FRAMES_MARKER  # SEC-PATH-OK: this module's constant filename
+            marker.write_text(
+                f"{reason}\nThese frames are kept for 24 hours. To make the video from them, run in this folder:\n"
+                "ffmpeg -f concat -safe 0 -i frames.txt -c:v libx264 -pix_fmt yuv420p timelapse.mp4\n"
+            )
+            return True
+        except OSError as e:
+            logger.warning("Could not mark timelapse frames as kept: %s", e)
+            return False
+
+    def keep_frames(self) -> None:
+        """Keep the frames of a failed stitch for a day instead of deleting them."""
+        if self.mark_kept("The timelapse could not be stitched."):
+            logger.warning(
+                "Timelapse frames for printer %s kept for 24 hours at %s (stitching failed)",
+                self.printer_id,
+                self.frames_dir,
+            )
+        else:
+            self.cleanup()
+
     def cleanup(self):
         """Remove temporary frames directory."""
         try:
@@ -280,39 +341,52 @@ async def on_print_complete(printer_id: int) -> Path | None:
     Returns:
         Path to stitched video, or None if no session or stitching failed
     """
-    session = _active_sessions.pop(printer_id, None)
+    session = _active_sessions.get(printer_id)
     if not session:
         return None
 
-    if session.frame_count == 0:
-        logger.info("No timelapse frames captured for printer %s", printer_id)
-        session.cleanup()
-        return None
-
-    # Create output path in parent of frames dir
-    output_path = session.frames_dir.parent / f"timelapse_{session.session_id}.mp4"
-
-    # The session is already out of _active_sessions, so mark it finalizing for
-    # the length of the stitch — otherwise a sweep running now sees a frames
-    # directory that matches no session and whose mtime is the last layer's
-    # write, which on a tall print's final layer is easily older than the age
-    # margin, and deletes ffmpeg's input from under it.
-    _finalizing_sessions[printer_id] = session.session_id
+    # Marked finalizing for the length of the stitch, and before it leaves
+    # _active_sessions (the janitor runs in a thread) — otherwise a sweep sees
+    # a frames directory that matches no session and whose mtime is the last
+    # layer's write, which on a tall print's final layer is easily older than
+    # the age margin, and deletes ffmpeg's input from under it.
+    _finalizing_sessions.setdefault(printer_id, set()).add(session.session_id)
+    _active_sessions.pop(printer_id, None)
     try:
+        if session.frame_count == 0:
+            logger.info("No timelapse frames captured for printer %s", printer_id)
+            session.cleanup()
+            return None
+
+        # Create output path in parent of frames dir
+        output_path = session.frames_dir.parent / f"timelapse_{session.session_id}.mp4"
         success = await session.stitch(output_path)
         if success:
             # Cleanup frames after successful stitch
             session.cleanup()
             return output_path
-        else:
-            session.cleanup()
-            return None
+        _keep_or_clean(session)
+        return None
     except Exception as e:
         logger.error("Timelapse completion failed: %s", e)
-        session.cleanup()
+        _keep_or_clean(session)
         return None
     finally:
-        _finalizing_sessions.pop(printer_id, None)
+        finalizing = _finalizing_sessions.get(printer_id)
+        if finalizing is not None:
+            finalizing.discard(session.session_id)
+            if not finalizing:
+                _finalizing_sessions.pop(printer_id, None)
+
+
+def _keep_or_clean(session: "TimelapseSession") -> None:
+    """After a failed stitch: keep the frames if ffmpeg ran on them (a slow
+    encode, an ffmpeg error), so the video can still be made; delete them if it
+    never could (no ffmpeg, no frames)."""
+    if session.ffmpeg_ran:
+        session.keep_frames()
+    else:
+        session.cleanup()
 
 
 def cancel_session(printer_id: int):
@@ -356,13 +430,15 @@ def cleanup_orphaned_timelapse_sessions(min_age_seconds: float = 300) -> int:
     * `_active_sessions` covers a session that is still capturing.
     * `_finalizing_sessions` covers the stitch window. on_print_complete drops
       the session from `_active_sessions` before handing frames_dir to ffmpeg,
-      so without this the directory matches no session for up to 300s while
-      being actively read.
+      so without this the directory matches no session for up to an hour
+      (`_stitch_timeout`) while being actively read.
     * `min_age_seconds` covers the remaining gap - a session in the middle of
       being created, and the stitched `.mp4` between ffmpeg finishing it and
       the caller attaching and unlinking it. Both are freshly written, so the
-      margin has real headroom there; it did NOT have any for the stitch
-      window, whose length is bounded by the same 300s.
+      margin has real headroom there; it has none for the stitch window.
+
+    Frames a failed stitch kept (`KEPT_FRAMES_MARKER`) stay for
+    `KEEP_FAILED_FRAMES_SECONDS`, counted from the marker.
 
     Returns the number of orphaned directories/files removed.
     """
@@ -381,10 +457,8 @@ def cleanup_orphaned_timelapse_sessions(min_age_seconds: float = 300) -> int:
             continue
 
         active_session = _active_sessions.get(printer_id)
-        in_use_session_ids = {
-            active_session.session_id if active_session else None,
-            _finalizing_sessions.get(printer_id),
-        } - {None}
+        in_use_session_ids = {active_session.session_id if active_session else None} - {None}
+        in_use_session_ids |= _finalizing_sessions.get(printer_id, set())
 
         for entry in printer_dir.iterdir():
             # Frame dirs are named "<session_id>/"; stitched-but-not-yet-
@@ -403,6 +477,9 @@ def cleanup_orphaned_timelapse_sessions(min_age_seconds: float = 300) -> int:
             try:
                 if now - entry.stat().st_mtime < min_age_seconds:
                     continue
+                marker = entry / KEPT_FRAMES_MARKER if entry.is_dir() else None
+                if marker is not None and marker.exists() and now - marker.stat().st_mtime < KEEP_FAILED_FRAMES_SECONDS:
+                    continue
             except OSError:
                 continue
             try:

backend/tests/unit/services/test_layer_timelapse.py

@@ -554,7 +554,7 @@ class TestCleanupOrphanedTimelapseSessions:
 
             # Exactly the state on_print_complete is in while ffmpeg runs.
             _active_sessions.pop(1, None)
-            _finalizing_sessions[1] = session.session_id
+            _finalizing_sessions[1] = {session.session_id}
 
             removed = cleanup_orphaned_timelapse_sessions(min_age_seconds=300)
 

backend/tests/unit/services/test_layer_timelapse_stitch_deadline.py

@@ -0,0 +1,324 @@
+"""A layer timelapse is not lost to a slow stitch (TO-C8).
+
+Stitching was killed at a flat 300 s, and the frames were then deleted: a
+1500-layer print on a Raspberry Pi 4 takes longer than that to encode, so its
+timelapse was gone for good. The deadline now grows with the frame count
+(half a second per frame, at least 5 minutes, at most an hour), and a stitch
+that fails keeps its frames for a day so the video can still be made.
+"""
+
+import asyncio
+import os
+import time
+from unittest.mock import AsyncMock, MagicMock, patch
+
+import pytest
+
+from backend.app.services import layer_timelapse
+from backend.app.services.layer_timelapse import TimelapseSession
+
+
+@pytest.fixture
+def base_dir(tmp_path, monkeypatch):
+    fake = MagicMock(base_dir=tmp_path)
+    monkeypatch.setattr(layer_timelapse, "settings", fake)
+    layer_timelapse._active_sessions.clear()
+    layer_timelapse._finalizing_sessions.clear()
+    yield tmp_path
+    layer_timelapse._active_sessions.clear()
+    layer_timelapse._finalizing_sessions.clear()
+
+
+def _session_with_frames(printer_id: int, frames: int) -> TimelapseSession:
+    session = TimelapseSession(printer_id=printer_id, archive_id=1, camera_url="http://cam/", camera_type="mjpeg")
+    for n in range(frames):
+        (session.frames_dir / f"layer_{n:05d}.jpg").write_bytes(b"\xff\xd8jpg\xff\xd9")
+    session.frame_count = frames
+    return session
+
+
+@pytest.mark.parametrize(("frames", "seconds"), [(10, 300), (600, 300), (1500, 750), (100_000, 3600)])
+def test_the_stitch_deadline_grows_with_the_frames(frames, seconds):
+    assert layer_timelapse._stitch_timeout(frames) == seconds
+
+
+@pytest.mark.asyncio
+async def test_a_stitch_past_its_deadline_is_killed_and_reaped(base_dir, monkeypatch):
+    session = _session_with_frames(1, 3)
+    proc = MagicMock()
+
+    async def never():
+        await asyncio.Event().wait()
+
+    proc.communicate = never
+    proc.wait = AsyncMock(return_value=-9)
+    monkeypatch.setattr(layer_timelapse, "_stitch_timeout", lambda _frames: 0.05)
+
+    with (
+        patch.object(layer_timelapse, "get_ffmpeg_path", return_value="/usr/bin/ffmpeg"),
+        patch("asyncio.create_subprocess_exec", AsyncMock(return_value=proc)),
+    ):
+        assert await session.stitch(base_dir / "out.mp4") is False
+
+    proc.kill.assert_called_once()
+    proc.wait.assert_awaited()
+
+
+@pytest.mark.asyncio
+async def test_a_failed_stitch_keeps_its_frames(base_dir):
+    session = _session_with_frames(1, 3)
+    layer_timelapse._active_sessions[1] = session
+
+    async def ffmpeg_failed(self, _output, fps=30):
+        self.ffmpeg_ran = True
+        return False
+
+    with patch.object(TimelapseSession, "stitch", ffmpeg_failed):
+        assert await layer_timelapse.on_print_complete(1) is None
+
+    assert len(list(session.frames_dir.glob("layer_*.jpg"))) == 3
+    marker = session.frames_dir / layer_timelapse.KEPT_FRAMES_MARKER
+    assert marker.exists() and "frames.txt" in marker.read_text()
+
+
+@pytest.mark.asyncio
+async def test_without_ffmpeg_the_frames_are_not_kept(base_dir):
+    """Nothing a later attempt could do differently: an install without ffmpeg
+    would otherwise keep every print's frames."""
+    session = _session_with_frames(1, 3)
+    layer_timelapse._active_sessions[1] = session
+
+    with patch.object(layer_timelapse, "get_ffmpeg_path", return_value=None):
+        assert await layer_timelapse.on_print_complete(1) is None
+
+    assert not session.frames_dir.exists()
+
+
+@pytest.mark.asyncio
+async def test_a_stitch_cut_short_by_a_restart_keeps_its_frames(base_dir):
+    """Marked before ffmpeg starts: at startup the janitor sees a kept folder."""
+    session = _session_with_frames(1, 3)
+    proc = MagicMock()
+    marked_while_running = []
+
+    async def interrupted():
+        marked_while_running.append((session.frames_dir / layer_timelapse.KEPT_FRAMES_MARKER).exists())
+        raise RuntimeError("process ended")
+
+    proc.communicate = interrupted
+    with (
+        patch.object(layer_timelapse, "get_ffmpeg_path", return_value="/usr/bin/ffmpeg"),
+        patch("asyncio.create_subprocess_exec", AsyncMock(return_value=proc)),
+    ):
+        assert await session.stitch(base_dir / "out.mp4") is False
+
+    assert marked_while_running == [True]
+
+
+@pytest.mark.asyncio
+async def test_a_failed_stitch_leaves_no_partial_video(base_dir, monkeypatch):
+    session = _session_with_frames(1, 3)
+    output = session.frames_dir.parent / "timelapse_x.mp4"
+    proc = MagicMock()
+    proc.returncode = 1
+
+    async def partial():
+        output.write_bytes(b"half a video")
+        return b"", b"encoder error"
+
+    proc.communicate = partial
+    with (
+        patch.object(layer_timelapse, "get_ffmpeg_path", return_value="/usr/bin/ffmpeg"),
+        patch("asyncio.create_subprocess_exec", AsyncMock(return_value=proc)),
+    ):
+        assert await session.stitch(output) is False
+
+    assert not output.exists()
+
+
+@pytest.mark.asyncio
+async def test_two_stitches_of_one_printer_are_both_protected(base_dir):
+    """A short print can finish while a long one still stitches; the janitor
+    now runs while Bambuddy does, so the first must stay protected."""
+    long_session = _session_with_frames(1, 3)
+    long_session.session_id = "20260101_000000"
+    short_session = _session_with_frames(1, 3)
+    short_session.session_id = "20260101_000500"
+    release = asyncio.Event()
+
+    async def slow(self, _output, fps=30):
+        if self is long_session:
+            await release.wait()
+        return True
+
+    with patch.object(TimelapseSession, "stitch", slow):
+        layer_timelapse._active_sessions[1] = long_session
+        first = asyncio.create_task(layer_timelapse.on_print_complete(1))
+        await asyncio.sleep(0)
+        layer_timelapse._active_sessions[1] = short_session
+        await layer_timelapse.on_print_complete(1)
+
+        assert layer_timelapse._finalizing_sessions[1] == {"20260101_000000"}
+        release.set()
+        await first
+    assert 1 not in layer_timelapse._finalizing_sessions
+
+
+@pytest.mark.asyncio
+async def test_a_successful_stitch_still_cleans_up(base_dir):
+    session = _session_with_frames(1, 3)
+    layer_timelapse._active_sessions[1] = session
+
+    with patch.object(TimelapseSession, "stitch", AsyncMock(return_value=True)):
+        assert await layer_timelapse.on_print_complete(1) is not None
+
+    assert not session.frames_dir.exists()
+
+
+def test_the_janitor_leaves_kept_frames_for_a_day(base_dir):
+    kept = base_dir / "timelapse_frames" / "1" / "20260101_000000"
+    kept.mkdir(parents=True)
+    (kept / "layer_00000.jpg").write_bytes(b"jpg")
+    (kept / layer_timelapse.KEPT_FRAMES_MARKER).write_text("stitch failed")
+    hour_ago = time.time() - 3600
+    os.utime(kept, (hour_ago, hour_ago))
+    os.utime(kept / layer_timelapse.KEPT_FRAMES_MARKER, (hour_ago, hour_ago))
+
+    assert layer_timelapse.cleanup_orphaned_timelapse_sessions() == 0
+    assert kept.exists()
+
+    old = time.time() - layer_timelapse.KEEP_FAILED_FRAMES_SECONDS - 60
+    os.utime(kept / layer_timelapse.KEPT_FRAMES_MARKER, (old, old))
+    assert layer_timelapse.cleanup_orphaned_timelapse_sessions() == 1
+    assert not kept.exists()
+
+
+def test_the_janitor_still_reaps_ordinary_orphans(base_dir):
+    orphan = base_dir / "timelapse_frames" / "1" / "20260101_000001"
+    orphan.mkdir(parents=True)
+    hour_ago = time.time() - 3600
+    os.utime(orphan, (hour_ago, hour_ago))
+
+    assert layer_timelapse.cleanup_orphaned_timelapse_sessions() == 1
+
+
+@pytest.mark.asyncio
+async def test_the_janitor_also_runs_while_bambuddy_does(monkeypatch):
+    """Hourly, so kept frames go after a day rather than at the next restart."""
+    from backend.app import main
+
+    sweeps = []
+    rounds = {"n": 0}
+
+    async def fake_sleep(_seconds):
+        rounds["n"] += 1
+        if rounds["n"] >= main.TIMELAPSE_CLEANUP_EVERY * 2:
+            raise asyncio.CancelledError
+
+    monkeypatch.setattr("backend.app.api.routes.camera.cleanup_orphaned_streams", AsyncMock())
+    monkeypatch.setattr(layer_timelapse, "cleanup_orphaned_timelapse_sessions", lambda: sweeps.append(1) or 0)
+    monkeypatch.setattr(main.asyncio, "sleep", fake_sleep)
+
+    with pytest.raises(asyncio.CancelledError):
+        await main._camera_cleanup_loop()
+
+    assert len(sweeps) == 2
+
+
+@pytest.mark.asyncio
+async def test_a_failed_attach_leaves_the_next_prints_session_alone():
+    """The completed print's session left _active_sessions when its stitch
+    began; the one there now is the next print's."""
+    import inspect
+
+    from backend.app import main
+
+    source = inspect.getsource(main)
+    i = source.index('logger.warning("[LAYER-TL] Failed: %s", e)')
+    assert 'if print_status != "completed":' in source[i : i + 600]
+
+
+@pytest.mark.asyncio
+async def test_ffmpeg_that_cannot_be_started_does_not_keep_the_frames(base_dir):
+    session = _session_with_frames(1, 3)
+    layer_timelapse._active_sessions[1] = session
+
+    with (
+        patch.object(layer_timelapse, "get_ffmpeg_path", return_value="/usr/bin/ffmpeg"),
+        patch("asyncio.create_subprocess_exec", AsyncMock(side_effect=PermissionError("not executable"))),
+    ):
+        assert await layer_timelapse.on_print_complete(1) is None
+
+    assert not session.frames_dir.exists()
+
+
+@pytest.mark.asyncio
+async def test_a_session_is_protected_before_it_leaves_the_active_list(base_dir, monkeypatch):
+    """The janitor runs in a thread: there must be no moment in which the
+    stitching session is in neither list."""
+    session = _session_with_frames(1, 3)
+    layer_timelapse._active_sessions[1] = session
+    seen = []
+
+    class Watched(dict):
+        def pop(self, key, default=None):
+            seen.append(session.session_id in layer_timelapse._finalizing_sessions.get(key, set()))
+            return super().pop(key, default)
+
+    monkeypatch.setattr(layer_timelapse, "_active_sessions", Watched(layer_timelapse._active_sessions))
+    with patch.object(TimelapseSession, "stitch", AsyncMock(return_value=True)):
+        await layer_timelapse.on_print_complete(1)
+
+    assert seen == [True]
+    assert 1 not in layer_timelapse._finalizing_sessions
+
+
+@pytest.mark.asyncio
+async def test_a_session_without_frames_is_not_left_protected(base_dir):
+    session = _session_with_frames(1, 0)
+    layer_timelapse._active_sessions[1] = session
+
+    assert await layer_timelapse.on_print_complete(1) is None
+    assert 1 not in layer_timelapse._finalizing_sessions
+    assert 1 not in layer_timelapse._active_sessions
+
+
+@pytest.mark.asyncio
+async def test_the_janitor_runs_even_when_the_stream_cleanup_fails(monkeypatch):
+    from backend.app import main
+
+    sweeps = []
+    rounds = {"n": 0}
+
+    async def fake_sleep(_seconds):
+        rounds["n"] += 1
+        if rounds["n"] >= main.TIMELAPSE_CLEANUP_EVERY:
+            raise asyncio.CancelledError
+
+    monkeypatch.setattr(
+        "backend.app.api.routes.camera.cleanup_orphaned_streams", AsyncMock(side_effect=RuntimeError("ps failed"))
+    )
+    monkeypatch.setattr(layer_timelapse, "cleanup_orphaned_timelapse_sessions", lambda: sweeps.append(1) or 0)
+    monkeypatch.setattr(main.asyncio, "sleep", fake_sleep)
+
+    with pytest.raises(asyncio.CancelledError):
+        await main._camera_cleanup_loop()
+
+    assert sweeps == [1]
+
+
+@pytest.mark.asyncio
+async def test_an_ffmpeg_error_keeps_the_frames_end_to_end(base_dir):
+    session = _session_with_frames(1, 3)
+    layer_timelapse._active_sessions[1] = session
+    proc = MagicMock()
+    proc.returncode = 1
+    proc.communicate = AsyncMock(return_value=(b"", b"encoder error"))
+
+    with (
+        patch.object(layer_timelapse, "get_ffmpeg_path", return_value="/usr/bin/ffmpeg"),
+        patch("asyncio.create_subprocess_exec", AsyncMock(return_value=proc)),
+    ):
+        assert await layer_timelapse.on_print_complete(1) is None
+
+    assert len(list(session.frames_dir.glob("layer_*.jpg"))) == 3

backend/tests/unit/test_updater_step_timeouts.py

@@ -0,0 +1,209 @@
+"""The in-app updater gives up on a step that hangs (TO-C7).
+
+Every git, pip and npm step was awaited with no deadline. A network stall in
+git fetch or npm install left the status at "downloading" or "installing" for
+good, and every retry answered "Update already in progress" until Bambuddy was
+restarted. Steps now have a deadline (git 300 s, pip and npm 900 s); the
+process is killed and the update ends with an error, except the optional
+frontend build, whose pre-built files ship with the release.
+"""
+
+import asyncio
+from unittest.mock import AsyncMock, MagicMock, patch
+
+import pytest
+
+from backend.app.api.routes import updates as updates_module
+
+
+def _fake_exec(hang_on: str, procs: list):
+    async def create(*args, **kwargs):
+        proc = MagicMock()
+        proc.returncode = 0
+        proc.killed = False
+
+        def kill():
+            proc.killed = True
+            proc.returncode = -9
+
+        proc.kill = kill
+        proc.terminate = kill
+        proc.wait = AsyncMock(return_value=-9)
+        if hang_on in args or (hang_on == "npm install" and args[:2] == ("/usr/bin/npm", "install")):
+
+            async def never():
+                await asyncio.Event().wait()
+
+            proc.communicate = never
+        else:
+            proc.communicate = AsyncMock(return_value=(b"git@github.com:maziggy/bambuddy.git\n", b""))
+        proc.args = args
+        procs.append(proc)
+        return proc
+
+    return create
+
+
+async def _update(tmp_path, hang_on: str, *, npm: bool = False):
+    app_dir = tmp_path / "app"
+    (app_dir / "frontend").mkdir(parents=True)
+    (app_dir / "requirements.txt").write_text("fastapi\n")
+    procs: list = []
+
+    def find(name):
+        return "/usr/bin/npm" if name == "npm" and npm else ("/usr/bin/git" if name == "git" else None)
+
+    with (
+        patch.object(updates_module.settings, "app_dir", app_dir),
+        patch.object(updates_module, "_find_executable", side_effect=find),
+        patch.object(updates_module.asyncio, "create_subprocess_exec", side_effect=_fake_exec(hang_on, procs)),
+        patch.object(updates_module, "GIT_STEP_TIMEOUT_SECONDS", 0.05),
+        patch.object(updates_module, "INSTALL_STEP_TIMEOUT_SECONDS", 0.05),
+    ):
+        await asyncio.wait_for(updates_module._perform_update("v9.9.9"), 5)
+    return updates_module._update_status, procs
+
+
+@pytest.fixture(autouse=True)
+def _idle_status():
+    updates_module._update_status = {"status": "idle", "progress": 0, "message": "", "error": None}
+    yield
+    updates_module._update_status = {"status": "idle", "progress": 0, "message": "", "error": None}
+
+
+@pytest.mark.asyncio
+async def test_a_stalled_git_fetch_ends_the_update_with_an_error(tmp_path):
+    status, procs = await _update(tmp_path, "fetch")
+
+    assert status["status"] == "error"
+    assert "timed out" in (status["error"] or "").lower()
+    stuck = [p for p in procs if "fetch" in p.args]
+    assert stuck and stuck[0].killed
+    stuck[0].wait.assert_awaited()
+
+
+@pytest.mark.asyncio
+async def test_a_stalled_pip_install_ends_the_update_with_an_error(tmp_path):
+    status, procs = await _update(tmp_path, "pip")
+
+    assert status["status"] == "error"
+    assert [p for p in procs if "pip" in p.args][0].killed
+    # The code has already moved: it must not be started without its dependencies.
+    assert "before restarting" in status["error"]
+
+
+@pytest.mark.asyncio
+async def test_a_stalled_frontend_build_falls_back_to_the_prebuilt_files(tmp_path):
+    status, procs = await _update(tmp_path, "build", npm=True)
+
+    assert status["status"] == "complete"
+    assert [p for p in procs if "build" in p.args][0].killed
+
+
+@pytest.mark.asyncio
+async def test_after_a_timeout_the_update_can_be_tried_again(tmp_path):
+    status, _ = await _update(tmp_path, "fetch")
+
+    assert status["status"] not in ("downloading", "installing")
+
+
+def test_the_deadlines_are_generous():
+    """Slow links and a Raspberry Pi: git 5 minutes, pip and npm 15."""
+    assert updates_module.GIT_STEP_TIMEOUT_SECONDS == 300
+    assert updates_module.INSTALL_STEP_TIMEOUT_SECONDS == 900
+
+
+@pytest.mark.asyncio
+async def test_a_stalled_npm_install_falls_back_to_the_prebuilt_files(tmp_path):
+    status, procs = await _update(tmp_path, "npm install", npm=True)
+
+    assert status["status"] == "complete"
+    assert [p for p in procs if p.args[:2] == ("/usr/bin/npm", "install")][0].killed
+
+
+@pytest.mark.asyncio
+async def test_a_step_in_its_own_session_is_killed_with_its_children(tmp_path):
+    """npm and pip start children of their own: killing only the parent would
+    leave them holding the pipes, and the wait would never end."""
+    import os
+    import sys
+
+    child_pid_file = tmp_path / "child.pid"
+    script = (
+        "import subprocess, sys, time;"
+        "c = subprocess.Popen([sys.executable, '-c', 'import time; time.sleep(60)']);"
+        f"open({str(child_pid_file)!r}, 'w').write(str(c.pid));"
+        "time.sleep(60)"
+    )
+    process = await asyncio.create_subprocess_exec(
+        sys.executable,
+        "-c",
+        script,
+        stdout=asyncio.subprocess.PIPE,
+        stderr=asyncio.subprocess.PIPE,
+        start_new_session=True,
+    )
+    for _ in range(100):
+        if child_pid_file.exists() and child_pid_file.read_text():
+            break
+        await asyncio.sleep(0.05)
+    child = int(child_pid_file.read_text())
+
+    with pytest.raises(updates_module._StepTimedOut):
+        await asyncio.wait_for(updates_module._communicate(process, 0.1, "npm install"), 15)
+
+    assert process.returncode is not None
+    for _ in range(100):
+        try:
+            os.kill(child, 0)
+        except ProcessLookupError:
+            break
+        await asyncio.sleep(0.05)
+    else:
+        os.kill(child, 9)
+        pytest.fail("the step's child outlived it")
+
+
+@pytest.mark.asyncio
+async def test_a_stalled_git_reset_says_the_code_may_be_partly_in_place(tmp_path):
+    status, _ = await _update(tmp_path, "reset")
+
+    assert status["status"] == "error"
+    assert "partly in place" in status["error"] and "before restarting" in status["error"]
+
+
+@pytest.mark.asyncio
+async def test_a_step_is_asked_to_stop_before_it_is_killed():
+    """git removes its lock files on SIGTERM; SIGKILL would leave
+    .git/index.lock behind and every retry would fail on it."""
+    process = MagicMock(pid=None)
+    process.communicate = asyncio.Event().wait
+    process.wait = AsyncMock(return_value=-15)
+
+    with pytest.raises(updates_module._StepTimedOut):
+        await updates_module._communicate(process, 0.05, "git reset")
+
+    process.terminate.assert_called_once()
+    process.kill.assert_not_called()
+
+
+@pytest.mark.asyncio
+async def test_a_step_that_ignores_the_request_is_killed(monkeypatch):
+    process = MagicMock(pid=None)
+    process.communicate = asyncio.Event().wait
+    waits = iter([TimeoutError(), -9])
+
+    async def wait():
+        result = next(waits)
+        if isinstance(result, Exception):
+            raise result
+        return result
+
+    process.wait = wait
+    monkeypatch.setattr(updates_module, "TERMINATE_GRACE_SECONDS", 0.05)
+
+    with pytest.raises(updates_module._StepTimedOut):
+        await updates_module._communicate(process, 0.05, "npm install")
+
+    process.terminate.assert_called_once()
+    process.kill.assert_called_once()