fix(camera): a frozen external RTSP camera is reconnected

An external RTSP camera that stops sending can leave ffmpeg repeating
its last picture, so the live view, /camera/status and everything that
reuses the live frame showed a picture that had stopped changing. Like
the printer's own camera (#3218), 20 s of identical frames now restarts
ffmpeg, a picture that really does not change is re-checked every
5 minutes, and a camera that is just slow to change waits three times
the longest gap it has shown (up to 5 minutes) instead of being
restarted every 20 s.

CHANGELOG.md

@@ -164,6 +164,7 @@ All notable changes to Bambuddy will be documented in this file.
 - **The frontend build no longer warns about `path` and `crypto` being externalized for the STEP previewer (#2976)** — `occt-import-js`, the Emscripten build behind STEP previews, requires both modules, but only inside its `ENVIRONMENT_IS_NODE` branches; in the browser it loads its `.wasm` from the URL the preview worker passes and draws randomness from `crypto.getRandomValues`. Vite still externalized both and printed two warnings on every build. `vite.config.ts` now drops exactly those two warnings for that one package through `build.rolldownOptions.onLog`, so an externalization anywhere else, or of any other module, still shows.
 
 ### Fixed
+- **A frozen external RTSP camera is reconnected** — When an external RTSP camera stops sending, `ffmpeg` can keep repeating the last picture, so the live view and `/camera/status` kept showing a picture that had stopped changing, and so did anything reusing the live frame (timelapse frames, finish photos, Obico). Like the printer's own camera, an external RTSP camera that sends the same picture for 20 seconds is now reconnected. A camera that is simply slow to change (a slow source, or a smart codec on a static night scene) is restarted at most once, after which Bambuddy waits three times the longest gap it has seen, up to 5 minutes; a picture that really does not change is re-checked every 5 minutes.
 - **A USB camera can be watched from several places at once** — Every viewer of an external camera opened its own connection to it. A USB camera can only be read by one program at a time, so opening the Camera page while a Cam Wall tile (or another browser) was already showing it gave a blank picture. External cameras now share one connection per printer, like the printer's own camera, which also means one connection per MJPEG or snapshot camera and one `ffmpeg` per RTSP or USB camera instead of one per viewer. As with the printer's own camera, the viewer who opens the connection sets its frame rate: with a Cam Wall tile already showing the camera, the Camera page gets the tile's 8 frames per second until everyone has closed it.
 - **Camera snapshots and the camera test no longer fail while someone watches the live view** — A USB camera can be read by one program at a time, so while a viewer was watching, every snapshot Home Assistant or the camera wall fetched from an external camera failed, and the external camera's **Test** button failed too. Snapshots now return the live view's frame, and the Test button reports the camera working from that frame. While the live view is still starting, a snapshot asks the caller to try again a moment later instead of taking the camera from the viewer (the camera wall and the camera page now retry once on their own); this now applies to the printer's own camera as well. An external camera with its own snapshot URL is still fetched from that URL.
 - **USB camera photos are no longer killed mid-capture** — A cleanup job that runs every minute stops leftover camera processes, and it did not know about the one-shot USB capture behind finish photos, timelapse frames and snapshots, so it sometimes killed one in progress. It also stopped any other program's `ffmpeg` reading a USB camera on the same machine; it now only stops Bambuddy's own.

backend/app/services/external_camera.py

@@ -15,6 +15,8 @@ import logging
 import re
 import shutil
 import socket
+import time
+import zlib
 from collections.abc import AsyncGenerator, Callable
 from pathlib import Path
 from urllib.parse import urljoin, urlparse
@@ -1036,8 +1038,11 @@ async def generate_mjpeg_stream(
 
     elif camera_type == "rtsp":
         # Use ffmpeg to convert RTSP to MJPEG, reconnecting after routine drops.
+        # One freeze watch for the whole stream: the still-scene rule compares a
+        # new session's first frame with the one the last session froze on.
+        freeze = _FreezeWatch()
         async with contextlib.aclosing(
-            _reconnecting(lambda: _stream_rtsp(url, fps, on_process=on_process), "RTSP")
+            _reconnecting(lambda: _stream_rtsp(url, fps, on_process=on_process, freeze=freeze), "RTSP")
         ) as frames:
             async for frame in frames:
                 yield _publish(frame)
@@ -1128,6 +1133,75 @@ async def _stream_mjpeg(url: str) -> AsyncGenerator[bytes, None]:
         logger.error("MJPEG stream error: %s", e)
 
 
+# How long an external RTSP stream may emit byte-identical frames before its
+# ffmpeg is restarted, and the longer re-check for a picture that really does
+# not change: the same rule as the printer's own RTSP stream (#3218, see
+# routes/camera._RTSP_FROZEN_SECONDS). ffmpeg's ``-r fps`` repeats its last
+# frame after the camera stops sending, so counting frames sees a live stream.
+_EXTERNAL_RTSP_FROZEN_SECONDS = 20.0
+_EXTERNAL_RTSP_STILL_RECHECK_SECONDS = 300.0
+
+
+class _FreezeWatch:
+    """Notices an RTSP picture that has stopped changing, across sessions.
+
+    A live camera rarely repeats a frame byte for byte (sensor noise), so a run
+    of identical frames usually means ffmpeg is repeating a frozen one. When the
+    first frame after the restart is that same picture again, the camera really
+    shows it (a dark, idle scene), and it is only re-checked at the longer
+    interval until it changes.
+
+    Unlike a Bambu camera, an external one can legitimately repeat frames for a
+    while: a source slower than ffmpeg's ``-r`` rate, or a surveillance camera's
+    smart codec on a static scene. So the limit is three times the longest gap
+    seen between real changes (a "freeze" after which the camera showed a new
+    picture counts as one), never below the 20 s of the printer's camera and
+    never above the still-scene interval.
+    """
+
+    def __init__(self) -> None:
+        self.frozen_crc: int | None = None
+        self.still_scene = False
+        self.unchanged_for = 0.0
+        self._last_crc: int | None = None
+        self._since = 0.0
+        self._longest_gap = 0.0
+
+    def start_session(self) -> None:
+        self._last_crc = None
+
+    def _limit(self) -> float:
+        if self.still_scene:
+            return _EXTERNAL_RTSP_STILL_RECHECK_SECONDS
+        return min(
+            max(_EXTERNAL_RTSP_FROZEN_SECONDS, 3 * self._longest_gap),
+            _EXTERNAL_RTSP_STILL_RECHECK_SECONDS,
+        )
+
+    def frozen(self, frame: bytes, now: float) -> bool:
+        crc = zlib.crc32(frame)
+        if self._last_crc is None:
+            if self.frozen_crc is not None and crc != self.frozen_crc:
+                # The picture had not frozen: the camera is just slow to change.
+                self._longest_gap = max(self._longest_gap, self.unchanged_for)
+            self.still_scene = self.frozen_crc is not None and crc == self.frozen_crc
+            self._last_crc = crc
+            self._since = now
+            return False
+        if crc != self._last_crc:
+            self._longest_gap = max(self._longest_gap, now - self._since)
+            self._last_crc = crc
+            self._since = now
+            self.still_scene = False
+            self.frozen_crc = None
+            return False
+        self.unchanged_for = now - self._since
+        if self.unchanged_for > self._limit():
+            self.frozen_crc = crc
+            return True
+        return False
+
+
 async def _end_stream_ffmpeg(process: asyncio.subprocess.Process) -> None:
     """Stop a streaming ffmpeg, bounded.
 
@@ -1147,6 +1221,7 @@ async def _stream_rtsp(
     fps: int,
     *,
     on_process: Callable[[asyncio.subprocess.Process], None] | None = None,
+    freeze: _FreezeWatch | None = None,
 ) -> AsyncGenerator[bytes, None]:
     """Stream frames from RTSP URL via ffmpeg.
 
@@ -1274,6 +1349,8 @@ async def _stream_rtsp(
         if state != "open":
             return
         log_watch = asyncio.create_task(watch_ffmpeg_log(process))
+        if freeze is not None:
+            freeze.start_session()
 
         buffer = b""
         jpeg_start = b"\xff\xd8"
@@ -1305,6 +1382,16 @@ async def _stream_rtsp(
                     frame = buffer[: end_idx + 2]
                     buffer = buffer[end_idx + 2 :]
                     yield frame
+                    if freeze is not None and freeze.frozen(frame, time.monotonic()):
+                        # Ending the session restarts ffmpeg (_reconnecting).
+                        if freeze.still_scene:
+                            logger.info("External RTSP still-scene re-check, restarting ffmpeg")
+                        else:
+                            logger.warning(
+                                "External RTSP picture frozen for %.0fs, restarting ffmpeg",
+                                freeze.unchanged_for,
+                            )
+                        return
 
             except TimeoutError:
                 logger.warning("RTSP stream read timeout")

backend/tests/unit/services/test_external_rtsp_frozen_frames.py

@@ -0,0 +1,175 @@
+"""A frozen external RTSP camera is reconnected (RC-C1).
+
+ffmpeg can go on repeating its last frame after the camera has stopped sending
+(`-r fps` duplicates it), so frame counting sees a healthy stream: status says
+live, and timelapse, finish photos and Obico reuse a picture that stopped
+hours ago. The printer's own RTSP stream restarts ffmpeg after 20 s of
+byte-identical frames (#3218); the external one now does the same, with the
+same still-scene rule for a camera that really shows an unchanging picture.
+"""
+
+import asyncio
+from unittest.mock import AsyncMock, MagicMock
+
+import pytest
+
+from backend.app.services import external_camera
+from backend.app.services.external_camera import _FreezeWatch
+
+A = b"\xff\xd8" + b"a" * 200 + b"\xff\xd9"
+B = b"\xff\xd8" + b"b" * 200 + b"\xff\xd9"
+
+
+class TestTheFreezeWatch:
+    def test_a_moving_picture_never_freezes(self):
+        watch = _FreezeWatch()
+        watch.start_session()
+        assert not any(watch.frozen(A if i % 2 else B, float(i)) for i in range(100))
+
+    def test_the_same_frame_for_twenty_seconds_is_frozen(self):
+        watch = _FreezeWatch()
+        watch.start_session()
+        assert not watch.frozen(A, 0.0)
+        assert not watch.frozen(A, 19.0)
+        assert watch.frozen(A, 21.0)
+
+    def test_the_same_picture_after_a_restart_is_a_still_scene(self):
+        """A dark, idle chamber can encode to the same frame every time: then
+        the stream only re-checks every few minutes."""
+        watch = _FreezeWatch()
+        watch.start_session()
+        watch.frozen(A, 0.0)
+        assert watch.frozen(A, 21.0)
+
+        watch.start_session()
+        assert not watch.frozen(A, 22.0)
+        assert not watch.frozen(A, 22.0 + 299.0)
+        assert watch.frozen(A, 22.0 + 301.0)
+
+    def test_a_changed_picture_ends_the_still_scene(self):
+        watch = _FreezeWatch()
+        watch.start_session()
+        watch.frozen(A, 0.0)
+        watch.frozen(A, 21.0)
+        watch.start_session()
+        watch.frozen(A, 22.0)
+
+        watch.frozen(B, 23.0)
+
+        assert not watch.frozen(B, 40.0)
+        assert watch.frozen(B, 44.0)
+
+
+class TestASlowOrQuietCamera:
+    """Unlike a Bambu camera, an external one can legitimately repeat frames for
+    longer than 20 s: a source slower than one picture per 20 s (ffmpeg's -r
+    repeats each until the next), or a surveillance camera's smart codec on a
+    static night scene. Restarting those every 20 s would break a camera that
+    worked; the limit grows to three times the longest gap seen between real
+    changes, up to the still-scene interval."""
+
+    def test_a_slow_source_is_restarted_once_then_left_alone(self):
+        watch = _FreezeWatch()
+        watch.start_session()
+        watch.frozen(A, 0.0)
+        assert watch.frozen(A, 21.0)  # looked frozen: no way to know yet
+
+        # After the restart the picture is a new one: it had not frozen, it is slow.
+        watch.start_session()
+        assert not watch.frozen(B, 23.0)
+        assert not watch.frozen(B, 23.0 + 60.0)  # 3 x the 21 s gap it showed
+        assert watch.frozen(B, 23.0 + 64.0)
+
+    def test_a_gap_seen_within_a_session_raises_the_limit(self):
+        watch = _FreezeWatch()
+        watch.start_session()
+        watch.frozen(A, 0.0)
+        watch.frozen(B, 15.0)  # a 15 s gap between real pictures
+        assert not watch.frozen(B, 15.0 + 44.0)
+        assert watch.frozen(B, 15.0 + 46.0)
+
+    def test_the_limit_never_exceeds_the_still_scene_interval(self):
+        watch = _FreezeWatch()
+        watch.start_session()
+        watch.frozen(A, 0.0)
+        watch.frozen(B, 200.0)  # 3 x 200 s would be 10 minutes
+        assert watch.frozen(B, 200.0 + 301.0)
+
+
+class _Ffmpeg:
+    """An ffmpeg that repeats one frame forever, as after a dead upstream."""
+
+    def __init__(self, frame: bytes) -> None:
+        self.pid = 53001
+        self.returncode = None
+        self.stdout = MagicMock()
+        self.stderr = asyncio.StreamReader()
+        self.stderr.feed_eof()
+        self._frame = frame
+
+        async def read(_n):
+            await asyncio.sleep(0.005)
+            return self._frame
+
+        self.stdout.read = read
+
+    def terminate(self) -> None:
+        self.returncode = 0
+
+    def kill(self) -> None:
+        self.returncode = -9
+
+    async def wait(self) -> int:
+        return 0
+
+
+@pytest.fixture
+def ffmpeg(monkeypatch):
+    spawned: list[_Ffmpeg] = []
+
+    async def spawn(*_args, **_kwargs):
+        proc = _Ffmpeg(A)
+        spawned.append(proc)
+        return proc
+
+    monkeypatch.setattr(external_camera, "get_ffmpeg_path", lambda: "/usr/bin/ffmpeg")
+    monkeypatch.setattr(external_camera, "read_ffmpeg_log_until_open", AsyncMock(return_value=("open", b"")))
+    monkeypatch.setattr(external_camera, "watch_ffmpeg_log", AsyncMock(return_value=None))
+    monkeypatch.setattr("asyncio.create_subprocess_exec", spawn)
+    monkeypatch.setattr(external_camera, "_EXTERNAL_RTSP_FROZEN_SECONDS", 0.1)
+    monkeypatch.setattr(external_camera.asyncio, "sleep", _fast_sleep(asyncio.sleep))
+    return spawned
+
+
+def _fast_sleep(real_sleep):
+    async def sleep(delay, *args, **kwargs):
+        # The reconnect pause; the frame pacing stays as it is.
+        return await real_sleep(min(delay, 0.01), *args, **kwargs)
+
+    return sleep
+
+
+@pytest.mark.asyncio
+async def test_a_frozen_session_ends(ffmpeg):
+    session = external_camera._stream_rtsp("rtsp://192.0.2.80/live", 5, freeze=_FreezeWatch())
+
+    frames = 0
+    async with asyncio.timeout(3):
+        async for _ in session:
+            frames += 1
+
+    assert frames > 1
+    # It ended on the freeze, not on its way out: ffmpeg was still running.
+    assert ffmpeg[0].returncode is not None
+
+
+@pytest.mark.asyncio
+async def test_the_stream_reconnects_a_frozen_camera(ffmpeg):
+    stream = external_camera.generate_mjpeg_stream("rtsp://192.0.2.80/live", "rtsp", 5)
+
+    async with asyncio.timeout(3):
+        while len(ffmpeg) < 2:
+            await anext(stream)
+    await stream.aclose()
+
+    assert len(ffmpeg) >= 2

backend/tests/unit/test_camera_reconnect_budget.py

@@ -162,7 +162,7 @@ def external(monkeypatch):
     opened: list[int] = []
     closed: list[int] = []
 
-    def _fake_stream_rtsp(_url, _fps, on_process=None):
+    def _fake_stream_rtsp(_url, _fps, on_process=None, freeze=None):
         frames = sessions.pop(0) if sessions else 0
         index = len(opened)
         opened.append(frames)
@@ -230,7 +230,7 @@ async def test_a_cancelled_viewer_is_not_redialled(monkeypatch):
     reading = asyncio.Event()
     swallow = [True]
 
-    def _fake_stream_rtsp(_url, _fps, on_process=None):
+    def _fake_stream_rtsp(_url, _fps, on_process=None, freeze=None):
         opened.append(len(opened))
 
         async def _gen():