feat(support): record process memory, threads and children in bundles (#2734)

A bundle described everything except the process it runs in. So a report of
memory climbing over days until the OOM killer fires arrives with no way to
act on it: the numbers that name the mechanism only exist while it is
happening, and by the time anyone asks, the container has been restarted.

The new `process` section carries what actually separates the candidates.
Resident against virtual memory: 650MB RSS with 12.9GB VMS is address
space — thread stacks or allocator arenas — not a heap full of live data,
and that reading is the opposite of the one the reporter drew from the same
figures. Thread count and child-process count then split those two apart,
and a census of live objects by type names what a growing heap is filling
up with. Open files, sockets and uptime round it out.

Three constraints worth keeping:

The heap census is skipped above 2GB. gc.get_objects() materialises every
tracked object, so it costs most on exactly the process that can least
afford it — a bundle generated to diagnose runaway memory must not be the
allocation that tips the host over. Everything else is still collected, and
the skip is recorded with its reason rather than silently omitted.

Children are recorded by executable name only. An ffmpeg command line
carries the camera URL, and with it the camera's password.

Collection runs off the event loop and every metric is independently
best-effort. psutil raises on hardened kernels and in restricted
containers, and the bundle is how someone reports a problem in the first
place — it has to be produced even when half the numbers are unavailable.

This does not fix #2734, and nothing here should be read as having found
its cause. The bundle's own evidence contradicts both proposed causes: the
orphan janitor ran 7 times in 26 days over 725 stream-ends and killed no
orphaned ffmpeg, which is not the #776 signature; and the 5 "database is
locked" errors all fall between two OOM kills, making them a symptom of the
memory pressure rather than a source of it.

CHANGELOG.md

@@ -25,6 +25,7 @@ All notable changes to Bambuddy will be documented in this file.
 - **Debug logs now record what the printer reports between the last layer and the end of a print (#2547, reporter @anthonyma94)** — The finish photo wants a moment that Bambu firmware does not obviously announce: printing done, toolhead parked, filament unload not yet started. Bambuddy has been driving that capture from `stg_cur=22` ("Filament unloading"), which turns out to fire on no model at all — across 247 support bundles there is not a single stage-22 capture, including the window in which it was the only trigger in the code, where all 104 captures on A1, A1 Mini, H2C, H2D, P1S, P2S, X1C and X2D fell through to the after-the-fact fallback. Choosing a replacement was not possible from the bundles we had, because outside `stg_cur` and `mc_print_sub_stage` every stage and action field the printers send is dropped unread, and the most promising candidates (`print_real_action`, `mc_action`, `mc_stage`) are absent from A1, A1 Mini and P1S payloads entirely. With debug logging enabled, Bambuddy now dumps those raw fields for the window between the last object layer and the end of the print — opening on the first end-of-print signal (last layer reached, progress at 99+, or no remaining time), logging only what changed frame to frame, and closing on the state transition — so a single debug bundle per model can show whether any firmware marks that moment. Diagnostics only: nothing reads these values, they are printer telemetry with nothing identifying in them, and at normal log levels the probe does no work at all. Covered by tests for the window boundaries, the frame budget and the guarantee that the probe cannot break status ingest.
 
 ### Added
+- **Support bundles now record Bambuddy's own memory, threads and child processes (#2734)** — A bundle described everything except the thing it runs in. That made reports of memory climbing over days impossible to act on: the numbers that identify what is actually growing only exist while it is happening, and by the time anyone asked, the container had been restarted. Bundles now carry resident and virtual memory, thread count, child processes by name, open files and sockets, process uptime, and a census of live objects by type. Those figures separate causes that look identical from outside — a large virtual size against a modest resident one is address space rather than data, a rising thread count points somewhere quite different from a rising child-process count, and the object census names what a growing heap is filling up with. The object census is skipped on processes already above 2 GB, because walking the heap costs most on exactly the process that can least afford it; everything else is still collected. Child processes are recorded by executable name only — an ffmpeg command line carries the camera URL and its password. Collection happens off the main loop and every metric is best-effort, so a hardened kernel or restricted container that refuses one of them still produces a complete bundle.
 - **Failure detection can now authenticate to a token-protected Obico ML API (#2733)** — Obico's `ml_api` container takes an optional `ML_API_TOKEN` environment variable; with it set, the container answers 401 to any detection request that doesn't carry that token, which is how you stop everything else on your network from using your inference server. Bambuddy never sent one, so the only way to use it was to remove the token from the server — a step the reporter had already taken for their Home Assistant setup and did not want to undo. **Settings → Failure Detection** now has an **ML API Token** field; leave it empty and requests go out exactly as before. Also worth knowing: this failed in the most confusing way possible, because Obico protects its detection endpoint but leaves its health endpoint open. Bambuddy's **Test** button pinged the open one, so it reported success against a server that was rejecting every real call, and detection just silently never fired. Test now checks both and says outright when a token is rejected, and the status card reports a rejected token as a rejected token rather than a bare HTTP error. Translated in all locales; wiki documents the token, the health-endpoint trap and how to recover from it.
 
 ### Fixed

backend/app/api/routes/support.py

@@ -9,6 +9,7 @@ import logging
 import os
 import platform
 import re
+import time
 import zipfile
 from datetime import datetime, timezone
 from pathlib import Path
@@ -300,6 +301,115 @@ def _get_container_memory_limit() -> int | None:
     return None
 
 
+# Above this RSS the heap census is skipped — see _collect_process_info.
+_GC_CENSUS_RSS_LIMIT = 2 * 1024**3
+
+
+def _collect_process_info() -> dict:
+    """Snapshot this process's resource usage, for reports about it growing.
+
+    Bundles used to carry nothing about Bambuddy's own footprint, which made
+    "memory climbs over days until the OOM killer fires" impossible to triage
+    from a bundle alone — the reporter of #2734 had to be asked to run commands
+    by hand, and the numbers that would have identified the mechanism could not
+    be recovered after the fact.
+
+    The four figures below separate the mechanisms that look identical from
+    outside:
+
+    * ``rss_bytes`` vs ``vms_bytes`` — a large virtual size against a modest
+      resident one is address space, not live data: thread stacks or allocator
+      arenas rather than a heap that keeps growing.
+    * ``num_threads`` — every leaked MQTT client reconnect would leave a paho
+      network thread behind, each reserving its stack.
+    * ``children`` — the ffmpeg-per-camera-stream leak class (#776).
+    * ``open_files`` / ``connections`` — descriptors held by streams or sockets
+      that were never closed.
+
+    Everything is best-effort: psutil raises on hardened kernels and inside
+    restricted containers, and a support bundle must still be produced when it
+    does. Child command lines are reduced to the executable name — a full
+    ffmpeg argv carries the camera URL, and with it the camera's password.
+    """
+    import psutil
+
+    out: dict = {}
+    try:
+        proc = psutil.Process()
+    except Exception:
+        return {"available": False}
+
+    out["available"] = True
+    try:
+        mem = proc.memory_info()
+        out["rss_bytes"] = mem.rss
+        out["rss_formatted"] = _format_bytes(mem.rss)
+        out["vms_bytes"] = mem.vms
+        out["vms_formatted"] = _format_bytes(mem.vms)
+    except Exception:
+        pass
+    try:
+        out["num_threads"] = proc.num_threads()
+    except Exception:
+        pass
+    try:
+        out["uptime_seconds"] = int(time.time() - proc.create_time())
+    except Exception:
+        pass
+    try:
+        out["open_files"] = len(proc.open_files())
+    except Exception:
+        pass
+    try:
+        out["connections"] = len(proc.net_connections(kind="inet"))
+    except Exception:
+        pass
+
+    # Children by executable name only. The count per name is what identifies a
+    # leak; the arguments would leak credentials.
+    try:
+        names: dict[str, int] = {}
+        for child in proc.children(recursive=True):
+            try:
+                names[child.name()] = names.get(child.name(), 0) + 1
+            except Exception:
+                names["<unknown>"] = names.get("<unknown>", 0) + 1
+        out["children_total"] = sum(names.values())
+        out["children_by_name"] = dict(sorted(names.items(), key=lambda kv: -kv[1]))
+    except Exception:
+        pass
+
+    # Live object counts by type, top 15. Identifies a heap that is growing and
+    # what it is growing with — the one thing RSS alone cannot say.
+    #
+    # Skipped above _GC_CENSUS_RSS_LIMIT. gc.get_objects() materialises a list
+    # of every tracked object, so the census costs most on exactly the process
+    # that can least afford it: a bundle generated to diagnose runaway memory
+    # must not be the allocation that tips the host over. The numbers that
+    # actually separate the mechanisms — RSS vs VMS, threads, children — are
+    # collected above and unaffected.
+    rss = out.get("rss_bytes")
+    if rss is not None and rss > _GC_CENSUS_RSS_LIMIT:
+        out["gc_census"] = (
+            f"skipped: process is using {_format_bytes(rss)}, above the "
+            f"{_format_bytes(_GC_CENSUS_RSS_LIMIT)} limit for walking the heap"
+        )
+        return out
+    try:
+        import gc
+
+        counts: dict[str, int] = {}
+        for obj in gc.get_objects():
+            name = type(obj).__name__
+            counts[name] = counts.get(name, 0) + 1
+        out["gc_tracked_objects"] = sum(counts.values())
+        out["gc_top_types"] = dict(sorted(counts.items(), key=lambda kv: -kv[1])[:15])
+    except Exception:
+        pass
+
+    return out
+
+
 def _format_bytes(size_bytes: int) -> str:
     """Format bytes into human-readable string."""
     if size_bytes < 1024:
@@ -699,6 +809,12 @@ async def _collect_support_info() -> dict:
         "database": {},
         "printers": [],
         "settings": {},
+        # Bambuddy's own footprint. Cheap to collect and the only thing that
+        # makes a "memory grows over days" report triageable from the bundle
+        # rather than a round trip of shell commands (#2734). Off the event
+        # loop: the heap census walks every tracked object, and a bundle
+        # request must not stall status ingest while it does.
+        "process": await asyncio.to_thread(_collect_process_info),
     }
 
     # Docker-specific info

backend/tests/unit/test_support_helpers.py

@@ -1472,3 +1472,104 @@ class TestSanitizePushStatusValues:
 
         assert raw == before, "input was mutated"
         assert out["tag_uid"] == "[SERIAL]"  # and the copy really was redacted
+
+
+class TestProcessInfo:
+    """Bambuddy's own footprint in the bundle (#2734).
+
+    Bundles carried nothing about the process itself, so "memory climbs over
+    days until the OOM killer fires" could not be triaged from a bundle — the
+    reporter had to run shell commands by hand, and the numbers that would have
+    named the mechanism were unrecoverable afterwards.
+    """
+
+    def test_reports_the_figures_that_separate_the_mechanisms(self):
+        """RSS vs VMS, threads and children distinguish a heap that is growing
+        from address space, a thread leak, and a child-process leak."""
+        from backend.app.api.routes.support import _collect_process_info
+
+        info = _collect_process_info()
+
+        assert info["available"] is True
+        for key in ("rss_bytes", "vms_bytes", "num_threads", "children_total"):
+            assert isinstance(info[key], int), key
+
+    def test_children_are_named_but_never_quoted(self):
+        """An ffmpeg argv carries the camera URL, and with it its password. The
+        count per executable is what identifies a leak; the arguments are not
+        needed and must not travel."""
+        from backend.app.api.routes.support import _collect_process_info
+
+        info = _collect_process_info()
+
+        for name in info.get("children_by_name", {}):
+            assert " " not in name, f"looks like a command line, not a name: {name!r}"
+            assert "://" not in name
+
+    def test_heap_census_is_skipped_on_a_large_process(self):
+        """gc.get_objects() materialises every tracked object, so the census
+        costs most on the process that can least afford it. A bundle generated
+        to diagnose runaway memory must not be the allocation that tips the
+        host over."""
+        from unittest.mock import MagicMock, patch
+
+        import backend.app.api.routes.support as support_module
+
+        fake = MagicMock()
+        fake.memory_info.return_value = MagicMock(rss=8 * 1024**3, vms=12 * 1024**3)
+        fake.num_threads.return_value = 40
+        fake.create_time.return_value = 0.0
+        fake.open_files.return_value = []
+        fake.net_connections.return_value = []
+        fake.children.return_value = []
+
+        with patch("psutil.Process", return_value=fake):
+            info = support_module._collect_process_info()
+
+        assert "gc_top_types" not in info
+        assert "skipped" in info["gc_census"]
+        # The discriminating numbers still come through — those are the point.
+        assert info["rss_bytes"] == 8 * 1024**3
+        assert info["num_threads"] == 40
+
+    def test_heap_census_runs_on_a_normal_process(self):
+        from backend.app.api.routes.support import _collect_process_info
+
+        info = _collect_process_info()
+
+        assert info["gc_tracked_objects"] > 0
+        assert len(info["gc_top_types"]) <= 15
+
+    def test_survives_a_hostile_psutil(self):
+        """psutil raises on hardened kernels and in restricted containers. A
+        support bundle must still be produced when it does — the bundle is how
+        someone reports the problem in the first place."""
+        from unittest.mock import patch
+
+        import backend.app.api.routes.support as support_module
+
+        with patch("psutil.Process", side_effect=RuntimeError("no /proc for you")):
+            info = support_module._collect_process_info()
+
+        assert info == {"available": False}
+
+    def test_partial_failures_do_not_lose_the_rest(self):
+        """One inaccessible metric must not cost the others."""
+        from unittest.mock import MagicMock, patch
+
+        import backend.app.api.routes.support as support_module
+
+        fake = MagicMock()
+        fake.memory_info.return_value = MagicMock(rss=100, vms=200)
+        fake.num_threads.side_effect = PermissionError("denied")
+        fake.create_time.return_value = 0.0
+        fake.open_files.side_effect = PermissionError("denied")
+        fake.net_connections.side_effect = PermissionError("denied")
+        fake.children.return_value = []
+
+        with patch("psutil.Process", return_value=fake):
+            info = support_module._collect_process_info()
+
+        assert info["rss_bytes"] == 100
+        assert "num_threads" not in info
+        assert info["children_total"] == 0