Browse Source

Long uploads: wait for the full file, no stale reconnect (#3538)

- An upload without a 226 confirmation is printed only once its size
  on the printer matches, read again for up to two minutes;
  otherwise it fails and the upload is retried
- MQTT silence while Bambuddy uploads to a printer, and within the
  stale timeout after, no longer forces a reconnect (queue and
  firmware uploads)
- Tests, changelog entry
maziggy 1 tuần trước cách đây
mục cha
commit
0262701dfb

+ 1 - 0
CHANGELOG.md

@@ -27,6 +27,7 @@ All notable changes to Bambuddy will be documented in this file.
 - **"Any model" jobs go to the printer with the fewest print hours (#3480, sponsor vote Q4 2026)** — When several printers of a model can take a job, the queue used to pick the first one in its list, so on a farm the same printer got most of the work. It now picks the one with the fewest print hours (the Maintenance counter, including baseline hours entered there), so jobs and wear spread across the farm. Preferred colours still come first, and the queue powers on the least used printer too when it has to switch one on with an Auto Power On plug.
 
 ### Fixed
+- **Long uploads to an A1 no longer end in a failed or unreadable print (#3538, reported by @SaadShaya7)** — Two problems at the handoff from a slow upload to the print command, seen with a 19 MB file that took three minutes. When the printer didn't confirm the upload, Bambuddy read the file size back, logged that the file was incomplete, and sent the print command anyway: the printer was still writing the file and either stayed idle or reported that it could not parse it. Bambuddy now keeps reading the size for up to two minutes, sends the print command once the file is complete, and otherwise retries the upload. And because the A1 sends no status while it receives a file, Bambuddy took the silence for a dead connection and reconnected just before the print command, which then failed the job. Silence during an upload, and in the minute after it, no longer counts as a dead connection; this also applies to firmware uploads.
 - **A failed slice for another printer is reported as failed (#3535, reported by @Jesse-Cachai)** — When the slicer failed on a 3MF with the picked printer, process and filament profiles, Bambuddy retried with the settings saved in the file and reported the slice as completed. Those settings are for the printer the file was made for, so a P1S project sliced for another printer came back as P1S G-code, with a believable print time and filament estimate. The retry now runs only when the file was made for the picked printer. For any other printer, or when either printer can't be told from the file or the profile, the slice fails with the slicer's error.
 - **A user moved out of Administrators is no longer an admin (#3533)** — Users carry an older admin role next to their groups, and either one made them an admin. The account created during setup has both, so moving it from Administrators to Operators left it with every admin right, still showing the Admin badge, and the user settings had no way to remove the role. The groups now decide: the role follows them wherever they change, in the user settings, on the groups, through the API and at LDAP or OIDC sign-in. Removing the last admin from Administrators is refused, as deactivating them already was. On the first start after the update, users who are not in Administrators lose the old admin role. If they are the only admins, they are added to Administrators instead.
 - **Spool assignments survive a printer reconnecting mid-print (#3526, reported by @SaadShaya7)** — After a short network drop, the first report of a reconnecting printer can show a slot with no filament type and colour `00000000`. Bambuddy only recognised an empty colour as a blank slot, so it read this as a different spool and removed the inventory assignment, even during a print. All four assignments on an A1 mini's AMS lite and an A2L's external spool were lost that way and had to be made again. A slot reporting `00000000` with no type now counts as blank: during a print the assignment is kept as a possible runout, otherwise it is unlinked only if the slot stays blank for two minutes. A clear filament, which also reports `00000000` but with its type, is not affected. The support package also no longer contains a virtual printer's serial number: for the A2L, P2S, H2 series and X2D models it appeared in full in the certificate log lines.

+ 55 - 17
backend/app/services/bambu_ftp.py

@@ -1300,9 +1300,16 @@ class BambuFTPClient:
                     e,
                     type(e).__name__,
                 )
-                if callback_exception is None:
-                    # A cancelled upload is deleted below; its size says nothing.
-                    self._check_unconfirmed_upload(remote_path, file_size)
+                # A cancelled upload is deleted below; its size says nothing.
+                if callback_exception is None and not self._check_unconfirmed_upload(remote_path, file_size):
+                    # Not complete on the printer: no print command for it.
+                    # NETWORK, so the caller's retry uploads it again (#3538).
+                    self.last_failure = FtpFailure(
+                        FtpFailureKind.NETWORK,
+                        f"{remote_path} did not reach its full size on the printer",
+                        None,
+                    )
+                    return False
 
             if callback_exception is not None:
                 cleanup_result: DeleteResult = DeleteResult.FAILED
@@ -1369,25 +1376,56 @@ class BambuFTPClient:
             self.last_failure = FtpFailure(kind, str(e), code)
             return False
 
-    def _check_unconfirmed_upload(self, remote_path: str, expected_size: int) -> None:
-        """Read an upload's size back when its 226 never came.
-
-        The upload still counts (an H2D can take 30+ s to answer), but a
-        printer that kept a partial file should not go unnoticed. A fresh
-        connection, because this one may be waiting for that late reply.
+    # How long an unconfirmed upload's size is watched for the printer to
+    # finish writing it out (#3538).
+    UNCONFIRMED_SIZE_POLL_INTERVAL = 5.0
+    UNCONFIRMED_SIZE_POLL_TIMEOUT = 120.0
+
+    def _check_unconfirmed_upload(self, remote_path: str, expected_size: int) -> bool:
+        """Read an upload's size back when its 226 never came. True if it may print.
+
+        A fresh connection, because this one may be waiting for that late
+        reply. An A1 can still be writing a large file out when the reply
+        times out: it read 0 bytes 2 s later and the full size minutes after,
+        and the print command sent in between made it refuse the file
+        (#3538). So a short size is read again until it matches or the time
+        runs out; only then is the upload a failure. A size that cannot be
+        read at all says nothing, and the upload counts as before.
         """
         server_size = _size_on_new_connection(self, remote_path)
         if server_size is None:
             logger.warning("FTP upload of %s could not be read back to check its size", remote_path)
-        elif server_size != expected_size:
-            logger.error(
-                "FTP upload of %s is %s bytes on the printer, expected %s: the file may be incomplete",
-                remote_path,
-                server_size,
-                expected_size,
-            )
-        else:
+            return True
+        if server_size == expected_size:
             logger.info("FTP upload of %s verified on a new connection (%s bytes)", remote_path, server_size)
+            return True
+        logger.info(
+            "FTP upload of %s is %s of %s bytes on the printer; waiting for it to finish writing",
+            remote_path,
+            server_size,
+            expected_size,
+        )
+        started = time.monotonic()
+        while time.monotonic() - started < self.UNCONFIRMED_SIZE_POLL_TIMEOUT:
+            time.sleep(self.UNCONFIRMED_SIZE_POLL_INTERVAL)
+            size = _size_on_new_connection(self, remote_path)
+            if size is not None:
+                server_size = size
+            if server_size == expected_size:
+                logger.info(
+                    "FTP upload of %s complete on the printer after %.0fs (%s bytes)",
+                    remote_path,
+                    time.monotonic() - started,
+                    server_size,
+                )
+                return True
+        logger.error(
+            "FTP upload of %s is %s bytes on the printer, expected %s: incomplete, not printing it",
+            remote_path,
+            server_size,
+            expected_size,
+        )
+        return False
 
     def delete_file(self, remote_path: str) -> DeleteResult:
         """Delete a file from the printer.

+ 24 - 2
backend/app/services/bambu_mqtt.py

@@ -1726,6 +1726,10 @@ class BambuMQTTClient:
         self._message_log: deque[MQTTLogEntry] = deque(maxlen=100)
         self._logging_enabled: bool = False
         self._last_message_time: float = 0.0  # Track when we last received a message
+        # Bambuddy's own FTP uploads to this printer (#3538): an A1 publishes
+        # nothing while it receives a large file, which is not a stale link.
+        self._uploads_in_flight: int = 0
+        self._last_upload_end: float = 0.0
         # Count of report-topic messages received since the last (re)connect.
         # Lets check_staleness() distinguish "printer never sent a status
         # report" (typically a wrong / mis-cased serial) from a normal quiet
@@ -1906,11 +1910,29 @@ class BambuMQTTClient:
     # Maximum time (seconds) without a message before considering connection stale
     STALE_TIMEOUT = 60.0
 
+    def begin_upload(self) -> None:
+        """Bambuddy starts an FTP upload to this printer (#3538)."""
+        self._uploads_in_flight += 1
+
+    def end_upload(self) -> None:
+        """The upload is over; the printer gets STALE_TIMEOUT from now to speak."""
+        self._uploads_in_flight = max(0, self._uploads_in_flight - 1)
+        self._last_upload_end = time.time()
+
     def is_stale(self) -> bool:
-        """Check if the connection is stale (no messages for too long)."""
+        """Check if the connection is stale (no messages for too long).
+
+        Not while Bambuddy is uploading to the printer, nor within
+        STALE_TIMEOUT after: an A1 says nothing over MQTT while it receives a
+        large file, and the forced reconnect that silence used to trigger
+        landed just before the print command, which then found the printer
+        disconnected and failed the job (#3538).
+        """
         if self._last_message_time == 0:
             return False  # Never received a message yet
-        time_since_last = time.time() - self._last_message_time
+        if self._uploads_in_flight > 0:
+            return False
+        time_since_last = time.time() - max(self._last_message_time, self._last_upload_end)
         return time_since_last > self.STALE_TIMEOUT
 
     def mark_power_off(self) -> bool:

+ 32 - 30
backend/app/services/firmware_update.py

@@ -353,36 +353,38 @@ class FirmwareUpdateService:
             # Get FTP retry settings
             ftp_retry_enabled, ftp_retry_count, ftp_retry_delay, ftp_timeout = await get_ftp_retry_settings()
 
-            if ftp_retry_enabled:
-                success = await with_ftp_retry(
-                    upload_file_async,
-                    ip_address,
-                    access_code,
-                    firmware_path,
-                    remote_path,
-                    progress_callback=on_upload_progress,
-                    socket_timeout=ftp_timeout,
-                    printer_model=model,
-                    # Someone pressed "update firmware" and is watching a
-                    # progress bar. Bounded and user-initiated, like a print
-                    # dispatch, so it does not spend its retries on a cool-off
-                    # meant for the background sweeps (#2898).
-                    respect_handshake_cooloff=False,
-                    max_retries=ftp_retry_count,
-                    retry_delay=ftp_retry_delay,
-                    operation_name=f"Upload firmware to printer {printer_id}",
-                )
-            else:
-                success = await upload_file_async(
-                    ip_address,
-                    access_code,
-                    firmware_path,
-                    remote_path,
-                    progress_callback=on_upload_progress,
-                    socket_timeout=ftp_timeout,
-                    printer_model=model,
-                    respect_handshake_cooloff=False,
-                )
+            # Quiet MQTT while the printer receives the file is no stale link (#3538).
+            with printer_manager.uploading(printer_id):
+                if ftp_retry_enabled:
+                    success = await with_ftp_retry(
+                        upload_file_async,
+                        ip_address,
+                        access_code,
+                        firmware_path,
+                        remote_path,
+                        progress_callback=on_upload_progress,
+                        socket_timeout=ftp_timeout,
+                        printer_model=model,
+                        # Someone pressed "update firmware" and is watching a
+                        # progress bar. Bounded and user-initiated, like a print
+                        # dispatch, so it does not spend its retries on a cool-off
+                        # meant for the background sweeps (#2898).
+                        respect_handshake_cooloff=False,
+                        max_retries=ftp_retry_count,
+                        retry_delay=ftp_retry_delay,
+                        operation_name=f"Upload firmware to printer {printer_id}",
+                    )
+                else:
+                    success = await upload_file_async(
+                        ip_address,
+                        access_code,
+                        firmware_path,
+                        remote_path,
+                        progress_callback=on_upload_progress,
+                        socket_timeout=ftp_timeout,
+                        printer_model=model,
+                        respect_handshake_cooloff=False,
+                    )
 
             if not success:
                 raise Exception("Failed to upload firmware to printer")

+ 31 - 28
backend/app/services/print_scheduler.py

@@ -9568,34 +9568,37 @@ class PrintScheduler:
         upload_failure = FtpFailureReport()
 
         try:
-            if ftp_retry_enabled:
-                uploaded = await with_ftp_retry(
-                    upload_file_async,
-                    printer.ip_address,
-                    printer.access_code,
-                    file_path,
-                    remote_path,
-                    socket_timeout=ftp_timeout,
-                    printer_model=printer.model,
-                    progress_callback=progress_bridge,
-                    respect_handshake_cooloff=False,
-                    failure=upload_failure,
-                    max_retries=ftp_retry_count,
-                    retry_delay=ftp_retry_delay,
-                    operation_name=f"Upload print to {printer.name}",
-                )
-            else:
-                uploaded = await upload_file_async(
-                    printer.ip_address,
-                    printer.access_code,
-                    file_path,
-                    remote_path,
-                    socket_timeout=ftp_timeout,
-                    printer_model=printer.model,
-                    progress_callback=progress_bridge,
-                    respect_handshake_cooloff=False,
-                    failure=upload_failure,
-                )
+            # The printer's MQTT goes quiet while it receives the file; the
+            # stale check must not reconnect it under the print command (#3538).
+            with printer_manager.uploading(item.printer_id):
+                if ftp_retry_enabled:
+                    uploaded = await with_ftp_retry(
+                        upload_file_async,
+                        printer.ip_address,
+                        printer.access_code,
+                        file_path,
+                        remote_path,
+                        socket_timeout=ftp_timeout,
+                        printer_model=printer.model,
+                        progress_callback=progress_bridge,
+                        respect_handshake_cooloff=False,
+                        failure=upload_failure,
+                        max_retries=ftp_retry_count,
+                        retry_delay=ftp_retry_delay,
+                        operation_name=f"Upload print to {printer.name}",
+                    )
+                else:
+                    uploaded = await upload_file_async(
+                        printer.ip_address,
+                        printer.access_code,
+                        file_path,
+                        remote_path,
+                        socket_timeout=ftp_timeout,
+                        printer_model=printer.model,
+                        progress_callback=progress_bridge,
+                        respect_handshake_cooloff=False,
+                        failure=upload_failure,
+                    )
         except UploadCancelled as e:
             uploaded = False
             upload_error = (

+ 18 - 1
backend/app/services/printer_manager.py

@@ -3,7 +3,8 @@ import logging
 import math
 import re
 import traceback
-from collections.abc import Callable
+from collections.abc import Callable, Iterator
+from contextlib import contextmanager
 
 from sqlalchemy import select
 from sqlalchemy.ext.asyncio import AsyncSession
@@ -982,6 +983,22 @@ class PrinterManager:
         """Get the MQTT client for a printer."""
         return self._clients.get(printer_id)
 
+    @contextmanager
+    def uploading(self, printer_id: int | None) -> Iterator[None]:
+        """Bambuddy is uploading to this printer: its MQTT silence is expected (#3538).
+
+        Ends on the client it began on, so a client replaced meanwhile is
+        neither left counting nor decremented.
+        """
+        client = self._clients.get(printer_id) if printer_id is not None else None
+        if client is not None:
+            client.begin_upload()
+        try:
+            yield
+        finally:
+            if client is not None:
+                client.end_upload()
+
     def mark_printer_offline(self, printer_id: int):
         """Mark a printer as offline and trigger status callback.
 

+ 25 - 7
backend/tests/unit/services/test_bambu_ftp.py

@@ -452,13 +452,25 @@ class TestUpload:
         assert result is False
         client.disconnect()
 
-    @pytest.mark.parametrize(("server_size", "logged_error"), [(100, True), (1024, False), (None, False)])
+    @pytest.mark.parametrize(
+        ("sizes", "uploaded", "logged_error"),
+        [
+            ([1024], True, False),  # complete at the first read
+            ([None], True, False),  # unreadable: nothing to judge by, proceeds as before
+            # #3538: an A1 still writing the file out reads short at first,
+            # then reaches the full size.
+            ([0, 0, 512, 1024], True, False),
+            # Never complete: no print command for it.
+            ([100], False, True),
+        ],
+    )
     def test_upload_without_a_226_is_checked_on_a_new_connection(
-        self, ftp_client_factory, ftp_server, tmp_path, caplog, server_size, logged_error
+        self, ftp_client_factory, ftp_server, tmp_path, caplog, sizes, uploaded, logged_error
     ):
         """No 226 within the wait (an H2D can take 30+ s) still lets the upload
-        proceed, but its size is read back on a fresh connection, since the
-        old one may be out of step, and a mismatch is logged as an error."""
+        proceed once its size is read back on a fresh connection, since the
+        old one may be out of step. A short size is read again until it
+        matches; one that never does fails the upload so it is retried."""
         import logging
 
         import backend.app.services.bambu_ftp as ftp_mod
@@ -467,6 +479,8 @@ class TestUpload:
         local.write_bytes(b"data" * 256)
         client = ftp_client_factory()
         client.connect()
+        client.UNCONFIRMED_SIZE_POLL_INTERVAL = 0
+        client.UNCONFIRMED_SIZE_POLL_TIMEOUT = 0.2
 
         def never_answers():
             raise TimeoutError("timed out")
@@ -476,16 +490,20 @@ class TestUpload:
 
         def fake_size(c, remote_path):
             checked.append(remote_path)
-            return server_size
+            return sizes[min(len(checked), len(sizes)) - 1]
 
         with (
             patch.object(ftp_mod, "_size_on_new_connection", fake_size),
             caplog.at_level(logging.INFO, logger="backend.app.services.bambu_ftp"),
         ):
-            assert client.upload_file(local, "/cache/test.bin") is True
+            assert client.upload_file(local, "/cache/test.bin") is uploaded
         client.disconnect()
 
-        assert checked == ["/cache/test.bin"]
+        assert set(checked) == {"/cache/test.bin"}
+        if uploaded:
+            assert len(checked) == len(sizes)
+        else:
+            assert client.last_failure.kind is ftp_mod.FtpFailureKind.NETWORK
         errors = [r for r in caplog.records if r.levelno >= logging.ERROR]
         assert bool(errors) is logged_error
 

+ 141 - 0
backend/tests/unit/test_upload_quiet_not_stale_3538.py

@@ -0,0 +1,141 @@
+"""A printer receiving Bambuddy's upload is quiet, not stale (#3538).
+
+An A1 publishes nothing over MQTT while it receives a large FTPS upload. A
+19 MB file took three minutes, the stale check force-closed the connection
+42 ms before the print command, and the command found the printer
+disconnected and failed the job. Silence during an upload, and within
+STALE_TIMEOUT after it, no longer counts.
+"""
+
+import time
+from contextlib import ExitStack
+from unittest.mock import AsyncMock, MagicMock, patch
+
+import pytest
+
+from backend.app.services.bambu_mqtt import BambuMQTTClient
+from backend.app.services.printer_manager import PrinterManager
+from backend.tests.unit.test_scheduler_busy_defer_2598 import dispatch_case  # noqa: F401 - fixture
+
+
+def _silent_client(silence: float) -> BambuMQTTClient:
+    client = BambuMQTTClient(ip_address="10.0.0.1", serial_number="A1", access_code="12345678")
+    client.state.connected = True
+    client._last_message_time = time.time() - silence
+    return client
+
+
+class TestStaleCheck:
+    def test_a_silent_printer_is_still_stale(self):
+        assert _silent_client(BambuMQTTClient.STALE_TIMEOUT + 10).is_stale()
+
+    def test_not_stale_while_an_upload_is_running(self):
+        client = _silent_client(BambuMQTTClient.STALE_TIMEOUT + 120)
+        client.begin_upload()
+
+        assert not client.is_stale()
+
+    def test_the_timeout_counts_from_the_end_of_the_upload(self):
+        client = _silent_client(BambuMQTTClient.STALE_TIMEOUT + 120)
+        client.begin_upload()
+        client.end_upload()
+
+        assert not client.is_stale(), "the printer gets STALE_TIMEOUT after the upload to speak"
+
+        client._last_upload_end = time.time() - BambuMQTTClient.STALE_TIMEOUT - 1
+        assert client.is_stale(), "silence long after the upload is staleness again"
+
+    def test_no_forced_reconnect_under_the_upload(self):
+        client = _silent_client(BambuMQTTClient.STALE_TIMEOUT + 120)
+        client.force_reconnect_stale_session = MagicMock()
+        client._client = MagicMock()
+        client.begin_upload()
+
+        assert client.check_staleness() is True
+        assert client.state.connected
+        client._client.disconnect.assert_not_called()
+
+    def test_overlapping_uploads_keep_it_quiet_until_the_last_ends(self):
+        client = _silent_client(BambuMQTTClient.STALE_TIMEOUT + 120)
+        client.begin_upload()
+        client.begin_upload()
+        client.end_upload()
+
+        assert client._uploads_in_flight == 1
+        assert not client.is_stale()
+
+
+class TestUploadingContext:
+    def test_marks_the_printers_client_for_the_upload(self):
+        manager = PrinterManager()
+        client = _silent_client(0)
+        manager._clients[7] = client
+
+        with manager.uploading(7):
+            assert client._uploads_in_flight == 1
+        assert client._uploads_in_flight == 0
+        assert client._last_upload_end > 0
+
+    def test_ends_on_the_client_it_began_on(self):
+        manager = PrinterManager()
+        old, new = _silent_client(0), _silent_client(0)
+        manager._clients[7] = old
+
+        with manager.uploading(7):
+            manager._clients[7] = new  # reconnect replaced the client meanwhile
+        assert old._uploads_in_flight == 0
+        assert new._uploads_in_flight == 0
+
+    def test_ends_when_the_upload_raises(self):
+        manager = PrinterManager()
+        client = _silent_client(0)
+        manager._clients[7] = client
+
+        with pytest.raises(RuntimeError), manager.uploading(7):
+            raise RuntimeError("upload failed")
+        assert client._uploads_in_flight == 0
+
+    def test_an_unknown_printer_is_a_no_op(self):
+        with PrinterManager().uploading(None), PrinterManager().uploading(99):
+            pass
+
+
+@pytest.mark.asyncio
+async def test_the_queue_marks_its_upload(dispatch_case):
+    """The queue's upload runs inside the uploading window of its printer."""
+    from backend.app.models.print_queue import PrintQueueItem
+    from backend.app.services.print_scheduler import PrintScheduler, printer_manager
+    from backend.tests.unit.test_scheduler_busy_defer_2598 import _base_patches
+
+    scheduler = PrintScheduler()
+    client = _silent_client(0)
+    seen: list[int] = []
+
+    async def upload(*args, **kwargs):
+        seen.append(client._uploads_in_flight)
+        return True
+
+    get_status = MagicMock(return_value=MagicMock(state="IDLE", subtask_id=None, gcode_file=None))
+    async with dispatch_case.session_maker() as db:
+        item = await db.get(PrintQueueItem, dispatch_case.ids.item_id)
+        with ExitStack() as stack:
+            for p in _base_patches(scheduler, dispatch_case, AsyncMock(side_effect=upload), MagicMock(), get_status):
+                stack.enter_context(p)
+            stack.enter_context(patch.dict(printer_manager._clients, {dispatch_case.ids.printer_id: client}))
+            await scheduler._start_print(db, item)
+
+    assert seen == [1]
+    assert client._uploads_in_flight == 0
+
+
+@pytest.mark.asyncio
+async def test_the_firmware_upload_marks_its_upload():
+    """The firmware update's upload is quiet for the printer the same way."""
+    import inspect
+
+    from backend.app.services import firmware_update
+
+    source = inspect.getsource(firmware_update)
+    upload_at = source.index("success = await with_ftp_retry(")
+    window_at = source.rindex("with printer_manager.uploading(printer_id):", 0, upload_at)
+    assert upload_at - window_at < 200