Queue: a refused card or access code no longer eats the queue

- A 553/552 (storage) or rejected access code failed the job, but left
  the printer idle, so the next job went to it and failed the same
  way. Both now join the #3210 requeue: the job stays pending and the
  printer is out of dispatch for 5 min, doubling up to an hour.
- A 553 for a file name no card can hold still fails the job, so one
  badly named file cannot block every printer of its model.
- "Any <model>" jobs refused by their printer go back to the pool for
  another printer of the model. Only the assignment is cleared.
- Rows say why: "refused to store the file - check its SD card or USB
  drive", or the access code; a released model job names the refusing
  printer instead of "Busy".
- Job Waiting is sent by the refusal: once per run, again if the reason
  changes, none for a job cancelled during the upload.
- The 553 message names the USB drive too (H2D).

CHANGELOG.md

@@ -200,6 +200,12 @@ All notable changes to Bambuddy will be documented in this file.
 - **The frontend build no longer warns about `path` and `crypto` being externalized for the STEP previewer (#2976)** — `occt-import-js`, the Emscripten build behind STEP previews, requires both modules, but only inside its `ENVIRONMENT_IS_NODE` branches; in the browser it loads its `.wasm` from the URL the preview worker passes and draws randomness from `crypto.getRandomValues`. Vite still externalized both and printed two warnings on every build. `vite.config.ts` now drops exactly those two warnings for that one package through `build.rolldownOptions.onLog`, so an externalization anywhere else, or of any other module, still shows.
 
 ### Fixed
+- **A printer whose SD card or USB drive refuses the file no longer eats the queue (#3409, reported by @simonkreuchauff-oss)** — When a printer refused an upload with 553 or 552 (storage missing, full or unusable), or rejected the access code, the job failed. The printer was still idle, so the next job went to it and failed the same way: an H2D with an unusable USB stick put seven jobs in History within an hour.
+  - **The job stays in the queue:** these refusals now work like a printer whose file service doesn't answer (#3210). The printer gets no jobs for five minutes, then the wait doubles with each refusal in a row, up to an hour, and the job is sent again when the wait is over. A job assigned to that printer shows what to check: "refused to store the file — check its SD card or USB drive", or its access code on the Printers page.
+  - **"Any <model>" jobs move on:** a job for any printer of a model now goes back to the pool when its printer refuses the file, so another printer of that model can take it. This also applies to the refusals #3210 already kept in the queue. While it waits, its row names the refusing printer and why, instead of "Busy".
+  - **A file name no card can hold still fails the job:** a 553 for a name with a character such as `:` or `?` would be refused on every printer, so that job fails with a request to rename the file, and the printer stays in service.
+  - **Notifications:** a **Job Waiting** notification goes out on the first refusal of a run, also when no job is left waiting on that printer, and again if the run turns into something to fix (a refused card after a dropped connection). None for a job cancelled during the upload. These refusals used to send **Job Failed**.
+  - The upload error now says "SD card or USB drive", as an H2D's external storage is a USB drive. Wiki updated.
 - **Spoolman: the stock forecast no longer borrows a built-in spool's usage history (#3406, reported by @ojimpo)** — The Forecast view read Bambuddy's own usage history in every mode. That history belongs to the built-in inventory and is keyed by its spool ids, so after switching to Spoolman, a Spoolman spool whose id matched an old built-in spool took over its prints: a spool using 2.5 g a day could show 46 g a day ("Trend") and run out in days. In Spoolman mode the forecast now works from each spool's own consumption, as the stock alerts already did.
 - **Spoolman: an external-spool print on an X2D or H2D is charged to the holder it ran from (#3405, reported by @juandelarossa)** — A print command that carries no `ams_mapping2` (#3166) writes the external spool as -1. In Spoolman mode that charged the left holder (254) whenever the printer listed it, which the X2D always does, empty or not, and in the daily builds no spool at all, so a spool on the right holder (255) was never charged. The built-in inventory treats such a mapping as no mapping and lets the printer's `tray_now` decide, and Spoolman mode now does the same, for finished and for aborted prints. In both modes `tray_now` 255 now also counts as the right-hand holder ahead of the last tray loaded before the print, which a print from that holder never updates, so the AMS tray or left holder an earlier print used was charged instead.
 - **A printer that does not answer the K-profile read is no longer a printer without K-profiles (#3404, reported by @Spegeli)** — When the printer did not answer any of the three requests, the read returned an empty table, the same as a printer with no profiles for that nozzle. The K-Profiles page then said "No K-Profiles" and offered to create the first one, and the Configure AMS Slot dialog offered only the default K. Worse, a Git restore matched the backed-up profiles against that empty table and sent every one of them as a new profile, next to the originals still on the printer. The read is now an error: the API answers HTTP 504 "The printer did not answer the K-profile request", the K-Profiles page says the profiles could not be read and offers Refresh, and the slot dialog says so above the default. A restore skips that nozzle, counts its profiles failed and says the printer did not answer. The read-back after an edit no longer has to treat an empty table as a possible timeout.

backend/app/services/bambu_ftp.py

@@ -1288,7 +1288,8 @@ class BambuFTPClient:
             # 452 is "insufficient storage" -- a 4xx, so ftplib raises it as
             # error_temp rather than error_perm, but it is the printer talking
             # about its card just as 552/553 are. As NETWORK it would read as a
-            # connection problem, and the queue would retry it forever (#3210).
+            # connection problem, and the queue would say it retries by itself
+            # instead of asking for the card to be checked (#3210, #3409).
             is_storage = isinstance(e, ftplib.Error) and code == "452"
             kind = FtpFailureKind.STORAGE if is_storage else FtpFailureKind.NETWORK
             self.last_failure = FtpFailure(kind, str(e), code)
@@ -1450,7 +1451,7 @@ def describe_upload_failure(failure: FtpFailure | None) -> str:
     if failure.kind is FtpFailureKind.STORAGE:
         return (
             f"The printer refused to store the file ({failure.code or 'storage error'}). Check that its SD card "
-            "is inserted, has space free, and is formatted FAT32 or exFAT."
+            "or USB drive is inserted, has space free, and is formatted FAT32 or exFAT."
         )
     if failure.kind is FtpFailureKind.HANDSHAKE:
         return (

backend/app/services/print_scheduler.py

@@ -76,7 +76,7 @@ from backend.app.utils.ams_humidity import ams_humidity_percent
 from backend.app.utils.archive_paths import archive_photos_dir
 from backend.app.utils.color_utils import perceptual_color_distance
 from backend.app.utils.filament_types import canonical_filament_type
-from backend.app.utils.filename import derive_remote_filename
+from backend.app.utils.filename import InvalidFilenameError, derive_remote_filename, validate_print_filename
 from backend.app.utils.library_paths import move_library_photos
 from backend.app.utils.local_time import utcnow_naive
 from backend.app.utils.printer_models import (
@@ -375,14 +375,60 @@ DISPATCH_MAX_ATTEMPTS = 3
 # item, which failed the same way. One P2S whose file service was out of
 # connection slots ate 43 queued jobs in ten minutes that way.
 #
-# Not here: AUTH (a wrong access code needs the user), STORAGE (a full or
-# missing card does too), NOT_FOUND (a Bambuddy-side path problem), UNKNOWN,
-# and an upload that overran its deadline -- each of those would fail the same
-# way on every retry.
+# STORAGE (a missing, full or unusable card or USB drive) and AUTH (a rejected
+# access code) were left out at first, because they need the user. But
+# failing the item leaves the printer just as idle, and #3409's H2D with an
+# unusable USB stick ate seven jobs within an hour. Every job sent to that
+# printer fails the same way, so they wait too; the retry finds the fix once
+# the user has made it.
+#
+# A 553 can also be this one file: a name the card's filesystem cannot hold.
+# That one fails the item, see `_start_print`.
+#
+# Not here: NOT_FOUND (a path problem, which can be this one file), UNKNOWN,
+# and an upload that overran its deadline -- each retry of that costs a whole
+# upload that would be just as slow.
 _UPLOAD_REQUEUE_KINDS: frozenset[FtpFailureKind] = frozenset(
-    {FtpFailureKind.HANDSHAKE, FtpFailureKind.COOLOFF, FtpFailureKind.TIMEOUT, FtpFailureKind.NETWORK}
+    {
+        FtpFailureKind.HANDSHAKE,
+        FtpFailureKind.COOLOFF,
+        FtpFailureKind.TIMEOUT,
+        FtpFailureKind.NETWORK,
+        FtpFailureKind.STORAGE,
+        FtpFailureKind.AUTH,
+    }
 )
 
+
+def _refused_upload_hold_reason(printer_label: str, kind: FtpFailureKind | None) -> str:
+    """What a job waiting on a printer that refused its last upload shows.
+
+    Storage and the access code need the user to do something, so they say
+    what; the rest clear on their own.
+    """
+    if kind is FtpFailureKind.STORAGE:
+        return f"{printer_label} refused to store the file — check its SD card or USB drive. {_REFUSED_UPLOAD_TAIL}"
+    if kind is FtpFailureKind.AUTH:
+        return (
+            f"{printer_label} refused the file transfer — check its access code on the Printers page. "
+            f"{_REFUSED_UPLOAD_TAIL}"
+        )
+    return f"{printer_label} is not accepting files — {_REFUSED_UPLOAD_TAIL}"
+
+
+_REFUSED_UPLOAD_TAIL = "Bambuddy will retry automatically"
+
+
+def _asks_nothing_new(waiting_reason: str) -> bool:
+    """A model job's reason made only of busy printers and printers that refused an upload.
+
+    The refusal sent its own notification (`_requeue_after_upload_refused`),
+    so this one has nothing to add.
+    """
+    parts = [p.strip() for p in waiting_reason.split(" | ")]
+    return all(p.startswith("Busy:") or p.endswith(_REFUSED_UPLOAD_TAIL) for p in parts)
+
+
 # How long a printer whose upload was put back stays out of dispatch (#3210).
 # The first window matches the FTP client's own handshake cool-off, so the
 # retry lands after the client would talk to the printer again anyway. Each
@@ -1233,12 +1279,14 @@ class PrintScheduler:
         # Refused uploads in a row per printer, which sets the next backoff
         # window. Cleared by a successful upload to that printer.
         self._upload_refusals: dict[int, int] = {}
-        # Printers whose current run of refusals has already sent its one
-        # "job waiting" notification. Each retry clears the item's waiting
-        # reason and the next refusal sets it again, which `hold_item` reads as
-        # a new reason; without this, every retry would notify. Cleared with
-        # `_upload_refusals`.
-        self._upload_refusal_notified: set[int] = set()
+        # Why each printer in `_upload_backoff` refused its last upload, for
+        # the reason its jobs show. Cleared with `_upload_refusals`.
+        self._upload_refusal_kind: dict[int, FtpFailureKind] = {}
+        # The "job waiting" text last sent for each printer's current run of
+        # refusals. A refusal notifies only when its text differs, so a run
+        # notifies once, and again only if it turns into something the user
+        # has to fix. Cleared with `_upload_refusals`.
+        self._upload_refusal_notified: dict[int, str] = {}
         # Track which printers are currently auto-drying (printer_id -> start timestamp)
         self._drying_in_progress: dict[int, float] = {}
         # Per-AMS memory of the auto-drying cycles WE armed, keyed by
@@ -1570,6 +1618,8 @@ class PrintScheduler:
         for keyed in (
             self._upload_backoff,
             self._upload_refusals,
+            self._upload_refusal_kind,
+            self._upload_refusal_notified,
             self._wake_failures,
             self._drying_in_progress,
             self._dispatch_holds,
@@ -1578,7 +1628,6 @@ class PrintScheduler:
             self._terminal_since,
         ):
             keyed.pop(printer_id, None)
-        self._upload_refusal_notified.discard(printer_id)
         for keyed in (self._auto_dry_units, self._auto_dry_above):
             for key in [key for key in keyed if key[0] == printer_id]:
                 del keyed[key]
@@ -1886,26 +1935,21 @@ class PrintScheduler:
             now_mono = time.monotonic()
             # In backoff this pass. Kept apart from busy_printers so keep-warm
             # can leave them out: there is no print to keep a bed warm for.
+            # Their holds are written silently: the refusal that put a printer
+            # here sent the "job waiting" notification for its run (#3409).
             backoff_printers: set[int] = set()
-            # In backoff, and the "job waiting" notification for this run of
-            # refusals has gone out already. Their holds are written silently.
-            silent_hold_printers: set[int] = set()
             for backoff_pid, retry_at in list(self._upload_backoff.items()):
                 if now_mono >= retry_at:
                     del self._upload_backoff[backoff_pid]
                     continue
                 backoff_printers.add(backoff_pid)
-                if backoff_pid in self._upload_refusal_notified:
-                    silent_hold_printers.add(backoff_pid)
-                else:
-                    self._upload_refusal_notified.add(backoff_pid)
                 mark_busy(
                     backoff_pid,
                     f"its file service refused the last upload; retrying in {retry_at - now_mono:.0f}s",
                 )
                 item_hold_reasons.setdefault(
                     backoff_pid,
-                    f"{printer_label(backoff_pid)} is not accepting files — Bambuddy will retry automatically",
+                    _refused_upload_hold_reason(printer_label(backoff_pid), self._upload_refusal_kind.get(backoff_pid)),
                 )
 
             # Printers held by a Home Assistant sensor interlock (#1148) — an
@@ -2105,7 +2149,7 @@ class PrintScheduler:
                         await hold_item(
                             item,
                             item_hold_reasons.get(item.printer_id) or f"Busy: {printer_label(item.printer_id)}",
-                            notify=item.printer_id not in silent_hold_printers,
+                            notify=item.printer_id not in backoff_printers,
                         )
                         continue
 
@@ -2401,6 +2445,7 @@ class PrintScheduler:
                                 is_short=is_short,
                                 drying_ids=drying_held,
                                 unmatched_on=unmatched_on,
+                                refused_uploads={pid: self._upload_refusal_kind.get(pid) for pid in backoff_printers},
                             )
                             # "Only if the previous print succeeded" is about the
                             # printer the job lands on: one whose last print failed
@@ -2474,7 +2519,7 @@ class PrintScheduler:
                         if (
                             waiting_reason
                             and (not was_waiting or self._is_busy_only(previous_reason or ""))
-                            and not self._is_busy_only(waiting_reason)
+                            and not _asks_nothing_new(waiting_reason)
                         ):
                             job_name = await self._get_job_name(db, item)
                             await notification_service.on_queue_job_waiting(
@@ -2833,52 +2878,106 @@ class PrintScheduler:
         printer: Printer,
         error_msg: str,
         toast_uid: int | None,
+        kind: FtpFailureKind,
     ) -> None:
         """Put an item back in the queue after its file never reached the printer (#3210).
 
-        The item keeps its printer. Its AMS mapping was computed against that
-        printer's trays, and nothing on the row says whether a mapping was
-        computed or set by the user, so moving it to a sibling could print from
-        the wrong slots. The printer goes into `_upload_backoff` instead, which
-        keeps every other item away from it; this one waits there and is the
-        only thing that knocks again, once per backoff window. The window
-        grows with each refusal in a row; see ``_upload_backoff_seconds``.
+        The printer goes into `_upload_backoff`, which keeps every item away
+        from it until the window ends. The window grows with each refusal in a
+        row; see ``_upload_backoff_seconds``.
+
+        An item pinned to this printer keeps it, waits there, and is the one
+        that knocks again. An item for "Any <model>" goes back to the pool
+        instead, so another printer of the model can take it (#3409). Only the
+        assignment goes: the next model assignment drops whatever on the row
+        was worked out for one printer, as it does for every model job.
+
+        The first refusal in a run sends the one "Job waiting" notification,
+        and a later one only if it asks for something else (a card after a
+        dropped connection). It cannot wait for the hold the way it used to: a
+        released item is not held on this printer, so nothing would say why
+        it stopped. A job cancelled or moved during the upload sends none.
 
         `dispatch_attempts` is not charged: that budget bounds a printer that
         takes the file and then never starts, which this is not.
         """
         refusals = self._upload_refusals.get(printer.id, 0) + 1
         self._upload_refusals[printer.id] = refusals
+        self._upload_refusal_kind[printer.id] = kind
         backoff = _upload_backoff_seconds(refusals)
         self._upload_backoff[printer.id] = time.monotonic() + backoff
-        # The row is still `pending` -- it only moves to `printing` after a
-        # successful upload -- so there is no status to write back. Writing one
-        # anyway would undo a cancel that landed during the upload.
-        if item.error_message:
+        released = False
+        if item.target_model:
+            # Only while it is still pending on this printer: a cancel, or a
+            # move the user made during the upload, stands.
+            result = cast(
+                CursorResult,
+                await db.execute(
+                    update(PrintQueueItem)
+                    .where(PrintQueueItem.id == item.id)
+                    .where(PrintQueueItem.status == "pending")
+                    .where(PrintQueueItem.printer_id == printer.id)
+                    .values(printer_id=None, error_message=None)
+                    .execution_options(synchronize_session=False)
+                ),
+            )
+            await db.commit()
+            released = result.rowcount > 0
+        elif item.error_message:
+            # The row is still `pending` -- it only moves to `printing` after a
+            # successful upload -- so there is no status to write back. Writing
+            # one anyway would undo a cancel that landed during the upload.
             item.error_message = None
             await db.commit()
         logger.warning(
-            "Queue item %s: upload to printer %s (%s) never reached it — %s Kept in the queue; "
+            "Queue item %s: upload to printer %s (%s) never reached it — %s Kept in the queue%s; "
             "refusal %d in a row, so the printer is out of dispatch for %ds.",
             item.id,
             printer.id,
             printer.name,
             error_msg,
+            " and returned to the pool" if released else "",
             refusals,
             backoff,
         )
+        reason = _refused_upload_hold_reason(printer.name or f"printer {printer.id}", kind)
+        if self._upload_refusal_notified.get(printer.id) != reason and await self._still_waiting_on(
+            db, item.id, None if released else printer.id
+        ):
+            self._upload_refusal_notified[printer.id] = reason
+            try:
+                await notification_service.on_queue_job_waiting(
+                    job_name=await self._get_job_name(db, item),
+                    target_model=item.target_model or printer.model or "",
+                    waiting_reason=reason,
+                    db=db,
+                    printer_id=printer.id,
+                )
+            except Exception as e:
+                # The job is safe in the queue either way.
+                logger.debug("Waiting notification failed for item %s: %s", item.id, e)
         try:
             # Closes the dispatch toast for this attempt. Without it the toast
             # keeps spinning on an upload that has ended.
             await ws_manager.send_queue_item_failed(
                 user_id=toast_uid,
                 queue_item_id=item.id,
-                printer_id=item.printer_id,
+                printer_id=printer.id,
                 reason="upload_failed",
             )
         except Exception:
             pass  # toast is best-effort
 
+    @staticmethod
+    async def _still_waiting_on(db: AsyncSession, item_id: int, printer_id: int | None) -> bool:
+        """Whether the item is still pending, on *printer_id* (None: unassigned)."""
+        row = (
+            await db.execute(
+                select(PrintQueueItem.status, PrintQueueItem.printer_id).where(PrintQueueItem.id == item_id)
+            )
+        ).one_or_none()
+        return row is not None and row.status == "pending" and row.printer_id == printer_id
+
     def _rollback_unconfirmed_print_user(self, item_id: int) -> None:
         printer_id = self._unconfirmed_print_user.pop(item_id, None)
         if printer_id is not None:
@@ -3307,6 +3406,7 @@ class PrintScheduler:
         is_short: Callable[[int], Awaitable[bool]] | None = None,
         drying_ids: set[int] | None = None,
         unmatched_on: Callable[[int], Awaitable[list[str]]] | None = None,
+        refused_uploads: Mapping[int, FtpFailureKind | None] | None = None,
     ) -> tuple[int | None, str | None]:
         """Find an idle, connected printer matching the model with compatible filaments.
 
@@ -3336,6 +3436,9 @@ class PrintScheduler:
                           passed over like one missing a filament type: the checks
                           above only ask whether each type and forced colour is
                           loaded somewhere, not on the right nozzle or once per slot.
+            refused_uploads: Excluded printers out of dispatch after a refused upload,
+                             with why (#3409). Each is named with the reason its own
+                             queue rows show, not as "Busy".
 
         Returns:
             Tuple of (printer_id, waiting_reason):
@@ -3355,6 +3458,9 @@ class PrintScheduler:
 
         # Track reasons for skipping printers
         printers_busy = []
+        # Printers out of dispatch after a refused upload, worded as the
+        # printer's own queue rows word it: "Busy" would read as printing (#3409).
+        printers_refused: list[str] = []
         printers_offline = []
         printers_offline_no_plug = []
         printers_missing_filament: list[tuple[str, list[str]]] = []
@@ -3364,6 +3470,9 @@ class PrintScheduler:
         short_fallback: int | None = None
 
         for printer in printers:
+            if printer.id in exclude_ids and refused_uploads and printer.id in refused_uploads:
+                printers_refused.append(_refused_upload_hold_reason(printer.name, refused_uploads[printer.id]))
+                continue
             if printer.id in exclude_ids:
                 # Printer is already claimed by another job in this scheduling run.
                 # For force-color jobs, still check if the color would match — if not,
@@ -3515,7 +3624,7 @@ class PrintScheduler:
                 # by itself, so the job is not actually waiting on anybody to change a
                 # spool (#2876 — offline printers reach this list now that a switched-off
                 # printer's own filament is read).
-                if not printers_busy and not printers_offline:
+                if not printers_busy and not printers_offline and not printers_refused:
                     all_missing = sorted({c for _, cols in printers_missing_filament for c in cols})
                     return None, f"No matching material/color. Waiting on {', '.join(all_missing)}"
                 # else: fall through — the self-resolving entries are appended below
@@ -3524,6 +3633,7 @@ class PrintScheduler:
                     f"{name} (needs {', '.join(missing)})" for name, missing in printers_missing_filament
                 ]
                 reasons.append(f"Waiting for filament: {'; '.join(names_and_missing)}")
+        reasons.extend(printers_refused)
         if printers_busy:
             reasons.append(f"Busy: {', '.join(printers_busy)}")
         if printers_offline:
@@ -9283,8 +9393,17 @@ class PrintScheduler:
             # handshake that never reached the printer's filesystem (#2899).
             error_msg = upload_error or describe_upload_failure(upload_failure.failure)
             failure = upload_failure.failure
+            if failure is not None and failure.kind is FtpFailureKind.STORAGE:
+                try:
+                    validate_print_filename(remote_filename)
+                except InvalidFilenameError as e:
+                    # This 553 is the file's name, which no card can hold, not
+                    # the printer's storage: it would be refused on every retry
+                    # and every printer, so this item fails and the queue goes on.
+                    upload_error = f"The printer could not create the file {remote_filename!r}: {e}. Rename the file."
+                    error_msg = upload_error
             if upload_error is None and failure is not None and failure.kind in _UPLOAD_REQUEUE_KINDS:
-                await self._requeue_after_upload_refused(db, item, printer, error_msg, toast_uid)
+                await self._requeue_after_upload_refused(db, item, printer, error_msg, toast_uid, failure.kind)
                 return
             if not await self._fail_dispatch(db, item, error_msg):
                 await self._close_cancelled_dispatch_toast(item, toast_uid)
@@ -9319,7 +9438,8 @@ class PrintScheduler:
         # The printer took a file, so any run of refused uploads is over: the
         # next refusal starts again at the shortest backoff, and notifies (#3210).
         self._upload_refusals.pop(printer.id, None)
-        self._upload_refusal_notified.discard(printer.id)
+        self._upload_refusal_kind.pop(printer.id, None)
+        self._upload_refusal_notified.pop(printer.id, None)
 
         # Parse AMS mapping if stored
         ams_mapping = None

backend/tests/unit/services/test_upload_failure_reason_2899.py

@@ -292,6 +292,11 @@ class TestTheWording:
         assert "553" in message
         assert "FAT32" in message
 
+    def test_the_storage_message_names_the_usb_drive_too(self):
+        """An H2D's external storage is a USB drive, not a card (#3409)."""
+        message = describe_upload_failure(FtpFailure(FtpFailureKind.STORAGE, "553 Could not create file.", "553"))
+        assert "SD card or USB drive" in message
+
 
 # ---------------------------------------------------------------------------
 # End to end: what the queue entry says
@@ -403,20 +408,17 @@ class TestWhatTheQueueEntrySays:
     # A handshake failure used to be this class's lead case. Since #3210 it no
     # longer fails the item at all -- the file never reached the printer, so the
     # item goes back in the queue (test_scheduler_upload_requeue_3210.py). Its
-    # wording is still pinned by TestTheWording above.
+    # wording is still pinned by TestTheWording above. Since #3409 the same
+    # goes for a 553: the job waits for the card instead of failing.
 
-    async def test_a_553_still_gets_the_card_advice(self, dispatch_case):
-        """The advice was written for this case and belongs to it.
-
-        Removing it everywhere would trade one wrong message for a vaguer one;
-        the point is to attach it where the printer actually said storage.
-        """
+    async def test_a_550_says_what_it_is_and_not_the_card(self, dispatch_case):
+        """A failure that still fails the item gets its own sentence, on the row and in the notification."""
         message, reason = await _dispatch_failing_with(
-            dispatch_case, FtpFailure(FtpFailureKind.STORAGE, "553 Could not create file.", "553")
+            dispatch_case, FtpFailure(FtpFailureKind.NOT_FOUND, "550 Permission denied.", "550")
         )
 
-        assert "FAT32" in message, message
-        assert "553" in message, message
+        assert "Bambuddy-side" in message, message
+        assert "FAT32" not in message, message
         assert reason == message
 
     async def test_an_unclassified_failure_points_at_the_log(self, dispatch_case):
@@ -454,7 +456,7 @@ class TestAnOwnerDeletedDuringTheUpload:
                 await admin_db.commit()
                 admin_db.add(User(id=alice_id, username="bob", password_hash="x", role="user"))
                 await admin_db.commit()
-            kwargs["failure"].failure = FtpFailure(FtpFailureKind.STORAGE, "553 Could not create file.", "553")
+            kwargs["failure"].failure = FtpFailure(FtpFailureKind.NOT_FOUND, "550 Permission denied.", "550")
             return False
 
         failed = AsyncMock()

backend/tests/unit/test_forget_printer.py

@@ -224,12 +224,14 @@ async def test_the_plate_references_of_a_printer_go(tmp_path, monkeypatch):
 
 
 async def test_the_scheduler_and_live_activities_forget_it():
+    from backend.app.services.bambu_ftp import FtpFailureKind
     from backend.app.services.notify_live_activities import notify_live_activities
     from backend.app.services.print_scheduler import scheduler
 
     scheduler._upload_backoff[GONE] = 1e12
     scheduler._upload_refusals[GONE] = 6
-    scheduler._upload_refusal_notified.add(GONE)
+    scheduler._upload_refusal_notified[GONE] = "printer is not accepting files"
+    scheduler._upload_refusal_kind[GONE] = FtpFailureKind.STORAGE
     scheduler._wake_failures[GONE] = 1e12
     scheduler._dispatch_holds[GONE] = (1.0, "IDLE", None)
     scheduler._auto_dry_units[(GONE, 0)] = {}
@@ -242,6 +244,7 @@ async def test_the_scheduler_and_live_activities_forget_it():
     assert GONE not in scheduler._upload_backoff
     assert GONE not in scheduler._upload_refusals
     assert GONE not in scheduler._upload_refusal_notified
+    assert GONE not in scheduler._upload_refusal_kind
     assert GONE not in scheduler._wake_failures
     assert GONE not in scheduler._dispatch_holds
     assert (GONE, 0) not in scheduler._auto_dry_units

backend/tests/unit/test_scheduler_upload_requeue_3210.py

@@ -10,9 +10,9 @@ gone in ten minutes, none printed on that printer.
 Two changes, pinned here:
 
 * An upload whose file never reached the printer (handshake refused, cool-off,
-  timeout, dropped connection) puts the item back in the queue. Failures that
-  would repeat on every retry -- a rejected access code, a full card -- still
-  fail it.
+  timeout, dropped connection, and since #3409 a refused card or access code)
+  puts the item back in the queue. A path problem, an unknown failure and an
+  upload too slow to finish still fail it.
 * The printer is out of dispatch for a backoff window, so nothing else is
   sent to it, and the item that was put back waits there with a reason.
 """
@@ -69,14 +69,16 @@ async def farm(tmp_path):
 
     counter = iter(range(1000))
 
-    async def add_item(*, printer_id: int | None = None, target_model: str | None = None) -> int:
+    async def add_item(
+        *, printer_id: int | None = None, target_model: str | None = None, filename: str | None = None
+    ) -> int:
         n = next(counter)
         async with session_maker() as db:
             archive_rel = Path("archives") / f"job-{n}.3mf"
             (base_dir / archive_rel).write_bytes(b"archive payload")
             archive = PrintArchive(
                 printer_id=printer_id,
-                filename=f"job-{n}.3mf",
+                filename=filename or f"job-{n}.3mf",
                 file_path=str(archive_rel),
                 file_size=15,
                 status="completed",
@@ -199,7 +201,14 @@ class TestOneFailedUpload:
     @pytest.mark.asyncio
     @pytest.mark.parametrize(
         "kind",
-        [FtpFailureKind.HANDSHAKE, FtpFailureKind.COOLOFF, FtpFailureKind.TIMEOUT, FtpFailureKind.NETWORK],
+        [
+            FtpFailureKind.HANDSHAKE,
+            FtpFailureKind.COOLOFF,
+            FtpFailureKind.TIMEOUT,
+            FtpFailureKind.NETWORK,
+            FtpFailureKind.STORAGE,
+            FtpFailureKind.AUTH,
+        ],
     )
     async def test_a_file_that_never_arrived_keeps_the_item(self, farm, kind):
         item_id = await farm.add_item(printer_id=farm.ids.refusing)
@@ -220,16 +229,14 @@ class TestOneFailedUpload:
     @pytest.mark.parametrize(
         "failure",
         [
-            FtpFailure(FtpFailureKind.AUTH, "530 Login incorrect.", "530"),
-            FtpFailure(FtpFailureKind.STORAGE, "553 Could not create file.", "553"),
             FtpFailure(FtpFailureKind.NOT_FOUND, "550 Permission denied.", "550"),
             FtpFailure(FtpFailureKind.UNKNOWN, "500 what"),
             None,
         ],
-        ids=["auth", "storage", "not_found", "unknown", "unreported"],
+        ids=["not_found", "unknown", "unreported"],
     )
     async def test_a_failure_that_would_repeat_still_fails_the_item(self, farm, failure):
-        """Retrying a wrong access code or a full card every five minutes helps no one."""
+        """A 550 can be this one file's path, and an unknown failure says nothing about the printer."""
         item_id = await farm.add_item(printer_id=farm.ids.refusing)
 
         async def upload(*_args, **kwargs):
@@ -287,9 +294,10 @@ class TestTheQueueIsNotDrained:
     async def test_any_model_items_stop_going_to_the_refusing_printer(self, farm):
         """#3210's shape: "any P2S" items, one P2S refusing every upload.
 
-        Pass 1 sends one item to each printer; the refusing one keeps its item.
-        Pass 2 has the healthy printer busy printing and the refusing one in
-        backoff, so the rest wait instead of being fed to it one by one.
+        Pass 1 sends one item to each printer; the refusing one gives its item
+        back to the pool (#3409). Pass 2 has the healthy printer busy printing
+        and the refusing one in backoff, so the rest wait instead of being fed
+        to it one by one.
         """
         ids = [await farm.add_item(target_model="P2S") for _ in range(5)]
         upload = _upload_refused_by(REFUSING_IP, HANDSHAKE)
@@ -304,11 +312,9 @@ class TestTheQueueIsNotDrained:
         items = [await _item(farm, i) for i in ids]
         assert [i.status for i in items].count("failed") == 0, [(i.id, i.status, i.error_message) for i in items]
         assert [i.status for i in items].count("printing") == 1
-        held = [i for i in items if i.printer_id == farm.ids.refusing]
-        assert len(held) == 1, "exactly one item waits on the refusing printer"
-        assert held[0].status == "pending"
+        assert not [i for i in items if i.printer_id == farm.ids.refusing], "nothing waits on the refusing printer"
         waiting = [i for i in items if i.status == "pending" and i.printer_id is None]
-        assert len(waiting) == 3, "the rest stay unassigned, free for any printer that comes free"
+        assert len(waiting) == 4, "the rest stay unassigned, free for any printer that comes free"
 
     @pytest.mark.asyncio
     async def test_the_item_kept_on_the_printer_says_why(self, farm):
@@ -516,3 +522,236 @@ class TestA452IsAStorageReply:
     def test_a_socket_error_is_never_read_as_a_reply_code(self, tmp_path):
         failure = self._upload_raising(OSError("452 looks like a code but is not a reply"), tmp_path)
         assert failure.kind is FtpFailureKind.NETWORK
+
+
+# ---------------------------------------------------------------------------
+# #3409: a printer whose card refuses the file, or whose access code is wrong
+# ---------------------------------------------------------------------------
+STORAGE = FtpFailure(FtpFailureKind.STORAGE, "553 Could not create file.", "553")
+AUTH = FtpFailure(FtpFailureKind.AUTH, "530 Login incorrect.", "530")
+
+
+def _counting(upload):
+    """*upload*, counting the calls that reached the refusing printer."""
+    calls = []
+
+    async def _upload(ip_address, *args, **kwargs):
+        if ip_address == REFUSING_IP:
+            calls.append(ip_address)
+        return await upload(ip_address, *args, **kwargs)
+
+    _upload.calls = calls
+    return _upload
+
+
+class TestARefusedCard:
+    @pytest.mark.asyncio
+    async def test_jobs_pinned_to_the_printer_wait_instead_of_failing_one_by_one(self, farm):
+        """#3409's shape: an H2D with an unusable USB stick, seven jobs failed within an hour."""
+        ids = [await farm.add_item(printer_id=farm.ids.refusing) for _ in range(4)]
+        upload = _counting(_upload_refused_by(REFUSING_IP, STORAGE))
+        scheduler = PrintScheduler()
+
+        for _ in range(4):
+            async with _scheduler(farm, upload, scheduler=scheduler):
+                await scheduler.check_queue()
+
+        items = [await _item(farm, i) for i in ids]
+        assert [i.status for i in items] == ["pending"] * 4, [(i.id, i.status, i.error_message) for i in items]
+        assert len(upload.calls) == 1, "one refusal, then the printer is out of dispatch"
+        assert items[0].waiting_reason == (
+            "P2S-8 refused to store the file — check its SD card or USB drive. Bambuddy will retry automatically"
+        )
+
+    @pytest.mark.asyncio
+    async def test_a_wrong_access_code_says_where_to_fix_it(self, farm):
+        item_id = await farm.add_item(printer_id=farm.ids.refusing)
+        upload = _upload_refused_by(REFUSING_IP, AUTH)
+        scheduler = PrintScheduler()
+
+        for _ in range(2):
+            async with _scheduler(farm, upload, scheduler=scheduler):
+                await scheduler.check_queue()
+
+        item = await _item(farm, item_id)
+        assert item.status == "pending"
+        assert "access code on the Printers page" in item.waiting_reason
+
+    @pytest.mark.asyncio
+    async def test_the_job_is_sent_once_the_card_is_back(self, farm):
+        item_id = await farm.add_item(printer_id=farm.ids.refusing)
+        scheduler = PrintScheduler()
+
+        async with _scheduler(farm, _upload_refused_by(REFUSING_IP, STORAGE), scheduler=scheduler):
+            await scheduler.check_queue()
+        _expire_backoff(scheduler, farm.ids.refusing)
+        async with _scheduler(farm, AsyncMock(return_value=True), scheduler=scheduler):
+            await scheduler.check_queue()
+
+        assert (await _item(farm, item_id)).status == "printing"
+        assert farm.ids.refusing not in scheduler._upload_refusal_kind
+
+
+class TestAnyModelJobsMoveOn:
+    @pytest.mark.asyncio
+    async def test_the_job_goes_to_another_printer_of_the_model(self, farm):
+        """ "Any H2D" means any: the refused job should not wait for the broken one."""
+        item_id = await farm.add_item(target_model="P2S")
+        upload = _upload_refused_by(REFUSING_IP, STORAGE)
+        scheduler = PrintScheduler()
+
+        # Only the refusing printer is free, so it gets the job and refuses it.
+        async with _scheduler(farm, upload, scheduler=scheduler, busy={farm.ids.healthy}):
+            await scheduler.check_queue()
+        item = await _item(farm, item_id)
+        assert item.status == "pending"
+        assert item.printer_id is None, "back in the pool, not waiting on the refusing printer"
+
+        # The healthy printer comes free.
+        async with _scheduler(farm, upload, scheduler=scheduler):
+            await scheduler.check_queue()
+        item = await _item(farm, item_id)
+        assert item.status == "printing"
+        assert item.printer_id == farm.ids.healthy
+
+    @pytest.mark.asyncio
+    async def test_what_the_job_carries_for_any_printer_stays(self, farm):
+        """Only the assignment goes; the next model assignment sorts out the rest, as for every model job."""
+        item_id = await farm.add_item(target_model="P2S")
+        async with farm.session_maker() as db:
+            row = await db.get(PrintQueueItem, item_id)
+            row.skip_filament_check = True
+            row.nozzle_rack_choice = '{"0": 1}'
+            row.nozzle_mapping = "[0, 1]"
+            await db.commit()
+
+        async with _scheduler(farm, _upload_refused_by(REFUSING_IP, STORAGE), busy={farm.ids.healthy}) as scheduler:
+            await scheduler.check_queue()
+
+        item = await _item(farm, item_id)
+        assert item.printer_id is None
+        assert item.skip_filament_check is True
+        assert item.nozzle_rack_choice == '{"0": 1}'
+        assert item.nozzle_mapping == "[0, 1]"
+
+    @pytest.mark.asyncio
+    async def test_the_released_job_says_why_rather_than_busy(self, farm):
+        """With the refusing printer in backoff, "Busy: P2S-8" would read as printing."""
+        item_id = await farm.add_item(target_model="P2S")
+        upload = _upload_refused_by(REFUSING_IP, STORAGE)
+        scheduler = PrintScheduler()
+        waiting = AsyncMock()
+
+        for _ in range(3):
+            async with _scheduler(farm, upload, scheduler=scheduler, busy={farm.ids.healthy}, waiting_notify=waiting):
+                await scheduler.check_queue()
+
+        item = await _item(farm, item_id)
+        assert item.waiting_reason == (
+            "P2S-8 refused to store the file — check its SD card or USB drive. Bambuddy will retry automatically"
+            " | Busy: P2S-9"
+        )
+        assert waiting.await_count == 1, "the refusal said it already; the row's reason adds nothing new"
+
+    @pytest.mark.asyncio
+    async def test_a_cancel_during_the_upload_is_not_undone(self, farm):
+        item_id = await farm.add_item(target_model="P2S")
+
+        async def upload(ip_address, *_args, **kwargs):
+            async with farm.session_maker() as other:
+                row = await other.get(PrintQueueItem, item_id)
+                row.status = "cancelled"
+                await other.commit()
+            kwargs["failure"].failure = STORAGE
+            return False
+
+        async with _scheduler(farm, upload, busy={farm.ids.healthy}) as scheduler:
+            await scheduler.check_queue()
+
+        item = await _item(farm, item_id)
+        assert item.status == "cancelled"
+        assert item.printer_id == farm.ids.refusing, "a cancelled row is left as the cancel wrote it"
+
+    @pytest.mark.asyncio
+    async def test_a_released_job_still_says_why_once(self, farm):
+        """Nothing waits on the printer to carry the notification, so the refusal sends it."""
+        await farm.add_item(target_model="P2S")
+        upload = _upload_refused_by(REFUSING_IP, STORAGE)
+        scheduler = PrintScheduler()
+        waiting = AsyncMock()
+
+        for _ in range(3):
+            async with _scheduler(farm, upload, scheduler=scheduler, busy={farm.ids.healthy}, waiting_notify=waiting):
+                await scheduler.check_queue()
+            _expire_backoff(scheduler, farm.ids.refusing)
+
+        assert waiting.await_count == 1
+        kwargs = waiting.await_args.kwargs
+        assert "SD card or USB drive" in kwargs["waiting_reason"]
+        assert kwargs["printer_id"] == farm.ids.refusing
+
+
+class TestNotifyingARun:
+    @pytest.mark.asyncio
+    async def test_a_run_that_turns_into_a_card_problem_says_so(self, farm):
+        """ "Will retry automatically" asks nothing of anyone; a refused card does."""
+        await farm.add_item(printer_id=farm.ids.refusing)
+        scheduler = PrintScheduler()
+        waiting = AsyncMock()
+
+        for failure in (HANDSHAKE, FtpFailure(FtpFailureKind.COOLOFF, "cooloff"), STORAGE, STORAGE):
+            async with _scheduler(
+                farm, _upload_refused_by(REFUSING_IP, failure), scheduler=scheduler, waiting_notify=waiting
+            ):
+                await scheduler.check_queue()
+            _expire_backoff(scheduler, farm.ids.refusing)
+
+        reasons = [c.kwargs["waiting_reason"] for c in waiting.await_args_list]
+        assert len(reasons) == 2, reasons
+        assert "not accepting files" in reasons[0]
+        assert "SD card or USB drive" in reasons[1]
+
+    @pytest.mark.asyncio
+    @pytest.mark.parametrize("target_model", [None, "P2S"], ids=["pinned", "any_model"])
+    async def test_a_job_cancelled_during_the_upload_sends_none(self, farm, target_model):
+        item_id = await farm.add_item(
+            printer_id=farm.ids.refusing if target_model is None else None, target_model=target_model
+        )
+
+        async def upload(ip_address, *_args, **kwargs):
+            async with farm.session_maker() as other:
+                row = await other.get(PrintQueueItem, item_id)
+                row.status = "cancelled"
+                await other.commit()
+            kwargs["failure"].failure = STORAGE
+            return False
+
+        waiting = AsyncMock()
+        async with _scheduler(farm, upload, busy={farm.ids.healthy}, waiting_notify=waiting) as scheduler:
+            await scheduler.check_queue()
+
+        waiting.assert_not_awaited()
+        assert farm.ids.refusing in scheduler._upload_backoff, "the printer still refused"
+
+
+class TestAFileNameNoCardCanHold:
+    @pytest.mark.asyncio
+    @pytest.mark.parametrize("target_model", [None, "P2S"], ids=["pinned", "any_model"])
+    async def test_its_553_fails_the_job_and_leaves_the_printer_in_service(self, farm, target_model):
+        """A ':' is refused by every card on every printer: waiting would block the queue for good."""
+        bad = await farm.add_item(
+            printer_id=farm.ids.refusing if target_model is None else None,
+            target_model=target_model,
+            filename="Bracket: v2.3mf",
+        )
+        upload = _upload_refused_by(REFUSING_IP, STORAGE)
+
+        async with _scheduler(farm, upload, busy={farm.ids.healthy}) as scheduler:
+            await scheduler.check_queue()
+
+        item = await _item(farm, bad)
+        assert item.status == "failed"
+        assert "Rename the file" in item.error_message
+        assert "contains invalid character: :" in item.error_message
+        scheduler.failed_notify.assert_awaited_once()
+        assert farm.ids.refusing not in scheduler._upload_backoff