fix(db): give older PostgreSQL databases the column types a new install has

On PostgreSQL databases older than the columns, migrations created
active_print_spoolman.tray_remain_start as TEXT where the model has JSON,
and spool.weight_used_baseline and smart_plugs.rest_energy_total_multiplier
as REAL where the model has double precision. The TEXT column read back as
a string, so reporting usage to Spoolman failed after the 3MF charge: prints
without a 3MF charged nothing, and archives got no colour, material or cost
from their Spoolman spools.

A new migration converts the columns while they still have the old type.
Values that aren't valid JSON (NaN included) are cleared first, so they
can't stop the start. REAL goes through text with extra_float_digits = 1,
so 0.001 stays 0.001 instead of carrying float4's error.

The upgrade check gets a scenario that seeds valid and invalid JSON and
float values under extra_float_digits = 0 and checks them after the
upgrade.

CHANGELOG.md

@@ -162,6 +162,7 @@ All notable changes to Bambuddy will be documented in this file.
 
 ### Fixed
 - **PostgreSQL databases from older versions get the links a new install has** — A database created before v0.2.4.1 (or before v1.2.5.2 for the last one) was missing three links: from a print log entry to its archive and to the user who started it, and from an archive to the library file it was printed from. Deleting an archive, a user or a library file left their ID behind instead of clearing it, so such entries didn't show up under "no user", for example. The links are added on the next start, and IDs of things already deleted are cleared. A print log entry without an archive also stays that way now: every start used to attach it to the newest archive with the same name on the same printer, which could be someone else's. Apart from that, SQLite installs are unchanged.
+- **Spoolman: colour, material and cost from the spool weren't applied on older PostgreSQL installs** — On a PostgreSQL database created before v0.2.4.8, the start snapshot of the AMS levels was stored as text instead of JSON and read back as text. Reporting usage to Spoolman then failed after charging the spools from the 3MF, with "Spoolman usage reporting failed" in the log. Prints without a 3MF charged nothing, and no archive got the colour, material and cost of its Spoolman spools. The column is converted on the next start. Two number columns also only had single precision on PostgreSQL databases older than them and are widened, keeping the values as entered: a spool's "Reset usage" baseline (databases created before v0.2.4.2) and a REST smart plug's energy total multiplier (before v1.2.5). SQLite installs aren't affected.
 - **Every install now gets the dependency versions that were tested** — Bambuddy's Python dependencies only had minimum versions, so each Docker build, install and update took whatever was newest that day: a fresh install could run library versions nobody had tested Bambuddy with (SQLAlchemy 2.1 arrived that way). They are now pinned in `requirements.lock`, which Docker, the installers, the update scripts and the tests all use, and newer versions are taken on deliberately, after testing. Nothing changes for an install today: the pins are the versions a fresh install gets now. A native install moves to the pinned versions with the update after the one that brings this change, since an update still runs the previous version of the update script.
 - **Native installs need Python 3.11 or newer** — Since v0.2.4.8 Bambuddy hasn't started on Python 3.10, while the installer, README and wiki still said 3.10 was enough. The requirement now says 3.11. The installer uses the system's `python3` when it is new enough and otherwise looks for a newer Python next to it: on Ubuntu 22.04, whose `python3` is 3.10, it installs and uses the `python3.11` package. An environment left from an older Python is rebuilt. The update scripts check the install's Python first and, if it is too old, stop before changing anything and link to the steps for moving it to 3.11. Docker and the Windows installer ship Python 3.13 and are not affected. Wiki updated.
 - **The webhook endpoints listed in Settings didn't exist** — **Settings → API Keys → Webhook Endpoints** listed endpoints for the status of all printers, pausing and resuming that Bambuddy doesn't have, and two others under the wrong path. It now lists the real ones: add to the queue, start the next queued job, stop, cancel, printer status and queue status. Wiki updated.

backend/app/core/database.py

@@ -5423,6 +5423,86 @@ async def _run_migrations(conn):
 
     await _migrate_auth_datetimes_to_timestamp(conn)
     await _migrate_add_missing_set_null_fks(conn)
+    await _migrate_pg_column_types(conn)
+
+
+# Columns a migration added with another type than the model declares, so a
+# PostgreSQL database older than the column has the wrong one:
+# (table, column, the type it has there, the model's type, USING expression).
+# REAL goes through text, which PostgreSQL renders as the shortest decimal that
+# reads back as the same float4: 0.001 stays 0.001 instead of becoming
+# 0.0010000000474974513 as a double.
+_PG_COLUMN_TYPE_FIXES = (
+    ("active_print_spoolman", "tray_remain_start", "text", "JSON", "CAST(tray_remain_start AS JSON)"),
+    (
+        "spool",
+        "weight_used_baseline",
+        "real",
+        "DOUBLE PRECISION",
+        "CAST(CAST(weight_used_baseline AS TEXT) AS DOUBLE PRECISION)",
+    ),
+    (
+        "smart_plugs",
+        "rest_energy_total_multiplier",
+        "real",
+        "DOUBLE PRECISION",
+        "CAST(CAST(rest_energy_total_multiplier AS TEXT) AS DOUBLE PRECISION)",
+    ),
+)
+
+
+def _reject_non_json_constant(name: str):
+    """json.loads accepts NaN and Infinity; PostgreSQL's json type doesn't."""
+    raise ValueError(f"{name} is not JSON")
+
+
+async def _migrate_pg_column_types(conn) -> None:
+    """Convert the columns in _PG_COLUMN_TYPE_FIXES to the model's type.
+
+    PostgreSQL only: on SQLite REAL is already a double, and a JSON column is
+    text that SQLAlchemy decodes. Only a column that still has the old type is
+    converted. On PostgreSQL a TEXT column declared JSON in the model reads
+    back as a string, not a dict.
+
+    A JSON value that doesn't parse would fail the conversion, and with it
+    every start, so such values are cleared first. tray_remain_start only
+    holds the start snapshot of prints running at the time.
+    """
+    import json
+
+    from sqlalchemy import text
+
+    if is_sqlite():
+        return
+    for table, column, old_type, new_type, using in _PG_COLUMN_TYPE_FIXES:
+        current = (
+            await conn.execute(
+                text(
+                    "SELECT data_type FROM information_schema.columns "
+                    "WHERE table_schema = current_schema() AND table_name = :t AND column_name = :c"
+                ),
+                {"t": table, "c": column},
+            )
+        ).scalar_one_or_none()
+        if current != old_type:
+            continue
+        if new_type == "JSON":
+            rows = await conn.execute(text(f"SELECT id, {column} FROM {table} WHERE {column} IS NOT NULL"))
+            for row_id, value in rows.all():
+                try:
+                    json.loads(value, parse_constant=_reject_non_json_constant)
+                except ValueError:
+                    logger.warning("Clearing %s.%s of row %s: not valid JSON", table, column, row_id)
+                    await conn.execute(text(f"UPDATE {table} SET {column} = NULL WHERE id = :id"), {"id": row_id})
+        # Float to text gives the shortest exact decimal only with
+        # extra_float_digits >= 1, PostgreSQL's default since 12, which a server,
+        # database or role setting can lower: at 0 text keeps 6 digits.
+        digits = (await conn.execute(text("SHOW extra_float_digits"))).scalar_one()
+        await conn.execute(text("SELECT set_config('extra_float_digits', '1', true)"))
+        try:
+            await _safe_execute(conn, f"ALTER TABLE {table} ALTER COLUMN {column} TYPE {new_type} USING {using}")
+        finally:
+            await conn.execute(text("SELECT set_config('extra_float_digits', :d, true)"), {"d": digits})
 
 
 # Foreign keys that the migrations adding these columns left out, so a

scripts/check_postgres_upgrade.py

@@ -106,6 +106,60 @@ SCENARIOS = (
                 OR NOT EXISTS (SELECT 1 FROM print_archives WHERE id = 1 AND library_file_id = 1)
         """,
     ),
+    Scenario(
+        name="column types",
+        # v1.2.5.7 has these as TEXT and REAL in a v0.2.3 database, where a
+        # fresh install has JSON and DOUBLE PRECISION. The values must survive
+        # the conversion as the user saw them, and invalid JSON values must not
+        # stop the upgrade (they are cleared). extra_float_digits = 0, which a
+        # DBA may set, must not cost the floats their digits.
+        chain=("v0.2.3", "v1.2.5.7"),
+        seed="""
+            DO $$ BEGIN
+                EXECUTE format('ALTER DATABASE %I SET extra_float_digits = 0', current_database());
+            END $$;
+            INSERT INTO printers (id, name, serial_number, ip_address, access_code, auto_archive,
+                                  awaiting_plate_clear, camera_rotation, external_camera_enabled, is_active,
+                                  nozzle_count, plate_detection_enabled, print_hours_offset, runtime_seconds)
+            VALUES (1, 'p', 'S1', '192.0.2.1', 'x', true, false, 0, false, true, 1, false, 0, 0);
+            INSERT INTO print_archives (id, printer_id, filename, file_path, file_size, status, quantity, is_favorite)
+            VALUES (1, 1, 'a.3mf', 'a/a.3mf', 1, 'completed', 1, false),
+                   (2, 1, 'b.3mf', 'a/b.3mf', 1, 'completed', 1, false),
+                   (3, 1, 'c.3mf', 'a/c.3mf', 1, 'completed', 1, false);
+            INSERT INTO active_print_spoolman (id, printer_id, archive_id, ams_trays, tray_remain_start)
+            VALUES (1, 1, 1, '{}', '{"0-1": {"remain": 80}}'),
+                   (2, 1, 2, '{}', 'not json'),
+                   (3, 1, 3, '{}', '{"0-1": {"remain": 80, "tray_uuid": NaN}}');
+            INSERT INTO smart_plugs (id, name, plug_type, auto_off, auto_off_executed, auto_off_pending,
+                                     auto_off_persistent, auto_on, enabled, mqtt_energy_multiplier, mqtt_multiplier,
+                                     mqtt_power_multiplier, off_delay_minutes, off_delay_mode, off_temp_threshold,
+                                     power_alert_enabled, schedule_enabled, show_in_switchbar, show_on_printer_card,
+                                     rest_energy_total_multiplier)
+            VALUES (1, 'plug', 'rest', false, false, false, false, false, true, 1, 1, 1, 0, 'time', 70,
+                    false, false, false, false, 0.001);
+            INSERT INTO spool (id, material, core_weight, label_weight, weight_locked, weight_used,
+                               weight_used_baseline)
+            VALUES (1, 'PLA', 250, 1000, false, 1234.567, 1234.567);
+        """,
+        check="""
+            SELECT 'active_print_spoolman.tray_remain_start is ' || pg_typeof(tray_remain_start) || ', not json'
+            FROM active_print_spoolman WHERE id = 1 AND pg_typeof(tray_remain_start)::text <> 'json'
+            UNION ALL
+            SELECT 'active_print_spoolman.tray_remain_start changed: ' || tray_remain_start::text
+            FROM active_print_spoolman
+            WHERE id = 1 AND tray_remain_start::text::jsonb IS DISTINCT FROM '{"0-1": {"remain": 80}}'::jsonb
+            UNION ALL
+            SELECT 'an invalid tray_remain_start was kept: ' || tray_remain_start::text
+            FROM active_print_spoolman WHERE id IN (2, 3) AND tray_remain_start IS NOT NULL
+            UNION ALL
+            SELECT 'smart_plugs.rest_energy_total_multiplier is ' || rest_energy_total_multiplier::text
+            FROM smart_plugs
+            WHERE id = 1 AND rest_energy_total_multiplier IS DISTINCT FROM CAST(0.001 AS double precision)
+            UNION ALL
+            SELECT 'spool.weight_used_baseline is ' || weight_used_baseline::text
+            FROM spool WHERE id = 1 AND weight_used_baseline IS DISTINCT FROM CAST(1234.567 AS double precision)
+        """,
+    ),
 )
 
 _INIT_DB = """

scripts/postgres_schema_drift.txt

@@ -8,7 +8,6 @@
 # Comment lines start with "#".
 
 # columns
-column active_print_spoolman.tray_remain_start | fresh install: json (udt json) NULL default None; upgraded: text (udt text) NULL default None
 column api_keys.can_access_cloud | fresh install: boolean (udt bool) NOT NULL default None; upgraded: boolean (udt bool) NULL default false
 column api_keys.can_manage_archives | fresh install: boolean (udt bool) NOT NULL default None; upgraded: boolean (udt bool) NULL default true
 column api_keys.can_manage_inventory | fresh install: boolean (udt bool) NOT NULL default None; upgraded: boolean (udt bool) NULL default true
@@ -54,8 +53,8 @@ column printers.camera_light_auto | fresh install: boolean (udt bool) NOT NULL d
 column smart_plugs.auto_off_after_drying | fresh install: boolean (udt bool) NOT NULL default false; upgraded: boolean (udt bool) NULL default false
 column smart_plugs.controls_printer_power | fresh install: boolean (udt bool) NOT NULL default true; upgraded: boolean (udt bool) NULL default true
 column smart_plugs.off_delay_after_drying_minutes | fresh install: integer (udt int4, precision 32, scale 0) NOT NULL default 10; upgraded: integer (udt int4, precision 32, scale 0) NULL default 10
-column smart_plugs.rest_energy_total_multiplier | fresh install: double precision (udt float8, precision 53) NOT NULL default '1'::double precision; upgraded: real (udt float4, precision 24) NULL default 1.0
-column spool.weight_used_baseline | fresh install: double precision (udt float8, precision 53) NOT NULL default None; upgraded: real (udt float4, precision 24) NULL default 0
+column smart_plugs.rest_energy_total_multiplier | fresh install: double precision (udt float8, precision 53) NOT NULL default '1'::double precision; upgraded: double precision (udt float8, precision 53) NULL default 1.0
+column spool.weight_used_baseline | fresh install: double precision (udt float8, precision 53) NOT NULL default None; upgraded: double precision (udt float8, precision 53) NULL default 0
 column virtual_printers.gcode_injection | fresh install: boolean (udt bool) NOT NULL default false; upgraded: boolean (udt bool) NULL default false
 column virtual_printers.queue_force_color_match | fresh install: boolean (udt bool) NOT NULL default false; upgraded: boolean (udt bool) NULL default false
 column virtual_printers.save_ams_mapping | fresh install: boolean (udt bool) NOT NULL default false; upgraded: boolean (udt bool) NULL default false