Ver Fonte

Backup: a first Git backup to an empty GitHub repository works (#3565)

- GitHub answers every Git Data call on a repository with no commits
  with 409 "Git Repository is empty"; the branch lookup treated that as
  a failure, and the blob/tree initial-commit path would have hit the
  same 409
- The first commit now goes through the Contents API with one backup
  file, written as push_files writes it, and push_files then pushes the
  rest; a single-file backup is done by that commit
- Should GitHub refuse to create the backup branch in an empty
  repository, the first commit goes to the default branch and the
  backup branch is made from it
- The empty-repository 409 is recognised on the default-branch lookup
  too; any other 409 is still reported, and a repository still empty
  after the first commit is not seeded twice
maziggy há 4 dias atrás
pai
commit
b69fd34591

+ 1 - 0
CHANGELOG.md

@@ -14,6 +14,7 @@ All notable changes to Bambuddy will be documented in this file.
 - **Choose which printer an "Any model" job takes (#2915, requested by @bryanmahin, contributed by @Keybored02 in #3547)** — Several printers of a model can take an "Any model" job, and the one with the fewest print hours always got it. **Settings > Workflow > Printer Order for Any-Model Jobs** now chooses: the fewest print hours (the default, as before), the most (use up the machines near replacement first), the printer idle the longest (by its last finished print; one that never printed goes first), or the printers' own priority, a number in **Edit Printer** (lowest first, none last). The order only ranks printers that can take the job: a busy, offline or filament-short printer is still passed over, and preferred colours still come first. Powering a printer on with an Auto Power On plug follows the same order.
 
 ### Fixed
+- **A first Git backup to an empty GitHub repository works (#3565, reported by @pike00)** — GitHub answers a repository with no commits with HTTP 409 "Git Repository is empty", and Bambuddy stopped there with "Failed to get branch ref", although Test Connection passed. Its fallback for an empty repository could not have helped either: it builds the first commit from blobs and trees, and GitHub refuses those too until the repository has a commit. The first backup now makes that commit through GitHub's Contents API, which writes one file, and pushes the rest on top, so no README or placeholder file is needed. Any other 409 is still reported as an error. Wiki updated.
 - **A printer that reconnects no longer waits out its upload backoff (#3563, reported by @Costyv95)** — When a printer's file service stops answering, Bambuddy sends it no jobs for a while, five minutes after the first refused upload and doubling up to an hour. Restarting the printer fixed its file service, but the queue still waited out the rest of that hour, and only restarting Bambuddy sent the job. A printer that reconnects is now tried again on the next pass, and the FTP cool-off from before the reconnect no longer blocks that try. If it still refuses, the next wait is the longer one and no second Job Waiting notification is sent. Wiki updated.
 - **Spoolman mode reads the subtype and colour from a filament named after its product (#3561, reported by @ARGACE)** — Spoolman has no subtype or colour-name field, so Bambuddy reads both from the filament's name, and it only understood names it writes itself ("PLA Matte"). A filament named "Matte Scarlet Red", "Silk+ Baby Blue" or "Gold - Basic PLA" got its whole name as the subtype and no colour, so the spool form left the colour blank. The finish words (Basic, Matte, Silk, HF, ...) are now the subtype and the rest of the name is the colour, wherever the material stands in it. Saving a spool also renamed its Spoolman filament to "material subtype" on every save; the name is now only rebuilt when the material or subtype is changed.
 - **Spool labels print Chinese, Japanese and Korean (#3560, reported by @mohan456-wqs)** — Since labels embed Liberation Sans (#3553), every character was drawn in that font, which has no Chinese, Japanese or Korean, so names in those languages came out as empty boxes in the PDF and in the preview. Installing fonts in the container could not help, because labels only use the fonts Bambuddy embeds. A character Liberation Sans lacks is now drawn in Droid Sans Fallback (Chinese and Japanese) or Noto Sans KR (Korean), and only the characters a label uses are embedded. Long names are shortened by their real printed width. These fonts have no bold or italic, so CJK text on a bold or italic line prints in regular weight.

+ 103 - 1
backend/app/services/git_providers/github.py

@@ -5,6 +5,7 @@ import json
 import logging
 import re
 from datetime import datetime, timezone
+from urllib.parse import quote
 
 import httpx
 
@@ -378,7 +379,8 @@ class GitHubBackend(GitProviderBackend):
 
             ref_response = await client.get(f"{api_base}/repos/{owner}/{repo}/git/refs/heads/{branch}", headers=headers)
 
-            if ref_response.status_code == 404:
+            empty = self._is_empty_repository(ref_response)
+            if ref_response.status_code == 404 or empty:
                 if not _allow_branch_create:
                     return {
                         "status": "failed",
@@ -387,6 +389,10 @@ class GitHubBackend(GitProviderBackend):
                             "The next scheduled backup will retry."
                         ),
                     }
+                if empty:
+                    return await self._seed_empty_repository(
+                        client, headers, api_base, owner, repo, branch, files, repo_url, token
+                    )
                 return await self._create_branch_and_push(
                     client, headers, api_base, owner, repo, branch, files, repo_url, token
                 )
@@ -565,6 +571,10 @@ class GitHubBackend(GitProviderBackend):
             ref_response = await client.get(
                 f"{api_base}/repos/{owner}/{repo}/git/refs/heads/{default_branch}", headers=headers
             )
+            if self._is_empty_repository(ref_response):
+                return await self._seed_empty_repository(
+                    client, headers, api_base, owner, repo, branch, files, repo_url, token
+                )
             if ref_response.status_code != 200:
                 return await self._create_initial_commit(client, headers, api_base, owner, repo, branch, files)
 
@@ -591,6 +601,98 @@ class GitHubBackend(GitProviderBackend):
             logger.exception("_create_branch_and_push failed for %s/%s branch=%s", owner, repo, branch)
             return {"status": "failed", "message": str(e), "error": str(e)}
 
+    @staticmethod
+    def _is_empty_repository(response: httpx.Response) -> bool:
+        """Whether GitHub refused a Git Data call because the repository has no commits (#3565)."""
+        if response.status_code != 409:
+            return False
+        try:
+            body = response.json()
+        except ValueError:
+            return False
+        message = body.get("message") if isinstance(body, dict) else None
+        return isinstance(message, str) and "repository is empty" in message.lower()
+
+    async def _seed_empty_repository(
+        self,
+        client: httpx.AsyncClient,
+        headers: dict,
+        api_base: str,
+        owner: str,
+        repo: str,
+        branch: str,
+        files: dict,
+        repo_url: str,
+        token: str,
+    ) -> dict:
+        """Give a repository with no commits its first one, then push the rest (#3565).
+
+        GitHub answers every Git Data call -- refs, blobs, trees -- with 409 until a
+        repository has a commit, so the blob/tree path cannot make the first one.
+        The Contents API can: it writes one file and creates the commit and the
+        branch. The other files then go through push_files like any later backup.
+        """
+        if not files:
+            return {"status": "skipped", "message": "No changes to commit", "commit_sha": None, "files_changed": 0}
+        try:
+            path = next(iter(files))
+            content_bytes = json.dumps(files[path], indent=2, default=str).encode("utf-8")
+            payload = {
+                "message": f"Initial Bambuddy backup - {datetime.now(timezone.utc).strftime('%Y-%m-%d %H:%M:%S UTC')}",
+                "content": base64.b64encode(content_bytes).decode(),
+                "branch": branch,
+            }
+            url = f"{api_base}/repos/{owner}/{repo}/contents/{quote(path)}"
+            response = await client.put(url, headers=headers, json=payload)
+            on_backup_branch = True
+            if response.status_code in (404, 422):
+                # Should GitHub not create a named branch in a repository that
+                # has none yet, the first commit goes to the default branch and
+                # push_files then branches the backup branch off it.
+                logger.info(
+                    "_seed_empty_repository %s/%s: branch %s refused (HTTP %s), seeding the default branch",
+                    owner,
+                    repo,
+                    branch,
+                    response.status_code,
+                )
+                default_branch_payload = {k: v for k, v in payload.items() if k != "branch"}
+                response = await client.put(url, headers=headers, json=default_branch_payload)
+                on_backup_branch = False
+            if response.status_code != 201:
+                msg = f"Failed to create the first commit (HTTP {response.status_code}): {self._truncated_response_text(response)}"
+                logger.warning("_seed_empty_repository %s/%s: %s", owner, repo, msg)
+                return {"status": "failed", "message": msg}
+
+            commit_sha, err = self._read_sha(response, "commit", "sha")
+            if err:
+                msg = f"Malformed contents response ({err}): {self._truncated_response_text(response)}"
+                logger.warning("_seed_empty_repository %s/%s: %s", owner, repo, msg)
+                return {"status": "failed", "message": msg}
+
+            seeded = {
+                "status": "success",
+                "message": f"Initial backup created - {len(files)} files",
+                "commit_sha": commit_sha,
+                "files_changed": len(files),
+            }
+            if len(files) == 1 and on_backup_branch:
+                return seeded
+
+            logger.info("Re-entering push_files after seeding %s/%s", owner, repo)
+            result = await self.push_files(
+                repo_url, token, branch, files, client, _allow_branch_create=not on_backup_branch
+            )
+            if result.get("status") == "skipped":
+                return seeded
+            if result.get("status") == "success":
+                return {**seeded, "commit_sha": result.get("commit_sha")}
+            return result
+
+        except Exception as e:
+            logger.exception("_seed_empty_repository failed for %s/%s branch=%s", owner, repo, branch)
+            return {"status": "failed", "message": str(e), "error": str(e)}
+
     async def _create_initial_commit(
         self,
         client: httpx.AsyncClient,

+ 175 - 0
backend/tests/unit/test_github_empty_repo_backup_3565.py

@@ -0,0 +1,175 @@
+"""A first Git backup to a GitHub repository with no commits (#3565).
+
+GitHub answers every Git Data call -- the branch lookup, but also blobs and
+trees -- with 409 "Git Repository is empty" until the repository has a commit.
+Bambuddy read the 409 on the branch lookup as a failure, and its blob/tree
+initial-commit path could not have started the repository either. The first
+commit now goes through the Contents API, and the rest follows as usual.
+"""
+
+import base64
+import json
+from unittest.mock import AsyncMock
+
+import pytest
+
+from backend.app.services.git_providers.github import GitHubBackend
+from backend.tests.unit.test_git_providers import _blob_sha, _make_mock_response
+
+pytestmark = pytest.mark.unit
+
+REPO_URL = "https://github.com/owner/repo"
+EMPTY = {"message": "Git Repository is empty.", "status": "409"}
+FILES = {"settings.json": {"a": 1}, "printers.json": [{"name": "X1C"}]}
+
+
+def _empty_409():
+    return _make_mock_response(409, EMPTY)
+
+
+def _client(*, get, put=(), post=(), patch=None):
+    client = AsyncMock()
+    client.get = AsyncMock(side_effect=list(get))
+    client.put = AsyncMock(side_effect=list(put))
+    client.post = AsyncMock(side_effect=list(post))
+    client.patch = AsyncMock(return_value=patch or _make_mock_response(200, {}))
+    return client
+
+
+def _branch_with(*paths):
+    """The three reads push_files makes on a branch holding these backup files."""
+    tree = [{"type": "blob", "path": p, "sha": _blob_sha(FILES[p])} for p in paths]
+    return [
+        _make_mock_response(200, {"object": {"sha": "c0"}}),
+        _make_mock_response(200, {"tree": {"sha": "t0"}}),
+        _make_mock_response(200, {"tree": tree}),
+    ]
+
+
+def _seeded(sha="c0"):
+    return _make_mock_response(201, {"commit": {"sha": sha}, "content": {}})
+
+
+async def _push(client, files=FILES, branch="main"):
+    return await GitHubBackend().push_files(REPO_URL, "ghp_token", branch, files, client)
+
+
+class TestTheFirstBackup:
+    @pytest.mark.asyncio
+    async def test_the_reported_case_creates_the_first_commit_and_pushes_the_rest(self):
+        client = _client(
+            get=[_empty_409(), *_branch_with("settings.json")],
+            put=[_seeded()],
+            post=[
+                _make_mock_response(201, {"sha": "blob-printers"}),
+                _make_mock_response(201, {"sha": "t1"}),
+                _make_mock_response(201, {"sha": "c1"}),
+            ],
+        )
+
+        result = await _push(client)
+
+        assert result["status"] == "success"
+        assert result["files_changed"] == 2
+        assert result["commit_sha"] == "c1"
+        # The first commit through the Contents API, on the configured branch,
+        # with the file exactly as push_files writes it, so it is not re-sent.
+        url = client.put.await_args.args[0]
+        body = client.put.await_args.kwargs["json"]
+        assert url == "https://api.github.com/repos/owner/repo/contents/settings.json"
+        assert body["branch"] == "main"
+        assert base64.b64decode(body["content"]) == json.dumps(FILES["settings.json"], indent=2).encode()
+        # The second file is the only blob: the seeded one was recognised.
+        blob_posts = [c for c in client.post.await_args_list if c.args[0].endswith("/git/blobs")]
+        assert len(blob_posts) == 1
+
+    @pytest.mark.asyncio
+    async def test_a_one_file_backup_is_done_by_the_first_commit(self):
+        client = _client(get=[_empty_409()], put=[_seeded()])
+
+        result = await _push(client, files={"settings.json": FILES["settings.json"]})
+
+        assert result == {
+            "status": "success",
+            "message": "Initial backup created - 1 files",
+            "commit_sha": "c0",
+            "files_changed": 1,
+        }
+        client.post.assert_not_awaited()
+
+    @pytest.mark.asyncio
+    async def test_a_backup_branch_github_will_not_create_branches_off_the_default_one(self):
+        files = {"settings.json": FILES["settings.json"]}
+        client = _client(
+            get=[
+                _empty_409(),
+                # After the commit on the default branch: backup branch missing,
+                # so it is branched off the default one and pushed to again.
+                _make_mock_response(404, {}),
+                _make_mock_response(200, {"default_branch": "main"}),
+                _make_mock_response(200, {"object": {"sha": "c0"}}),
+                *_branch_with("settings.json"),
+            ],
+            put=[_make_mock_response(404, {"message": "Branch bambuddy-backup not found"}), _seeded()],
+            post=[_make_mock_response(201, {"ref": "refs/heads/bambuddy-backup"})],
+        )
+
+        result = await _push(client, files=files, branch="bambuddy-backup")
+
+        assert result["status"] == "success"
+        first, second = client.put.await_args_list
+        assert first.kwargs["json"]["branch"] == "bambuddy-backup"
+        assert "branch" not in second.kwargs["json"]
+        assert client.post.await_args.kwargs["json"] == {"ref": "refs/heads/bambuddy-backup", "sha": "c0"}
+
+    @pytest.mark.asyncio
+    async def test_an_empty_default_branch_behind_a_missing_backup_branch_is_seeded_too(self):
+        client = _client(
+            get=[
+                _make_mock_response(404, {}),
+                _make_mock_response(200, {"default_branch": "main"}),
+                _empty_409(),
+            ],
+            put=[_seeded()],
+        )
+
+        result = await _push(client, files={"settings.json": FILES["settings.json"]}, branch="bambuddy-backup")
+
+        assert result["status"] == "success"
+        client.post.assert_not_awaited()
+
+
+class TestWhatStaysAFailure:
+    @pytest.mark.asyncio
+    async def test_another_409_is_reported_not_seeded(self):
+        client = _client(get=[_make_mock_response(409, {"message": "Conflict"}, text="Conflict")])
+
+        result = await _push(client)
+
+        assert result["status"] == "failed"
+        assert "Failed to get branch ref (HTTP 409)" in result["message"]
+        client.put.assert_not_awaited()
+
+    @pytest.mark.asyncio
+    async def test_a_refused_first_commit_says_why(self):
+        client = _client(
+            get=[_empty_409()],
+            put=[_make_mock_response(403, {}, text="Resource not accessible by personal access token")],
+        )
+
+        result = await _push(client)
+
+        assert result["status"] == "failed"
+        assert "HTTP 403" in result["message"]
+        assert "personal access token" in result["message"]
+
+    @pytest.mark.asyncio
+    async def test_a_repository_still_empty_after_the_first_commit_is_not_seeded_twice(self):
+        """GitHub lagging behind its own write: the next backup retries."""
+        client = _client(get=[_empty_409(), _empty_409()], put=[_seeded()])
+
+        result = await _push(client)
+
+        assert result["status"] == "failed"
+        assert "replication lag" in result["message"]
+        assert client.put.await_count == 1