docs: document manual deploy for web and desktop releases

.claude/skills/releasing-healthchat/SKILL.md

@@ -9,25 +9,51 @@ Releases are driven by git tags of the form `healthchat-v<semver>` pushed to
 `origin`. The tag triggers two workflows in parallel:
 
 - `.github/workflows/build-healthchat.yaml` (web) — builds the HealthChat Docker
-  image, publishes it to ghcr.io under `healthchat-web:<version>`, `:stable`, and
+  image, publishes it to ghcr.io under `healthchat-web:<version>`, `:latest`, and
   `:<sha>`, and creates a GitHub Release whose body is the matching section of
   `HEALTHCHAT_CHANGES.md`.
+
+  **`:latest` is the tag production deploys from.** It only moves for a clean
+  `X.Y.Z` **tag push** — never for `-rc1`/`-hotfix1`/`-dryrun1` tags, and never for
+  a manual `workflow_dispatch` run. Those builds still publish `:<version>` and
+  `:<sha>`, so you can deploy them deliberately by overriding the image ref (see
+  "Deploying the release"), but they can never become production by accident.
 - `.github/workflows/build-healthchat-desktop.yaml` (desktop) — builds the Electron
   app for macOS (universal, signed+notarised) and Windows (x64/arm64, signed), and
   attaches the installers (.dmg/.zip/.exe/.msi) to the **same** GitHub Release. The two
   workflows create-if-missing then upload, so they race safely. (Linux is intentionally
   not built — the upstream Linux reusable pulls a build-env Docker image keyed by commit
   SHA that only exists in the upstream element-hq registry, so it cannot succeed in this
-  fork.) The `release` job additionally **mirrors the Squirrel update feed and the public
-  installers to `https://healthchat.ch/packages/desktop/` over FTPS** (the private GitHub
-  Release can serve neither end users nor the auto-updater). This is what makes installed
-  apps self-update; see `apps/desktop/element.io/healthchat/README` for the feed layout and
+  fork.) The `release` job also **stages** the Squirrel update feed and public installers and
+  attaches them to the Release as `healthchat-desktop-feed-<version>.tar.gz` — but it
+  **does not publish them**. See "Deploying the desktop release" below.
+- `.github/workflows/deploy-healthchat-desktop.yaml` (desktop deploy) — **manual
+  dispatch only.** Downloads that tarball and mirrors it to
+  `https://healthchat.ch/packages/desktop/` over FTPS (the private GitHub Release can
+  serve neither end users nor the auto-updater). This is what makes installed apps
+  self-update; see `apps/desktop/element.io/healthchat/README` for the feed layout and
   the two caveats (first install is manual; Windows auto-update needs the Setup.exe path,
   not the .msi).
 
 Publish target: **ghcr.io Docker image + GitHub Release (web bundle + desktop
 installers).** No npm, no mobile fan-out.
 
+## Release ≠ deploy
+
+Cutting a tag never changes what any user is running. Both halves of the deployment are
+separate, manual, and need no code change:
+
+| Component | Released by | Deployed by |
+| --- | --- | --- |
+| Web (ghcr.io image) | `healthchat-v*` tag push → `:latest` moves | `gh workflow run deploy.yml` on the ansible repo |
+| Desktop (update feed) | `healthchat-v*` tag push → feed tarball on the Release | `gh workflow run deploy-healthchat-desktop.yaml` |
+
+Writing the desktop feed pushes an update to **every installed client**, so it is a
+deployment, not a publication step — hence the same manual gate as the server side.
+Deploy the two together unless you have a reason not to: a desktop client talks to the
+web deployment, so leaving one behind means users run a client built against a different
+release.
+
 The desktop app version is taken from `apps/desktop/package.json` (the per-job
 `version` input cannot set it — Windows squirrel mangles it into a nightly string,
 and on macOS it only feeds the update manifest). So `apps/desktop/package.json`
@@ -99,6 +125,116 @@ Examples:
 The workflow triggers automatically. Watch it via `gh run watch` or GitHub
 Actions UI.
 
+## Deploying the release
+
+Publishing the image does **not** deploy it. Deployment is a separate manual
+dispatch against the ansible repo — but it needs **no code change anywhere**: the
+inventory pins `matrix_client_element_container_image` to
+`ghcr.io/novaloop-ag/element-web/healthchat-web:latest`, and the ansible role pulls
+with `pull: always`, so it picks up whatever `:latest` now points at and restarts
+the container only if the digest actually moved.
+
+Wait for the release workflow to go green first (the deploy pulls whatever is in the
+registry at that moment), then:
+
+```bash
+gh workflow run deploy.yml \
+  --repo Novaloop-AG/matrix-docker-ansible-deploy \
+  --ref release/healthchat \
+  -f command="install-service client-element"
+```
+
+Watch it: `gh run watch --repo Novaloop-AG/matrix-docker-ansible-deploy`.
+
+Or via the UI: GitHub → `Novaloop-AG/matrix-docker-ansible-deploy` → Actions →
+"Deploy HealthChat (Synapse, Web app etc.)" → Run workflow → **command:**
+`install-service client-element`.
+
+Add `-f check_mode=true` for a dry run (`--check --diff`).
+
+### Deploying a specific version instead of `:latest`
+
+RCs, hotfix tags and dispatch builds do not move `:latest`. To ship one — or to roll
+back — pass an explicit image ref through `extra_args`, which the workflow forwards
+verbatim to `ansible-playbook`. **No inventory edit, no commit:**
+
+```bash
+gh workflow run deploy.yml \
+  --repo Novaloop-AG/matrix-docker-ansible-deploy \
+  --ref release/healthchat \
+  -f command="install-service client-element" \
+  -f extra_args="-e matrix_client_element_container_image=ghcr.io/novaloop-ag/element-web/healthchat-web:1.5.0"
+```
+
+Note this is a one-shot override: the *next* plain deploy returns to `:latest`. For a
+sustained hold, pin the version in the inventory instead.
+
+### Confirming what actually shipped
+
+Because the inventory tracks `:latest`, git no longer records the deployed version —
+check the running app, not the repo:
+
+```bash
+curl -fsS https://app.healthchat.ch/version
+```
+
+That serves the build version baked in by `scripts/docker-package.sh` (e.g.
+`healthchat-v1.5.0`). If it still reports the old version, the deploy either did not
+run or `:latest` had not moved yet.
+
+## Deploying the desktop release
+
+The desktop build attaches `healthchat-desktop-feed-<version>.tar.gz` to the Release but
+leaves `https://healthchat.ch/packages/desktop/` untouched. Until you run this, installed
+apps keep whatever version they have and the public download links still serve the old
+installers.
+
+Wait for "Build HealthChat Desktop" to go green, then:
+
+```bash
+gh workflow run deploy-healthchat-desktop.yaml \
+  --ref release/healthchat \
+  -f version=1.5.0
+```
+
+Or: GitHub → Actions → "Deploy HealthChat Desktop" → Run workflow → **version:** `1.5.0`.
+
+Dry run first if you want to see what would be written without touching the server:
+
+```bash
+gh workflow run deploy-healthchat-desktop.yaml \
+  --ref release/healthchat \
+  -f version=1.5.0 -f dry_run=true
+```
+
+The workflow refuses to publish a partial feed — it fails before contacting the server if
+`releases.json`, either arch's `RELEASES`, or the macOS `*-mac.zip` payload is missing. A
+half-written feed breaks auto-update for everyone already installed, which is much worse
+than not deploying.
+
+On success it curls the five load-bearing URLs and fails if any is not 200, so a green run
+means the feed is genuinely live. Installed apps pick it up on their next update check.
+
+### Deploying an RC or rolling back
+
+`version` accepts any released version, including `1.5.0-rc1`. Because the feed lives on
+the Release (assets never expire, unlike workflow artifacts), **re-running this workflow
+with an older version republishes that feed** — no rebuild, no re-signing. That is the
+desktop rollback path:
+
+```bash
+gh workflow run deploy-healthchat-desktop.yaml --ref release/healthchat -f version=1.4.1
+```
+
+Caveat: `mirror -R` does not delete, so rolling back rewrites the manifests but leaves the
+newer `*.nupkg` files in place. That is intentional — Squirrel.Windows needs the delta
+history — and harmless, since clients follow `RELEASES`/`releases.json`, not directory
+listings.
+
+Releases cut **before** this release/deploy split carry no feed tarball. The workflow says
+so explicitly and lists the assets it did find; re-run "Build HealthChat Desktop" for that
+version to produce one.
+
 ## Manual / retry path (workflow_dispatch)
 
 Use when retrying a failed run, testing the pipeline, or rebuilding an image
@@ -112,7 +248,10 @@ without moving tags.
      lookup.
 3. Run. Docker image is pushed. **No GitHub Release is created on the dispatch
    path** — only the tag-push path creates Releases (keeps manual retries from
-   clobbering the release history).
+   clobbering the release history). **`:latest` is not moved either**, for the same
+   reason: a retry or pipeline test must not become production. The build still
+   publishes `:<version>` and `:<sha>`; deploy it with the explicit-version
+   override in "Deploying the release" if that is what you want.
 
 The desktop workflow ("Build HealthChat Desktop") has its own workflow_dispatch
 with the same inputs; like the web path, its `release` job only publishes on a
@@ -161,22 +300,43 @@ warning. Prefer fixing the changelog over relying on the fallback.
 ## After the release
 
 - [ ] Workflow run is green in GitHub Actions.
-- [ ] `docker pull ghcr.io/<owner>/healthchat-web:<version>` succeeds.
+- [ ] `docker pull ghcr.io/novaloop-ag/element-web/healthchat-web:<version>` succeeds.
+      (Note the image path contains the repo segment — the workflow builds it as
+      `format('{0}/healthchat-web', github.repository)`.)
 - [ ] GitHub Release exists at
       `https://github.com/<owner>/<repo>/releases/tag/healthchat-v<version>`
       with the correct body.
-- [ ] The `:stable` tag on ghcr.io now points at the new digest
-      (`docker buildx imagetools inspect ghcr.io/<owner>/healthchat-web:stable`).
+- [ ] The `:latest` tag on ghcr.io now points at the new digest, and matches
+      `:<version>`:
+
+      ```bash
+      docker buildx imagetools inspect ghcr.io/novaloop-ag/element-web/healthchat-web:latest
+      docker buildx imagetools inspect ghcr.io/novaloop-ag/element-web/healthchat-web:<version>
+      ```
+
+      For an `-rc`/`-hotfix`/`-dryrun` tag the opposite is expected: `:latest` must
+      be **unchanged**.
 - [ ] The desktop workflow run ("Build HealthChat Desktop") is green and the
       GitHub Release has the desktop installers attached (.dmg, *-mac.zip, *.msi,
-      "* Setup*.exe").
-- [ ] The update feed + installers published to healthchat.ch: `curl -fsI`
-      `https://healthchat.ch/packages/desktop/update/macos/releases.json`,
-      `.../update/win32/x64/RELEASES`, and the public installer URLs
-      `.../install/macos/HealthChat.dmg` / `.../install/win32/x64/HealthChat-Setup.exe`
-      all return 200.
-- [ ] Smoke-check signing: macOS `spctl -a -vvv -t install <app>` is accepted;
-      Windows installer shows a valid publisher (no SmartScreen block).
+      "* Setup*.exe") **plus** `healthchat-desktop-feed-<version>.tar.gz`. Without that
+      tarball the desktop deploy has nothing to publish.
+- [ ] Confirm the live feed is still on the **previous** version — releasing must not have
+      moved it. `curl -fsS https://healthchat.ch/packages/desktop/update/macos/releases.json`
+- [ ] Smoke-check signing before shipping to users: macOS `spctl -a -vvv -t install <app>`
+      is accepted; Windows installer shows a valid publisher (no SmartScreen block).
+
+## After the deployment
+
+Two independent deploys — do both (see "Deploying the release" and "Deploying the desktop
+release"):
+
+- [ ] Web deployed and `curl -fsS https://app.healthchat.ch/version` reports the new version.
+- [ ] Desktop feed deployed: the "Deploy HealthChat Desktop" run is green. Its own verify
+      step already asserts the five feed URLs return 200, so a green run is the check.
+- [ ] `releases.json` now advertises the new version:
+      `curl -fsS https://healthchat.ch/packages/desktop/update/macos/releases.json | jq -r '.releases[0].updateTo.version'`
+- [ ] Spot-check auto-update on a real installed client (or at least confirm the app's
+      update check reaches the feed) before announcing.
 
 ## Required secrets
 
@@ -186,7 +346,7 @@ Configured in repo Settings → Secrets and variables → Actions:
 |---|---|
 | `HEALTHCHAT_WORKSPACE_TOKEN` | PAT with read access to `Novaloop-AG/healthchat-workspace`. Needed by `actions/checkout` for the private plugin repo. |
 | `GITHUB_TOKEN` | Auto-provided. Used for ghcr.io login and GitHub Release creation. No manual setup required. |
-| `FTP_SERVER` / `FTP_USERNAME` / `FTP_PASSWORD` | Credentials for the healthchat.ch FTP(S) account. The desktop `release` job uses them to publish the update feed + public installers under `/public_html/healthchat.ch/packages/desktop/`. `FTP_SERVER` is the hostname only (explicit FTPS / AUTH TLS). Without them the FTP publish step fails (but the GitHub Release still gets the installers). |
+| `FTP_SERVER` / `FTP_USERNAME` / `FTP_PASSWORD` | Credentials for the healthchat.ch FTP(S) account. Used **only** by `deploy-healthchat-desktop.yaml` to publish the update feed + public installers under `/public_html/healthchat.ch/packages/desktop/`. `FTP_SERVER` is the hostname only (explicit FTPS / AUTH TLS). The release workflows never touch FTP, so a missing/expired credential breaks the deploy but never the release. |
 
 Desktop signing additionally requires two GitHub **environments** (Settings →
 Environments). Without them the desktop builds still run but produce
@@ -206,9 +366,26 @@ fork must stay **internal** (not public). Add branch/environment protection so o
 
 ## Rollback
 
+- **Bad image already deployed** → redeploy the previous version immediately using the
+  explicit-version override in "Deploying the release" (no commit, no inventory edit).
+  That restores service; then fix forward with a hotfix tag. Remember the override is
+  one-shot — the next plain deploy goes back to `:latest`, so land the real fix before
+  anyone runs `setup-all`.
 - **Bad image, notes fine** → fix the cause, commit, tag `healthchat-v<ver>-hotfix1`.
+  Note a `-hotfix1` tag does **not** move `:latest`; once you have verified it, either
+  deploy it via the explicit-version override or cut a clean `X.Y.Z` tag to promote it.
+- **Bad desktop build already deployed to the feed** → re-run "Deploy HealthChat Desktop"
+  with the previous version (see "Deploying an RC or rolling back"). Clients that already
+  auto-updated stay on the bad version until the next release — the feed only controls what
+  is offered next, it cannot recall an installed build. That asymmetry is exactly why the
+  desktop deploy is gated behind a manual dispatch.
+- **Bad desktop build caught before deploying** → nothing to undo. Do not run the deploy
+  workflow; fix forward and cut a new tag.
 - **Bad release notes** → edit `HEALTHCHAT_CHANGES.md` and the GitHub Release
-  body manually (GitHub UI). The ghcr.io tag is immutable, don't try to move it.
+  body manually (GitHub UI). The ghcr.io tag is immutable, don't try to move it. Note the
+  desktop feed bakes the notes in at **build** time (into `releases.json` and `notes.txt`),
+  so a post-hoc edit to `HEALTHCHAT_CHANGES.md` does not reach the update dialog — rebuild
+  that version if the notes matter to desktop users.
 - **Completely wrong tag** → delete the GitHub Release (UI). Do **not** delete
   the ghcr.io image tag — treat it as burned. Re-tag with a new version.
 
@@ -220,6 +397,7 @@ during rebase):
 
 - `.github/workflows/build-healthchat.yaml`
 - `.github/workflows/build-healthchat-desktop.yaml`
+- `.github/workflows/deploy-healthchat-desktop.yaml`
 - `HEALTHCHAT_CHANGES.md`
 - `.healthchat-workspace-version`
 - `.claude/skills/releasing-healthchat/SKILL.md`

CLAUDE.md

@@ -133,4 +133,4 @@ Conventions unique to this fork that span multiple files:
 `pnpm link` is **not** used (it mutates `package.json`/lockfile). Instead, the `postinstall` hook runs `scripts/pnpm-link.ts`, which reads a `.link-config` file of `DEPENDENCY=PATH` lines and replaces those `node_modules` entries with symlinks. To develop against a local SDK, add a line like `matrix-js-sdk=/abs/path/to/matrix-js-sdk` to `.link-config` (not committed) and re-run `pnpm install`.
 
 ### Releasing HealthChat
-Releases are cut by pushing a `healthchat-v<semver>` tag against `release/healthchat`, which triggers `.github/workflows/build-healthchat.yaml` (web) and `build-healthchat-desktop.yaml` (desktop). Release notes come from `HEALTHCHAT_CHANGES.md` (kept separate from upstream `CHANGELOG.md`); the healthchat-workspace plugin ref is pinned in `.healthchat-workspace-version`. Full runbook: `releasing-healthchat` skill.
+Releases are cut by pushing a `healthchat-v<semver>` tag against `release/healthchat`, which triggers `.github/workflows/build-healthchat.yaml` (web) and `build-healthchat-desktop.yaml` (desktop). Release notes come from `HEALTHCHAT_CHANGES.md` (kept separate from upstream `CHANGELOG.md`); the healthchat-workspace plugin ref is pinned in `.healthchat-workspace-version`. The image is published to `ghcr.io/novaloop-ag/element-web/healthchat-web` as `:<version>`, `:<sha>`, and `:latest` — where `:latest` moves **only** for a clean `X.Y.Z` tag push, never for `-rc`/`-hotfix` tags or manual dispatch runs. **Deploying is a separate manual step for both halves** and needs no version bump anywhere: the web side is a `gh workflow run deploy.yml` against the ansible repo (whose inventory tracks `:latest`), and the desktop side is `deploy-healthchat-desktop.yaml`, which publishes the update feed to healthchat.ch. Cutting a tag never pushes an update to installed desktop clients — `build-healthchat-desktop.yaml` only attaches a staged `healthchat-desktop-feed-<version>.tar.gz` to the Release. Full runbook: `releasing-healthchat` skill.