04a8b907

By: Michael Lynch <git@mtlynch.io>

Replace production databases with or without a Fly volume

Backport the upload safeguards from refactoring-english-webhooks commits
 df7a6a7, a342b16, and 460337d. Uploading a Litestream snapshot alone does
not replace a database that survives on the machine because the entrypoint
only restores when the database is absent.

Disable the app during replacement, clear the old database and Litestream
state, then restore the original entrypoint and wait for machine and app
readiness. Accept already-absent files on ephemeral storage rather than
requiring a mounted volume or a separate workflow. Check remote exit codes
so failed deletion or inspection cannot report success.

Keep failed replacements disabled, print the original init settings for
recovery, and reject retries that would preserve the temporary sleep
entrypoint. Add jq and GNU coreutils for machine JSON and bounded health
requests, plus an offline regression target covering both storage cases
and failure paths.

Live Fly startup and restoration remain unverified because this environment
has no production credentials.

Total

1m21s

Time to Start Worker time Duration Time to finish Idle
Config 0s 1s 1s 1s 0s
Eval 2s 53s 53s 55s 0s
Build 48s 26s 33s 1m22s 0s
Suite 0s 1m21s 1m21s 1m22s 0s

Timeline

Config Eval Build
0s10s20s30s40s50s1m1m10s1m20s