Author: Michael Lynch <git@mtlynch.io>
Committer: Michael Lynch <mtlynch@noreply.codeberg.org>
Widen the shutdown budget so the final sync can finish (#327)
Litestream's shutdown-sync-timeout defaulted to 30s and fly.toml's
kill_timeout was also 30s. Those two numbers being equal meant Fly's
SIGKILL was scheduled to land at exactly the moment Litestream was still
allowed to be uploading. Any final sync that took the full budget lost
the race, and Fly won.
That race matters more here than it would in most deployments, for two
reasons that compound:
- sync-interval is 10m, so at any given moment up to ten minutes of
writes exist only in the local WAL and have never been sent to
object storage.
- There is no Fly Volume by design, so the machine's disk disappears
with the machine. The final sync is not an optimization; it is the
only thing standing between a routine deploy and ten minutes of lost
comments, uploads, and sessions.
Raise shutdown-sync-timeout to 60s and kill_timeout to 90s. The budget
is now 15s of request draining plus 60s of syncing, leaving 15s of slack
for process teardown inside the 90s Fly allows.
This costs nothing in the common case. Shutdown normally completes in a
second or two; the larger ceiling only gets used when object storage is
slow, which is precisely the situation where giving up early is most
expensive. The tradeoff is that a pathological deploy takes longer to
roll, which is acceptable for a single-machine app that already accepts
brief downtime during deploys.
Both values carry comments pointing at each other, since changing either
one in isolation reintroduces the race.
Co-Authored-By: Claude <noreply@anthropic.com>
Reviewed-on: https://codeberg.org/mtlynch/little-moments/pulls/327