Author: Michael Lynch <git@mtlynch.io>
Committer: Michael Lynch <mtlynch@noreply.codeberg.org>
Shut down the server gracefully on SIGINT and SIGTERM (#328)
The server ran as log.Fatal(http.ListenAndServe(...)), so a shutdown
signal killed it mid-request with no chance to finish in-flight work or
close the database. Replace that with signal handling, a bounded drain
via http.Server.Shutdown, and store.Close.
Handle SIGTERM as well as SIGINT. fly.toml sends SIGINT, but the e2e
suite stops the server with SIGTERM, so both paths are exercised.
The drain is capped at 15s rather than left unbounded. Uploads transcode
video synchronously inside the request handler and accept files up to
5 GB, so an unbounded Shutdown could run for minutes. That time would
come straight out of the window Fly grants before SIGKILL, starving
Litestream's final sync. Abandoning one long transcode is a much smaller
loss than abandoning every write since the last sync.
The subtle part, and the reason the shutdown path deliberately violates
the usual "fail hard on unexpected errors" rule:
Litestream runs this process as a child via -exec. On receiving a signal
it forwards that signal to the child, waits, and then performs its final
sync. But the wait is guarded by:
if err := <-c.execCh; err != nil &&
!strings.HasPrefix(err.Error(), "signal:") {
return fmt.Errorf("cannot wait for exec process: %w", err)
}
A child that exits non-zero produces "exit status 1", which does not
match that prefix, so Litestream returns early and never reaches
Close(), which is where the final sync lives.
Until now this was masked. With no signal handler the Go runtime killed
the process, Wait reported "signal: interrupt", the prefix matched, and
the sync ran. Adding a handler moves us off that accidental path, so any
log.Fatal on the shutdown path would silently discard up to 10 minutes
of writes -- precisely the failure this change exists to prevent.
Verified both directions against Litestream 0.5.11 with a file replica:
exit 0: "sending signal to exec process" -> app drains -> final
"replica sync" -> "litestream shut down", exit code 0
exit 1: "cannot wait for exec process: exit status 1", no final sync,
no shutdown line, exit code 1
So shutDown logs failures and returns instead of calling log.Fatal, and
carries a comment explaining why. log.Fatal is still correct for a
startup failure from ListenAndServe, which is handled separately.
Signal registration happens after startup completes rather than at the
top of main, so that the log.Fatal calls during startup cannot bypass
cleanup. The stop function is discarded because the process exits as
soon as the select returns.
Co-Authored-By: Claude <noreply@anthropic.com>
Reviewed-on: https://codeberg.org/mtlynch/little-moments/pulls/328