Social LiveDocumentationPLATFORM DOCS

Backup & Restore

Nightly database and uploads backups, rotation, the offsite copy, and the quarterly restore drill.

Last updated Thu Jul 16 2026 00:00:00 GMT+0000 (Coordinated Universal Time)

TL;DR -Every night the server backs up the database and media, keeps a rotating set of copies, and (when configured) ships a copy offsite. Once a quarter we prove the backups actually restore.

Who this is for -DevOps & Operations: the schedule, the retention, and the restore commands. Executive & Product: assurance that data loss is bounded to at most a day and that we test recovery. Engineering: the exact artifacts, volumes, and sanity checks.

A backup is a saved snapshot you can go back to if something breaks or data is lost. Social Live takes one automatically every night. The catch that most teams learn the hard way: a backup you've never restored isn't really a backup -so we also rehearse the restore on a schedule.

Backups run nightly on the VPS via /etc/cron.d/social-live-backup (03:30 UTC), executing scripts/db-backup.sh. Everything lands in /root/social-live-backups/.

What gets backed up

The database is the critical one (it holds users, wallets, and the money ledger); the media volumes hold older uploaded files. "Retention" means how many past copies are kept before the oldest is deleted.

ArtifactHowRetention
Postgres (social_live)pg_dump -Fc from the postgres container, integrity-checked with pg_restore --list7 daily + 4 weekly (Sundays) + 3 monthly (1st)
Uploads volume (social-live_social_live_uploads) -public avatars/bannerstar via throwaway alpine container7 daily
Chat media volume (social-live_social_live_chat_media) -private DM photos/videostar via throwaway alpine container7 daily

Since object storage went live (R2 bucket social-live-media), new media uploads go to R2, whose durability/versioning replaces nightly tars for them; the two volumes above still hold pre-R2 files and the dev-fallback path, so their backups continue unchanged. | .env.production | copy, mode 600 | last 5 |

Every run writes last-backup-status (OK <stamp> or FAILED <stamp>: why) and appends to backup.log (self-trimmed to 2,000 lines). Monitoring check: if last-backup-status doesn't start with OK or is older than 26 hours, backups are broken.

Offsite copy

A backup that only lives on the same server dies with that server. The offsite copy sends a duplicate to remote storage so we survive losing the box entirely.

If BACKUP_RCLONE_REMOTE is set in .env.production (e.g. b2:social-live-backups) and rclone is configured for root, each run also uploads the day's dump, uploads archive, and env copy. Until an offsite destination is configured, backups die with the VPS disk -the script logs a warning every night to keep this visible.

Restore

Recovering from a backup. There are two modes: a safe rehearsal that touches nothing real, and a destructive one that replaces the live database.

scripts/db-restore.sh has two modes:

# Safe drill -restores into a throwaway DB, checks row counts, drops it:
scripts/db-restore.sh --test /root/social-live-backups/daily/<file>.dump

# DESTRUCTIVE -stops backend, replaces the production DB, restarts:
scripts/db-restore.sh --real /root/social-live-backups/daily/<file>.dump

--real requires typing the database name to confirm, terminates open connections, and runs the same row-count sanity check (User, Wallet, LedgerTransaction) before restarting the backend.

Restoring the media volumes is rarely needed since those assets are non-financial; the command below follows the same pattern for uploads and chat-media:

docker run --rm -v social-live_social_live_uploads:/data \
  -v /root/social-live-backups/uploads:/backup alpine \
  sh -c "cd /data && tar xzf /backup/<file>.tgz"

The quarterly restore drill

This is the "prove it works" step. Once a quarter, restore the newest backup into a throwaway copy and confirm it looks like production.

An untested backup is not a backup. Once a quarter:

  1. scripts/db-restore.sh --test against the newest daily dump.
  2. Confirm RESTORE TEST OK and that the sanity row counts look like production.
  3. Check last-backup-status age and the offsite destination actually contains recent files.

Loss envelope

How much data we could lose in the worst case, and why that's acceptable for now. RPO ("recovery point objective") is the maximum amount of recent data a restore might not include.

Nightly dumps mean up to 24h RPO. Acceptable at current volume; when real money throughput grows, tighten by raising cron frequency (the script is idempotent and cheap -~2s at today's size) or moving to WAL archiving. The 72h diamond settlement hold means most in-flight earnings survive a restore within the hold window anyway.