A rebuild usually starts with the container. That is the easy part: the compose file is in git, the image pulls, the process starts. Then the service asks for the key that signs its sessions, and that key lived in an environment file on the disk that died. The rebuild stops while you look for a copy.
The order is what trips people up. The parts that cannot be re-created have to come first, because everything after them depends on those bytes.
Put the kit somewhere else
The repository password is the first item, because nothing in the backup can be read without it. Restic encrypts the repository, and the key command manages the passwords that open it. A second key stored away from the backup host is cheap insurance: restic key list shows the keys that exist, and restic key add adds another.
Then the small files that never made it into the data backup because they lived in the wrong place: the compose file or the systemd unit, the environment file, the application key that encrypts stored secrets, the TLS private key if certificates are not issued automatically, and the DNS provider credentials. Keep them in a password manager or on paper, somewhere other than the machine they describe.
Write down the repository URL and the restic version beside the password. A restore run months later, from a different machine, is easier when the notes say which binary and which bucket.
Restore the data into a scratch path
restic -r /srv/restic-repo snapshots
restic -r /srv/restic-repo restore 79766175 --target /srv/restore --include /srv/app--path selects which snapshot to restore; it does not filter the files inside it. --include does that, and the path is the one inside the snapshot, which is what restic ls prints. When a single file is missing rather than the whole host, restic find locates the snapshot that holds it.
latest means the newest snapshot, not the newest snapshot that contains the file you want. In a repository with more than one host or job those are often different snapshots, so list first and restore a specific ID. A drill that restores latest looks successful until the file turns out to be missing.
Restore into a scratch directory rather than over the live path. The default is --overwrite always, which writes over whatever is already there, and you want a look at what came back before it touches anything. --dry-run reports what would be restored without writing it; the file-by-file listing needs --verbose=2 as well.
Then check ownership. ls -ln on the restored tree prints the numeric UID and GID the files came back with, and on a rebuilt host those numbers may belong to a different account. A container running as 1000 against a directory restored as root fails at the first write.
Start it on the old address
Bring the stack up without touching DNS:
docker compose up -d
docker compose ps
docker compose logs -f app
curl -fsS http://127.0.0.1:8080/healthUse whatever path the application exposes for health. If it has no health endpoint, the compose file’s healthcheck block can run the check the image’s own maintainers document, and docker compose ps will then show the health state next to the container status.
Do not point the public name at a service you have not reached locally. The DNS change is the moment readers arrive, and a proxy returning 502 is harder to diagnose with traffic on it.
Databases come back as dumps
A PostgreSQL data directory copied out of a volume is a copy taken at an unknown moment. The portable form is a custom-format dump, and pg_restore loads it back. PostgreSQL’s documentation is explicit that, except in simple cases, pg_dump is generally not the right choice for taking regular backups of production databases. For a recovery, though, the dump is the thing you can load into a rebuild.
pg_dump -Fc -d appdb -f /srv/backup/appdb.dump
pg_restore -l /srv/backup/appdb.dump
pg_restore -d appdb /srv/backup/appdb.dumppg_restore -l lists the archive’s contents, which is how you confirm the dump is the one you meant before loading it. Restore into an empty database: the archive carries the schema, and loading it over an existing one produces duplicate-object errors that read like corruption.
SQLite is different. A database file copied while a transaction is open can contain some old content and some new content. SQLite’s own documentation on corruption says such a copy “might contain some old and some new content, and thus be corrupt”, and lists VACUUM INTO, the backup API and sqlite3_rsync as the safe ways to copy a live database. During a recovery the simplest safe route is to stop the application first and then copy the file.
The volume tar, if that is all you have
Docker’s documentation backs up a volume with a temporary container that mounts it and tars the contents. The two --volumes-from commands need a container that already mounts the volume, which is what Docker’s example makes with docker run -v /dbdata --name dbstore ubuntu /bin/bash.
docker run --rm --volumes-from dbstore -v "$(pwd)":/backup ubuntu tar cvf /backup/backup.tar /dbdataThe matching restore un-tars it into a new container’s volume:
docker run --rm --volumes-from dbstore2 -v "$(pwd)":/backup ubuntu bash -c "cd /dbdata && tar xvf /backup/backup.tar --strip 1"If the volume has a name rather than living inside a container, mount it by name:
docker run --rm -v app_db:/dbdata -v "$(pwd)":/backup ubuntu tar cvf /backup/backup.tar /dbdataThe restore is the same container with tar xvf /backup/backup.tar --strip 1 run inside /dbdata.
That is a file copy of the volume, so it has the same weakness as a raw database copy when the application was running. Stop the stack before restoring into a volume in use, and check the ownership inside the archive before trusting it.
DNS is the last switch
Lower the record’s TTL a day before a planned move. The TTL is the number of seconds a resolver may keep the record before asking again, so a short value means the old address falls out of caches quickly. For an unplanned recovery you cannot go back and lower it, so read the current value first and expect the old answer to persist for up to that long.
Change the record, then ask two resolvers that are not yours:
dig +short app.example.com @1.1.1.1
dig +short app.example.com @8.8.8.8A single dig on the host that made the change proves the zone file, not propagation. When both answers match the new address, request the site by name and look at the certificate. A proxy still pointing at the old backend, or a certificate for the wrong name, shows up here and not in the A record.
The check that proves it
Open the service and do the thing it exists for: log in, create an item, read it back. Then restart the container and read it again. A service that answers 200 but has forgotten yesterday’s write is usually reading an empty or wrong volume, and the page looked fine until the restart.
Finish with the plain checks. The backup job should be scheduled and have run since the restore, and monitoring should be back on. The certificate deserves a look as well, because an expiry two days after a recovery turns into a second outage.
Source: the restic documentation on restoring from a snapshot and managing repository keys; Docker’s documentation on volumes and docker compose ps; PostgreSQL’s pg_dump and pg_restore; SQLite’s How To Corrupt An SQLite Database File; and RFC 1035 for the TTL field.