On this page
Reliability means important work does not depend on memory
The dangerous period may come months after a website's first deployment: nobody notices a certificate warning, disk use grows slowly, a backup job has been failing, or a former colleague still has access. A maintenance checklist gives infrequent but important work a rhythm. An incident runbook means responders do not have to search old chat messages for the right command.
Keep the checklist short and verifiable. “Check server security” is hard to act on. “Review pending security updates, decide whether a restart is needed, and record the last patch date” produces an observable result.
Daily: watch signals that affect service now
Automatically check external HTTPS, key health endpoints, unusual 5xx responses, critical disk use, repeated service restarts, and the age of the last successful backup. Summarize normal operation rather than sending a success message every five minutes. An exception should name the service, time, and first troubleshooting step.
People need to handle alerts and important content feedback. Manually connecting over SSH to look at top every day is not monitoring; it can create false confidence that yesterday's healthy snapshot still applies today.
Weekly: updates, logs, and capacity trends
Review operating-system security updates and important dependency notices, evaluate their impact, and schedule an appropriate window. Look for login failures, unusual 5xx patterns, out-of-memory events, service restarts, and database errors over the week. Compare the growth of disk use, database size, uploads, and backups, then estimate when a threshold may be reached.
Spot-check the integrity of a recent backup and confirm that its off-site copy arrived. After content publication, check that the sitemap, page responses, and search visibility have not developed a template-wide problem.
Monthly: access, renewals, cost, and recovery readiness
Review administrator access, SSH keys, cloud accounts, and third-party service accounts. Revoke access no longer needed. Check renewal dates for domains, certificates, servers, and storage. Automatic certificates should renew, but their remaining lifetime still needs monitoring.
Review releases and incidents: Which alerts led to no useful action? Which problems did users discover first? Could the team roll back within its target time? Before cleaning old backups or logs, list the candidates and confirm that retention rules protect the latest successful copy.
Quarterly: perform a real restore rehearsal
In an isolated environment, restore the database and uploads, then follow the documentation to start the app, proxy, and required configuration. Record how long it takes. If the runbook omits a credential source or software version, correct it immediately. A rehearsal can also reveal that a growing backup has pushed recovery time beyond what the site can tolerate.
Review the support lifecycle of the operating system, database, and main runtime. Treat a major-version upgrade as its own tested change, not an improvised addition to a routine patch window.
Put the first things a responder needs at the top of the runbook
Start with a list of services and domains, a simple architecture map, contacts, status or notification routes, stop conditions, and the location of recent releases. Then organize by symptom: entire site unavailable, 502/504, certificate error, full disk, database unavailable, or suspected credential leak.
Use the same pattern for each symptom: what it looks like, confirmation commands, data-protection actions, steps to restore service, when to escalate, verification, and evidence to retain. Use example variables rather than real passwords. Put prerequisites and rollback instructions before dangerous actions.
Use incident records to improve the system, not assign blame
Record when impact began, when it was detected, mitigation, recovery, and confirmation. Separate the direct trigger from conditions that allowed it to cause an outage. Give each improvement an owner and due date. “Be more careful” is not an action; a disk alert, corrected release check, or restore rehearsal is.
When the same problem recurs, automate verifiable steps instead of adding more manual approvals. Automation still needs a dry run, narrow targets, and a stop on failure; convenience does not justify broad deletion or restarts.
What should this series leave behind?
From the architecture map and systemd/Caddy/Compose setup through secrets, backups, releases, monitoring, troubleshooting, and the runbook, each part should have live evidence and a recovery path. Mark operational documents with their last verified date and update them when configuration changes. One server can still fail, but it no longer has to be a black box maintained from one person's memory.
To build the first runbook, work backward from the 502, 504, and HTTPS troubleshooting steps: record the accounts, logs, commands, and contacts needed at each layer. After three months, ask someone who did not deploy the site to follow the runbook through a restore. The places where they get stuck are the places the documentation needs to improve.
Sources
- journalctlsystemd
- SQLite Online Backup APISQLite Documentation
- PostgreSQL Backup and RestorePostgreSQL Documentation
Next steps
Continue with the next useful task; you do not need to read everything at once.