Backup and Recovery Procedure

Implements: Business Continuity Policy Section 4 (Backup and Recovery) Owner: Security Lead Last Updated: 21 May 2026 Next Review: 21 November 2026

Backup Architecture

Automated Backup (provii-backup)

The provii-backup is a Cloudflare Worker running on a cron schedule that:

1. Lists all keys in each configured KV namespace
2. Exports key-value pairs as JSON
3. Uploads the export to Cloudflare R2 with a timestamped filename
4. Logs success/failure to the worker’s log output

Verifying Backups

# Check recent backup files in R2
wrangler r2 object list provii-backups --remote

# Verify a specific backup is readable
wrangler r2 object get provii-backups/kv-backup-YYYY-MM-DD.json --remote --file /tmp/backup-check.json

Run this verification monthly or after any infrastructure changes.

Recovery Procedures

Scenario 1: Corrupted KV Data

1. Identify which namespace and keys are affected
2. Download the most recent clean backup from R2:

   wrangler r2 object get provii-backups/kv-backup-YYYY-MM-DD.json --remote --file /tmp/restore.json

3. Parse the JSON and write the affected keys back:

   wrangler kv key put KEY VALUE --namespace-id=XXX --remote

4. Verify the restored data via API calls

Scenario 2: Signing Key Loss

1. Signing keys are stored in Cloudflare Workers Secrets
2. If secrets are lost, redeploy with wrangler secret put SIGNING_KEY
3. Generate new keys if the previous key material is unrecoverable
4. If issuer signing keys are rotated, all existing credentials become invalid. Users must re-issue.
5. Update JWKS endpoint with new public key

Scenario 3: Full Service Reconstruction

If Cloudflare account access is lost:

1. All source code is in GitHub. clone all 15 repositories
2. Create a new Cloudflare account
3. Run wrangler deploy for each Worker using checked-in wrangler.toml
4. Restore secrets (from secure offline storage)
5. Restore KV data from R2 backups (if R2 is accessible) or accept data loss
6. Update DNS records to point to new Cloudflare account
7. Estimated RTO: 4 hours

RPO and RTO

Testing

Backup recovery is tested annually:

1. Download a backup from R2
2. Restore selected keys to a sandbox KV namespace
3. Verify data integrity
4. Document results in the management review