Backup and Recovery Procedure

Implements: Business Continuity Policy Section 4 (Backup and Recovery) Owner: Security Lead Last Updated: 19 June 2026 Next Review: 21 November 2026

Scope

This procedure covers backup and recovery across both Maelstrom AI platforms — Provii (zero-knowledge age verification) and downpipes (no-custody backup & disaster recovery for the Cloudflare data layer):

• Maelstrom’s own Cloudflare estate (the production infrastructure behind Provii and our other Workers) is backed up by our own production instance of downpipes — we run downpipes as a customer of our own product. This is the live, finished state. See Cloudflare Backup Evidence.
• The downpipes product itself runs in each customer’s own Cloudflare account, under the customer’s keys (no-custody). Maelstrom holds no customer keys, data or Cloudflare tokens, so the customer’s backup and recovery operations are complementary user-entity controls (CUECs); the customer-side posture is summarised at the end of this document.

Backup Architecture

Maelstrom AI’s entire Cloudflare estate is backed up by our own production instance of downpipes. The engine runs as a routeless Worker inside our own Cloudflare account, reads the sources below on a schedule, seals each archive with post-quantum hybrid encryption under operator-held keys, and fans out (3-2-1) to two or more Maelstrom-controlled destinations.

Automated Backup (self-hosted downpipes)

Our own production instance of downpipes runs the backups for Maelstrom’s Cloudflare estate. On its schedule the engine:

1. Reads the configured sources (Workers KV, D1, R2, Secrets Store, and the 51 zone/account configuration surfaces)
2. Seals each archive with post-quantum hybrid encryption under operator-held keys (no archive is readable without those keys)
3. Fans the sealed archive out (3-2-1) to two or more Maelstrom-controlled destinations
4. Emits signed run reports and a content-free advisory beacon to the vendor-side control-plane (which holds no keys, data or Cloudflare tokens and is fail-open — it never gates backup or restore)

Verifying Backups

Recoverability is proven continuously by the product, not merely asserted:

• Hourly canary integrity flights confirm that archives are well-formed and decryptable.
• Scheduled in-account restore drills prove that a real restore succeeds, not just that a backup was written.
• Signed reports from the engine evidence restore-test recency and run success; review them in the downpipes console (console.downpipes.io, our own instance).
• Scored security posture with regression alerts surfaces any drift; alerts page the operator.

Review the posture and the most recent restore-drill report monthly or after any infrastructure change.

Recovery Procedures

Scenario 1: Corrupted KV Data

1. Identify which namespace and keys are affected
2. Select the most recent clean archive in the downpipes console (console.downpipes.io, our own instance) and request a restore; restores require operator approval before they run
3. The engine decrypts the sealed archive under the operator-held key and restores the affected source in-account
4. If the console or the engine is unavailable, recover with the MIT-licensed downpipe offline reader — it reconstructs the data from the archive bytes plus the operator’s offline key alone, with no vendor and no network
5. Verify the restored data via API calls

Scenario 2: Signing Key Loss

1. Signing keys are stored in Cloudflare Workers Secrets
2. If secrets are lost, redeploy with wrangler secret put SIGNING_KEY
3. Generate new keys if the previous key material is unrecoverable
4. If issuer signing keys are rotated, all existing credentials become invalid. Users must re-issue.
5. Update JWKS endpoint with new public key

Scenario 3: Full Service Reconstruction

If Cloudflare account access is lost:

1. All source code is in GitHub; clone the platform repositories
2. Create a new Cloudflare account
3. Run wrangler deploy for each Worker using checked-in wrangler.toml
4. Restore secrets (from secure offline storage)
5. Restore the Cloudflare data layer (Workers KV, D1, R2, Secrets Store, and zone/account configuration) from the most recent downpipes archives. Because the archives are sealed under operator-held keys and fanned out 3-2-1 to independent destinations, recovery does not depend on the original Cloudflare account being reachable: the downpipe offline reader restores from the archive bytes plus the operator’s offline key alone
6. Update DNS records to point to new Cloudflare account
7. Estimated RTO: 4 hours

RPO and RTO

Testing

Backup recovery is tested continuously by the product and reviewed periodically:

1. Hourly canary integrity flights confirm archives are well-formed and decryptable
2. Scheduled in-account restore drills restore selected sources and prove a real restore succeeds
3. Verify data integrity against the restored sources
4. Document the signed restore-drill reports in the management review

downpipes Product — Customer-Operated Recovery (CUECs)

The recovery procedures above describe how Maelstrom recovers its own Cloudflare estate. When a customer runs downpipes, the engine and console run in the customer’s own Cloudflare account, under the customer’s keys, and Maelstrom holds no customer keys, data or Cloudflare tokens. The following are therefore the customer’s responsibility (complementary user-entity controls):

• Key custody. Customers generate break-glass, operational and archive-signer keys through the guided in-browser key ceremony, retain the printed recovery sheet, and store keys offline (encrypted USB, password manager, or Shamir split). The default two-recipient posture applies unless the customer explicitly opts into the strict break-glass-only mode.
• Destinations, retention and restore cadence. Customers choose their 3-2-1 destinations, set retention, and run their own restore drills.
• Restore approval. Restores are requested in the customer’s own console and require operator approval before they run.
• Vendor-independent recovery. The MIT-licensed downpipe offline reader restores from the archive bytes plus the customer’s offline key alone — no vendor, no network — built from a frozen public archive-format specification.

Maelstrom’s responsibility for the product is limited to building and maintaining the software and operating the vendor-side control-plane (fail-open) and the signature-pinned update channel (updates.downpipes.io, pull-only, no phone-home). The control-plane never gates a customer’s backup or restore.

See the ISMS Scope Statement shared-responsibility model and the Cloudflare Backup Evidence for Maelstrom’s own self-hosted instance.