Change Management Policy

Controlled changes to production systems

Public

Purpose

This procedure is designed to help ensure that changes to Maelstrom AI’s production systems are controlled, tested, and documented to maintain security and availability while enabling rapid innovation.

Scope

This policy applies to changes across both Maelstrom AI platforms — Provii (zero-knowledge age verification) and downpipes (no-custody backup & disaster recovery for the Cloudflare data layer). The change types, review gates and approvals below apply uniformly to both.

Applies to all repositories in our GitHub organisation, including:

  • Rust backend services (provii-verifier including hosted mode, provii-issuer)
  • Rust libraries (provii-crypto, provii-mobile-sdk, shared-rate-limit)
  • TypeScript portals and workers (admin-portal, provii-management, provii-credit-management, provii-status, shared-portal-lib)
  • TypeScript SDKs and sites (provii-agegate, provii-website, provii-docs)
  • Mobile applications (provii-mobile)
  • Integration and demo projects (provii-demos)
  • downpipes components (engine, console, control-plane, the MIT offline reader, and the signature-pinned update channel)
  • Infrastructure configuration (wrangler.toml, KV settings)
  • Security controls
  • Cryptographic parameters
  • Access permissions

For downpipes, a key boundary applies. Maelstrom builds and maintains the software and operates the vendor-side surfaces — the control-plane (fail-open) and the signature-pinned update channel — through this same process. The downpipes engine and console run in each customer’s own Cloudflare account, under the customer’s keys (no-custody), so Maelstrom cannot wrangler deploy into a customer tenant; how a change reaches those customer-operated components is governed by the downpipes Release Channel below, and the decision to apply an update is a complementary user-entity control (CUEC) of the customer.

Change Types

Standard Changes

Definition: Routine, low-risk changes following established procedures

Examples:

  • Dependency updates (non-breaking)
  • Documentation updates
  • Configuration tweaks
  • Bug fixes (non-security)

Approval: Automated via CI/CD passing

Process:

  1. Create feature branch
  2. Implement change
  3. CI/CD tests pass (required)
  4. Code review approval (2 reviewers minimum)
  5. Merge to main
  6. Automatic deployment

Timeline: Minutes to hours

Normal Changes

Definition: Significant changes requiring review but not emergency

Examples:

  • New features
  • Refactoring
  • Third-party integrations
  • Schema changes

Approval: Code review + Security Lead signoff (if security-relevant)

Process:

  1. Create feature branch
  2. Threat model (if security-relevant)
  3. Implement with tests
  4. Code review
  5. CI/CD validation
  6. Merge to main
  7. Monitor post-deployment

Timeline: Days to weeks

Emergency Changes

Definition: Urgent changes to resolve critical incidents

Examples:

  • Security vulnerability patches
  • Production outages
  • Signing key rotations (if compromised)
  • Critical bug fixes

Approval: ISMS Owner or Security Lead

Process:

  1. Assess urgency and risk
  2. Document change and justification
  3. Implement fix
  4. Expedited review (can be post-deployment for P0)
  5. Deploy immediately
  6. Retrospective within 24 hours

Timeline: Minutes to hours

Change Workflow

1. Planning

For Normal/Emergency Changes:

  • Define scope and objectives
  • Identify affected systems
  • Assess risks and impacts
  • Plan rollback strategy
  • Determine testing requirements
  • Schedule change window (if applicable)

2. Development

All changes:

  • Create Git branch (git checkout -b feature/description)
  • Implement change
  • Write/update tests
  • Run locally (wrangler dev --local)
  • Commit with descriptive message

Security-Relevant Changes:

  • Threat model documented
  • Security testing included
  • Security Lead review requested

3. Testing

Automated Testing (runs on every push):

# .github/workflows/ci.yml
- Security scanning (CodeQL, cargo audit, npm audit)
- Unit tests
- Integration tests
- Linting and formatting
- Build validation

Manual Testing:

  • Smoke testing for major changes
  • E2E testing for user-facing features
  • Performance testing (if relevant)

4. Review

Code Review Requirements:

  • Self-review via pull request with passing CI status checks before merge. Two-person approval is not feasible for a sole operator (accepted limitation, see the risk register).
  • Reviewers check:
  • Code quality and style
  • Security implications
  • Test coverage
  • Documentation updates
  • Approver cannot be author

Security Review Triggers:

  • Changes to authentication/authorisation
  • Cryptographic code changes
  • New third-party dependencies
  • Infrastructure configuration
  • Access control changes

5. Approval

Automated Approval:

  • CI/CD passes ✅
  • Code review approval ✅
  • Dismiss stale reviews on new pushes ✅

Manual Approval (if required):

  • Security Lead: Security-relevant changes
  • ISMS Owner: Emergency changes, major releases

6. Deployment

Automated Deployment (standard/normal):

# Triggered by merge to main
git checkout main
git merge feature/description
git push origin main
# GitHub Actions runs wrangler deploy

Manual Deployment (emergency):

# From local machine (requires Cloudflare credentials)
cd provii-verifier  # or provii-issuer/worker
wrangler deploy --env production

Deployment Verification:

  • Health check passes
  • Smoke tests pass
  • Monitoring shows normal operation
  • Rollback prepared if needed

7. Post-Deployment

Monitoring (first 30 minutes):

  • Error rates (Cloudflare Workers Logs in Grafana Loki)
  • Response times
  • Authentication success rates
  • Any anomalous behaviour

Rollback Triggers:

  • Error rate >5%
  • Critical functionality broken
  • Security issue introduced
  • Performance degradation >50%

Communication:

  • Internal: Team notification
  • External: Status page (if customer-impacting)

Rollback Procedures

Automatic Rollback

Not currently implemented. Manual rollback required.

Manual Rollback

Process:

# 1. Identify last known good commit
git log --oneline

# 2. Revert the problematic commit
git revert <commit-hash>
git push origin main

# 3. Or deploy specific good version
git checkout <good-commit>
cd provii-verifier
wrangler deploy --env production

# 4. Verify rollback successful
curl https://verify.provii.app/health

Timeline: Target <5 minutes for critical rollbacks

downpipes Release Channel

The downpipes engine and console do not run in Maelstrom’s account — they run in each customer’s own Cloudflare account, under the customer’s keys, with no standing inbound path (“runs dark”). Maelstrom cannot wrangler deploy into a customer tenant, so changes reach those customer-operated components through the signature-pinned update channel rather than the deployment steps above.

Release Pipeline (vendor side)

  1. The code change is proposed, reviewed and merged through the Standard/Normal/Emergency process above
  2. The release artefact is built and published as a static update to the signature-pinned R2 bucket at updates.downpipes.io
  3. The update is signed with the downpipes Update Signer, a key held offline by the operator and never deployed to any Worker
  4. The update channel PULLS — updates are never pushed into a tenant. The engine fetches the update on its own schedule and verifies the signature against a pinned public key before applying anything
  5. Application is canary-gated (safe-apply) with automatic rollback on failure; there is no phone-home

Vendor-side surfaces (control-plane)

The downpipes control-plane — which mints licence tokens and a content-free advisory beacon, holds no customer keys, data or Cloudflare tokens, and is fail-open (it never gates backup or restore) — is vendor-operated and changes through the same Cloudflare Workers process as Provii’s Workers (Standard/Normal/Emergency, CI deploy via wrangler deploy, rollback via wrangler rollback). Because the control-plane is fail-open by design, a failed control-plane change can never block a customer’s backup or restore.

Customer responsibility (CUEC)

Applying an update to a customer’s engine or console is the customer’s decision and is a complementary user-entity control. Maelstrom’s responsibility is limited to building, signing and publishing the update correctly; the offline-held Update Signer and the engine’s pinned-key signature verification protect the integrity of every release before a customer applies it.

Maelstrom’s own Cloudflare estate is backed up by our own production instance of downpipes, which receives engine and console updates through this same signature-pinned, pull-only channel — so we operate the release process as a customer of our own product. See Cloudflare Backup Evidence.

Change Documentation

Git Commit Messages

Format:

<type>: <brief description>

<detailed explanation>
<why the change was needed>
<what alternatives were considered>

Fixes: #issue-number

Types: feat, fix, refactor, docs, test, chore, security

Example:

security: Implement rate limiting for challenge creation

Add token bucket rate limiting to prevent API abuse.
Limit: 10 challenges per minute per IP.

Fixes: #123

Change Log

For Major Releases:

  • Update CHANGELOG.md
  • Follow Keep a Changelog format
  • Group by Added/Changed/Fixed/Security

Special Change Types

Cryptographic Changes

Examples: Provii’s zero knowledge proof and credential-issuance primitives; downpipes’ post-quantum hybrid archive sealing and the archive format spec.

Extra requirements:

  • Cryptographer review (if available) or thorough self-review
  • Extensive testing (including property-based, fuzz testing)
  • Backward compatibility considered
  • Documentation updated
  • May require coordinated release with dependent services

For downpipes, archives are sealed under customer-held keys (break-glass, operational and archive-signer keys), so a cryptographic change must preserve the ability of the MIT offline reader to restore from archive bytes plus the customer’s offline key alone. Changes to the public, frozen archive format spec are treated as cryptographic changes and require the same coordinated, backward-compatible handling. Changes to the vendor-side downpipes Update Signer (held offline, never deployed to any Worker) follow the key-rotation procedure rather than the standard deployment path.

Infrastructure Changes

Examples: wrangler.toml, KV namespace changes, Durable Object migrations; for downpipes, the vendor-operated control-plane and the signature-pinned update channel (updates.downpipes.io).

Requirements:

  • Test in development environment first
  • Document expected impact
  • Plan for rollback (KV changes may not be easily reversible)
  • ISMS Owner approval

The downpipes engine and console run in the customer’s own Cloudflare account; Maelstrom does not make infrastructure changes inside a customer tenant — those changes reach customers only via the downpipes Release Channel.

Access Control Changes

Examples: GitHub permissions, Cloudflare roles, API key changes

Requirements:

  • Documented justification
  • Principle of least privilege verified
  • Security Lead approval
  • Audit log check post-change

Change Calendar

Change Windows

Preferred Windows:

  • Standard changes: Any time (automated)
  • Normal changes: Business hours (easier monitoring)
  • Major changes: Tuesday-Thursday (avoid Fridays/Mondays)

Change Freeze:

  • None currently (continuous deployment)
  • May implement for major events (e.g., days before certification audit)

Metrics and KPIs

Tracked Metrics:

  • Deployment frequency (current: multiple per week)
  • Lead time for changes (commit to production)
  • Change failure rate (target: <5%)
  • Mean time to recovery (target: <1 hour)

Quarterly Review:

  • Analyse metrics
  • Identify improvement opportunities
  • Update procedures as needed

Emergency Change Authorisation

Who Can Authorise:

  1. ISMS Owner - Any emergency change
  2. Security Lead - Security incidents
  3. On-call Engineer - P0 incidents (retrospective approval within 24h)

Documentation:

  • Record in incident ticket
  • Explain urgency and risk
  • Document testing performed
  • Post-mortem within 24 hours

Document Information

  • Version. 1.2
  • Effective Date. 2025-01-13
  • Last Updated. 2026-06-19
  • Owner. ISMS Owner
  • Maintained By. Security Lead
  • Review Frequency. Annually
  • Next Review. 2026-11-21
  • Classification. Public
  • Change (1.2). downpipes added as second in-scope platform; the legacy backup worker retired in favour of self-hosted downpipes. Added the downpipes release channel (signature-pinned, pull-only update channel; fail-open control-plane; CUEC for applying updates) and extended the cryptographic- and infrastructure-change requirements to cover both platforms.