Skip to content

STD-DATA-064: Data Retention & Archival

Owner Classification Review Date Status
CDO Office Internal April 2027 Active

STD-DATA-064: Data Retention & Archival

Field Value
Standard STD-DATA-064
Title Data Retention & Archival
Status Draft
Owner Data Lead
Created 2026-04-03
Review Quarterly

Purpose

Define retention periods, archival tiers and automated lifecycle management for all data in Simpaisa's payment gateway. Regulatory requirements across PK (SBP), BD (Bangladesh Bank), NP (NRB) and IQ (CBI) mandate long retention of transaction records. Simultaneously, operational costs demand that data is tiered appropriately rather than kept hot indefinitely.

Scope

All data generated, processed or stored by Simpaisa: transaction records, API logs, PII, audit trails, analytics data, configuration data and backups. Applies across all storage systems: SurrealDB, S3, Glacier and any future data stores.

Current State

  • Transaction data retained indefinitely in primary database (SurrealDB).

  • API logs retained for ~6 months in logging pipeline before manual cleanup.

  • No automated archival — hot storage growing linearly with transaction volume.

  • PII retention not formally bounded — data persists until manual deletion.

  • Legal hold capability does not exist.

Gaps

  1. No defined retention periods per data classification.

  2. No automated archival pipeline — everything stays in hot storage.

  3. No legal hold mechanism to override normal retention.

  4. Storage costs increasing linearly — no tiering strategy.

  5. Retrieval SLAs undefined — no guarantee on archived data access time.

Target State

  • Defined retention periods per data class, aligned to regulatory maximums.

  • Automated archival pipeline (Temporal workflows): hot → warm → cold.

  • Legal hold capability overrides normal retention when activated.

  • Retrieval SLAs defined and met for each storage tier.

  • Automated cleanup job enforces retention limits.

Retention Periods

Data Class Retention Period Regulatory Basis
Transaction records 10 years SBP requirements (longest mandate)
Settlement/recon records 10 years SBP requirements
Audit trails 10 years PCI DSS, SBP, Bangladesh Bank
API request/response logs 2 years Operational + dispute resolution
PII (customer data) Until account closure + 7 years Multi-jurisdiction max
Analytics / behavioural 3 years Business requirement
System/infra logs 1 year Operational requirement
Debug/trace logs 30 days Operational requirement
Backups 90 days DR requirement

Storage Tiers

Tier Technology Access Latency Use Case Cost Profile
Hot SurrealDB <100ms Active queries, real-time ops $$$$
Warm S3 <1 hour Recent historical, dispute lookup $$
Cold Glacier <24 hours Regulatory archive, legal hold $

Archival Pipeline

Automated via Temporal scheduled workflows:

  1. Hot → Warm (SurrealDB → S3):

    • Transaction records: after 12 months.

    • API logs: after 6 months.

    • Audit trails: after 24 months.

    • Format: Parquet files, partitioned by date and market.

    • Validation: row count and checksum verification before deletion from hot.

  2. Warm → Cold (S3 → Glacier):

    • Transaction records: after 3 years.

    • API logs: after 18 months.

    • Audit trails: after 5 years.

    • Format: same Parquet, Glacier Deep Archive class.

  3. Cold → Deletion :

    • After retention period expires.

    • Legal hold check before any deletion.

    • Deletion logged in audit trail (itself retained for 10 years).

Retrieval SLAs

Tier Retrieval SLA Method
Hot <100ms Direct database query
Warm <1 hour Temporal workflow triggers S3 read
Cold <24 hours Glacier restore + S3 read

Retrieval requests tracked in Beads with tag data-retrieval.

  • Legal hold can be placed on any data set by CDO or Compliance Team.

  • Hold overrides normal retention — data is not archived or deleted.

  • Hold tracked in Beads with tag legal-hold.

  • Quarterly review of active legal holds at ARB.

  • Hold removal requires CDO approval.

Automated Cleanup

  • Temporal workflow runs daily, scanning for data past retention period.

  • Pre-deletion checks: legal hold status, archive verification.

  • Deletion is soft-delete for 7 days, then hard-delete.

  • All deletions logged with: data class, volume, date range, executor.

Actions

# Action Owner Deadline
1 Build Temporal archival workflow (hot → warm) Data Team 2026-Q2
2 Build Temporal archival workflow (warm → cold) Data Team 2026-Q3
3 Implement legal hold mechanism in Beads Platform Team 2026-Q2
4 Build automated cleanup job Data Team 2026-Q3
5 Migrate existing historical data to warm/cold tiers Data Team 2026-Q3

References

  • PII-HANDLING-STANDARD.md

  • DATA-ARCHITECTURE.md

  • SECURITY-ARCHITECTURE.md

  • CROSS-BORDER-COMPLIANCE-FRAMEWORK.md