Secret Management Standard

Owner	Classification	Review Date	Status
Security	Confidential	April 2027	Active

Secret Management Standard

Organisation: Simpaisa Holdings
Document Owner: Daniel O'Reilly, Chief Digital Officer
Classification: Confidential
Version: 1.0
Date: 3 April 2026
Status: Active

---

Table of Contents

1. Purpose

2. Current State

3. Secret Classification

4. Secret Storage

5. Secret Injection

6. Secret Scanning

7. Access Control

8. Rotation Procedures

9. Emergency Rotation

10. Monitoring & Alerting

11. Implementation Checklist

---

1. Purpose

Zero secrets in code, configuration, or logs. This is the single governing principle of this standard.

Simpaisa processes $1B+ in transactions annually. A single leaked secret — an operator API key, a signing key, a database credential — could enable:

• Unauthorised payment initiation or disbursement via compromised operator credentials.

• Transaction data theft via compromised database credentials.

• Infrastructure compromise via leaked AWS access keys.

• Merchant impersonation via leaked webhook signing keys.

This standard defines how every secret across Simpaisa's platform is classified, stored, accessed, rotated, and monitored. It applies to all environments (production, staging, sandbox), all services, and all personnel.

---

2. Current State

2.1 Current State Assessment

Aspect	Status	Score
Secret storage	AWS Parameter Store (SecureString) — mostly working	6/10
Secret rotation	No documented rotation schedule; no automated rotation	2/10
Secret scanning	No pre-commit hooks; no CI pipeline scanning	1/10
Access control	AWS IAM policies per service — reasonable but undocumented	5/10
Monitoring	CloudWatch basic logging; no anomaly detection	3/10

2.2 Known Exposures

Exposure	Severity	Status	Action Required
Surge deployment token indeploy.sh	CRITICAL	Known — documented in Security Architecture review	Rotate immediately ; remove from source control; move to Parameter Store; scan git history
Operator credentials encryption method undocumented	HIGH	Unknown whether operator_credentials table uses application-level encryption or relies solely on RDS encryption	Document and verify encryption method; migrate to KMS envelope encryption if not already implemented
No evidence of credential rotation history	HIGH	No audit trail of when secrets were last rotated	Establish rotation schedule; begin tracking rotation dates
AWS access keys — rotation cadence unknown	MEDIUM	IAM access keys may not have been rotated per AWS 90-day best practice	Audit all IAM access keys; rotate any older than 90 days

---

3. Secret Classification

3.1 Secret Types and Rotation Schedule

Type	Examples	Classification	Rotation Cadence	Overlap Period	Storage
RSA signing keys	Merchant API request signing keys (RSA-SHA256)	Restricted	Annual	30-day dual-key validation period	AWS Parameter Store (SecureString)
HMAC webhook keys	Per-merchant webhook signing secrets (HMAC-SHA256)	Restricted	Annual (auto); on merchant request	72-hour dual-secret validation window	AWS Parameter Store (SecureString)
Operator API keys	Easypaisa, JazzCash, bKash, Nagad, eSewa, Khalti API credentials	Restricted	Quarterly	Coordinated with operator; 7-day overlap	AWS Parameter Store (SecureString)
Database credentials	RDS MySQL (primary + read replicas), SurrealDB	Restricted	Quarterly	Zero-downtime: new credential activated before old deactivated	AWS Parameter Store (SecureString)
Service-to-service keys	Internal API authentication between Go services	Confidential	Quarterly (auto-rotate)	24-hour dual-key period	AWS Parameter Store (SecureString)
AWS access keys	IAM access keys for service accounts	Restricted	90-day rotation (AWS best practice)	7-day overlap; old key deactivated (not deleted) for rollback	AWS IAM + Parameter Store
Encryption keys (DEK)	AES-256 data encryption keys for PII column encryption	Restricted	Annual via KMS automatic rotation	Seamless — KMS handles key versioning	AWS KMS
Encryption keys (CMK)	KMS Customer Master Keys	Restricted	Annual via KMS automatic rotation	Seamless — previous versions remain available for decryption	AWS KMS
TLS certificates	Server certificates for mTLS, API endpoints	Confidential	Annual (or per CA expiry)	30-day pre-expiry renewal	AWS Certificate Manager / Cloudflare
OAuth client secrets	ControlPlane.com client credentials	Restricted	Quarterly	24-hour overlap	AWS Parameter Store (SecureString)
Redis auth tokens	ElastiCache authentication tokens	Confidential	Quarterly	Zero-downtime: ElastiCache supports dual-token authentication	AWS Parameter Store (SecureString)
Deployment tokens	CI/CD pipeline tokens (Bitbucket Pipelines)	Confidential	Quarterly	Immediate rotation; pipeline updated before old token revoked	Bitbucket Pipeline Variables (secured)

3.2 Classification Rules

Rule	Detail
Any credential that can initiate a payment is Restricted	Operator API keys, signing keys, database credentials with write access
Any credential that can read PII is Restricted	Database credentials, API keys with PII scope
Any credential that can modify infrastructure is Restricted	AWS access keys, deployment tokens, Cloudflare API keys
Internal service keys are Confidential minimum	Even internal keys can be exploited for lateral movement
No secret is ever classified below Confidential	By definition, a secret requires protection

---

4. Secret Storage

4.1 Current State

Aspect	Implementation
Primary store	AWS Systems Manager Parameter Store — SecureString parameters encrypted with KMS
Encryption	KMS-managed key per parameter (aws/ssm default or custom CMK)
Versioning	Parameter Store maintains version history
Access control	IAM policies restrict access per service role
Audit logging	CloudTrail logs all Parameter Store API calls

4.2 Target State

Aspect	Current	Target	Timeline
Primary store	AWS Parameter Store	AWS Parameter Store (continue) + evaluate ControlPlane.com secrets	Q2 2026 evaluation
Encryption	Default SSM key	Custom CMK per secret classification (Restricted vs Confidential)	Q2 2026
Automated rotation	None	AWS Lambda-based rotation for database credentials; custom rotation for operator keys	Q3 2026
Secret versioning	Parameter Store versions	Maintain last 3 versions for rollback; older versions marked for deletion	Q2 2026
Cross-region	Single region	Per-market Parameter Store instances (data residency compliance)	Q3 2026

4.3 Evaluation: ControlPlane.com

Criterion	Requirement	ControlPlane.com
Secrets management	Store, version, rotate secrets	Evaluate capabilities
Access control	Per-service, per-environment access policies	Identity-based access model
Audit logging	Complete audit trail of secret access	Built-in audit logging
Multi-jurisdiction	Secrets stored per market for data residency	Evaluate region support
Integration	Go SDK, Kubernetes integration	Evaluate Go SDK maturity

Decision timeline: Evaluation complete by Q2 2026. If ControlPlane.com meets requirements, begin migration Q3 2026. AWS Parameter Store remains the fallback.

4.4 Evaluation: HashiCorp Vault

Criterion	Requirement	HashiCorp Vault
Secrets management	Store, version, rotate secrets	Full capabilities (KV v2, dynamic secrets)
Access control	Per-service, per-environment	Policy-based; AppRole, Kubernetes auth
Audit logging	Complete audit trail	Audit backend (file, syslog, socket)
Operational overhead	Minimal self-management	Concern: Vault requires dedicated operational expertise; high availability setup is complex
Cost	Within budget	HCP Vault (managed) vs self-hosted

Decision: Evaluate alongside ControlPlane.com. Operational overhead is the primary concern for a small team.

4.5 Where Secrets Must NOT Be Stored

Location	Status	Enforcement
Source code (any file in git)	PROHIBITED	Pre-commit hook (gitleaks); CI pipeline scan
Environment files committed to git (.env, .env.production)	PROHIBITED	.gitignore rule; pre-commit hook
CI/CD pipeline logs	PROHIBITED	Mask all secret variables in Bitbucket Pipelines
Application logs	PROHIBITED	Log masking (see PII-HANDLING-STANDARD.md Section 6)
Docker images / container layers	PROHIBITED	Multi-stage builds; secrets injected at runtime, not build time
Wiki / Confluence / Slack	PROHIBITED	Security awareness training; DLP scanning
Local developer machines (plain text)	PROHIBITED	Use aws ssm get-parameter for local dev; never copy secrets to files
Terraform state files (unencrypted)	PROHIBITED	Terraform state encrypted in S3 with KMS; state access restricted
CLI arguments (visible in process list)	PROHIBITED	Pass via environment variables or mounted files, never CLI args

---

5. Secret Injection

5.1 Injection Methods

Method	Use Case	Implementation
Environment variables	Primary method for all services	Service reads from environment at startup; values injected by container orchestrator from Parameter Store
Mounted files	TLS certificates, large keys	Secret mounted as read-only file in /run/secrets/; permissions 0400 (owner read only)
AWS SDK	Dynamic secret retrieval	Service uses AWS SDK to fetch from Parameter Store at startup; cached in memory with TTL

5.2 Injection Rules

Rule	Detail
Never in code	No secret values in source code, configuration files, or build scripts — ever
Never in CLI arguments	Secrets passed via environment variables, not command-line arguments (which appear in /proc and process listings)
Never in Docker build	Secrets not available during docker build; injected at container runtime only
Immutable after injection	Once a service starts with a secret, it does not change during the process lifetime; rotation requires service restart or graceful reload
Memory protection	Secrets stored in memory should be cleared (zeroed) when no longer needed; Go memguard or equivalent

5.3 Local Development

Scenario	Approach
Developers needing secrets	Use aws ssm get-parameter --name /simpaisa/dev/{secret} to fetch from dev Parameter Store; never copy to files
Sandbox environment	Sandbox uses separate secrets from production; distinct Parameter Store path (/simpaisa/sandbox/)
CI/CD pipelines	Bitbucket Pipeline secured variables; masked in logs; per-environment variable sets

---

6. Secret Scanning

6.1 Pre-Commit Scanning

Aspect	Implementation
Tool	gitleaks (preferred) or git-secrets
Scope	Every commit across all Simpaisa repositories
Installation	Mandatory pre-commit hook installed via team onboarding checklist
Bypass	--no-verify is prohibited by team policy; CI will catch anything that bypasses the hook
Custom rules	Simpaisa-specific patterns: operator API key formats, AWS key patterns, RSA key markers

6.2 CI Pipeline Scanning

Aspect	Implementation
Tool	gitleaks integrated into Bitbucket Pipelines
Trigger	Every pull request; every merge to main / master
Action on detection	Pipeline fails; PR cannot be merged; security team notified
False positive handling	.gitleaksignore file for documented false positives (reviewed quarterly)

6.3 Retrospective Scanning

Aspect	Implementation
Scope	Full git history of all Simpaisa repositories
Tool	gitleaks with --no-git flag scanning all historical commits
Frequency	One-time comprehensive scan (immediate priority); then quarterly
Known findings	Surge token in deploy.sh — rotate immediately
Action on discovery	Secret rotated immediately; git history rewritten if feasible; incident documented

6.4 Continuous Monitoring

Aspect	Implementation
GitHub/Bitbucket secret scanning	Enable Bitbucket's built-in secret scanning if available
Third-party monitoring	Evaluate services that monitor public paste sites, dark web for leaked credentials
Internal scanning	Weekly automated scan of all deployed container images for embedded secrets

---

7. Access Control

7.1 Principle of Least Privilege

Rule	Detail
Per-service access	Each service has its own IAM role with access to only the secrets it needs
No shared credentials	No two services use the same secret; if two services need the same upstream credential, create separate copies with separate access policies
No human access to production secrets	Engineers do not have direct access to production secrets; access requires break-glass procedure
Environment isolation	Production, staging, and sandbox secrets are in separate Parameter Store paths with separate IAM policies

7.2 IAM Policy Structure

/simpaisa/
  ├── production/
  │   ├── payin/          ← Pay-In service role only
  │   │   ├── db-credentials
  │   │   ├── easypaisa-api-key
  │   │   ├── jazzcash-api-key
  │   │   └── signing-key
  │   ├── payout/         ← Pay-Out service role only
  │   │   ├── db-credentials
  │   │   ├── bank-api-keys
  │   │   └── signing-key
  │   ├── remittance/     ← Remittance service role only
  │   │   ├── db-credentials
  │   │   ├── corridor-api-keys
  │   │   └── signing-key
  │   ├── cards/          ← Cards service role only
  │   │   ├── db-credentials
  │   │   ├── processor-api-key
  │   │   └── signing-key
  │   └── shared/         ← Gateway and infrastructure roles
  │       ├── redis-auth-token
  │       ├── nsq-auth-token
  │       └── krakend-config-key
  ├── staging/            ← Same structure, different IAM policies
  └── sandbox/            ← Same structure, different IAM policies

7.3 Break-Glass Access

Step	Action	Owner
1	Engineer submits break-glass request with justification (incident ticket reference)	Requesting engineer
2	CDO or delegate approves request	CDO / Security Lead
3	Temporary IAM policy attached to engineer's role (1-hour TTL)	Platform Engineering
4	Engineer accesses secret; all access logged via CloudTrail	Engineer
5	Temporary policy automatically expires; access revoked	AWS IAM (TTL-based)
6	Post-access review — verify access was appropriate; determine if secret needs rotation	Security team

7.4 Audit Logging

Event	Logged	Source
Secret read	Yes — who, when, which secret, from which IP	CloudTrail
Secret write/update	Yes — who, when, which secret, old version archived	CloudTrail
Secret delete	Yes — who, when, which secret	CloudTrail
Failed access attempt	Yes — who, when, which secret, denial reason	CloudTrail
Break-glass access	Yes — request, approval, access, and revocation	CloudTrail + internal audit log
Rotation event	Yes — automated or manual, old version archived	CloudTrail + rotation job logs

---

8. Rotation Procedures

8.1 Database Credentials (RDS MySQL, SurrealDB)

Step	Action	Downtime
1	Generate new credentials (strong random password, 32+ characters)	None
2	Create new database user with identical permissions	None
3	Store new credentials in Parameter Store (new version)	None
4	Rolling restart of service instances to pick up new credentials	None (rolling)
5	Verify all instances connected with new credentials (connection pool monitoring)	None
6	Disable old database user (do not delete — retain for 7 days for rollback)	None
7	Delete old database user after 7-day rollback window	None

Automation target: AWS Lambda function triggered by EventBridge schedule (quarterly).

8.2 Operator API Keys (Easypaisa, JazzCash, bKash, etc.)

Step	Action	Downtime
1	Request new API credentials from operator partner (coordination required)	None
2	Receive new credentials from operator via secure channel (not email)	None
3	Store new credentials in Parameter Store (new version)	None
4	Update KrakenD / service configuration to use new credentials	None
5	Validate by sending test transaction through the operator channel	None
6	Confirm old credentials deactivated by operator	None
7	Mark old credentials as rotated in audit log	None

Coordination: Operator key rotation requires scheduling with the operator partner. Minimum 2-week lead time. Coordinated overlap period of 7 days where both old and new credentials are active.

8.3 RSA Signing Keys (Merchant API Signing)

Step	Action	Downtime
1	Generate new RSA key pair (4096-bit minimum)	None
2	Store new private key in Parameter Store; publish new public key to merchant	None
3	Begin dual-key validation period: accept signatures from both old and new keys	None
4	Notify merchants to update to new public key (30-day migration window)	None
5	Monitor: track which merchants are still using old key	None
6	After 30 days: disable old key validation; reject signatures from old key	None
7	Archive old key pair (encrypted) for signature verification of historical data if needed	None

8.4 Webhook Signing Secrets (Per-Merchant HMAC Keys)

Step	Action	Downtime
1	Generate new HMAC secret (256-bit random)	None
2	Store new secret in Parameter Store	None
3	Begin dual-secret period: send webhooks with signatures from both old and new secrets (72-hour window)	None
4	Notify merchant of new secret via secure channel (DevEx portal, not email)	None
5	Merchant updates their verification to use new secret	None
6	After 72 hours: stop signing with old secret	None
7	Delete old secret from Parameter Store	None

8.5 AWS Access Keys

Step	Action	Downtime
1	Create new access key for the IAM user/role	None
2	Update all services/pipelines using the old key to use the new key	None
3	Deactivate old access key (do not delete — 7-day rollback window)	None
4	After 7 days with no access attempts on old key: delete old key	None

Best practice: Prefer IAM roles (instance profiles, task roles) over access keys wherever possible. Access keys should only exist where role assumption is not feasible.

8.6 Encryption Keys (KMS)

Step	Action	Downtime
1	KMS automatic rotation generates new key material annually	None
2	New data encrypted with new key material; old data decryptable with previous key material	None
3	No application changes required — KMS handles key versioning transparently	None

Note: KMS key rotation is seamless. No manual intervention required. Monitor via CloudTrail for rotation events.

---

9. Emergency Rotation

9.1 Triggers

Emergency rotation is required when:

Trigger	Example	Response Time
Secret confirmed exposed	Credential found in public git repository, paste site, or logs	Immediate — within 1 hour
Secret suspected compromised	Anomalous access patterns; employee departure without credential handoff	Urgent — within 4 hours
Upstream breach notification	Operator partner reports their systems compromised	Urgent — within 4 hours
Internal security incident	Unauthorised access to Parameter Store detected	Immediate — within 1 hour

9.2 Emergency Rotation Procedure

Step	Action	Owner	Timeline
1	Assess scope — identify all secrets potentially compromised and all systems using them	Security team	15 minutes
2	Revoke immediately — disable/revoke the compromised secret at the source	Security team	30 minutes
3	Generate new secret — create replacement credential	Security team + Platform Engineering	30 minutes
4	Deploy new secret — update Parameter Store; rolling restart affected services	Platform Engineering	1 hour
5	Verify — confirm all services operational with new credentials; no transaction failures	Platform Engineering	30 minutes
6	Investigate — determine how the secret was exposed; address root cause	Security team	24 hours
7	Document — incident report with timeline, impact, root cause, and preventive measures	Security team	48 hours
8	Notify — inform CDO, affected merchants (if applicable), regulators (if required)	Compliance + Security	Per breach notification timelines

9.3 Immediate Actions for Known Exposure: Surge Token

Action	Detail	Owner	Timeline
Rotate Surge token	Generate new Surge deployment token	Platform Engineering	Immediate
Remove from deploy.sh	Delete hardcoded token from source code	Platform Engineering	Immediate
Store in Bitbucket Pipeline Variables	New token stored as secured pipeline variable	Platform Engineering	Immediate
Scan git history	Use git filter-repo or BFG to remove token from git history	Platform Engineering	Within 24 hours
Verify removal	Confirm token no longer appears in any file or git history	Security team	Within 48 hours

---

10. Monitoring & Alerting

10.1 Alerts

Alert	Condition	Severity	Action
Anomalous secret access	Secret accessed from unexpected IP, role, or at unusual time	High	Security team investigates; potential compromise
Failed access attempts

3 failed attempts to access a secret within 5 minutes

| Medium| Verify requesting service; check for misconfiguration or attack
Secret approaching rotation deadline| Secret within 7 days of scheduled rotation date| Warning| Platform Engineering schedules rotation
Secret past rotation deadline| Secret has not been rotated by scheduled date| High| Escalate to CDO; rotate immediately
New secret created| Any new secret added to Parameter Store| Info| Verify creation was authorised; ensure proper classification
Secret deleted| Any secret removed from Parameter Store| High| Verify deletion was authorised; check for service impact
Break-glass access| Any break-glass access request| High| CDO notified; post-access review scheduled
Pre-commit hook bypass| Commit pushed without pre-commit scan passing| High| CI pipeline blocks merge; security team notified

10.2 Dashboard (Grafana)

Panel	Description
Secret access heatmap	Access frequency per secret per hour (identify anomalies)
Rotation compliance	Table of all secrets with last rotation date, next rotation date, status (compliant/overdue)
Failed access attempts	Time series of denied access attempts
Break-glass events	Log of all break-glass access requests and outcomes
Secret age distribution	Histogram showing age of all secrets (identify stale credentials)

10.3 Metrics (OpenTelemetry)

Metric	Type	Labels
simpaisa.secrets.access.total	Counter	secret_name, service, result (success/denied)
simpaisa.secrets.rotation.age_days	Gauge	secret_name, secret_type
simpaisa.secrets.rotation.overdue	Gauge	secret_name (1 if overdue, 0 if compliant)
simpaisa.secrets.breakglass.total	Counter	requester, secret_name

---

11. Implementation Checklist

11.1 Immediate (P0)

• Rotate Surge token — remove from deploy.sh, store in Bitbucket Pipeline Variables
• Retrospective git scan — run gitleaks against all repository histories
• Document operator credential encryption — verify and document how operator_credentials table is encrypted
• Audit AWS access key ages — rotate any access keys older than 90 days

11.2 Short-Term (Q2 2026)

• Pre-commit hooks (gitleaks) installed across all repositories
• CI pipeline secret scanning integrated into Bitbucket Pipelines
• Secret rotation schedule documented for all secret types
• Per-service IAM policies reviewed and tightened (least privilege)
• Parameter Store path structure reorganised per Section 7.2
• Custom CMKs provisioned per secret classification
• Break-glass procedure documented and tested
• Grafana dashboard for secret management operational

11.3 Medium-Term (Q3–Q4 2026)

• Automated database credential rotation via AWS Lambda
• Automated service-to-service key rotation
• ControlPlane.com evaluation complete; migration decision made
• Per-market Parameter Store instances for data residency compliance
• Secret age monitoring and alerting operational
• Quarterly retrospective scans scheduled
• All operator key rotations on quarterly schedule

11.4 Ongoing

• Rotation schedule maintained — no secret exceeds its cadence
• Pre-commit hooks and CI scanning verified monthly
• Break-glass access reviewed within 24 hours of each event
• Secret inventory reconciled quarterly (no orphaned or unused secrets)
• New services — secret requirements reviewed during architecture review

---

Cross-References

Document	Relevance
SECURITY-ARCHITECTURE.md	Section 10: Credential & Secret Management
INFRASTRUCTURE-STANDARDS.md	Section 11: Secret Management; Section 15: CI/CD Pipeline Standards
PII-HANDLING-STANDARD.md	Section 5: Encryption Standards (key management)
DATA-ARCHITECTURE.md	Section 9: Encryption Standards
INCIDENT-RESPONSE-PLAYBOOK.md	Emergency rotation triggers incident response
VENDOR-INTEGRATION-REGISTER.md	Operator API key details and rotation coordination contacts