Skip to content

Webhook Delivery Workflow

Temporal Workflow Definition Service: notification-svc Markets: PK, BD, NP, IQ, EG Scale: 270M+ transactions, $1B+ processed


Overview

The Webhook Delivery Workflow guarantees at-least-once delivery of event notifications to merchant-configured HTTPS endpoints. It implements exponential backoff with a fixed retry schedule, HMAC-SHA256 payload signing, and dead-letter queue (DLQ) escalation on exhaustion.

Idempotency is ensured by using the eventId as the Temporal Workflow ID — duplicate events are automatically deduplicated by the Temporal server.


Workflow Definition

package workflows

import (
    "time"

    "go.temporal.io/sdk/temporal"
    "go.temporal.io/sdk/workflow"
)

// WebhookPayload represents the event to be delivered.
type WebhookPayload struct {
    EventID     string                 // Unique event identifier (CloudEvents id)
    EventType   string                 // e.g. payin.success, payout.failed
    MerchantID  string
    Timestamp   time.Time
    Data        map[string]interface{} // Event-specific payload
}

// MerchantWebhookConfig holds the merchant's webhook settings.
type MerchantWebhookConfig struct {
    WebhookID   string
    MerchantID  string
    URL         string   // HTTPS endpoint
    Secret      string   // HMAC-SHA256 signing secret
    Version     string   // Payload version (e.g. 2024-01-01)
    EventTypes  []string // Subscribed event types
}

// WebhookDeliveryResult is the final outcome.
type WebhookDeliveryResult struct {
    EventID      string
    Delivered    bool
    Attempts     int
    FinalStatus  string // DELIVERED, DLQ, CANCELLED
    LastError    string
    CompletedAt  time.Time
}

// retrySchedule defines the fixed backoff intervals for webhook delivery.
// Total span: ~91 hours (approximately 3.8 days).
var retrySchedule = []time.Duration{
    1 * time.Minute,
    5 * time.Minute,
    30 * time.Minute,
    2 * time.Hour,
    12 * time.Hour,
    24 * time.Hour,
    72 * time.Hour,
}

// WebhookDeliveryWorkflow delivers a webhook payload to a merchant endpoint.
//
// Workflow ID: webhook-{eventId}  (ensures idempotency per event)
// Task Queue:  webhook-worker
func WebhookDeliveryWorkflow(ctx workflow.Context, payload WebhookPayload, config MerchantWebhookConfig) (*WebhookDeliveryResult, error) {

    logger := workflow.GetLogger(ctx)

    // -----------------------------------------------------------------
    // Activity options
    // -----------------------------------------------------------------
    serialiseOpts := workflow.ActivityOptions{
        StartToCloseTimeout: 5 * time.Second,
        RetryPolicy: &temporal.RetryPolicy{
            MaximumAttempts: 3,
        },
    }

    signOpts := workflow.ActivityOptions{
        StartToCloseTimeout: 5 * time.Second,
        RetryPolicy: &temporal.RetryPolicy{
            MaximumAttempts: 3,
        },
    }

    deliverOpts := workflow.ActivityOptions{
        StartToCloseTimeout: 30 * time.Second,
        RetryPolicy: &temporal.RetryPolicy{
            MaximumAttempts: 1, // We manage retries ourselves via the schedule
        },
    }

    dlqOpts := workflow.ActivityOptions{
        StartToCloseTimeout: 30 * time.Second,
        RetryPolicy: &temporal.RetryPolicy{
            MaximumAttempts: 5,
            BackoffCoefficient: 2.0,
        },
    }

    // -----------------------------------------------------------------
    // Query handler: delivery status
    // -----------------------------------------------------------------
    deliveryStatus := &DeliveryStatus{
        EventID:  payload.EventID,
        Status:   "PENDING",
        Attempts: 0,
    }

    err := workflow.SetQueryHandler(ctx, "delivery-status", func() (*DeliveryStatus, error) {
        return deliveryStatus, nil
    })
    if err != nil {
        logger.Error("Failed to register query handler", "error", err)
    }

    // -----------------------------------------------------------------
    // Step 1: Serialise Payload
    // -----------------------------------------------------------------
    serialiseCtx := workflow.WithActivityOptions(ctx, serialiseOpts)
    var serialisedPayload []byte
    err = workflow.ExecuteActivity(serialiseCtx, SerialisePayload, SerialiseInput{
        Payload: payload,
        Version: config.Version,
    }).Get(ctx, &serialisedPayload)
    if err != nil {
        return &WebhookDeliveryResult{
            EventID:     payload.EventID,
            Delivered:   false,
            FinalStatus: "SERIALISATION_FAILED",
            LastError:   err.Error(),
            CompletedAt: workflow.Now(ctx),
        }, nil
    }

    // -----------------------------------------------------------------
    // Step 2: Sign Payload (HMAC-SHA256)
    // -----------------------------------------------------------------
    signCtx := workflow.WithActivityOptions(ctx, signOpts)
    var signature string
    err = workflow.ExecuteActivity(signCtx, SignPayload, SignPayloadInput{
        Payload: serialisedPayload,
        Secret:  config.Secret,
    }).Get(ctx, &signature)
    if err != nil {
        return &WebhookDeliveryResult{
            EventID:     payload.EventID,
            Delivered:   false,
            FinalStatus: "SIGNING_FAILED",
            LastError:   err.Error(),
            CompletedAt: workflow.Now(ctx),
        }, nil
    }

    // -----------------------------------------------------------------
    // Step 3: Attempt Delivery with Retry Schedule
    // -----------------------------------------------------------------
    deliverCtx := workflow.WithActivityOptions(ctx, deliverOpts)
    maxAttempts := 1 + len(retrySchedule) // initial attempt + retries
    var lastError string

    for attempt := 0; attempt < maxAttempts; attempt++ {
        deliveryStatus.Attempts = attempt + 1
        deliveryStatus.Status = "DELIVERING"

        var response DeliveryResponse
        err = workflow.ExecuteActivity(deliverCtx, DeliverWebhook, DeliverWebhookInput{
            URL:       config.URL,
            Payload:   serialisedPayload,
            Signature: signature,
            EventID:   payload.EventID,
            EventType: payload.EventType,
            Attempt:   attempt + 1,
        }).Get(ctx, &response)

        if err == nil {
            // ---------------------------------------------------------
            // Step 4: Verify Response
            // ---------------------------------------------------------
            verifyCtx := workflow.WithActivityOptions(ctx, serialiseOpts)
            var verified bool
            _ = workflow.ExecuteActivity(verifyCtx, VerifyResponse, VerifyResponseInput{
                StatusCode: response.StatusCode,
                Body:       response.Body,
            }).Get(ctx, &verified)

            if verified {
                deliveryStatus.Status = "DELIVERED"
                return &WebhookDeliveryResult{
                    EventID:     payload.EventID,
                    Delivered:   true,
                    Attempts:    attempt + 1,
                    FinalStatus: "DELIVERED",
                    CompletedAt: workflow.Now(ctx),
                }, nil
            }
        }

        lastError = ""
        if err != nil {
            lastError = err.Error()
        }
        deliveryStatus.LastError = lastError

        // Wait before next retry (unless this was the last attempt)
        if attempt < len(retrySchedule) {
            deliveryStatus.Status = "WAITING_RETRY"
            _ = workflow.Sleep(ctx, retrySchedule[attempt])
        }
    }

    // -----------------------------------------------------------------
    // Step 5: Send to Dead Letter Queue (all retries exhausted)
    // -----------------------------------------------------------------
    deliveryStatus.Status = "DLQ"
    dlqCtx := workflow.WithActivityOptions(ctx, dlqOpts)
    _ = workflow.ExecuteActivity(dlqCtx, SendToDLQ, DLQInput{
        EventID:    payload.EventID,
        EventType:  payload.EventType,
        MerchantID: payload.MerchantID,
        Payload:    serialisedPayload,
        Attempts:   maxAttempts,
        LastError:  lastError,
        URL:        config.URL,
    }).Get(ctx, nil)

    return &WebhookDeliveryResult{
        EventID:     payload.EventID,
        Delivered:   false,
        Attempts:    maxAttempts,
        FinalStatus: "DLQ",
        LastError:   lastError,
        CompletedAt: workflow.Now(ctx),
    }, nil
}

Activities Summary

Activity Description Idempotent
SerialisePayload Converts event payload to versioned JSON format Yes
SignPayload Computes HMAC-SHA256 signature using merchant secret Yes
DeliverWebhook HTTP POST to merchant endpoint with signed payload and headers Yes (at-least-once)
VerifyResponse Validates HTTP response (2xx = success) Yes
SendToDLQ Persists failed delivery to dead-letter queue for manual inspection Yes

HTTP Delivery Headers

POST {merchant_url}
Content-Type: application/json
X-Simpaisa-Event-ID: {eventId}
X-Simpaisa-Event-Type: {eventType}
X-Simpaisa-Signature: sha256={hmac_hex}
X-Simpaisa-Timestamp: {unix_timestamp}
X-Simpaisa-Delivery-Attempt: {attempt_number}
User-Agent: Simpaisa-Webhooks/1.0

Retry Schedule

Attempt Delay After Failure Cumulative Time
1 Immediate 0
2 1 minute 1 minute
3 5 minutes 6 minutes
4 30 minutes 36 minutes
5 2 hours 2h 36m
6 12 hours 14h 36m
7 24 hours 38h 36m
8 72 hours 110h 36m (~4.6 days)
DLQ N/A Exhausted

Query Handlers

Query Response Purpose
delivery-status DeliveryStatus{EventID, Status, Attempts, LastError} Check current delivery state from merchant dashboard or support tooling

Idempotency

  • Workflow ID = webhook-{eventId} — Temporal rejects duplicate starts for the same event
  • Delivery is at-least-once — merchants must handle duplicate deliveries gracefully
  • The X-Simpaisa-Event-ID header allows merchants to deduplicate on their side

Failure Modes

Failure Behaviour
Serialisation fails Workflow completes with SERIALISATION_FAILED status
Signing fails Workflow completes with SIGNING_FAILED status
Merchant returns non-2xx Retried per schedule
Merchant endpoint unreachable Retried per schedule
All retries exhausted Sent to DLQ, merchant notified via dashboard
Merchant disables webhook Workflow cancelled via Temporal API