Designing Workflows That Don't Break

Every workflow node should be idempotent — processing the same input twice produces the same output. This is critical because retries, network glitches, and duplicate webhooks are facts of life in production. Design for exactly-once semantics, but implement for at-least-once delivery.

When a node fails after all retries, the message shouldn't disappear. Dead-letter queues capture failed messages for inspection, manual resolution, or reprocessing. Without them, silent failures corrupt your data integrity and you won't know until a customer complains.

We implement dead-letter queues at every node boundary. Each DLQ includes the original message, error details, retry count, and timestamp. This gives operators full context for debugging.

Not every failure requires a full stop. Design workflows to degrade gracefully: if the enrichment API is down, proceed with partial data. If the notification service fails, queue the notification for later. The goal is maximum throughput with minimum data loss.

Alert on business outcomes, not technical symptoms. 'Order processing latency > 30s' matters more than 'CPU > 80%'. Build dashboards that show throughput, error rates, and processing times per node. Set alerts on SLO breaches, not arbitrary thresholds.