Building for recovery - To Redis or not?
A decision framework for Redis vs. orchestrators failures
At Swym, folks probably know one of the hills I’ll die on is that there is still nothing like Redis queues for baseline throughput and cognitive simplicity.
And no matter what is promised, once a broker or Temporal-class orchestrator enters the stack, we are buying a different operating model, not just a better queue. But for seasoned builders and engineers, the useful version is: it depends on which costs we want now vs later, and where we can tolerate ambiguity.
The mistake is pretending that queue choice is an architecture decision once and done. It is an operating cost decision that keeps compounding.
Here is the non-gyaan version (maybe a bit of TLDR;):
Redis buys speed now and pushes coordination cost into application code later.
Temporal/brokers buy explicit workflow semantics now and charge platform tax immediately.
Both work.
Both hurt.
There is a sweet spot between hurt and gains that will match the team’s failure appetite.
So “it depends” is lifting a lot of weight clearly, let’s talk through what “it” “depends” on.
Some notes before going into it
This is not about our AI-first ops, those fit better on separate detailed posts
This post focuses on the Redis vs. Orchestrator extremes. If you are reaching for SQS/SNS or Kafka, you are generally landing in the 'Redis-like' bucket—optimizing for throughput and eventual consistency—but with a different set of platform tradeoffs. We treat them as 'Redis-first' for the purpose of this argument because the fundamental trade-off remains: platform-provided durability vs. application-logic-defined flow.
Redis - Still the fastest way to a useful system
A quick reality check: 'Redis-first' is only 'cognitive simplicity' if you already have a mature Redis management strategy. Once you hit scale, persistence requirements, and replication lag, maintaining a high-throughput Redis cluster can become an operational tax of its own. Do not assume it is free infrastructure.
We already know the upside:
Minimal bootstrap time
Familiar operational model
Tight feedback loops in local/dev
Low ceremony for producer/consumer patterns
For narrow, high-throughput, short-lived jobs, Redis is often the right default.
What gets missed is that this default silently expands from “queue” to “execution semantics” over time.
We start with enqueue/dequeue. We end up hand-building retry policy, poison message handling, delayed jobs, idempotency, visibility, backoff, dead-letter behavior, and failure observability.
At that point, we did not avoid orchestration complexity. We just forced it into application code.
Temporal/Brokers - Complexity relocated
The common debate is framed as simplicity vs complexity. That’s not as well connected to any real first principles for us.
The real trade-off is:
Local simplicity (Redis-first): fewer moving parts early, more custom semantics later
Systemic explicitness (Temporal/broker-first): more moving parts early, fewer implicit behaviors later
Temporal/broker ecosystems buy us explicit workflow state, durable execution history, replay/debug affordances, and stronger long-running control primitives.
They also introduce:
Platform dependency
New operational surface area
Team-wide learning tax
Higher blast radius when orchestration infra degrades
At this stage, it sounds fair to say neither is “better” by default.
Receipts from our own systems
This is not theory, here are real issues (recorded on our internal tracking, but more generalized)
Manual trigger gaps turn into compliance risk when workflow ownership is implicit.
Missing fail paths in queue processors silently drop obligations.
Small reliability defaults become large incident multipliers at scale.
Storage runs out with a tsunami spike - affects both brokered queues and redis-based queues
Cross-repo write/read split creates real execution drag in production fixes.
No audit trail on template mutation is not “small tooling debt”, it is operational blindness.
Even service-to-service auth details can fail in ways that look trivial but hurt recovery speed.
Temporal setup is explicitly heavier, even in docs, because you are buying orchestration primitives not just a queue.
Event infrastructure with Kafka/Event Hubs has clear setup and operational surface area beyond app logic.
We already encode “fail fast vs degrade gracefully” differently across repos. That alone tells you cost is contextual, not ideological.
Cost Model: What Actually Bites in Production
1. Infra Cost
Redis-first:
Cheap baseline infra footprint
Predictable memory/network profile initially
Costs spike when adding sidecar services for delayed/retry/scheduling metrics
Temporal/broker-first:
Higher fixed platform cost from day one
Additional storage for event history/state
More cross-service network chatter
Often more expensive in low-volume workloads, relatively cheaper in high-coordination workloads
Hidden cost in both:
Observability spend (logs, traces, cardinality explosions, retention)
Peak-capacity headroom for incident scenarios, not average traffic
2. Maintenance Cost
Redis-first:
Policy logic drifts across services
Retry/backoff semantics fork over time
“One-off” processors accumulate bespoke behavior
Documentation lags actual queue behavior
Temporal/broker-first:
Versioning workflows/activities safely is non-trivial
Backward compatibility and migration discipline required
SDK/runtime upgrades can create broad churn
Platform ownership becomes a standing function, not a task
3. Debugging Cost
Redis-first:
Great for simple failure modes
Painful when failures are multi-step and asynchronous
Hard to reconstruct causal chains across retries and partial side effects
Temporal/broker-first:
Better execution history and traceability
Better post-facto reasoning for long workflows
New class of bugs: determinism/replay/versioning edge cases, activity timeouts, worker lifecycle behaviors
Bottom line:
Redis debugging pain is usually missing context.
Temporal debugging pain is usually semantic correctness under platform rules.
4. Scaling Cost
Redis-first:
Scales throughput quickly
Coordination semantics degrade first under scale (dedupe, ordering, retries, race windows)
More load reveals consistency gaps, not just capacity gaps
Temporal/broker-first:
Better scaling for multi-step orchestration and delayed/compensating flows
Throughput still bounded by worker design, activity latency, and downstream rate limits
Cluster sizing and partitioning strategy become first-order concerns
5. On-call and Incident Cost
Redis-first incidents:
Faster to triage initially
Longer tail for “why did this happen this way in this order?”
More manual compensations and data patching
Temporal/broker-first incidents:
Better forensic trail
More infra+app joint incidents
Requires stronger runbook quality and clearer ownership boundaries
Competing Contexts (Where “Just Use X” Fails)
Use Redis-first when:
Jobs are short-lived and reversible
Throughput matters more than long-lived coordination
Failure impact is bounded and compensations are cheap
Team cannot yet justify orchestration platform ownership
Use Temporal/broker-first when:
Workflows are long-lived, stateful, or compliance-sensitive
We need explicit retries, timeouts, cancellation, and auditability
Cross-service coordination is core, not incidental
Failure cost is high enough to justify platform overhead
Use hybrid intentionally, not accidentally:
Redis for high-volume fire-and-forget lanes
Orchestrator for critical, multi-step, high-blast-radius paths
Shared invariants: idempotency keys, correlation IDs, retry taxonomy, dead-letter strategy
The Hard Rule: If you find yourself building a complex state machine inside your Redis consumer logic—tracking status, retries, and conditions—you have already lost the bet. Port it to the Orchestrator, or simplify the requirement. Don't build a poor man's orchestrator in your queue code
A practical lens for other mid to senior teams
Before choosing tooling, answer these clearly:
What is the maximum acceptable ambiguity during incident reconstruction?
Which failures must be automatically recoverable vs manually compensable?
Where do we need strict audit trails?
What is the cost of being wrong: latency, money, trust, or compliance?
If these are vague, architecture choice will not resemble engineering, more ideology.
The Decision Matrix
Final Take
Redis queues are still the best baseline for many systems. Temporal/brokers are not over-engineering by default; they are explicit bets on failure semantics and recoverability.
The wrong move is not as much choosing one over the other but it is pretending either comes without an operating cost.
Coming back to us - Our failure mode dictates what infrastructure we pick, in our case it has been both. As a first principle, we don’t optimize for architecture aesthetics at Swym, we optimize for recoverability. In other words - what’s the worst failure to handle before they happen as much as possible before we ship and proceed with iterative coverage.
Our undeniable belief to drive decisions - shipping isn't the win, owning the failure mode is the only path to win (even if it leads to failure some times).
References
https://github.com/taoensso/carmine → our primary clojure x redis library
http://oldblog.antirez.com/post/250 → old blog exploring lua based server-side scripting to handle queues over redis