Engineering
How I build systems that keep working when things fail: what I focus on, how a reliable pipeline is put together, and the principles behind it.
Focus
Selected engineering
Async processing
Queues, schedulers and webhooks that keep long-running jobs moving, with retries, dead-letter queues and reconcilers.
AWS Serverless
Amplify Gen 2, AppSync, Lambda, DynamoDB, SQS and EventBridge Scheduler in production.
AI in products
LLM features inside real workflows: AI-scored websites, generated emails and LLM-assisted scheduling checked by deterministic rules.
Complex frontends
React and Next.js interfaces for data-heavy workflows, with live status through subscriptions.
How systems survive production
Anatomy of a reliable system
Six patterns that keep an event-driven system correct when calls fail, messages repeat and traffic spikes.
System diagram: Requests are acknowledged at the API and buffered in the queue; two workers drain it at their own pace. 01 / 06
Accept fast, process async
The API validates the request, writes it to a durable queue and answers 202 Accepted within milliseconds. Slow or flaky work happens later, in workers that pull at their own pace. The caller’s latency no longer depends on the slowest downstream dependency.
In practice: Return a job or event ID with the 202 so clients can poll or subscribe for the result.
System diagram: A message fails twice, waits with a growing backoff and succeeds on the third attempt. 02 / 06
Retries with backoff
Most failures are transient: a throttled API, a timeout, a dropped connection. A failed message is not deleted; it becomes visible again after the visibility timeout and is retried. Exponential backoff with jitter spreads the attempts, so retries don’t hit a struggling dependency as one synchronized wave.
In practice: Keep the visibility timeout well above the worker timeout (about 6× for Lambda), or messages are redelivered mid-flight.
System diagram: A message fails three times and moves to the dead-letter queue; the alarm fires and the message is redriven after a fix. 03 / 06
Dead-letter queue
Some messages will never succeed: invalid data, a bug, a missing record. After maxReceiveCount attempts the queue moves the message to a dead-letter queue, so one poison message can’t block the rest or burn capacity. The failure is parked, not lost: an alarm fires, the cause is fixed, the message is redriven.
In practice: Alarm on DLQ depth > 0. A dead-letter queue nobody watches is just a slower way to lose data.
System diagram: The same message is delivered twice; the second write finds the existing key and becomes a no-op. 04 / 06
Idempotency
Queues deliver at least once, and clients retry too, so the same message will eventually arrive twice. Each message carries an idempotency key; the worker records it with a conditional write and only performs the side effect if the key is new. The duplicate is absorbed, and the outcome is the same as a single delivery.
In practice: In DynamoDB, attribute_not_exists(pk) on the key: one conditional write, no read-then-write race.
System diagram: A burst fills the queue while only three worker slots run; the backlog drains at a steady rate. 05 / 06
Backpressure and concurrency limits
A traffic spike is absorbed by the queue instead of being passed through. Reserved concurrency caps how many workers run at once, so the database or a partner API sees the steady load it was sized for. The backlog grows briefly, then drains: nothing is dropped, it just takes a little longer.
In practice: Size the cap by what the downstream can handle, not by what the spike demands.
System diagram: Every component emits logs and metrics; alarms watch DLQ depth and the age of the oldest message. 06 / 06
Observability
Asynchronous systems fail quietly: the API keeps answering 202 while messages pile up behind it. Structured logs with a correlation ID let you follow one message across services; metrics show throughput, errors and queue depth. Alarms go on the symptoms that matter: DLQ depth and the age of the oldest message.
In practice: Age of the oldest message catches a stuck consumer even when traffic is too low for depth to look alarming.
How I work
Principles I build by
Conclusions from my own articles. They cover the part many demos skip, which is also the part that makes a distributed system operable in production.
- 01
Expect the retry
Lambda will retry, and my architecture should be ready for it. Every handler decides on purpose which errors it throws and which it handles, and running it twice must not cause harm.
- 02
Reconcile, don’t just retry
Retries are not a replacement for reconciliation. A regular check compares what should have happened with what did, so a lost message or a missed schedule does not stay lost.
- 03
Mutations coordinate workflows
CRUD models state; mutations coordinate workflows. The browser stays responsible for intent, and the backend becomes responsible for orchestration.
- 04
Real-time, with a fallback
Real-time when it works, resilient when it doesn’t. I use subscriptions first for live status and keep polling as the fallback, so progress still arrives when a connection drops.