Why I became more careful with DynamoDB Streams

Context

I'm the main developer of a production platform at Formilo, built with Amplify Gen 2, AppSync, Lambda and DynamoDB. It discovers companies, scrapes and AI-scores their websites, and runs scheduled outreach email campaigns.

Website scraping is the heaviest asynchronous part. A run creates one task per website, and the jobs we measured had between roughly 2,700 and 3,600 websites each. Every task is a slow, failure-prone network job against a site we don't control.

What I tried first

At first, the architecture looked very clean:

1. create a task in DynamoDB,
2. capture the INSERT event via DynamoDB Streams,
3. trigger Lambda to start processing automatically.

Simple, elegant, and fully serverless. On an earlier product in 2025, that's exactly how I built it: five DynamoDB Stream → Lambda mappings, INSERT events only, batches of 10. None of them had a retry limit, bisect on error or an on-failure destination.

What I learned

Under real production load and failure scenarios, some important details started to surface.

According to the AWS documentation, the DynamoDB Streams + Lambda integration works with at-least-once delivery:

A Lambda consumer for a DynamoDB stream doesn't guarantee exactly once delivery and may lead to occasional duplicates.

So the same event can be processed more than once, idempotency becomes critical, and retry behavior has to be designed from the beginning. In failure scenarios, Lambda may retry the entire batch, which leads to duplicate processing, repeated batch retries, growing iterator age and awkward partial batch failures. With the defaults, a failing batch is retried until its records expire from the stream, and the shard waits behind it.

Looking back at that 2025 code, I found two mistakes that made this worse:

• One handler returned batchItemFailures: [], but its mapping never enabled ReportBatchItemFailures. The return value had no effect at all.
• Another handler caught its errors and carried on. Lambda saw a success, so the failure never reached any retry or dead-letter mechanism.

The stream was doing two jobs at once: telling the system that work exists, and deciding how that work is retried and paced. It's very good at the first. For the second, I wanted a tool that was designed for it.

Solution

In January 2026 I moved website scraping off stream-triggered processing. DynamoDB stays the source of truth; SQS takes over durable queuing and retries:

1. A custom AppSync mutation creates one task row per website, in state Pending.
2. A dispatcher Lambda reads the table's stream in batches of 10. It only looks at INSERT events, only dispatches tasks that are still Pending, and sends one SQS message per task. It does no scraping itself.
3. The scraper Lambda consumes the queue with batch size 1: one task, one invocation.
4. Messages that keep failing go to a dead-letter queue.

The first version had a visibility timeout of 16 minutes (the Lambda timeout of 15 minutes plus a buffer), maxReceiveCount 3 and a DLQ that keeps messages for 14 days. Try the scenarios below; "Worker fails", "Duplicate delivery" and "Traffic spike" replay the failure modes against the current design.

The queue configuration today

The numbers have been tuned since, partly by me and partly by the team as we measured real runs. This is the current setup, simplified:

const queue = new Queue(stack, "ScrapeQueue", {
  // AWS guidance: at least 6x the Lambda timeout
  visibilityTimeout: Duration.minutes(90),
  retentionPeriod: Duration.days(4),
  deadLetterQueue: { queue: scrapeDlq, maxReceiveCount: 3 }, // DLQ keeps 14 days
});

scraper.addEventSource(
  new SqsEventSource(queue, {
    batchSize: 1, // 1 task = 1 Lambda invocation
    maxBatchingWindow: Duration.seconds(0),
    maxConcurrency: 140, // reserved concurrency is also 140
    // Without this, batchItemFailures is ignored and messages
    // are deleted even on failure.
    reportBatchItemFailures: true,
  }),
);

The scraper runs with a 420-second timeout and 1,536 MB of memory. An alarm on the DLQ fires at the first message.

Fatal errors and retryable errors

Retrying everything is as wrong as retrying nothing. The scraper handler decides per message:

• Unparseable message: fail without retry. Retrying can't fix a broken body.
• Task already Finished or Failed: acknowledge and skip. This is what makes a duplicate delivery harmless.
• FatalScrapeError (for example an invalid URL): mark the task Failed, no retry.
• Anything else: report the message in batchItemFailures, so SQS delivers it again after the visibility timeout.

Inside one task, a single URL is fetched at most three times before the scraper gives up on it.

Retries are not recovery

SQS redelivery used to be the only healer, and it stops after maxReceiveCount. A worker that dies after setting a task to Running leaves a row that no retry will ever touch. So EventBridge rules now run every 5 minutes: they re-dispatch tasks stuck in Pending and terminalize tasks stuck in Running.

The team later replaced the dispatcher Lambda with an EventBridge Pipe (stream → SQS, filtered to tasks that become Pending, with bisect on error and its own DLQ) and added an atomic claim, so only one delivery can move a task to Running. The skeleton didn't change, which is the point: once storage and processing are separated, the piece in between is replaceable.

Trade-offs

• More moving parts. A dispatcher, a queue, a DLQ, alarms and a reconciler replace a single stream subscription. Each one is simple, but there's more to deploy and monitor.
• Retries are slow on purpose. A retried website comes back after the visibility timeout, not seconds later. For a background job that's acceptable; for user-facing work it wouldn't be.
• A capped pool takes longer. At most 140 scrapers run at once, so a big job takes longer than an unbounded fan-out. Predictable load is worth more to us than finishing a few minutes earlier.
• Idempotency is still on us. SQS standard queues deliver at least once, too. The terminal-state check is what keeps a duplicate from doing the work twice.

DynamoDB Streams are still an excellent tool: for change data capture, audit pipelines, cache invalidation, projections and analytics. But when a database insert becomes the primary trigger for an async business workflow, retries, idempotency and recovery need much more attention than it first seems.

What I'd do differently

• Design retry behavior on day one. Every stream mapping should have a retry limit and a failure destination, and every handler that returns batchItemFailures needs ReportBatchItemFailures enabled. Otherwise it's just a comment in code.
• Never swallow errors to make a function look green. A swallowed error is a lost event.
• Build the reconciler together with the queue, not after the first stuck task.

Result

• Failures are isolated per website: a retryable error retries only its own message, and a fatal one fails its task immediately.
• Poison messages land in a DLQ after three receives, with an alarm at the first one.
• Concurrency is a setting (140), not a side effect of how the stream is sharded.
• Jobs of roughly 2,700 to 3,600 websites per run go through this pipeline; the largest we measured had 3,575.