Context
The earlier stories covered work that runs inside our own AWS account: a queue, a scheduler and custom mutations as the entry point. On the production platform I work on at Formilo, some of the heavy work happens outside our system: external job providers handle maps discovery, email generation and scoring.
Those jobs still need to fit the same loop:
command (mutation) → durable task created → external job started → status update → state updated in the database → UI reacts live
Problem
When the work runs elsewhere, the provider has to tell us when something changed. Webhooks are the natural channel, but they come with their own failure modes:
- the same event can arrive more than once,
- events can arrive late or out of order,
- an event can refer to a task we no longer have, or to a job that's already done,
- if we answer with an error, the provider retries,
- and sometimes a webhook simply never arrives.
Solution
The loop
- A custom mutation validates the request, creates the task records and starts the external job.
- DynamoDB keeps the durable task or batch records: state, progress, timestamps, errors. The provider's job id is stored on the task.
- The external service processes the job.
- It sends status updates to a webhook endpoint: API Gateway REST endpoints, one per kind of job.
- A webhook handler Lambda validates the payload, maps the provider's status to our task state, updates the task records and loads results if needed.
- The UI reacts live through
onUpdate/observeQuerysubscriptions.
Notify, then fetch
When a job is reported as completed, the handler treats the webhook as a notification, not as the result. It then fetches the job result through the provider's API itself. The webhook only has to say "something changed".
Idempotent answers
The handler answers with a 2xx for everything it has understood, including events it decides to ignore. It returns an explicit "ignored" result with a reason:
task_not_found,already_completed,duplicate_completed,invalid_transition.
Stale, completed and failed events are all acknowledged with a 2xx, so the provider doesn't retry them. A duplicate or late webhook becomes a no-op instead of an error loop, and a task never moves backwards because an old event arrived last.
Batch jobs report in one payload with a batch_id and a jobs[] list, so one request can update many tasks.
A safety net for missed webhooks
Webhooks are not a guarantee. For batch jobs, a polling fallback runs every minute, specifically for webhook delivery issues: it checks the external jobs through the provider's API and repairs drift from missed webhooks, failed deliveries or out-of-order events.
Trade-offs
- An extra API call per completion. Notify-then-fetch costs one more request, in exchange for one clear way to load a result.
- A 2xx doesn't mean "applied". Because ignored events are acknowledged too, the provider can't tell whether we used an update. The reason we return and our own logs have to carry that information.
- Polling isn't free. The fallback runs every minute. It's cheap, but it exists only because webhooks alone weren't enough.
- More endpoints to secure. Each provider gets a public endpoint, which needs its own validation.
Result
- External jobs use the same task model and lifecycle as internal ones; the UI doesn't care where the work ran.
- Duplicate, stale and out-of-order webhooks are acknowledged and ignored instead of corrupting state or triggering retries.
- Missed webhooks are caught by the polling fallback within minutes.
This closed the loop: whether the work runs in a queue, a scheduler or an external service, the entry point is a command, the state lives in DynamoDB, and the UI reacts to state transitions.