Queues and Async Jobs
Use queues for fan-out, batch processing, retries, and rate-controlled AI work.
Key takeaways
- Queues turn bursty AI work into controlled pipelines, useful for batch summaries, embeddings, notifications, retries, and background enrichment.
- Core patterns are fan-out, batch, retry, delay, and dead letter for preserving failed jobs.
- Make jobs idempotent and track correlation IDs from request to job to output.
- Define retry limits and poison-message handling, and rate-limit model and external API calls.
- Use Workflow for orchestration and Queues for distributed execution rather than forcing one primitive to do both.
Queues turn bursty AI work into controlled pipelines. They are especially useful for batch summaries, embeddings, notifications, retries, and background enrichment.
Queue Patterns
| Pattern | Use |
|---|---|
| Fan-out | Split one event into many independent tasks |
| Batch | Process many records with controlled throughput |
| Retry | Recover from transient model or API failures |
| Delay | Schedule follow-up work |
| Dead letter | Preserve failed jobs for inspection |
Operating Rules
- Make jobs idempotent.
- Track correlation IDs from request to job to output.
- Define retry limits and poison-message handling.
- Rate-limit model and external API calls.
- Expose queue depth, age, failure rate, and cost.
Workflow Combination
Use Workflow for orchestration and Queues for distributed execution. Do not force one primitive to handle both coordination and high-volume work.