Distributed Task Orchestrator

Problem

The company outgrew its homegrown cron system. Critical financial workflows ran on a fragile mix of shell scripts and database triggers with no central observability.

Solution

• Designed an opinionated orchestrator with clear separation between control plane (scheduling, retries, state) and data plane (workers).
• Used PostgreSQL with SKIP LOCKED for fair dispatch and Redis for ephemeral coordination.
• Built an explicit idempotency contract: every task carries a stable key, guaranteed at-least-once with deterministic dedup at the worker.

Result

• 10x throughput on the same hardware with linear scaling until the database became the bottleneck.
• Mean time to recover from bad deploys dropped from hours to minutes.
• Platform became the default substrate for new internal workflows.