Skip to content

Distributed Worker Task Lifecycle

Status: proposed task lifecycle RFC v1.

This document defines the proposed task granularity, lease lifecycle, acquisition transaction, result acceptance, retry, and cancellation semantics for Distributed Workers.

Current execution constraints

CURRENT:

  • API scan creation currently enqueues a Redis message through backend/internal/scans/service.go:171.
  • Worker consumes Redis with BRPop in backend/internal/scans/worker.go:97.
  • Worker runs all plugins for a scan in backend/internal/scans/worker.go:382.
  • Worker creates one scan_jobs row per plugin at backend/internal/scans/worker.go:505.
  • Worker executes the plugin at backend/internal/scans/worker.go:538.
  • Worker processes successful plugin results at backend/internal/scans/worker.go:588.
  • Worker enriches subsequent plugin scope after each successful plugin at backend/internal/scans/worker.go:592.
  • Worker updates final scan status at backend/internal/scans/worker.go:603.

These constraints mean task execution cannot be moved remotely by only forwarding the current Redis message. Result ingestion and scan orchestration must be centralized.

Task granularity decision

PROPOSED: one durable Worker task represents one scanner plugin execution.

Relationship:

scan
  -> scan_job
       -> worker_task
            -> worker_task_attempts
            -> worker_result_batches

Rationale:

  • Current scan_jobs already represent plugin executions.
  • Plugin retries and manual plugin runs are naturally job-scoped.
  • A plugin task can be leased, retried, cancelled, and audited independently.
  • A whole-scan task would keep DB-backed orchestration and result ingestion inside the remote Worker.

Tasks should be created lazily by a central scan orchestrator/Gateway as each plugin becomes ready. This preserves existing sequential profile execution and runtime scope enrichment.

State ownership

Layer Owns Notes
scans.status aggregate scan lifecycle pending, running, success, failed, cancelled
scan_jobs.status user-visible plugin job status queued, running, success, failed, timeout, cancelled
worker_tasks.state Gateway delivery lifecycle pending, leased, running, retry_wait, completed
worker_task_attempts.status one Worker execution attempt leased, running, succeeded, failed, expired

Avoid duplicating all states across all layers. Gateway updates user-visible scan/job state as a consequence of task state transitions.

Task state machine

stateDiagram-v2
    [*] --> pending
    pending --> leased: acquire
    retry_wait --> leased: acquire after available_at
    leased --> running: progress or first result
    running --> running: renew lease
    leased --> retry_wait: lease expired and attempts remain
    running --> retry_wait: lease expired and attempts remain
    leased --> failed_terminal: lease expired and attempts exhausted
    running --> failed_terminal: fail terminal or attempts exhausted
    running --> completed: complete with valid lease
    leased --> cancelling: cancel requested
    running --> cancelling: cancel requested
    pending --> cancelled: cancel before acquire
    retry_wait --> cancelled: cancel before acquire
    cancelling --> cancelled: worker acknowledges or server terminates
    completed --> [*]
    failed_terminal --> [*]
    cancelled --> [*]
Transition Actor Preconditions Mutation Idempotency
pending -> leased Gateway acquire Worker enabled, assigned, capable, capacity available create attempt, set lease one transaction with row lock
retry_wait -> leased Gateway acquire available_at <= now() and attempts remain create new attempt one transaction with row lock
leased -> running Worker progress/result valid active lease set task running, attempt running repeated progress is idempotent
running -> running Worker renews lease valid active lease, deadline not exceeded extend lease_expires_at safe to repeat
running -> completed Worker complete valid active lease, accepted final result mark attempt succeeded, task completed, scan job success idempotency key required
running -> retry_wait Gateway sweeper lease expired, attempts remain mark attempt expired, set backoff safe under task lock
running -> failed_terminal Worker/Gateway valid lease or attempts exhausted mark failure repeated failure returns current terminal state
pending -> cancelled API/Gateway scan/job cancelled before lease mark cancelled repeated cancel is no-op
running -> cancelling API/Gateway cancel requested set cancel timestamp/state repeated cancel is no-op
cancelling -> cancelled Worker/Gateway cancel acknowledged or deadline expired mark cancelled repeated cancel is no-op

Lease duration and execution deadline

PROPOSED:

  • default lease duration: short and renewable, for example 60 seconds;
  • execution deadline: plugin-specific hard cap, for example from existing plugin timeout settings;
  • Worker renews lease periodically while the subprocess is active;
  • Gateway may reject renewal if the task is cancelled, Worker revoked, assignment revoked, or deadline exceeded.

Lease expiration is a recovery mechanism. Execution deadline is a policy limit.

Atomic task acquisition

PROPOSED transaction outline using PostgreSQL row locking.

The Worker must already be authenticated from mTLS. The transaction must not trust organization IDs from the Worker request.

Illustrative SQL:

BEGIN;

-- Lock and validate Worker identity.
SELECT id, enabled, revoked_at, max_concurrent_tasks, capabilities
FROM workers
WHERE id = $worker_id
FOR UPDATE;

-- Check active capacity.
SELECT COUNT(*)
FROM worker_task_attempts
WHERE worker_id = $worker_id
  AND status IN ('leased', 'running')
  AND lease_expires_at > NOW();

-- Select one eligible task. Organization assignment is checked in the same query.
WITH candidate AS (
    SELECT t.id
    FROM worker_tasks t
    JOIN organization_workers ow
      ON ow.organization_id = t.organization_id
     AND ow.worker_id = $worker_id
     AND ow.enabled = true
     AND ow.revoked_at IS NULL
    WHERE t.state IN ('pending', 'retry_wait')
      AND t.available_at <= NOW()
      AND t.attempt_count < t.max_attempts
      AND t.task_type = ANY($allowed_task_types)
      -- Capability match is implemented by application SQL or a helper function.
      -- Example placeholder:
      AND worker_has_required_capabilities($worker_id, t.required_capabilities)
    ORDER BY t.priority DESC, t.created_at ASC
    FOR UPDATE SKIP LOCKED
    LIMIT 1
),
updated_task AS (
    UPDATE worker_tasks t
    SET state = 'leased',
        attempt_count = attempt_count + 1,
        updated_at = NOW()
    FROM candidate c
    WHERE t.id = c.id
    RETURNING t.*
),
created_attempt AS (
    INSERT INTO worker_task_attempts (
        task_id, worker_id, attempt_number, lease_token, status, lease_expires_at
    )
    SELECT id, $worker_id, attempt_count, $lease_token, 'leased', NOW() + $lease_duration
    FROM updated_task
    RETURNING *
)
SELECT * FROM updated_task JOIN created_attempt ON created_attempt.task_id = updated_task.id;

COMMIT;

Race with assignment removal:

  • If the assignment row is disabled or revoked before the transaction selects the task, no task is returned.
  • If assignment removal races after acquisition, the revocation policy determines whether the active lease is cancelled.
  • New leases always observe committed assignment state.

Acquire one task per request for v1. Multiple-task acquisition can be added later, but single acquisition simplifies fairness, capacity enforcement, and cancellation.

Reliable delivery model

Recommended architecture:

  • PostgreSQL stores task state, attempts, leases, and accepted batches.
  • Redis is optional notification only.
  • Workers long-poll Gateway if no task is available.

If Redis notification is lost, Workers still poll and acquire tasks from PostgreSQL. Redis is not a correctness dependency.

Failure behavior:

Failure Behavior
Gateway restart leases remain in PostgreSQL; Workers reconnect and renew or leases expire.
Worker crash lease expires; task retries if attempts remain.
Network partition renewals fail; task becomes retryable after lease expiration.
PostgreSQL restart task acquisition pauses; no task is lost if transaction did not commit.
Redis unavailable long polling continues against PostgreSQL.
Duplicate acquire requests FOR UPDATE SKIP LOCKED prevents two Workers from leasing the same task.
Stale attempt completes lease token and attempt ID validation rejects it.

Result submission and idempotency

PROPOSED v1: support incremental result batches, but allow a Worker to submit one final batch for simple plugins.

Each batch includes:

  • task_id;
  • attempt_id;
  • lease_token;
  • sequence_number;
  • idempotency_key;
  • payload_sha256;
  • plugin result payload;
  • final flag.

Acceptance transaction:

1. Begin transaction.
2. Lock task and attempt.
3. Validate Worker identity, active attempt, lease token, and non-expired lease.
4. Insert worker_result_batches row.
5. If duplicate idempotency key with same checksum, return previous acknowledgement.
6. If duplicate idempotency key with different checksum, reject.
7. Persist batch through centralized ingestion.
8. Mark batch persisted.
9. Commit.

Existing idempotency boundaries:

  • Asset upsert is idempotent by existing asset uniqueness.
  • Threat upsert is idempotent for non-empty external IDs.
  • Threat detection is idempotent by (threat_id, scan_job_id).

Non-idempotent or partial-idempotent areas:

  • file metadata creation;
  • exposure-change recording;
  • empty external ID threats;
  • raw output and artifact registration.

Therefore, a durable batch acceptance table is required before retry-safe remote result submission.

Cancellation

Cancellation sources:

  • user cancels scan;
  • administrator disables Worker;
  • administrator removes Worker assignment;
  • Worker disconnects;
  • Worker crashes;
  • Gateway shuts down;
  • execution deadline expires.

Recommended assignment revocation default:

  • new leases stop immediately;
  • active tasks are cancelled for security-driven revocation;
  • routine reassignment may allow active tasks to finish if explicitly requested.

Outbound-only cancellation delivery:

  • Worker receives cancellation on lease renewal response, long-poll response, or a future server-stream event.
  • If Worker does not receive cancellation, Gateway rejects later renewals/results after cancellation state is committed.

Server-side rejection is the authoritative safety boundary. Cooperative process cancellation improves speed but is not trusted as the only enforcement.

Scheduler ownership

CURRENT:

  • every Worker starts scheduler in backend/cmd/worker/main.go:143;
  • scheduler loop is backend/internal/scans/worker.go:1234;
  • due schedules are queried in backend/internal/scans/repository.go:243;
  • current query does not claim rows transactionally.

PROPOSED v1:

  • Host scheduler in Gateway or API process, not scanner Workers.
  • Use PostgreSQL transactional schedule claiming with FOR UPDATE SKIP LOCKED or a PostgreSQL advisory lock.
  • Create scan and first durable task in the same transaction or a recoverable transaction sequence.

Smallest safe option:

BEGIN;

SELECT *
FROM scheduled_scans
WHERE enabled = true
  AND next_run_at <= NOW()
ORDER BY next_run_at ASC
FOR UPDATE SKIP LOCKED
LIMIT $batch_size;

-- For each claimed schedule:
-- 1. create scan
-- 2. update last_run_at and next_run_at
-- 3. create initial orchestration/task state

COMMIT;

This avoids a new scheduler microservice while preventing duplicate schedule creation across Gateway replicas.

Data migration and compatibility

Migration from Redis list queue:

  1. Add new tables with no behavior change.
  2. Add repositories and Gateway code behind feature flags.
  3. Keep legacy Worker consuming Redis for existing scans.
  4. Enable Gateway task creation for selected organizations or scan profiles.
  5. Drain legacy Redis queue before switching an organization fully to Gateway mode.
  6. Disable legacy Worker scheduler once central scheduler is enabled.
  7. Retire BRPop scan execution after all scan creation paths create durable tasks.

Rollback:

  • disable Gateway execution flag;
  • keep existing Worker path until durable task execution is proven;
  • avoid running both legacy and Gateway execution for the same scan/job.

Prevention of dual acquisition:

  • a scan/job must be marked as either legacy-executed or Gateway-executed;
  • Gateway-created worker_tasks should be unique for scan_job_id;
  • legacy Worker must ignore Gateway-managed jobs once migration reaches mixed mode.