Skip to content

Distributed Workers Migration Plan

Status: proposed implementation plan.

This plan migrates hxEASM from the current direct-infrastructure Worker model to Gateway-connected Workers without a big-bang rewrite.

Principles

  • Preserve existing scans during migration.
  • Preserve existing scanner plugins.
  • Preserve current asset and threat persistence behavior.
  • Preserve scan profiles.
  • Keep rollback possible at every phase.
  • Avoid permanent dual execution architectures.
  • Do not introduce separate customer-only scanner logic.

Phase 1: Separate execution from ingestion

Goal: separate scanner plugin execution from result persistence inside the current Worker code.

CURRENT references:

  • Plugin execution: backend/internal/scans/worker.go:538
  • Artifact upload: backend/internal/scans/worker.go:621
  • Result ingestion: backend/internal/scans/worker.go:698

Likely affected packages:

  • backend/internal/scans
  • backend/internal/assets
  • backend/internal/vulnerabilities
  • backend/internal/files

New packages: optional internal extraction such as backend/internal/scans/ingestion.

Database changes: none expected.

Acceptance criteria:

  • Legacy Worker behavior remains unchanged.
  • Existing asset/threat/file persistence tests still pass.
  • Ingestion can be called without executing scanner subprocesses in the same function.

Rollback:

  • Revert extraction while keeping current Worker path.

Phase 2: Reliable task delivery

Goal: replace lossy Redis list semantics with durable task state and leases.

CURRENT references:

  • Redis queue key: backend/internal/scans/service.go:18
  • Enqueue with LPush: backend/internal/scans/service.go:171
  • Consume with BRPop: backend/internal/scans/worker.go:97

PROPOSED:

  • PostgreSQL is source of truth for task state.
  • Redis may be used only for wake-up notifications.
  • Task attempts have leases, deadlines, and idempotency keys.
  • Stale attempts cannot complete or overwrite newer attempts.

Database changes: required, but schema is not finalized in this document.

The proposed task, attempt, lease, result-batch, and artifact-upload persistence model is documented in data-model.md and task-lifecycle.md. Those schemas are illustrative and must be implemented through future migrations only after approval.

Acceptance criteria:

  • Worker crash after task acquisition does not permanently lose the task.
  • Duplicate result submission is rejected or idempotently ignored.
  • Lease expiration returns task to pending/retry state when allowed.

Rollback:

  • Keep legacy Redis list path behind configuration until lease model is stable.

Phase 3: Worker identity and organization assignments

Goal: introduce authenticated Worker identity and many-to-many organization assignments.

ACCEPTED DESIGN:

  • Organization-to-Worker is many-to-many.
  • Assignments are enforced server-side.
  • Worker self-declared organization IDs are not trusted.

Database changes: required later for Worker identities, certificates, and assignments.

The proposed assignment table is organization_workers, with a many-to-many relationship between organizations and Workers. Worker Pools remain optional and are not required for v1 authorization.

Acceptance criteria:

  • Gateway can authenticate a Worker.
  • Gateway can determine which organizations the Worker is assigned to.
  • Revoked assignments stop new task leases.
  • Assignment and denial decisions are auditable.

Rollback:

  • Disable Gateway leasing and keep legacy local Worker path.

Phase 4: Worker Gateway binary and gRPC protocol

Goal: introduce backend/cmd/worker-gateway/main.go.

PROPOSED responsibilities:

  • gRPC server;
  • mTLS Worker authentication;
  • task acquisition;
  • lease renewal;
  • progress reporting;
  • result submission;
  • artifact upload coordination;
  • centralized ingestion through existing services.

Acceptance criteria:

  • A Scanner Worker can acquire and execute a task through Gateway.
  • The Worker does not require PostgreSQL or Redis credentials for that task.
  • Gateway persists results through existing services.

Rollback:

  • Stop Gateway and use existing Worker deployment.

Phase 5: Artifact handling

Goal: remove permanent platform S3 credentials from remote Workers.

CURRENT references:

  • Worker reads local artifact paths: backend/internal/scans/worker.go:628
  • File service uploads to S3: backend/internal/files/service.go:56
  • S3 storage uses static credentials: backend/internal/files/storage.go:40

PROPOSED:

  • Gateway issues task-authorized upload sessions.
  • Large uploads use short-lived presigned URLs where possible.
  • Gateway registers artifacts after upload.
  • Abandoned upload sessions are cleaned up.

Acceptance criteria:

  • Remote Worker can submit screenshots/evidence without permanent S3 credentials.
  • Unauthorized Worker cannot register artifacts for another organization's task.

Phase 6: Scheduler migration

Goal: stop scheduling from every scanner Worker.

CURRENT references:

  • Worker starts scheduler: backend/cmd/worker/main.go:143
  • Scheduler loop: backend/internal/scans/worker.go:1234
  • Due schedule query: backend/internal/scans/repository.go:243

PROPOSED:

  • Central scheduler responsibility.
  • Use PostgreSQL transactional claims or advisory locking.
  • Scheduler creates durable task records.

Acceptance criteria:

  • Multiple Gateway or scheduler replicas do not create duplicate scheduled scans for the same due schedule.

Phase 7: Remote Worker deployment

Goal: support remote physical servers, Kubernetes, and customer infrastructure.

Required capabilities:

  • bootstrap config with Gateway endpoint and trust root;
  • unique Worker identity;
  • mTLS certificate enrollment and rotation;
  • capability advertisement;
  • template bundle distribution;
  • local cache directories;
  • controlled concurrency.

Acceptance criteria:

  • Remote Worker connects outbound only.
  • Remote Worker receives tasks only for assigned organizations.
  • Remote Worker can be revoked.

Phase 8: AI Worker compatibility

Goal: reuse shared infrastructure for future AI Workers without coupling them to scanner logic.

FUTURE:

  • AI Workers share identity, transport, heartbeat, leases, and task ownership.
  • AI execution schemas and model access remain separate.

Smallest useful vertical slice

The first useful implementation slice is:

  1. Extract result ingestion so Gateway can call it.
  2. Add durable task and lease records for one scanner plugin task.
  3. Add Gateway binary with minimal gRPC task acquisition and result submission.
  4. Add Scanner Worker Gateway mode that does not initialize PostgreSQL or Redis.
  5. Execute one plugin task through Gateway and persist the result through existing services.

This proves the architecture without claiming full production readiness.