Skip to content

Distributed Workers

Status: architecture documentation v1.1.

This section documents the agreed direction for distributed Worker execution in hxEASM. It distinguishes the current Worker implementation from the accepted target architecture so implementation work can proceed incrementally without changing existing scan behavior in one large rewrite.

Purpose

hxEASM Workers currently execute scanner plugins and also connect directly to PostgreSQL, Redis, and S3-compatible storage. The distributed Worker architecture introduces a dedicated Worker Gateway so scanner Workers can run locally or remotely without direct access to PostgreSQL or Redis.

The target architecture supports:

  • local Docker Compose Workers;
  • Workers on separate physical servers;
  • Workers in Kubernetes;
  • Workers in customer-controlled infrastructure;
  • private-network and public-network scanning;
  • future AI Workers as a separate execution type.

Scope

This documentation covers:

  • the dedicated Worker Gateway;
  • unified scanner Worker behavior;
  • organization-to-Worker many-to-many assignment;
  • scanner capability reporting;
  • task delivery and leases;
  • result ingestion through existing backend services;
  • artifact handling without permanent platform S3 credentials on remote Workers;
  • migration from the current direct-infrastructure Worker model.

It does not define final database schemas, protobuf files, or deployment manifests. Those remain implementation tasks.

Status labels

  • CURRENT: implemented in the repository today.
  • ACCEPTED DESIGN: agreed architectural direction.
  • PROPOSED: recommended but not finalized in implementation detail.
  • FUTURE: outside the initial scanner Worker implementation.

Architecture overview

CURRENT:

  • Worker startup is in backend/cmd/worker/main.go.
  • Workers install tools, sync private Nuclei templates, connect to PostgreSQL and Redis, configure S3 storage, then run both scan consumption and scheduling.
  • Scan tasks are pushed to Redis with LPush and consumed with BRPop.
  • Workers persist assets, threats, files, scan job output, and scan state directly.

ACCEPTED DESIGN:

  • Worker Gateway is a separate Go binary: backend/cmd/worker-gateway/main.go.
  • Workers initiate outbound gRPC connections to Gateway.
  • Production Worker connections use TLS and mutual TLS with unique Worker identities.
  • Scanner Workers do not need direct PostgreSQL or Redis access.
  • Remote Workers do not need permanent platform S3 credentials.
  • Gateway reuses existing backend services and repositories for persistence.
  • Gateway does not execute scanner plugins.

Key documents