Skip to content

Distributed Workers Architecture

Status: architecture documentation v1.1.

Current Worker architecture

CURRENT: The Worker is a backend peer with direct infrastructure access.

Important source references:

  • Worker startup: backend/cmd/worker/main.go
  • Tool installation: backend/cmd/worker/main.go:49
  • Private Nuclei template sync: backend/cmd/worker/main.go:54
  • PostgreSQL connection: backend/cmd/worker/main.go:61
  • Redis connection: backend/cmd/worker/main.go:67
  • S3 storage setup: backend/cmd/worker/main.go:87
  • Worker run loop: backend/internal/scans/worker.go:87
  • Redis BRPop: backend/internal/scans/worker.go:97
  • Worker scheduler: backend/internal/scans/worker.go:1234
  • Scan enqueue with LPush: backend/internal/scans/service.go:171
  • Artifact upload: backend/internal/scans/worker.go:621
  • Result processing: backend/internal/scans/worker.go:698
  • Threat persistence: backend/internal/vulnerabilities/service.go:128
  • File persistence: backend/internal/files/service.go:56
flowchart TD
    API[hxEASM API] --> PG[(PostgreSQL)]
    API --> Redis[(Redis)]
    API --> S3[(S3 / MinIO)]
    API -->|LPush| Redis

    Worker[Current Worker] -->|BRPop| Redis
    Worker --> PG
    Worker --> S3
    Worker --> Tools[Scanner tools]
    Worker --> Templates[Nuclei templates]
    Worker --> Scheduler[Scheduler loop]
    Scheduler --> PG
    Scheduler -->|LPush| Redis

CURRENT limitations:

  • Redis list consumption removes tasks before execution completes.
  • Every Worker process runs the scheduler.
  • Remote Workers require PostgreSQL, Redis, and S3 credentials.
  • Scanner subprocess execution and result persistence are mixed in backend/internal/scans/worker.go.
  • Plugin artifacts are represented by local filesystem paths in backend/internal/plugins/model.go.

Target Worker Gateway architecture

ACCEPTED DESIGN: Worker Gateway is a separate Go binary:

backend/cmd/worker-gateway/main.go

Gateway responsibilities:

  • authenticate Workers;
  • authorize tasks against Worker identity and organization assignments;
  • issue and renew task leases;
  • receive progress and results;
  • ingest results through existing backend services;
  • coordinate artifact upload and registration;
  • update scan and job state;
  • support horizontal Gateway replicas.

Gateway must not:

  • execute scanner plugins;
  • duplicate asset, threat, scan, or file persistence logic;
  • act as a generic SQL-over-gRPC or Redis-over-gRPC proxy.
flowchart TD
    API[hxEASM API] --> PG[(PostgreSQL)]
    API --> Redis[(Redis notifications)]
    API --> S3[(S3 / MinIO)]

    Scheduler[Central Scheduler] --> PG
    Scheduler --> Redis

    Gateway[Worker Gateway] --> PG
    Gateway --> Redis
    Gateway --> S3
    Gateway --> Ingestion[Existing ingestion services]

    Ingestion --> Assets[Asset service]
    Ingestion --> Threats[Threat service]
    Ingestion --> Files[File service]

    Scanner[Unified Scanner Worker] -->|Outbound gRPC + mTLS| Gateway
    Scanner --> Tools[Scanner plugins and tools]
    Scanner --> Targets[Scan targets]

    AI[Future AI Worker] -->|Outbound gRPC + mTLS| Gateway

Component responsibilities

hxEASM API

CURRENT: Creates scans through backend/internal/scans/service.go and currently enqueues Redis messages.

ACCEPTED DESIGN:

  • keep user-facing HTTP API and RBAC;
  • create scan and task records;
  • do not execute scanner plugins;
  • do not route tasks based on Worker self-declared organization IDs.

Worker Gateway

ACCEPTED DESIGN:

  • expose versioned gRPC APIs;
  • verify mTLS Worker identity;
  • enforce organization-to-Worker assignments;
  • lease tasks only to eligible Workers;
  • reject stale attempts;
  • receive plugin results and artifacts;
  • call existing asset, threat, scan, and file services.

Unified Scanner Worker

ACCEPTED DESIGN: There is one functional Scanner Worker implementation.

The same Scanner Worker supports:

  • local Docker Compose;
  • remote physical servers;
  • Kubernetes;
  • customer-controlled infrastructure;
  • private-network scanning;
  • public-network scanning.

Deployment ownership does not inherently restrict supported asset or threat categories. Customer-hosted Scanner Workers can discover every asset and threat category supported by their installed plugins, tools, configuration, and assigned tasks.

Future AI Worker

FUTURE: AI Workers are a separate execution type. They may reuse Gateway transport, identity, heartbeat, task leasing, and observability, but they must not share scanner-specific plugin execution logic.

Organization-to-Worker assignments

ACCEPTED DESIGN: organization-to-Worker assignment is many-to-many.

erDiagram
    ORGANIZATION ||--o{ ORGANIZATION_WORKER_ASSIGNMENT : has
    WORKER ||--o{ ORGANIZATION_WORKER_ASSIGNMENT : assigned

Rules:

  • An organization can have multiple assigned Workers.
  • A Worker can be assigned to multiple organizations.
  • The Gateway selects eligible Workers using centrally stored assignments.
  • A Worker cannot self-authorize by sending an organization ID.
  • Assignment checks happen server-side using authenticated Worker identity.

Implications:

  • Task routing must filter by organization assignment and capability.
  • Organization isolation is enforced by Gateway before leasing a task.
  • Assignment changes affect future task acquisition.
  • Running tasks during revocation need explicit policy: allow current lease to finish, cancel immediately, or mark for cancellation. The recommended default is to stop new leases immediately and cancel active leases when revocation is security-driven.
  • Audit logs should record assignment changes, task lease decisions, revocations, and denied access attempts.

Worker Pools are optional future grouping. They are not mandatory for v1 task authorization.

Scanner capabilities

ACCEPTED DESIGN: Capabilities are not identity and not deployment location.

Capabilities describe what a Worker can execute:

  • plugin availability;
  • scanner binary availability;
  • runtime configuration;
  • template bundle versions;
  • tool versions;
  • OS/platform constraints;
  • safe concurrency limits.

Gateway routing must consider both:

  • authorization: organization-to-Worker assignment;
  • capability: can this Worker execute this task.

Task routing

PROPOSED routing inputs:

  • organization ID;
  • scan profile;
  • plugin name;
  • required capabilities;
  • Worker assignment;
  • Worker availability and capacity;
  • lease status;
  • task priority.

Routing must not use Worker-supplied organization IDs as authorization proof.

The concrete task data model and lifecycle are documented in data-model.md and task-lifecycle.md. The current recommendation is one durable Worker task per scanner plugin execution, linked to the existing scan_jobs row. This is PROPOSED, not implemented.

Result ingestion

ACCEPTED DESIGN: Gateway performs centralized ingestion.

Scanner Workers submit plugin result data. Gateway validates task ownership and uses existing backend services:

  • assets through backend/internal/assets;
  • threats through backend/internal/vulnerabilities;
  • files through backend/internal/files;
  • scan/job state through backend/internal/scans.

This preserves:

  • asset deduplication;
  • threat deduplication;
  • threat reclassification;
  • metadata.threat_type handling;
  • exposure-change behavior;
  • detection history;
  • organization isolation.

Artifact handling

PROPOSED:

  • Remote Workers must not receive permanent platform S3 credentials.
  • Gateway should issue task-authorized upload sessions.
  • Large artifacts should use short-lived presigned upload URLs where possible.
  • Small artifacts or restricted networks may use gRPC streaming through Gateway.
  • Gateway registers uploaded artifacts through the existing file service.

Current local artifact paths in PluginArtifact.Path remain Worker-local implementation details.

Deployment scenarios

ACCEPTED DESIGN: the same Scanner Worker implementation supports all scanner deployments.

Scenario Connection model Notes
Local Docker Compose outbound to Gateway on internal network can coexist with legacy Worker during migration
Remote server outbound gRPC/mTLS no direct PostgreSQL or Redis
Kubernetes outbound gRPC/mTLS from pods Gateway may be internal or external Service
Customer infrastructure outbound gRPC/mTLS task authorization limited by assignment
Private network scanning Worker reaches local targets Gateway does not need target network access
Public network scanning Worker reaches public targets same Worker logic

Nuclei templates

CURRENT:

  • Standard Nuclei templates are synchronized under /opt/hxeasm/exp-nuclei-templates.
  • The private template repository and branch are defined in backend/internal/nucleitemplates/sync.go.
  • The custom Nuclei plugin uses a separate hxresearch template path.

PROPOSED:

  • Distributed Workers should receive approved template bundles without receiving the private GitHub PAT.
  • Bundles should be versioned and checksummed.
  • Standard and custom template sets remain separate.
  • Tasks should record or require the template bundle version used.

AI Worker extensibility

FUTURE:

  • AI Workers can reuse Worker identity, mTLS, heartbeat, task ownership, and observability.
  • AI task schemas, model/provider access, GPU scheduling, data access policy, and AI result ingestion remain separate from scanner plugin execution.