Skip to content

Native flow queue/batch execution with configurable concurrency limit #298

Description

@JAPACX

Target Component

Core Services (Frontend UI/Backend API)

Enhancement Description

Problem
When integrating PentAGI into automated pipelines (e.g. weekly security reviews across multiple repositories), there is no way to control how many flows run concurrently. Calling createFlow N times in sequence starts all N flows immediately, spinning up N Kali Linux containers simultaneously and saturating host resources.
Currently the only workaround is implementing an external polling loop:

  1. Call createFlow for a batch of N
  2. Poll GET /api/v1/flows/{id} until status == completed | failed
  3. Launch next batch manually
    This adds significant complexity to any external orchestration layer.

Proposed Solution
Two complementary additions:

  1. MAX_CONCURRENT_FLOWS environment variable
    A simple cap on how many flows can be in active status simultaneously. Flows created beyond the cap would enter a queued status at the Flow level and be promoted automatically as slots free up.
  2. Webhook / callback on flow completion
    A configurable WEBHOOK_URL that PentAGI calls with a POST when a flow transitions to completed or failed. This eliminates the need for external polling loops entirely.

Current Workaround
External batch orchestration via polling — functional but requires building and maintaining a custom polling loop outside PentAGI.

Use Case
Automated weekly security pipeline across 30+ repositories:

  • Each repo triggers a PentAGI flow with its findings
  • Without concurrency control, 30 Kali containers start simultaneously
  • With MAX_CONCURRENT_FLOWS=5, PentAGI would self-manage the queue and process them in controlled batches

Technical Details

No response

Designs and Mockups

No response

Alternative Solutions

No response

Verification

  • I have checked that this enhancement hasn't been already proposed
  • This enhancement aligns with PentAGI's goal of autonomous penetration testing
  • I have considered the security implications of this enhancement
  • I have provided clear use cases and benefits
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions