Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,19 @@ All notable changes to PythonScriptRunner are documented in this file. The forma
[Semantic Versioning](https://semver.org/spec/v2.0.0.html). The NuGet packages and the container images share one
version; the `psrunner` SDK has its own version on PyPI.

## [Unreleased]

### Added

- Guides in English and Portuguese (`guides/`): getting started and the ways to use the library, stages, pipelines,
the Python SDK, persistence, queue and scheduling, the dashboard, hosting, a configuration reference and how it
works.

### Changed

- The README covers every capability with short examples and links to the guides. The `psrunner` README, the PyPI
page, is now a complete reference of the SDK.

## [0.1.0] - 2026-10-21

The first release.
Expand Down Expand Up @@ -58,4 +71,5 @@ The first release.
|---------------------------------------|------|----------|------------|--------|
| 0.1.0 | 10 | 1 | 0.1.x | 3.10–3.13 |

[Unreleased]: https://github.com/ScriptRunnerKit/PythonScriptRunner/compare/v0.1.0...HEAD
[0.1.0]: https://github.com/ScriptRunnerKit/PythonScriptRunner/releases/tag/v0.1.0
50 changes: 48 additions & 2 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,8 +29,54 @@ cd python/psrunner && uv sync && uv run pytest
cd panel && npm ci && npm test
```

The [README](README.md#development) explains each test project, the environment variables the tests read, and how to run
the local environment with .NET Aspire.
To run everything locally with .NET Aspire, see [Hosting](guides/en/hosting.md#local-environment-net-aspire).

### Test projects

| Project | Needs | Notes |
|---------|-------|-------|
| `PythonScriptRunner.Core.Tests` | uv | Unit tests and integration with real Python processes. |
| `PythonScriptRunner.EntityFrameworkCore.Tests`, `.RabbitMQ.Tests`, `.Worker.Tests`, `.Dashboard.Tests` | Docker | PostgreSQL 16, RabbitMQ 3.13 and Mailpit with Testcontainers. |
| `PythonScriptRunner.Containers.Tests` | Docker | Builds both host images and runs them as in production (read-only, tmpfs volumes). |
| `PythonScriptRunner.AppHost.Tests` | Docker, Node.js and uv | Starts the Aspire AppHost and runs the sample pipelines. |
| `PythonScriptRunner.Panel.E2E.Tests` | Docker and the embedded UI | Drives Chromium with Playwright against a real dashboard, worker, PostgreSQL and RabbitMQ. |

Details of each project:
- **Containers.Tests:**
- The compose test needs the Aspire CLI (`PSR_ASPIRE_CLI` points to it if it is not on the PATH).
- To test images you built yourself, set `PSR_TEST_WORKER_IMAGE` and `PSR_TEST_DASHBOARD_IMAGE`.
- When a test fails, the logs of its containers are saved to `TestResults/container-logs/<test>/` next to the test
assembly, and the end of the worker and dashboard logs goes to the test output.
- **Panel.E2E.Tests:**
- Build it with `-p:PsrBuildPanel=true` to embed the UI.
- Run it on its own: containers starting or stopping in other test projects change the network interfaces, and Chromium
then aborts requests with `ERR_NETWORK_CHANGED`.
- `PSR_E2E_HEADED=1` shows the browser.
- A failed test saves its Playwright trace to `TestResults/traces` (`npx playwright show-trace <file>.zip`).
- **Dashboard performance test:** it uses about 200,000 runs; `PSR_PERF_RUNS_PER_DAY=50000` runs the full average
volume (90 days).
- **Dashboard OpenAPI document:** it is compared with
`tests/PythonScriptRunner.Dashboard.Tests/Snapshots/openapi-v1.verified.json`; after you review an intended change,
`PSR_ACCEPT_SNAPSHOTS=1` updates the snapshot.

### The dashboard UI

The UI lives in `panel/` (Node.js 24). Debug builds of `PythonScriptRunner.Dashboard` embed a placeholder page; in Release
(or with `-p:PsrBuildPanel=true`) MSBuild runs `npm ci` and `npm run build` and embeds `panel/dist`.

```bash
cd panel
npm run dev # Vite at http://localhost:5173/; /api and /hubs go to the DashboardHost at PSR_DASHBOARD_URL (default http://localhost:5080)
npm test # Vitest
npm run lint # ESLint and Prettier
npm run build # type check, build and the budget of the initial bundle (500 KB gzip)
npm run api # regenerates src/api/schema.d.ts from the OpenAPI snapshot
```

### Documentation

The guides live in `guides/en` and `guides/pt-br`, with the same pages in both languages. A change in behavior updates
both, and the README when the overview changes.

## What the CI checks

Expand Down
684 changes: 201 additions & 483 deletions README.md

Large diffs are not rendered by default.

4 changes: 4 additions & 0 deletions guides/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
# PythonScriptRunner guides

- [English](en/README.md)
- [Português](pt-br/README.md)
22 changes: 22 additions & 0 deletions guides/en/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
# PythonScriptRunner guides

English | [Português](../pt-br/README.md)

PythonScriptRunner runs Python scripts from .NET with a typed contract. These guides go from the smallest setup (the
library alone, with no database and no message broker) to the complete one, with a queue, scheduling and an embedded
dashboard.

| Guide | What it covers |
|-------|----------------|
| [Getting started](getting-started.md) | The ways to use it, what to install, and your first stage in a few minutes |
| [Stages](stages.md) | The contract (inputs, outputs, the 12 field types), `stage.json`, stages in C#, limits, retry and secrets |
| [Pipelines](pipelines.md) | Chaining stages, bindings, pipeline outputs and failure handling |
| [Python SDK](python-sdk.md) | Writing scripts with `psrunner`, files, errors, and testing with pytest and the CLI |
| [Persistence (PostgreSQL)](persistence.md) | Versioned catalog, run history, retention and audit |
| [Queue, worker and scheduling (RabbitMQ)](queue-and-worker.md) | Background runs, retry, cron, events and integration with other systems |
| [Dashboard](dashboard.md) | The embedded web UI, access modes and roles |
| [Hosting](hosting.md) | Container images, volumes, memory limits, Kubernetes, .NET Aspire and Docker Compose |
| [Configuration reference](configuration.md) | Every option with its default value |
| [How it works](how-it-works.md) | Process isolation, Python environments, limits, security model and observability |

The [README](../../README.md) has an overview of everything on one page.
108 changes: 108 additions & 0 deletions guides/en/configuration.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,108 @@
# Configuration reference

English | [Português](../pt-br/configuration.md)

Options are set in code, in the callback of each `Add...` method, or bound from configuration. The ready-made hosts bind
these sections, and you can do the same in your application:

```csharp
var configuration = builder.Configuration;
builder.Services
.AddPythonScriptRunner(o => configuration.GetSection("PythonScriptRunner").Bind(o))
.AddEntityFrameworkCore(o =>
{
o.UseNpgsql(configuration.GetConnectionString("psr")!);
configuration.GetSection("PythonScriptRunner:EntityFrameworkCore").Bind(o);
})
.AddRabbitMq(o =>
{
o.ConnectionString = configuration.GetConnectionString("rabbitmq");
configuration.GetSection("PythonScriptRunner:RabbitMQ").Bind(o);
})
.AddWorker(o => configuration.GetSection("PythonScriptRunner:Worker").Bind(o));
```

In environment variables, sections are separated by `__`: `PythonScriptRunner__Worker__MaxConcurrency=8`. Time values use
the `hh:mm:ss` format (`00:10:00`).

## Runner (`PythonScriptRunner`)

| Option | Default | Description |
|--------|---------|-------------|
| `WorkRoot` | `%TEMP%/psr` | Folder for the work and result folders of the attempts. |
| `EnvironmentCacheRoot` | `%LOCALAPPDATA%/psr` | Folder where Python environments and lock files are cached. |
| `UvPath` | `uv` | Path or name of the `uv` executable (searched in the PATH). |
| `DefaultPythonVersion` | `3.12` | Python version of stages that do not declare one. |
| `DefaultTimeout` | 5 minutes | Timeout of stages that do not declare one. |
| `DefaultMemoryLimitMb` | `1024` | Memory limit of stages that do not declare one. |
| `MaxFileBytes` | 50 MB | Maximum size of one file. |
| `MaxRunFilesBytes` | 200 MB | Maximum size of all files of one run. |
| `MaxLogBytes` | 1 MB | Bytes of stdout and stderr kept per attempt (the tail). |
| `MaxConcurrentProcesses` | processor count | Python processes running at the same time in this process. |
| `ThreadsPerProcess` | not set | Threads of the numeric libraries in each process. When not set: 1 on Linux when the memory limit falls back to `RLIMIT_AS`, no limit otherwise. See [Memory limits](hosting.md#memory-limits). |
| `PackageIndexUrl` | PyPI | Package index (pip `--index-url`). |
| `ExtraIndexUrls` | none | Extra package indexes (pip `--extra-index-url`). |
| `PsrunnerRequirement` | `psrunner==0.1.0` | The SDK installed in every environment: a pip requirement or the path of a local wheel. |
| `OrphanRetention` | 24 hours | Age after which leftover work folders are deleted. |

Secrets are read from `PythonScriptRunner:Secrets:<NAME>`.

## Persistence (`PythonScriptRunner:EntityFrameworkCore`)

| Option | Default | Description |
|--------|---------|-------------|
| `Retention:Artifacts` | 7 days | How long file contents are kept. |
| `Retention:Runs` | 90 days | How long runs, steps, attempts, logs and file metadata are kept. |
| `Retention:Audit` | 365 days | How long audit events are kept. |
| `Maintenance:Enabled` | `true` | Runs the maintenance job (partitions, retention, cleanups) in this process. |
| `Maintenance:Interval` | 1 hour | How often the maintenance job runs. |
| `Maintenance:BatchSize` | `1000` | Rows handled per batch by the maintenance job. |
| `IdempotencyWindow` | 24 hours | How long an idempotency key returns the same run. |
| `LatestVersionCacheDuration` | 30 seconds | How long "latest published version" lookups are cached in this process. |
| `UseDatabaseClock` | `true` | Uses the PostgreSQL clock for leases, retry delays, schedules and the outbox, so hosts with skewed clocks agree. |

The connection is set in code with `o.UseNpgsql(connectionString)`.

## RabbitMQ (`PythonScriptRunner:RabbitMQ`)

| Option | Default | Description |
|--------|---------|-------------|
| `ConnectionString` | none | AMQP URI: `amqp://user:password@host:5672/vhost`, or `amqps://` for TLS. |
| `DeclareTopology` | `true` | Declares exchanges and queues at startup. With `false`, startup only checks that they exist (for brokers managed by an operator). |
| `RunOutboxRelay` | `true` | Publishes the outbox from this process. Several instances can run it. |
| `EnablePublicCommands` | `true` | Consumes the public `StartPipelineRun` command. |
| `InstanceId` | machine name and process id | Names the per-instance queues of this process. |
| `RelayInterval` | 250 ms | How often the relay looks for messages when the outbox is idle. |
| `RelayBatchSize` | `100` | Messages published per relay batch. |
| `DeliveryLimit` | `5` | Delivery attempts of a command before it goes to `psr.dead-letter`. |
| `StartupConnectTimeout` | 10 seconds | How long startup waits for the broker before continuing in the background. |

## Worker (`PythonScriptRunner:Worker`)

| Option | Default | Description |
|--------|---------|-------------|
| `WorkerId` | the RabbitMQ instance id | Identifier recorded on the attempts this worker runs. |
| `MaxConcurrency` | processor count | Steps executed at the same time by this worker. |
| `LeaseDuration` | 60 seconds | Lease of a running attempt; another worker recovers it when not renewed. |
| `LeaseRenewInterval` | 15 seconds | How often the lease of a running attempt is renewed. |
| `ReaperInterval` | 15 seconds | How often expired leases are looked for. |
| `MaxLeaseRecoveries` | `3` | Recoveries of one step after lost workers before it fails with `WorkerLost`. |
| `ShutdownGracePeriod` | 5 minutes | How long running steps may finish when the host stops. The platform must allow at least this long. |
| `WorkFolderCleanupInterval` | 1 hour | How often leftover work folders are removed. |
| `Scheduler:Enabled` | `true` | Fires schedules from this worker. Several workers never fire the same occurrence. |
| `Scheduler:PollInterval` | 5 seconds | How often due schedules are looked for (the firing precision). |
| `Scheduler:BatchSize` | `50` | Schedules fired per transaction. |

## Dashboard fixed login (`PythonScriptRunner:Dashboard:Login`)

| Option | Default | Description |
|--------|---------|-------------|
| `Users` | none | List of `{ Name, Role, PasswordHash }`. |
| `SlidingExpiration` | 8 hours | Session lifetime, renewed while the user is active. |
| `AbsoluteExpiration` | 24 hours | Maximum session lifetime. |
| `MaxFailedAttempts` | `5` | Failed logins in a row before the user is locked. |
| `LockoutDuration` | 15 minutes | How long a locked user waits. |
| `MaxAttemptsPerMinutePerIp` | `10` | Login attempts allowed per IP per minute. |

The same options exist in code on `UseFixedLogin(l => ...)`, which also has `PersistKeysInDatabase` (default `true`): it
keeps the cookie keys in PostgreSQL, so sessions work on every instance and survive restarts.
147 changes: 147 additions & 0 deletions guides/en/dashboard.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,147 @@
# Dashboard

English | [Português](../pt-br/dashboard.md)

`PythonScriptRunner.Dashboard` embeds a web UI in your ASP.NET Core application, in the style of Hangfire. It has an
internal API, real-time updates over SignalR and a Vue interface, all inside the package: there is nothing to host
separately. The dashboard needs [persistence](persistence.md) and the [queue](queue-and-worker.md), because it stores
versions and history and runs tests and runs on the workers.

## Setup

```bash
dotnet add package PythonScriptRunner.Dashboard
```

```csharp
builder.Services
.AddPythonScriptRunner()
.AddEntityFrameworkCore(o => o.UseNpgsql(builder.Configuration.GetConnectionString("psr")!))
.AddRabbitMq(o => o.ConnectionString = builder.Configuration.GetConnectionString("rabbitmq"))
.AddDashboard(o => o.UseFixedLogin(login => login.BindConfiguration())); // section PythonScriptRunner:Dashboard:Login

var app = builder.Build();
app.MapPythonScriptRunnerDashboard("/psr");
```

`MapPythonScriptRunnerDashboard("/psr")` serves:
- the UI at `/psr/`;
- the API at `/psr/api`;
- real-time updates at `/psr/hubs/runs`.

Without that call there is no dashboard. To run it somewhere other than your own application, use the ready-made
`DashboardHost` image (see [Hosting](hosting.md)).

## What you can do

The UI is in Brazilian Portuguese, with light and dark themes. An English version is planned.

**Edit a stage.** Write the script in a Monaco editor and build the contract with a field builder: name, type, required,
default value and description. The Dependencies tab sets the Python version and the requirements. You can test a draft
on a real worker before publishing it.

![Stage editor](../images/dashboard-stage-editor.png)

![Adding an output to the contract](../images/dashboard-contract.png)

**Build a pipeline.** Drag published stages onto a canvas and connect outputs to inputs. The handles are typed, and
problems are listed while you edit. A list view offers the same editing without the canvas.

![Visual pipeline editor](../images/dashboard-pipeline-editor.png)

**Run and follow.** Start a run from a form generated from the contract. The flow shows each step changing status live.
The tabs show the inputs, the outputs (tables included), the logs of every attempt, the files and a timeline. Failed runs
can be retried from the step that failed, and running ones can be cancelled.

![Run form](../images/dashboard-run-form.png)

![A run in progress](../images/dashboard-run-live.png)

![Outputs of a finished run](../images/dashboard-run-outputs.png)

The UI also has:
- an overview of the last 24 hours;
- run lists with filters;
- cron schedules, with a preview of the next occurrences;
- the audit log;
- an administration page for the `Admin` role.

## Access modes

Choose **one** main mode:

| Mode | Configuration | When to use |
|------|---------------|-------------|
| Fixed login | `o.UseFixedLogin(l => l.BindConfiguration())`, or `l.Users` in code | A few known users, no identity provider. |
| Host authentication | `o.UseHostAuthentication(r => { ... })` | Your application already authenticates users (cookies, OpenID Connect, Microsoft Entra ID). |
| Custom filter | `o.UseAuthorizationFilter<MyFilter>()` | Your own rule: an `IDashboardAuthorizationFilter` returns `DashboardUser(name, role)` or `null`. |
| Local requests | `o.AllowLocalRequests = true` | Development only (startup fails outside Development). It adds to a main mode. |

## Roles

The four roles are cumulative:

| Role | Can |
|------|-----|
| `Reader` | See everything: catalog, runs, logs, files, schedules. |
| `Executor` | Also start, cancel and retry runs. |
| `Author` | Also edit, test, publish, import and schedule. |
| `Admin` | Also delete, and manage retention, secrets status and the audit log. |

## Fixed login

Users live in configuration. The password is never stored, only its hash (PBKDF2, the ASP.NET Identity format):

```json
"PythonScriptRunner": {
"Dashboard": {
"Login": {
"Users": [
{ "Name": "ana", "Role": "Admin", "PasswordHash": "AQAAAAIAAYagAAAAE..." },
{ "Name": "bruno", "Role": "Reader", "PasswordHash": "AQAAAAIAAYagAAAAE..." }
]
}
}
}
```

Generate the hash with the `psr` tool:

```bash
dotnet tool install -g PythonScriptRunner.Tool
psr hash-password # asks for the password twice, without echo
echo "password" | psr hash-password --stdin # for scripts
```

How fixed login behaves:
- **Session:** an HttpOnly `psr.session` cookie (`SameSite=Lax`), sliding for 8 hours, 24 hours at most.
- **Lockout:** 5 failed attempts in a row lock the user for 15 minutes, and one IP can try 10 times per minute.
- **Configuration changes:** they apply without a restart. A removed user or a changed password ends the session, and a
new role applies at once.
- **Cookie keys:** they are stored in the database (`psr.dashboard_keys`), so a session works on every instance and
survives restarts.

## Host authentication

Map each role to an authorization policy of your application, and tell the dashboard where your login page is:

```csharp
.AddDashboard(o => o.UseHostAuthentication(r =>
{
r.Reader = p => p.RequireRole("psr-reader");
r.Executor = p => p.RequireRole("psr-executor");
r.Author = p => p.RequireRole("psr-author");
r.Admin = p => p.RequireRole("psr-admin");
r.LoginUrl = "/account/login?returnUrl={returnUrl}";
}));
```

## The API

The API is internal: it exists for the UI and can change between versions. To integrate other systems, use the .NET
library or the [RabbitMQ command and events](queue-and-worker.md#systems-written-in-other-languages).
- **Writes:** requests that change something require the header `X-PSR-Request: 1`.
- **Edits:** they use `ETag` and `If-Match`.
- **Errors:** they follow `ProblemDetails` with a `code`.

In Development, the OpenAPI document is at `/psr/api/openapi/v1.json`.
Loading
Loading