Skip to content

Commit be3358c

Browse files
authored
Merge pull request #17 from ScriptRunnerKit/fase-9/documentacao
docs: guias em inglês e português e READMEs completos
2 parents 4e868cf + 939a7b6 commit be3358c

34 files changed

Lines changed: 3227 additions & 494 deletions

‎CHANGELOG.md‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,19 @@ All notable changes to PythonScriptRunner are documented in this file. The forma
55
[Semantic Versioning](https://semver.org/spec/v2.0.0.html). The NuGet packages and the container images share one
66
version; the `psrunner` SDK has its own version on PyPI.
77

8+
## [Unreleased]
9+
10+
### Added
11+
12+
- Guides in English and Portuguese (`guides/`): getting started and the ways to use the library, stages, pipelines,
13+
the Python SDK, persistence, queue and scheduling, the dashboard, hosting, a configuration reference and how it
14+
works.
15+
16+
### Changed
17+
18+
- The README covers every capability with short examples and links to the guides. The `psrunner` README, the PyPI
19+
page, is now a complete reference of the SDK.
20+
821
## [0.1.0] - 2026-10-21
922

1023
The first release.
@@ -58,4 +71,5 @@ The first release.
5871
|---------------------------------------|------|----------|------------|--------|
5972
| 0.1.0 | 10 | 1 | 0.1.x | 3.10–3.13 |
6073

74+
[Unreleased]: https://github.com/ScriptRunnerKit/PythonScriptRunner/compare/v0.1.0...HEAD
6175
[0.1.0]: https://github.com/ScriptRunnerKit/PythonScriptRunner/releases/tag/v0.1.0

‎CONTRIBUTING.md‎

Lines changed: 48 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -29,8 +29,54 @@ cd python/psrunner && uv sync && uv run pytest
2929
cd panel && npm ci && npm test
3030
```
3131

32-
The [README](README.md#development) explains each test project, the environment variables the tests read, and how to run
33-
the local environment with .NET Aspire.
32+
To run everything locally with .NET Aspire, see [Hosting](guides/en/hosting.md#local-environment-net-aspire).
33+
34+
### Test projects
35+
36+
| Project | Needs | Notes |
37+
|---------|-------|-------|
38+
| `PythonScriptRunner.Core.Tests` | uv | Unit tests and integration with real Python processes. |
39+
| `PythonScriptRunner.EntityFrameworkCore.Tests`, `.RabbitMQ.Tests`, `.Worker.Tests`, `.Dashboard.Tests` | Docker | PostgreSQL 16, RabbitMQ 3.13 and Mailpit with Testcontainers. |
40+
| `PythonScriptRunner.Containers.Tests` | Docker | Builds both host images and runs them as in production (read-only, tmpfs volumes). |
41+
| `PythonScriptRunner.AppHost.Tests` | Docker, Node.js and uv | Starts the Aspire AppHost and runs the sample pipelines. |
42+
| `PythonScriptRunner.Panel.E2E.Tests` | Docker and the embedded UI | Drives Chromium with Playwright against a real dashboard, worker, PostgreSQL and RabbitMQ. |
43+
44+
Details of each project:
45+
- **Containers.Tests:**
46+
- The compose test needs the Aspire CLI (`PSR_ASPIRE_CLI` points to it if it is not on the PATH).
47+
- To test images you built yourself, set `PSR_TEST_WORKER_IMAGE` and `PSR_TEST_DASHBOARD_IMAGE`.
48+
- When a test fails, the logs of its containers are saved to `TestResults/container-logs/<test>/` next to the test
49+
assembly, and the end of the worker and dashboard logs goes to the test output.
50+
- **Panel.E2E.Tests:**
51+
- Build it with `-p:PsrBuildPanel=true` to embed the UI.
52+
- Run it on its own: containers starting or stopping in other test projects change the network interfaces, and Chromium
53+
then aborts requests with `ERR_NETWORK_CHANGED`.
54+
- `PSR_E2E_HEADED=1` shows the browser.
55+
- A failed test saves its Playwright trace to `TestResults/traces` (`npx playwright show-trace <file>.zip`).
56+
- **Dashboard performance test:** it uses about 200,000 runs; `PSR_PERF_RUNS_PER_DAY=50000` runs the full average
57+
volume (90 days).
58+
- **Dashboard OpenAPI document:** it is compared with
59+
`tests/PythonScriptRunner.Dashboard.Tests/Snapshots/openapi-v1.verified.json`; after you review an intended change,
60+
`PSR_ACCEPT_SNAPSHOTS=1` updates the snapshot.
61+
62+
### The dashboard UI
63+
64+
The UI lives in `panel/` (Node.js 24). Debug builds of `PythonScriptRunner.Dashboard` embed a placeholder page; in Release
65+
(or with `-p:PsrBuildPanel=true`) MSBuild runs `npm ci` and `npm run build` and embeds `panel/dist`.
66+
67+
```bash
68+
cd panel
69+
npm run dev # Vite at http://localhost:5173/; /api and /hubs go to the DashboardHost at PSR_DASHBOARD_URL (default http://localhost:5080)
70+
npm test # Vitest
71+
npm run lint # ESLint and Prettier
72+
npm run build # type check, build and the budget of the initial bundle (500 KB gzip)
73+
npm run api # regenerates src/api/schema.d.ts from the OpenAPI snapshot
74+
```
75+
76+
### Documentation
77+
78+
The guides live in `guides/en` and `guides/pt-br`, with the same pages in both languages. A change in behavior updates
79+
both, and the README when the overview changes.
3480

3581
## What the CI checks
3682

‎README.md‎

Lines changed: 201 additions & 483 deletions
Large diffs are not rendered by default.

‎guides/README.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
# PythonScriptRunner guides
2+
3+
- [English](en/README.md)
4+
- [Português](pt-br/README.md)

‎guides/en/README.md‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
# PythonScriptRunner guides
2+
3+
English | [Português](../pt-br/README.md)
4+
5+
PythonScriptRunner runs Python scripts from .NET with a typed contract. These guides go from the smallest setup (the
6+
library alone, with no database and no message broker) to the complete one, with a queue, scheduling and an embedded
7+
dashboard.
8+
9+
| Guide | What it covers |
10+
|-------|----------------|
11+
| [Getting started](getting-started.md) | The ways to use it, what to install, and your first stage in a few minutes |
12+
| [Stages](stages.md) | The contract (inputs, outputs, the 12 field types), `stage.json`, stages in C#, limits, retry and secrets |
13+
| [Pipelines](pipelines.md) | Chaining stages, bindings, pipeline outputs and failure handling |
14+
| [Python SDK](python-sdk.md) | Writing scripts with `psrunner`, files, errors, and testing with pytest and the CLI |
15+
| [Persistence (PostgreSQL)](persistence.md) | Versioned catalog, run history, retention and audit |
16+
| [Queue, worker and scheduling (RabbitMQ)](queue-and-worker.md) | Background runs, retry, cron, events and integration with other systems |
17+
| [Dashboard](dashboard.md) | The embedded web UI, access modes and roles |
18+
| [Hosting](hosting.md) | Container images, volumes, memory limits, Kubernetes, .NET Aspire and Docker Compose |
19+
| [Configuration reference](configuration.md) | Every option with its default value |
20+
| [How it works](how-it-works.md) | Process isolation, Python environments, limits, security model and observability |
21+
22+
The [README](../../README.md) has an overview of everything on one page.

‎guides/en/configuration.md‎

Lines changed: 108 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,108 @@
1+
# Configuration reference
2+
3+
English | [Português](../pt-br/configuration.md)
4+
5+
Options are set in code, in the callback of each `Add...` method, or bound from configuration. The ready-made hosts bind
6+
these sections, and you can do the same in your application:
7+
8+
```csharp
9+
var configuration = builder.Configuration;
10+
builder.Services
11+
.AddPythonScriptRunner(o => configuration.GetSection("PythonScriptRunner").Bind(o))
12+
.AddEntityFrameworkCore(o =>
13+
{
14+
o.UseNpgsql(configuration.GetConnectionString("psr")!);
15+
configuration.GetSection("PythonScriptRunner:EntityFrameworkCore").Bind(o);
16+
})
17+
.AddRabbitMq(o =>
18+
{
19+
o.ConnectionString = configuration.GetConnectionString("rabbitmq");
20+
configuration.GetSection("PythonScriptRunner:RabbitMQ").Bind(o);
21+
})
22+
.AddWorker(o => configuration.GetSection("PythonScriptRunner:Worker").Bind(o));
23+
```
24+
25+
In environment variables, sections are separated by `__`: `PythonScriptRunner__Worker__MaxConcurrency=8`. Time values use
26+
the `hh:mm:ss` format (`00:10:00`).
27+
28+
## Runner (`PythonScriptRunner`)
29+
30+
| Option | Default | Description |
31+
|--------|---------|-------------|
32+
| `WorkRoot` | `%TEMP%/psr` | Folder for the work and result folders of the attempts. |
33+
| `EnvironmentCacheRoot` | `%LOCALAPPDATA%/psr` | Folder where Python environments and lock files are cached. |
34+
| `UvPath` | `uv` | Path or name of the `uv` executable (searched in the PATH). |
35+
| `DefaultPythonVersion` | `3.12` | Python version of stages that do not declare one. |
36+
| `DefaultTimeout` | 5 minutes | Timeout of stages that do not declare one. |
37+
| `DefaultMemoryLimitMb` | `1024` | Memory limit of stages that do not declare one. |
38+
| `MaxFileBytes` | 50 MB | Maximum size of one file. |
39+
| `MaxRunFilesBytes` | 200 MB | Maximum size of all files of one run. |
40+
| `MaxLogBytes` | 1 MB | Bytes of stdout and stderr kept per attempt (the tail). |
41+
| `MaxConcurrentProcesses` | processor count | Python processes running at the same time in this process. |
42+
| `ThreadsPerProcess` | not set | Threads of the numeric libraries in each process. When not set: 1 on Linux when the memory limit falls back to `RLIMIT_AS`, no limit otherwise. See [Memory limits](hosting.md#memory-limits). |
43+
| `PackageIndexUrl` | PyPI | Package index (pip `--index-url`). |
44+
| `ExtraIndexUrls` | none | Extra package indexes (pip `--extra-index-url`). |
45+
| `PsrunnerRequirement` | `psrunner==0.1.0` | The SDK installed in every environment: a pip requirement or the path of a local wheel. |
46+
| `OrphanRetention` | 24 hours | Age after which leftover work folders are deleted. |
47+
48+
Secrets are read from `PythonScriptRunner:Secrets:<NAME>`.
49+
50+
## Persistence (`PythonScriptRunner:EntityFrameworkCore`)
51+
52+
| Option | Default | Description |
53+
|--------|---------|-------------|
54+
| `Retention:Artifacts` | 7 days | How long file contents are kept. |
55+
| `Retention:Runs` | 90 days | How long runs, steps, attempts, logs and file metadata are kept. |
56+
| `Retention:Audit` | 365 days | How long audit events are kept. |
57+
| `Maintenance:Enabled` | `true` | Runs the maintenance job (partitions, retention, cleanups) in this process. |
58+
| `Maintenance:Interval` | 1 hour | How often the maintenance job runs. |
59+
| `Maintenance:BatchSize` | `1000` | Rows handled per batch by the maintenance job. |
60+
| `IdempotencyWindow` | 24 hours | How long an idempotency key returns the same run. |
61+
| `LatestVersionCacheDuration` | 30 seconds | How long "latest published version" lookups are cached in this process. |
62+
| `UseDatabaseClock` | `true` | Uses the PostgreSQL clock for leases, retry delays, schedules and the outbox, so hosts with skewed clocks agree. |
63+
64+
The connection is set in code with `o.UseNpgsql(connectionString)`.
65+
66+
## RabbitMQ (`PythonScriptRunner:RabbitMQ`)
67+
68+
| Option | Default | Description |
69+
|--------|---------|-------------|
70+
| `ConnectionString` | none | AMQP URI: `amqp://user:password@host:5672/vhost`, or `amqps://` for TLS. |
71+
| `DeclareTopology` | `true` | Declares exchanges and queues at startup. With `false`, startup only checks that they exist (for brokers managed by an operator). |
72+
| `RunOutboxRelay` | `true` | Publishes the outbox from this process. Several instances can run it. |
73+
| `EnablePublicCommands` | `true` | Consumes the public `StartPipelineRun` command. |
74+
| `InstanceId` | machine name and process id | Names the per-instance queues of this process. |
75+
| `RelayInterval` | 250 ms | How often the relay looks for messages when the outbox is idle. |
76+
| `RelayBatchSize` | `100` | Messages published per relay batch. |
77+
| `DeliveryLimit` | `5` | Delivery attempts of a command before it goes to `psr.dead-letter`. |
78+
| `StartupConnectTimeout` | 10 seconds | How long startup waits for the broker before continuing in the background. |
79+
80+
## Worker (`PythonScriptRunner:Worker`)
81+
82+
| Option | Default | Description |
83+
|--------|---------|-------------|
84+
| `WorkerId` | the RabbitMQ instance id | Identifier recorded on the attempts this worker runs. |
85+
| `MaxConcurrency` | processor count | Steps executed at the same time by this worker. |
86+
| `LeaseDuration` | 60 seconds | Lease of a running attempt; another worker recovers it when not renewed. |
87+
| `LeaseRenewInterval` | 15 seconds | How often the lease of a running attempt is renewed. |
88+
| `ReaperInterval` | 15 seconds | How often expired leases are looked for. |
89+
| `MaxLeaseRecoveries` | `3` | Recoveries of one step after lost workers before it fails with `WorkerLost`. |
90+
| `ShutdownGracePeriod` | 5 minutes | How long running steps may finish when the host stops. The platform must allow at least this long. |
91+
| `WorkFolderCleanupInterval` | 1 hour | How often leftover work folders are removed. |
92+
| `Scheduler:Enabled` | `true` | Fires schedules from this worker. Several workers never fire the same occurrence. |
93+
| `Scheduler:PollInterval` | 5 seconds | How often due schedules are looked for (the firing precision). |
94+
| `Scheduler:BatchSize` | `50` | Schedules fired per transaction. |
95+
96+
## Dashboard fixed login (`PythonScriptRunner:Dashboard:Login`)
97+
98+
| Option | Default | Description |
99+
|--------|---------|-------------|
100+
| `Users` | none | List of `{ Name, Role, PasswordHash }`. |
101+
| `SlidingExpiration` | 8 hours | Session lifetime, renewed while the user is active. |
102+
| `AbsoluteExpiration` | 24 hours | Maximum session lifetime. |
103+
| `MaxFailedAttempts` | `5` | Failed logins in a row before the user is locked. |
104+
| `LockoutDuration` | 15 minutes | How long a locked user waits. |
105+
| `MaxAttemptsPerMinutePerIp` | `10` | Login attempts allowed per IP per minute. |
106+
107+
The same options exist in code on `UseFixedLogin(l => ...)`, which also has `PersistKeysInDatabase` (default `true`): it
108+
keeps the cookie keys in PostgreSQL, so sessions work on every instance and survive restarts.

‎guides/en/dashboard.md‎

Lines changed: 147 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,147 @@
1+
# Dashboard
2+
3+
English | [Português](../pt-br/dashboard.md)
4+
5+
`PythonScriptRunner.Dashboard` embeds a web UI in your ASP.NET Core application, in the style of Hangfire. It has an
6+
internal API, real-time updates over SignalR and a Vue interface, all inside the package: there is nothing to host
7+
separately. The dashboard needs [persistence](persistence.md) and the [queue](queue-and-worker.md), because it stores
8+
versions and history and runs tests and runs on the workers.
9+
10+
## Setup
11+
12+
```bash
13+
dotnet add package PythonScriptRunner.Dashboard
14+
```
15+
16+
```csharp
17+
builder.Services
18+
.AddPythonScriptRunner()
19+
.AddEntityFrameworkCore(o => o.UseNpgsql(builder.Configuration.GetConnectionString("psr")!))
20+
.AddRabbitMq(o => o.ConnectionString = builder.Configuration.GetConnectionString("rabbitmq"))
21+
.AddDashboard(o => o.UseFixedLogin(login => login.BindConfiguration())); // section PythonScriptRunner:Dashboard:Login
22+
23+
var app = builder.Build();
24+
app.MapPythonScriptRunnerDashboard("/psr");
25+
```
26+
27+
`MapPythonScriptRunnerDashboard("/psr")` serves:
28+
- the UI at `/psr/`;
29+
- the API at `/psr/api`;
30+
- real-time updates at `/psr/hubs/runs`.
31+
32+
Without that call there is no dashboard. To run it somewhere other than your own application, use the ready-made
33+
`DashboardHost` image (see [Hosting](hosting.md)).
34+
35+
## What you can do
36+
37+
The UI is in Brazilian Portuguese, with light and dark themes. An English version is planned.
38+
39+
**Edit a stage.** Write the script in a Monaco editor and build the contract with a field builder: name, type, required,
40+
default value and description. The Dependencies tab sets the Python version and the requirements. You can test a draft
41+
on a real worker before publishing it.
42+
43+
![Stage editor](../images/dashboard-stage-editor.png)
44+
45+
![Adding an output to the contract](../images/dashboard-contract.png)
46+
47+
**Build a pipeline.** Drag published stages onto a canvas and connect outputs to inputs. The handles are typed, and
48+
problems are listed while you edit. A list view offers the same editing without the canvas.
49+
50+
![Visual pipeline editor](../images/dashboard-pipeline-editor.png)
51+
52+
**Run and follow.** Start a run from a form generated from the contract. The flow shows each step changing status live.
53+
The tabs show the inputs, the outputs (tables included), the logs of every attempt, the files and a timeline. Failed runs
54+
can be retried from the step that failed, and running ones can be cancelled.
55+
56+
![Run form](../images/dashboard-run-form.png)
57+
58+
![A run in progress](../images/dashboard-run-live.png)
59+
60+
![Outputs of a finished run](../images/dashboard-run-outputs.png)
61+
62+
The UI also has:
63+
- an overview of the last 24 hours;
64+
- run lists with filters;
65+
- cron schedules, with a preview of the next occurrences;
66+
- the audit log;
67+
- an administration page for the `Admin` role.
68+
69+
## Access modes
70+
71+
Choose **one** main mode:
72+
73+
| Mode | Configuration | When to use |
74+
|------|---------------|-------------|
75+
| Fixed login | `o.UseFixedLogin(l => l.BindConfiguration())`, or `l.Users` in code | A few known users, no identity provider. |
76+
| Host authentication | `o.UseHostAuthentication(r => { ... })` | Your application already authenticates users (cookies, OpenID Connect, Microsoft Entra ID). |
77+
| Custom filter | `o.UseAuthorizationFilter<MyFilter>()` | Your own rule: an `IDashboardAuthorizationFilter` returns `DashboardUser(name, role)` or `null`. |
78+
| Local requests | `o.AllowLocalRequests = true` | Development only (startup fails outside Development). It adds to a main mode. |
79+
80+
## Roles
81+
82+
The four roles are cumulative:
83+
84+
| Role | Can |
85+
|------|-----|
86+
| `Reader` | See everything: catalog, runs, logs, files, schedules. |
87+
| `Executor` | Also start, cancel and retry runs. |
88+
| `Author` | Also edit, test, publish, import and schedule. |
89+
| `Admin` | Also delete, and manage retention, secrets status and the audit log. |
90+
91+
## Fixed login
92+
93+
Users live in configuration. The password is never stored, only its hash (PBKDF2, the ASP.NET Identity format):
94+
95+
```json
96+
"PythonScriptRunner": {
97+
"Dashboard": {
98+
"Login": {
99+
"Users": [
100+
{ "Name": "ana", "Role": "Admin", "PasswordHash": "AQAAAAIAAYagAAAAE..." },
101+
{ "Name": "bruno", "Role": "Reader", "PasswordHash": "AQAAAAIAAYagAAAAE..." }
102+
]
103+
}
104+
}
105+
}
106+
```
107+
108+
Generate the hash with the `psr` tool:
109+
110+
```bash
111+
dotnet tool install -g PythonScriptRunner.Tool
112+
psr hash-password # asks for the password twice, without echo
113+
echo "password" | psr hash-password --stdin # for scripts
114+
```
115+
116+
How fixed login behaves:
117+
- **Session:** an HttpOnly `psr.session` cookie (`SameSite=Lax`), sliding for 8 hours, 24 hours at most.
118+
- **Lockout:** 5 failed attempts in a row lock the user for 15 minutes, and one IP can try 10 times per minute.
119+
- **Configuration changes:** they apply without a restart. A removed user or a changed password ends the session, and a
120+
new role applies at once.
121+
- **Cookie keys:** they are stored in the database (`psr.dashboard_keys`), so a session works on every instance and
122+
survives restarts.
123+
124+
## Host authentication
125+
126+
Map each role to an authorization policy of your application, and tell the dashboard where your login page is:
127+
128+
```csharp
129+
.AddDashboard(o => o.UseHostAuthentication(r =>
130+
{
131+
r.Reader = p => p.RequireRole("psr-reader");
132+
r.Executor = p => p.RequireRole("psr-executor");
133+
r.Author = p => p.RequireRole("psr-author");
134+
r.Admin = p => p.RequireRole("psr-admin");
135+
r.LoginUrl = "/account/login?returnUrl={returnUrl}";
136+
}));
137+
```
138+
139+
## The API
140+
141+
The API is internal: it exists for the UI and can change between versions. To integrate other systems, use the .NET
142+
library or the [RabbitMQ command and events](queue-and-worker.md#systems-written-in-other-languages).
143+
- **Writes:** requests that change something require the header `X-PSR-Request: 1`.
144+
- **Edits:** they use `ETag` and `If-Match`.
145+
- **Errors:** they follow `ProblemDetails` with a `code`.
146+
147+
In Development, the OpenAPI document is at `/psr/api/openapi/v1.json`.

0 commit comments

Comments
 (0)