diff --git a/CHANGELOG.md b/CHANGELOG.md index 5ce0cef..d8baa34 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,19 @@ All notable changes to PythonScriptRunner are documented in this file. The forma [Semantic Versioning](https://semver.org/spec/v2.0.0.html). The NuGet packages and the container images share one version; the `psrunner` SDK has its own version on PyPI. +## [Unreleased] + +### Added + +- Guides in English and Portuguese (`guides/`): getting started and the ways to use the library, stages, pipelines, + the Python SDK, persistence, queue and scheduling, the dashboard, hosting, a configuration reference and how it + works. + +### Changed + +- The README covers every capability with short examples and links to the guides. The `psrunner` README, the PyPI + page, is now a complete reference of the SDK. + ## [0.1.0] - 2026-10-21 The first release. @@ -58,4 +71,5 @@ The first release. |---------------------------------------|------|----------|------------|--------| | 0.1.0 | 10 | 1 | 0.1.x | 3.10–3.13 | +[Unreleased]: https://github.com/ScriptRunnerKit/PythonScriptRunner/compare/v0.1.0...HEAD [0.1.0]: https://github.com/ScriptRunnerKit/PythonScriptRunner/releases/tag/v0.1.0 diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index e2155f9..83ff93b 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -29,8 +29,54 @@ cd python/psrunner && uv sync && uv run pytest cd panel && npm ci && npm test ``` -The [README](README.md#development) explains each test project, the environment variables the tests read, and how to run -the local environment with .NET Aspire. +To run everything locally with .NET Aspire, see [Hosting](guides/en/hosting.md#local-environment-net-aspire). + +### Test projects + +| Project | Needs | Notes | +|---------|-------|-------| +| `PythonScriptRunner.Core.Tests` | uv | Unit tests and integration with real Python processes. | +| `PythonScriptRunner.EntityFrameworkCore.Tests`, `.RabbitMQ.Tests`, `.Worker.Tests`, `.Dashboard.Tests` | Docker | PostgreSQL 16, RabbitMQ 3.13 and Mailpit with Testcontainers. | +| `PythonScriptRunner.Containers.Tests` | Docker | Builds both host images and runs them as in production (read-only, tmpfs volumes). | +| `PythonScriptRunner.AppHost.Tests` | Docker, Node.js and uv | Starts the Aspire AppHost and runs the sample pipelines. | +| `PythonScriptRunner.Panel.E2E.Tests` | Docker and the embedded UI | Drives Chromium with Playwright against a real dashboard, worker, PostgreSQL and RabbitMQ. | + +Details of each project: +- **Containers.Tests:** + - The compose test needs the Aspire CLI (`PSR_ASPIRE_CLI` points to it if it is not on the PATH). + - To test images you built yourself, set `PSR_TEST_WORKER_IMAGE` and `PSR_TEST_DASHBOARD_IMAGE`. + - When a test fails, the logs of its containers are saved to `TestResults/container-logs//` next to the test + assembly, and the end of the worker and dashboard logs goes to the test output. +- **Panel.E2E.Tests:** + - Build it with `-p:PsrBuildPanel=true` to embed the UI. + - Run it on its own: containers starting or stopping in other test projects change the network interfaces, and Chromium + then aborts requests with `ERR_NETWORK_CHANGED`. + - `PSR_E2E_HEADED=1` shows the browser. + - A failed test saves its Playwright trace to `TestResults/traces` (`npx playwright show-trace .zip`). +- **Dashboard performance test:** it uses about 200,000 runs; `PSR_PERF_RUNS_PER_DAY=50000` runs the full average + volume (90 days). +- **Dashboard OpenAPI document:** it is compared with + `tests/PythonScriptRunner.Dashboard.Tests/Snapshots/openapi-v1.verified.json`; after you review an intended change, + `PSR_ACCEPT_SNAPSHOTS=1` updates the snapshot. + +### The dashboard UI + +The UI lives in `panel/` (Node.js 24). Debug builds of `PythonScriptRunner.Dashboard` embed a placeholder page; in Release +(or with `-p:PsrBuildPanel=true`) MSBuild runs `npm ci` and `npm run build` and embeds `panel/dist`. + +```bash +cd panel +npm run dev # Vite at http://localhost:5173/; /api and /hubs go to the DashboardHost at PSR_DASHBOARD_URL (default http://localhost:5080) +npm test # Vitest +npm run lint # ESLint and Prettier +npm run build # type check, build and the budget of the initial bundle (500 KB gzip) +npm run api # regenerates src/api/schema.d.ts from the OpenAPI snapshot +``` + +### Documentation + +The guides live in `guides/en` and `guides/pt-br`, with the same pages in both languages. A change in behavior updates +both, and the README when the overview changes. ## What the CI checks diff --git a/README.md b/README.md index ccf6b9d..e6c1b8b 100644 --- a/README.md +++ b/README.md @@ -1,582 +1,300 @@ # PythonScriptRunner -A .NET 10 library for running Python in a typed, composable and observable way. **Stages** declare an input/output -contract, **pipelines** chain them, and RabbitMQ runs them asynchronously. Cron scheduling and an optional Vue -dashboard are included. +[![CI](https://github.com/ScriptRunnerKit/PythonScriptRunner/actions/workflows/ci.yml/badge.svg)](https://github.com/ScriptRunnerKit/PythonScriptRunner/actions/workflows/ci.yml) +[![NuGet](https://img.shields.io/nuget/v/PythonScriptRunner.Core?label=NuGet)](https://www.nuget.org/packages/PythonScriptRunner.Core) +[![PyPI](https://img.shields.io/pypi/v/psrunner?label=PyPI)](https://pypi.org/project/psrunner/) +[![License: MIT](https://img.shields.io/badge/license-MIT-blue)](https://github.com/ScriptRunnerKit/PythonScriptRunner/blob/main/LICENSE) -> The project is under active development, heading for its first release (0.1.0). This version ships the **Core Runner** -> (synchronous stages and pipelines), the **`psrunner`** Python SDK, **PostgreSQL persistence** (versioned catalog and -> history), **asynchronous execution** with RabbitMQ, a worker and cron scheduling, the **embeddable dashboard** (Vue UI, -> API, real time and configurable access), ready-made **hosts and container images** and a local environment on -> **.NET Aspire**. +Run Python from .NET with a typed contract. The calculation stays in a plain Python script (numpy, pandas, scipy, +anything on PyPI), written by whoever knows the subject. The .NET application receives a result that was already validated +against the contract. -## How it works +Start with the library alone, with no database and no message broker. Add PostgreSQL for history and versions, RabbitMQ +for background workers and cron schedules, and an embedded dashboard to edit, test and follow everything in the browser. -Each stage declares its inputs and outputs and has exactly one Python script. There are 12 field types: `number`, -`integer`, `decimal`, `string`, `bool`, `date`, `datetime`, `enum`, `json`, `table`, `file` and `list`. The runner -validates the inputs and runs the script in an isolated process: a clean environment, a timeout, a memory limit, and -secrets injected into that process only. It then validates the outputs and returns the typed result. +**Documentation:** [guides in English](https://github.com/ScriptRunnerKit/PythonScriptRunner/blob/main/guides/en/README.md) | +[guias em português](https://github.com/ScriptRunnerKit/PythonScriptRunner/blob/main/guides/pt-br/README.md) -```python -# stages/progression-terms/script.py -from psrunner import stage +![A pipeline run in the dashboard](https://raw.githubusercontent.com/ScriptRunnerKit/PythonScriptRunner/main/guides/images/dashboard-run-live.png) -@stage -def run(inputs, ctx): - a1, ratio, n = inputs["a1"], inputs["ratio"], inputs["n"] - terms = [a1 * ratio**i for i in range(n)] - ctx.log.info("generated %d terms", n) - return {"terms": terms} -``` +## Why -```jsonc -// stages/progression-terms/stage.json -{ - "$schema": "https://raw.githubusercontent.com/ScriptRunnerKit/PythonScriptRunner/main/schemas/stage.v1.json", - "key": "progression-terms", - "version": 1, - "name": "Progression terms", - "inputs": [ - { "name": "a1", "type": "number" }, - { "name": "ratio", "type": "number" }, - { "name": "n", "type": "integer" } - ], - "outputs": [ { "name": "terms", "type": "list", "itemType": "number" } ] -} -``` +Teams that do a lot of calculation often meet the same problem: the math lives in Python, but the system that needs the +result is .NET. The usual ways out are rewriting the formula in C#, or calling `python` with `Process.Start` and parsing +an unchecked JSON. PythonScriptRunner gives the two sides a contract instead: -```csharp -builder.Services - .AddPythonScriptRunner() - .AddFileCatalog("./psr"); // folder with stages//stage.json +- **Typed contract:** inputs and outputs have types (12 of them, including `decimal`, `date`, `table`, `file` and lists), + checked on both sides. +- **Isolated processes:** each run gets its own process, with a timeout, a memory limit and its own Python environment + created by uv. +- **Reproducible dependencies:** published versions lock their requirements with hashes. +- **Typed results:** the .NET side gets typed values or a clear error, never invalid data. -// ... -await using var result = await runner.RunStageAsync("progression-terms", new { a1 = 1, ratio = 2, n = 4 }); -result.EnsureSuccess(); -double[] terms = result.GetOutput("terms"); // [1, 2, 4, 8] -``` +## Ways to use it -Stages can also be defined in C# (`AddStage("key", s => s.Script(...).Input(...).Output(...))`) and tested locally with -pytest through `psrunner.testing.run_stage`, without .NET. +| Level | What you need | What you get | Guide | +|-------|---------------|--------------|-------| +| **1. Library only** | .NET 10 and uv | Run stages and pipelines from your code. No database, no broker, no dashboard. | [Getting started](https://github.com/ScriptRunnerKit/PythonScriptRunner/blob/main/guides/en/getting-started.md) | +| **2. With PostgreSQL** | PostgreSQL 16+ | A versioned catalog, the history of every run, files, retention and audit. | [Persistence](https://github.com/ScriptRunnerKit/PythonScriptRunner/blob/main/guides/en/persistence.md) | +| **3. With RabbitMQ** | PostgreSQL and RabbitMQ 3.13+ | Background runs on workers, retry, cron schedules, events, and a command for systems in other languages. | [Queue, worker and scheduling](https://github.com/ScriptRunnerKit/PythonScriptRunner/blob/main/guides/en/queue-and-worker.md) | +| **4. With the dashboard** | PostgreSQL and RabbitMQ | An embedded web UI to edit, test, run and follow stages and pipelines. | [Dashboard](https://github.com/ScriptRunnerKit/PythonScriptRunner/blob/main/guides/en/dashboard.md) | +| **Ready-made hosts** | Docker | Worker and dashboard container images, or the complete environment with one command on .NET Aspire. | [Hosting](https://github.com/ScriptRunnerKit/PythonScriptRunner/blob/main/guides/en/hosting.md) | -## Pipelines +## Quick start -A pipeline chains stages in order. Each input of a step takes one of these: +Install the [.NET 10 SDK](https://dotnet.microsoft.com/download) and [uv](https://docs.astral.sh/uv/) +(`winget install --id=astral-sh.uv -e` on Windows, `curl -LsSf https://astral.sh/uv/install.sh | sh` on macOS and Linux). +You do not need to install Python: uv downloads the version each stage asks for. -- an input of the pipeline (`$pipeline.field`); -- an output of any earlier step (`$steps.step.field`); -- a fixed value (`{"const": ...}`). +```bash +dotnet new console -n MyCalculation && cd MyCalculation +dotnet add package PythonScriptRunner.Core +dotnet add package Microsoft.Extensions.Hosting +``` -Without an explicit binding, the field with the same name from the previous step is used. The pipeline declares its own -outputs. Types and bindings are validated when the catalog loads. +A stage is a folder with a contract and a script. `psr/stages/average/stage.json`: -```jsonc -// pipelines/progression-calc.pipeline.json +```json { - "key": "progression-calc", "version": 1, "name": "Progression calculation", - "inputs": [ { "name": "a1", "type": "number" }, { "name": "ratio", "type": "number" }, - { "name": "n", "type": "integer" }, { "name": "coefficients", "type": "list", "itemType": "number" } ], - "steps": [ - { "key": "terms", "stage": "progression-terms@1" }, - { "key": "polynomial", "stage": "polynomial-eval@1", - "bindings": { "coefficients": "$pipeline.coefficients", "x": "$steps.terms.terms" } }, - { "key": "statistics", "stage": "statistics@1", "bindings": { "values": "$steps.polynomial.values" } } - ], - "outputs": [ { "name": "sum", "source": "$steps.statistics.sum" } ] + "key": "average", + "version": 1, + "name": "Average", + "inputs": [ { "name": "values", "type": "list", "itemType": "number" } ], + "outputs": [ { "name": "average", "type": "number" } ] } ``` -```csharp -await using var result = await runner.RunPipelineAsync("progression-calc", - new { a1 = 1, ratio = 2, n = 4, coefficients = new[] { 1.0, 0.0, 1.0 } }); -double sum = result.GetOutput("sum"); -double[] terms = result.GetStepOutput("terms", "terms"); // debugging -``` - -The same pipeline can be defined in C# with `AddPipeline("key", p => p.Input(...).Step(...).Output(...))`. If a step -fails, the following steps end as `Skipped` and the run ends as `Failed`. +`psr/stages/average/script.py`: -## Persistence (PostgreSQL) - -With the `PythonScriptRunner.EntityFrameworkCore` package, stages and pipelines live in the database (schema `psr`). -Their versions are immutable and move from draft to published to archived. **Every** run is recorded: steps, attempts, -the tail of stdout/stderr and files (stored in `bytea`, read and written as streams). +```python +from psrunner import stage -```csharp -builder.Services - .AddPythonScriptRunner() - .AddEntityFrameworkCore(o => - { - o.UseNpgsql(builder.Configuration.GetConnectionString("psr")!); // PostgreSQL 16+ - o.Retention.Artifacts = TimeSpan.FromDays(7); // files - o.Retention.Runs = TimeSpan.FromDays(90); // history - }); -builder.Services.AddHealthChecks().AddPythonScriptRunnerDatabase(); - -var app = builder.Build(); -await app.Services.MigratePythonScriptRunnerAsync(); // explicit; or GeneratePythonScriptRunnerMigrationScriptAsync() +@stage +def run(inputs, ctx): + values = inputs["values"] + ctx.log.info("averaging %d values", len(values)) + return {"average": sum(values) / len(values)} ``` -- **Catalog:** `ICatalogAdmin` handles the whole life of a version: - - creates drafts, with optimistic concurrency; - - publishes them, after checking the syntax with the declared Python and locking the dependencies with - `uv pip compile --generate-hashes`; - - archives, deletes and tests drafts; - - imports and exports zip files in the folder format. -- **Runs:** `IRunManager` queries runs (keyset pagination), cancels them and retries a failed run. Retried steps that - succeeded reuse their outputs; the step that failed gets a new attempt. `RunOptions.IdempotencyKey` returns the same - run for 24 h. -- **Retention:** the history is partitioned (by week for runs, by day for file contents, by month for the audit log). - A maintenance job creates partitions ahead of time and drops old ones. A pipeline can shorten its retention. -- **Audit:** human actions (`Actor.User(...)`) go to `psr.audit_events`; automatic triggers are recorded on the run - itself. +`Program.cs`: -## Queue, worker and scheduling (RabbitMQ) - -With `PythonScriptRunner.RabbitMQ` and `PythonScriptRunner.Worker`, runs can be queued. Each step is a message: - -1. A worker claims the step with a *lease* and runs the Python process. -2. It writes the result, and the message for the next step, in the same transaction (an outbox in PostgreSQL). +```csharp +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Hosting; +using PythonScriptRunner; -Enqueuing works even while the broker is down. +var builder = Host.CreateApplicationBuilder(args); +builder.Services.AddPythonScriptRunner().AddFileCatalog("psr"); -```csharp -builder.Services - .AddPythonScriptRunner() - .AddEntityFrameworkCore(o => o.UseNpgsql(builder.Configuration.GetConnectionString("psr")!)) - .AddRabbitMq(o => o.ConnectionString = builder.Configuration.GetConnectionString("rabbitmq")) // RabbitMQ 3.13+ - .AddWorker(o => o.MaxConcurrency = 8) // only in the processes that run steps - .AddRunEventHandler(s => s.Subscription("erp").On("run.succeeded.progression-calc", "run.failed.*")); -builder.Services.AddHealthChecks().AddPythonScriptRunnerChecks(); // database, broker and uv +using var host = builder.Build(); +await host.StartAsync(); -// ... -RunHandle run = await queue.EnqueuePipelineAsync("progression-calc", new { a1 = 1, ratio = 2, n = 4, coefficients = new[] { 1, 0, 1 } }); +var runner = host.Services.GetRequiredService(); +await using var result = await runner.RunStageAsync("average", new { values = new[] { 2.0, 4.0, 9.0 } }); +result.EnsureSuccess(); +Console.WriteLine(result.GetOutput("average")); // 5 ``` -- **Reliability:** - - automatic retry per stage, with exponential backoff; - - a lost worker is detected through its *lease*. The step resumes if the stage is idempotent or has retries left; - otherwise it fails with `WorkerLost` for a manual retry; - - duplicate messages never run a step twice. -- **Concurrency:** `MaxConcurrency` per worker and `max_concurrent_runs` per pipeline (the other runs wait as `Pending`, - in order). -- **Cancellation:** `IRunManager.CancelAsync` stops the process tree on the worker that runs the step. -- **Scheduling:** `ICatalogAdmin.CreateScheduleAsync` takes a cron expression and an IANA time zone (default - `America/Sao_Paulo`). A missed occurrence becomes one catch-up run, and several workers never fire the same occurrence. -- **External systems:** the public `StartPipelineRun` command (exchange `psr.public`) starts pipelines. The `run.*` and - `step.*` events (exchange `psr.events`) report progress and outputs. The contracts are in `schemas/messages`. -- **Observability:** `ActivitySource` and `Meter` named `PythonScriptRunner`; the `traceparent` follows the run all the - way to the worker. The relay polls the outbox every 250 ms, so the library logs its SQL commands at `Debug` - (`Microsoft.EntityFrameworkCore.Database.Command`). - -`hosts/PythonScriptRunner.WorkerHost` is a ready-made worker. It provides `/health/live`, `/health/ready`, JSON logs and -OTLP, and is configured with environment variables: - -- `ConnectionStrings__psr` and `ConnectionStrings__rabbitmq`; -- `PythonScriptRunner__Worker__MaxConcurrency`; -- `PythonScriptRunner__Secrets__`; -- `PSR_AUTO_MIGRATE=true`. +Add a `requirements.txt` next to the script to use packages such as numpy or pandas. More in +[Getting started](https://github.com/ScriptRunnerKit/PythonScriptRunner/blob/main/guides/en/getting-started.md) and +[Stages](https://github.com/ScriptRunnerKit/PythonScriptRunner/blob/main/guides/en/stages.md). -## Dashboard +## Stages and the contract -With `PythonScriptRunner.Dashboard`, the dashboard is embedded in your application, in the style of Hangfire. It has an -internal API, real-time updates (SignalR) and the Vue UI, served under the prefix you choose. Without -`MapPythonScriptRunnerDashboard` there is no dashboard. +A stage declares its inputs and outputs with one of 12 types: `number`, `integer`, `decimal`, `string`, `bool`, `date`, +`datetime`, `enum`, `json`, `table`, `file` and `list`. Values are converted on both sides. For example, `decimal` +becomes `decimal` in .NET and `Decimal` in Python, and `table` becomes a list of records or a `pandas.DataFrame`. -The UI is in Brazilian Portuguese, with light and dark themes. It covers: +Each stage can also declare: +- its Python version and requirements; +- a timeout and a memory limit; +- an automatic retry policy; +- the secrets it receives. -- an overview; -- **Stages**: a list you can filter by pipeline, a Monaco editor with a field builder, test runs through the queue, and - publishing; -- **Pipelines**: a visual editor with typed *handles* or an editable list, test runs, and publishing; -- live runs: the flow by status, logs, files, a timeline, cancel and retry; -- schedules with a cron preview; -- the audit log and administration. +Stages can be defined in `stage.json` files or in C# +(`AddStage("key", s => s.Script(...).Input(...).Output(...))`). See +[Stages](https://github.com/ScriptRunnerKit/PythonScriptRunner/blob/main/guides/en/stages.md). -The UI ships inside the package: there is nothing to host separately. +## Pipelines -```csharp -builder.Services - .AddPythonScriptRunner() - .AddEntityFrameworkCore(o => o.UseNpgsql(builder.Configuration.GetConnectionString("psr")!)) - .AddRabbitMq(o => o.ConnectionString = builder.Configuration.GetConnectionString("rabbitmq")) - .AddDashboard(o => o.UseFixedLogin(login => login.BindConfiguration())); // section PythonScriptRunner:Dashboard:Login +A pipeline runs stages in order. Each step input comes from a pipeline input (`$pipeline.field`), from an output of any +earlier step (`$steps.step.field`) or from a constant, and types are checked before anything runs. -var app = builder.Build(); -app.MapPythonScriptRunnerDashboard("/psr"); // API at /psr/api, real time at /psr/hubs/runs, UI at /psr/ +```csharp +await using var result = await runner.RunPipelineAsync("progression-calc", + new { a1 = 1, ratio = 2, n = 4, coefficients = new[] { 1.0, 0.0, 1.0 } }); +double sum = result.GetOutput("sum"); ``` -Choose **one** access mode. Roles are cumulative: - -1. `Reader` sees everything. -2. `Executor` also triggers, cancels and retries runs. -3. `Author` also edits, tests, publishes, imports and schedules. -4. `Admin` also deletes and manages retention, secrets and the audit log. +If a step fails, the following ones are skipped and the error says which step broke. See +[Pipelines](https://github.com/ScriptRunnerKit/PythonScriptRunner/blob/main/guides/en/pipelines.md). -| Mode | Configuration | -|------|---------------| -| Fixed login | `o.UseFixedLogin(l => l.BindConfiguration())` with users in the configuration (below) or `l.Users` in code | -| Custom filter | `o.UseAuthorizationFilter()`: an `IDashboardAuthorizationFilter` returns `DashboardUser(name, role)` or `null` | -| Host authentication | `o.UseHostAuthentication(r => { r.Reader = p => p.RequireRole("psr-reader"); /* Executor, Author, Admin */ r.LoginUrl = "/account/login?returnUrl={returnUrl}"; })` | -| Local only | `o.AllowLocalRequests = true`: Development only (outside it, startup fails); it adds to a main mode | +## Python SDK -```json -"PythonScriptRunner": { - "Dashboard": { - "Login": { - "Users": [ - { "Name": "ana", "Role": "Admin", "PasswordHash": "AQAAAAIAAYagAAAAE..." }, - { "Name": "bruno", "Role": "Reader", "PasswordHash": "AQAAAAIAAYagAAAAE..." } - ] - } - } -} -``` +The [`psrunner`](https://pypi.org/project/psrunner/) package is the Python side. The runner installs it in every +environment; script authors install it only to test locally, with pytest and without .NET: -The password never goes into the configuration, only its hash (PBKDF2, in the ASP.NET Identity format). The `psr` tool -generates it: +```python +from psrunner.testing import run_stage -```bash -dotnet tool install -g PythonScriptRunner.Tool -psr hash-password # type the password twice, without echo -echo "password" | psr hash-password --stdin # in scripts +def test_average(): + result = run_stage("psr/stages/average/script.py", {"values": [2, 4, 9]}, + stage_json="psr/stages/average/stage.json") + assert result.outputs == {"average": 5.0} ``` -How fixed login behaves: +Scripts also get a context with: +- a logger; +- the declared secrets; +- output files. -- **Session:** a `psr.session` cookie (HttpOnly, `SameSite=Lax`), sliding for 8 h, 24 h at most. -- **Lockout:** 5 failures in a row lock the user for 15 min; one IP can try 10 times per minute. -- **Configuration changes** apply without a restart. A removed user or a changed password ends the session; a new role - applies immediately. -- **Cookie keys** live in `psr.dashboard_keys`, so a session works on every instance and survives restarts. +Business errors are raised with `StageError(message, code=...)`. See the +[Python SDK guide](https://github.com/ScriptRunnerKit/PythonScriptRunner/blob/main/guides/en/python-sdk.md). -In the API, requests that change something require the `X-PSR-Request: 1` header. Edits use `ETag`/`If-Match`, and -errors follow `ProblemDetails` with a `code`. In Development, the (internal) OpenAPI document is at -`/psr/api/openapi/v1.json`. - -`hosts/PythonScriptRunner.DashboardHost` is a ready-made dashboard. It authenticates in one of two ways: +## Persistence (PostgreSQL) -- **Fixed login**, with users from environment variables (`PythonScriptRunner__Dashboard__Login__Users__0__Name`, - `__Role`, `__PasswordHash`). -- **OpenID Connect**, with `Oidc__Authority`, `Oidc__ClientId` and `Oidc__ClientSecret`. Roles come from the - `Dashboard__RoleClaim` claim (default `roles`), mapped with `Dashboard__Roles__Reader|Executor|Author|Admin`. +```csharp +builder.Services.AddPythonScriptRunner() + .AddEntityFrameworkCore(o => o.UseNpgsql(builder.Configuration.GetConnectionString("psr")!)); +await app.Services.MigratePythonScriptRunnerAsync(); +``` -It also accepts these variables: +- **Versions:** stages and pipelines have immutable versions (draft, published, archived). Publishing locks the + requirements with hashes. +- **Runs:** every run is recorded, with its steps, attempts, logs and files. +- **Management:** `IRunManager` queries runs, cancels them and retries a failed run from the step that failed. +- **History:** retention is partitioned and cleaned by a maintenance job. Human actions go to an audit log. -- `Dashboard__PathPrefix` (default `/psr`); -- `ASPNETCORE_FORWARDEDHEADERS_ENABLED=true` behind a proxy; -- `PSR_AUTO_MIGRATE=true`. +See [Persistence](https://github.com/ScriptRunnerKit/PythonScriptRunner/blob/main/guides/en/persistence.md). -## Packages +## Queue, worker and scheduling (RabbitMQ) -| Package | Contents | -|---------|----------| -| `PythonScriptRunner.Abstractions` | public contracts: field types, `StageDefinition`, `RunResult`, `PsrFile`, interfaces | -| `PythonScriptRunner.Core` | the runner: catalogs (code and folder), validation, uv environments, process executor | -| `PythonScriptRunner.EntityFrameworkCore` | PostgreSQL persistence: versioned catalog, history, files, audit, retention | -| `PythonScriptRunner.RabbitMQ` | queue, outbox relay, public command and events | -| `PythonScriptRunner.Worker` | runs queued steps; cron scheduling | -| `PythonScriptRunner.Dashboard` | embeddable dashboard (API, SignalR, Vue UI) | -| `PythonScriptRunner.Tool` | the `psr` dotnet tool (`psr hash-password`) | +```csharp +builder.Services.AddPythonScriptRunner() + .AddEntityFrameworkCore(o => o.UseNpgsql(builder.Configuration.GetConnectionString("psr")!)) + .AddRabbitMq(o => o.ConnectionString = builder.Configuration.GetConnectionString("rabbitmq")) + .AddWorker(o => o.MaxConcurrency = 4); -```bash -dotnet add package PythonScriptRunner.Core -dotnet add package PythonScriptRunner.EntityFrameworkCore # optional: PostgreSQL -dotnet add package PythonScriptRunner.RabbitMQ # optional: queue -dotnet add package PythonScriptRunner.Worker # optional: worker and cron -dotnet add package PythonScriptRunner.Dashboard # optional: dashboard +RunHandle run = await queue.EnqueuePipelineAsync("progression-calc", inputs); ``` -All 7 packages share one version and ship symbols with SourceLink. The runner installs the `psrunner` SDK from PyPI in -every Python environment it creates (`PythonScriptRunner:PsrunnerRequirement`, default `psrunner==0.1.0`). To use a -private index, set `PythonScriptRunner:PackageIndexUrl` and `PythonScriptRunner:ExtraIndexUrls`. - -## Containers +- **Transactional outbox:** enqueuing works even while the broker is down. +- **Retry:** failed attempts are retried with exponential backoff. +- **Lost workers:** a lost worker is detected by its lease and the step resumes elsewhere. +- **Cron schedules:** they run in any IANA time zone, and several workers never fire the same occurrence. +- **Other languages:** systems written in other languages start pipelines with a `StartPipelineRun` message and follow + them through `run.*` and `step.*` events. -The hosts are published as multi-arch images (amd64 and arm64), with the same version as the NuGet packages, plus -`latest`: - -| Image | Host | -|-------|------| -| `ghcr.io/scriptrunnerkit/psr-worker` | `WorkerHost` | -| `ghcr.io/scriptrunnerkit/psr-dashboard` | `DashboardHost` (the Vue UI embedded) | +See [Queue, worker and scheduling](https://github.com/ScriptRunnerKit/PythonScriptRunner/blob/main/guides/en/queue-and-worker.md). -Both images share these properties: +## Dashboard -- They are built on `mcr.microsoft.com/dotnet/aspnet:10.0` with `uv`. -- They run as the non-root user `psr` (uid 10001) and listen on port 8080, with `/health/live` and `/health/ready`. -- They include the `psrunner` wheel of the same commit: the Python environments do not download the SDK. The stage - dependencies still come from PyPI or from your index. -- They carry an SBOM and provenance attestations. +```csharp +builder.Services.AddPythonScriptRunner() + .AddEntityFrameworkCore(o => o.UseNpgsql(builder.Configuration.GetConnectionString("psr")!)) + .AddRabbitMq(o => o.ConnectionString = builder.Configuration.GetConnectionString("rabbitmq")) + .AddDashboard(o => o.UseFixedLogin(login => login.BindConfiguration())); -```bash -docker run -d --read-only \ - --tmpfs /tmp/psr:uid=10001,gid=10001,exec \ - -v psr-envs:/var/lib/psr/envs -v psr-python:/var/lib/psr/python \ - -e ConnectionStrings__psr="Host=pg;Database=psr;Username=psr_app;Password=..." \ - -e ConnectionStrings__rabbitmq="amqp://psr:...@rabbit:5672" \ - ghcr.io/scriptrunnerkit/psr-worker:0.1.0 +app.MapPythonScriptRunnerDashboard("/psr"); ``` -| Variable | Host | Example | -|----------|------|---------| -| `ConnectionStrings__psr` | both | `Host=pg;Database=psr;Username=psr_app;Password=…` | -| `ConnectionStrings__rabbitmq` | both | `amqps://psr:…@rabbit:5671/psr` | -| `PSR_AUTO_MIGRATE` | both | `true` applies the migrations at startup (default `false`); concurrent hosts wait on an advisory lock | -| `PythonScriptRunner__Worker__MaxConcurrency` | worker | `8` | -| `PythonScriptRunner__Secrets__` | worker | from the platform's secret store | -| `PythonScriptRunner__ThreadsPerProcess` | both | threads for the numeric libraries (see [Memory limits](#memory-limits)) | -| `PythonScriptRunner__PackageIndexUrl` | both | private package index | -| `PythonScriptRunner__Dashboard__Login__Users__N__Name` / `__Role` / `__PasswordHash` | dashboard | fixed-login users | -| `Oidc__Authority`, `Oidc__ClientId`, `Oidc__ClientSecret` | dashboard | optional: turns on OpenID Connect | -| `Dashboard__PathPrefix` | dashboard | `/psr` | -| `OTEL_EXPORTER_OTLP_ENDPOINT` | both | `http://otel-collector:4317` | -| `Serilog__MinimumLevel__Override__` | both | `Information` for `Microsoft.AspNetCore` brings back the request logs (default `Warning`) | - -### Volumes and read-only file system - -The images run with a read-only root file system. Python writes only to these volumes: - -| Path | Contents | If it is not persistent | -|------|----------|-------------------------| -| `/var/lib/psr/envs` | cached virtual environments (one per set of dependencies) and the uv cache | every start recreates the environments, so the first run of each stage is slower | -| `/var/lib/psr/python` | Python interpreters installed by uv | the interpreter is downloaded again | -| `/tmp/psr` | work folders of the runs and `TMPDIR` | expected: it only holds temporary files | - -A tmpfs works for any of them, but it must allow execution (`exec`), because the volumes run Python; Docker mounts tmpfs -`noexec` by default. Without access to PyPI, the environments fail with `EnvironmentSetupFailed`. In that case, point -`PythonScriptRunner__PackageIndexUrl` to a private index, or build the image with an interpreter already installed: +An embedded web UI, in the style of Hangfire: +- a Monaco editor for scripts and a builder for the contract; +- a visual pipeline editor; +- runs started from generated forms and followed live; +- logs, files and a timeline; +- cron schedules, audit and administration. -```bash -docker build --build-arg PSR_PREINSTALL_PYTHON=3.12 -f hosts/PythonScriptRunner.WorkerHost/Dockerfile -t my-psr-worker . -``` +Access is by fixed login (password hashes from the `psr` tool), by your application's own authentication (including +OpenID Connect), or by a custom filter, with four cumulative roles. See +[Dashboard](https://github.com/ScriptRunnerKit/PythonScriptRunner/blob/main/guides/en/dashboard.md). -### Memory limits - -Every Python process gets the memory limit of its stage (default 1024 MB). How it is enforced depends on the platform, -and the host logs the mode it uses at startup: - -- **Windows:** a Job Object per process. -- **Linux with a delegable cgroup v2:** each process runs in its own cgroup (`memory.max`, no swap). When the process - exceeds it, the kernel kills it and the attempt ends with `MemoryLimitExceeded` (code `OUT_OF_MEMORY`). The host needs - write access to its own cgroup: a systemd service with `Delegate=yes`, or a container running as root with a writable - cgroup file system. Log: `Memory limits of Python processes use cgroup v2 under …`. -- **Other Linux setups, including standard containers** (they mount the cgroup file system read-only): an address-space - limit (`RLIMIT_AS`). Numeric libraries (OpenBLAS, MKL, OpenMP, NumExpr, vecLib) reserve address space per thread, so in - this mode they get 1 thread per process. Log: `Memory limits of Python processes use RLIMIT_AS …`. - -`PythonScriptRunner:ThreadsPerProcess` sets the thread count in every mode. Raise it for parallel numeric work, with a -memory limit that has room for it. - -### Kubernetes - -There is no official Helm chart yet. This example of a worker `Deployment` shows the settings that matter: - -```yaml -apiVersion: apps/v1 -kind: Deployment -metadata: - name: psr-worker -spec: - replicas: 2 - selector: - matchLabels: { app: psr-worker } - template: - metadata: - labels: { app: psr-worker } - spec: - # Longer than PythonScriptRunner__Worker__ShutdownGracePeriod (default 5 min), so running steps can finish. - terminationGracePeriodSeconds: 330 - securityContext: - runAsNonRoot: true - runAsUser: 10001 - fsGroup: 10001 - containers: - - name: worker - image: ghcr.io/scriptrunnerkit/psr-worker:0.1.0 - env: - - name: ConnectionStrings__psr - valueFrom: { secretKeyRef: { name: psr, key: postgres } } - - name: ConnectionStrings__rabbitmq - valueFrom: { secretKeyRef: { name: psr, key: rabbitmq } } - ports: [ { containerPort: 8080 } ] - livenessProbe: { httpGet: { path: /health/live, port: 8080 } } - readinessProbe: { httpGet: { path: /health/ready, port: 8080 } } - securityContext: - readOnlyRootFilesystem: true - allowPrivilegeEscalation: false - resources: - limits: { memory: 4Gi } # room for MaxConcurrency processes at their stage limit - volumeMounts: - - { name: envs, mountPath: /var/lib/psr/envs } - - { name: python, mountPath: /var/lib/psr/python } - - { name: work, mountPath: /tmp/psr } - volumes: - - { name: envs, emptyDir: {} } # a PersistentVolumeClaim keeps the environments across restarts - - { name: python, emptyDir: {} } - - { name: work, emptyDir: {} } -``` +![Stage editor](https://raw.githubusercontent.com/ScriptRunnerKit/PythonScriptRunner/main/guides/images/dashboard-stage-editor.png) + +## Containers and .NET Aspire -If `terminationGracePeriodSeconds` is shorter than `ShutdownGracePeriod`, the kubelet kills attempts that are still -running. They then follow the lease rules: they resume on another worker or fail with `WorkerLost`. +| Image | Host | +|-------|------| +| `ghcr.io/scriptrunnerkit/psr-worker` | Runs queued steps and schedules | +| `ghcr.io/scriptrunnerkit/psr-dashboard` | The dashboard, with the UI embedded | -## Local environment (.NET Aspire) +The images are: +- multi-arch (amd64 and arm64); +- non-root, and they run on a read-only root file system; +- configured by environment variables; +- shipped with an SBOM and provenance attestations. -With Docker, the .NET 10 SDK, Node.js 24 and uv installed: +For a complete local environment (PostgreSQL, RabbitMQ, worker, dashboard and a test SMTP server) with traces in the +Aspire dashboard: ```bash +git clone https://github.com/ScriptRunnerKit/PythonScriptRunner && cd PythonScriptRunner dotnet run --project hosts/PythonScriptRunner.AppHost ``` -The AppHost starts PostgreSQL 16, RabbitMQ 3.13 (with the management UI), Mailpit (a test SMTP server), the WorkerHost, -the DashboardHost and the Vue UI in development mode (Vite, with hot reload). The Aspire dashboard -(`http://localhost:15000`) shows the resources, logs and traces of every run. The link **Painel** on the `dashboard` -resource opens the UI. - -The DashboardHost has one development user per role, all with the password `psr-dev`: - -| User | Role | -|------|------| -| `leitor` | Reader | -| `executor` | Executor | -| `autor` | Author | -| `admin` | Admin | - -The sample pipelines are in `tests/fixtures`. To import one: +`aspire publish` turns the same setup into a Docker Compose file. See +[Hosting](https://github.com/ScriptRunnerKit/PythonScriptRunner/blob/main/guides/en/hosting.md), which also covers +volumes, memory limits and a Kubernetes example. -1. Zip the contents of its folder, with `stages/` and `pipelines/` at the root of the zip. -2. Import it in the UI as `autor`. -3. Publish the stages, then the pipeline. +## Observability and security -`html-para-email` sends its e-mail to Mailpit. +- **Observability:** OpenTelemetry traces and metrics (`PythonScriptRunner` source and meter), health checks for the + database, the broker and uv, and structured logs. One trace follows a run from the HTTP request to every Python + process. +- **Security:** stages are code from trusted authors and run with the worker's permissions; it is not a sandbox for + untrusted code. Secrets reach only the stage that declares them and are masked in the logs. -## Docker Compose +See [How it works](https://github.com/ScriptRunnerKit/PythonScriptRunner/blob/main/guides/en/how-it-works.md) and the +[configuration reference](https://github.com/ScriptRunnerKit/PythonScriptRunner/blob/main/guides/en/configuration.md). -`aspire publish` turns the AppHost into a `docker-compose.yaml` that uses the published images: - -```bash -dotnet tool install --global Aspire.Cli --version 13.6.2 -aspire publish --apphost hosts/PythonScriptRunner.AppHost/PythonScriptRunner.AppHost.csproj -o deploy -``` - -Fill in `deploy/.env`, then start it with `docker compose up -d` in `deploy`: - -| Variable | Meaning | -|----------|---------| -| `POSTGRES_PASSWORD` | password of the `postgres` user | -| `RABBITMQ_PASSWORD` | password of the RabbitMQ `guest` user | -| `PSR_VERSION` | image version (default `latest`) | -| `PSR_DASHBOARD_PORT` | port of the dashboard on the host (default `8080`); the UI is at `/psr/` | - -The compose includes an Aspire dashboard for the traces. It is a starting point, and it keeps the development setup: +## Packages -- the four users with the password `psr-dev`; -- Mailpit; -- the test SMTP secrets. +| Package | Contents | +|---------|----------| +| [`PythonScriptRunner.Core`](https://www.nuget.org/packages/PythonScriptRunner.Core) | The runner: catalogs (folder and code), validation, uv environments, process executor | +| [`PythonScriptRunner.Abstractions`](https://www.nuget.org/packages/PythonScriptRunner.Abstractions) | Public contracts: field types, results, files, interfaces | +| [`PythonScriptRunner.EntityFrameworkCore`](https://www.nuget.org/packages/PythonScriptRunner.EntityFrameworkCore) | PostgreSQL persistence: versioned catalog, history, files, audit, retention | +| [`PythonScriptRunner.RabbitMQ`](https://www.nuget.org/packages/PythonScriptRunner.RabbitMQ) | Queue, outbox relay, public command and events | +| [`PythonScriptRunner.Worker`](https://www.nuget.org/packages/PythonScriptRunner.Worker) | Runs queued steps and cron schedules | +| [`PythonScriptRunner.Dashboard`](https://www.nuget.org/packages/PythonScriptRunner.Dashboard) | Embedded dashboard (API, SignalR, Vue UI) | +| [`PythonScriptRunner.Tool`](https://www.nuget.org/packages/PythonScriptRunner.Tool) | The `psr` dotnet tool (`psr hash-password`) | +| [`psrunner`](https://pypi.org/project/psrunner/) (PyPI) | The Python SDK | -Before you expose it, replace the `PasswordHash` values with hashes from `psr hash-password` (or configure OpenID -Connect), and remove what you do not use. +All NuGet packages share one version and ship symbols with SourceLink. ## Compatibility | PythonScriptRunner (NuGet and images) | .NET | Protocol | `psrunner` | Python | |---------------------------------------|------|----------|------------|--------| -| 0.1.x | 10 | 1 | 0.1.x | 3.10–3.13 | - -The `psrunner` SDK has its own version on PyPI. It stays compatible with protocol 1 for as long as protocol 1 is in use. -The infrastructure needs PostgreSQL 16+ and RabbitMQ 3.13+. - -## Requirements +| 0.1.x | 10 | 1 | 0.1.x | 3.10 to 3.13 | -- .NET 10 SDK. -- [uv](https://docs.astral.sh/uv/) on the PATH. It creates and caches one Python environment per set of dependencies. -- Docker, only for the tests and the local environment: the tests use Testcontainers. -- PostgreSQL 16+, only for the persistence package. -- RabbitMQ 3.13+, only for the queue. -- Node.js 24, only to build in Release or to pack `PythonScriptRunner.Dashboard`: the build embeds the UI from `panel/`. +The `psrunner` SDK has its own version on PyPI and stays compatible with protocol 1 for as long as that protocol is in +use. The infrastructure needs PostgreSQL 16+ and RabbitMQ 3.13+, only at the levels that use them. ## Repository layout | Folder | Contents | |--------|----------| -| `src/PythonScriptRunner.Abstractions` | public contracts (field types, `StageDefinition`, `RunResult`, `PsrFile`, interfaces) | -| `src/PythonScriptRunner.Core` | catalogs (code and folder), validation, uv environments, process executor, runner | -| `src/PythonScriptRunner.EntityFrameworkCore` | PostgreSQL: versioned catalog, history, files, idempotency, outbox, audit, retention, schedules and the lifecycle of queued runs | -| `src/PythonScriptRunner.RabbitMQ` | topology, outbox relay, `IRunQueue`, public command, events and subscriptions | -| `src/PythonScriptRunner.Worker` | runs queued steps: *lease*, retry, cancellation, cron | -| `src/PythonScriptRunner.Dashboard` | embeddable dashboard: internal API, SignalR hub, access modes (fixed login, filter, host, local) | -| `src/PythonScriptRunner.Tool` | the `psr` dotnet tool (`psr hash-password`) | -| `panel` | the dashboard UI: Vue 3, Vuetify 4, Vue Flow and Monaco (TypeScript, Vite, Vitest) | -| `hosts` | `WorkerHost` and `DashboardHost` (with their Dockerfiles), `ServiceDefaults` (Serilog, OpenTelemetry, health checks) and the Aspire `AppHost` | -| `python/psrunner` | the Python SDK (`@stage`, `Context`, `File`, `StageError`, `psrunner.testing`, CLI) | -| `schemas` | JSON Schemas for `stage.json`, `*.pipeline.json` and the public messages | -| `tests` | .NET tests (unit, integration with real Python, containers, Aspire and dashboard E2E with Playwright) and sample pipelines | +| `src/` | The NuGet packages (Abstractions, Core, EntityFrameworkCore, RabbitMQ, Worker, Dashboard, Tool) | +| `panel/` | The dashboard UI: Vue 3, Vuetify 4, Vue Flow and Monaco (TypeScript, Vite, Vitest) | +| `hosts/` | `WorkerHost` and `DashboardHost` (with their Dockerfiles), `ServiceDefaults` and the Aspire `AppHost` | +| `python/psrunner/` | The Python SDK | +| `schemas/` | JSON Schemas of `stage.json`, `*.pipeline.json` and the public messages | +| `guides/` | The documentation, in English and Portuguese | +| `tests/` | .NET tests (unit, integration with real Python, PostgreSQL and RabbitMQ, containers, Aspire and dashboard end-to-end with Playwright) and sample pipelines | ## Development ```bash dotnet build -dotnet test --project tests/PythonScriptRunner.Core.Tests -dotnet test --project tests/PythonScriptRunner.EntityFrameworkCore.Tests # needs Docker -dotnet test --project tests/PythonScriptRunner.RabbitMQ.Tests # needs Docker -dotnet test --project tests/PythonScriptRunner.Worker.Tests # needs Docker -dotnet test --project tests/PythonScriptRunner.Dashboard.Tests # needs Docker -dotnet test --project tests/PythonScriptRunner.AppHost.Tests # needs Docker, Node.js and uv - -cd python/psrunner -uv sync -uv run pytest -``` - -The UI lives in `panel/` (Node 24). Debug builds embed a placeholder page. In Release (or with `-p:PsrBuildPanel=true`), -MSBuild runs `npm ci` and `npm run build` and embeds `panel/dist`. - -```bash -cd panel -npm ci -npm run dev # Vite at http://localhost:5173/; /api and /hubs go to the DashboardHost at PSR_DASHBOARD_URL (default http://localhost:5080, prefix /psr) -npm test # Vitest -npm run lint # ESLint + Prettier -npm run build # type check, build and the budget of the initial bundle (≤ 500 KB gzip) -npm run api # regenerates src/api/schema.d.ts from the OpenAPI snapshot -``` - -The dashboard E2E tests need Docker and the embedded UI. They start the `DashboardHost` with 4 fixed-login users, a -worker, PostgreSQL and RabbitMQ, and drive Chromium with Playwright: - -```bash -dotnet build tests/PythonScriptRunner.Panel.E2E.Tests -p:PsrBuildPanel=true -dotnet test --project tests/PythonScriptRunner.Panel.E2E.Tests --no-build # PSR_E2E_HEADED=1 shows the browser -``` - -When a test fails, the Playwright trace is saved to `TestResults/traces` next to the test assembly -(`npx playwright show-trace .zip`). +dotnet test --solution tests/PythonScriptRunner.SemE2E.slnf # unit, integration and dashboard tests (needs Docker) -Run the E2E tests on their own. Containers starting or stopping in other test projects change the network interfaces, -and Chromium then aborts requests with `ERR_NETWORK_CHANGED`. The solution filter runs the unit, integration and -dashboard tests at once (the CI does the same; the container and AppHost tests run in their own jobs): - -```bash -dotnet test --solution tests/PythonScriptRunner.SemE2E.slnf +cd python/psrunner && uv sync && uv run pytest +cd panel && npm ci && npm test ``` -The container tests build both images and run them the way production does: read-only, with tmpfs volumes. They cover -the memory limit through cgroup v2 and the compose from `aspire publish`; the compose test needs the Aspire CLI -(`PSR_ASPIRE_CLI` points to it if it is not on the PATH). To test images you built yourself, set -`PSR_TEST_WORKER_IMAGE` and `PSR_TEST_DASHBOARD_IMAGE`: - -```bash -dotnet test --project tests/PythonScriptRunner.Containers.Tests # needs Docker -``` - -When a container test fails, it saves the logs of its containers to `TestResults/container-logs//` next to the test -assembly, and the end of the worker and dashboard logs goes to the test output. The CI uploads that folder as the -`container-logs` artifact. - -The dashboard performance test uses about 200,000 runs; `PSR_PERF_RUNS_PER_DAY=50000` runs the full average volume (90 -days). The dashboard's OpenAPI document is compared with -`tests/PythonScriptRunner.Dashboard.Tests/Snapshots/openapi-v1.verified.json`. After you review an intended change, -`PSR_ACCEPT_SNAPSHOTS=1` updates the snapshot. +[CONTRIBUTING.md](https://github.com/ScriptRunnerKit/PythonScriptRunner/blob/main/CONTRIBUTING.md) covers the +prerequisites, every test project, the checks the CI runs and how releases work. ## Contributing and security diff --git a/guides/README.md b/guides/README.md new file mode 100644 index 0000000..0d18d4d --- /dev/null +++ b/guides/README.md @@ -0,0 +1,4 @@ +# PythonScriptRunner guides + +- [English](en/README.md) +- [Português](pt-br/README.md) diff --git a/guides/en/README.md b/guides/en/README.md new file mode 100644 index 0000000..d99e2b3 --- /dev/null +++ b/guides/en/README.md @@ -0,0 +1,22 @@ +# PythonScriptRunner guides + +English | [Português](../pt-br/README.md) + +PythonScriptRunner runs Python scripts from .NET with a typed contract. These guides go from the smallest setup (the +library alone, with no database and no message broker) to the complete one, with a queue, scheduling and an embedded +dashboard. + +| Guide | What it covers | +|-------|----------------| +| [Getting started](getting-started.md) | The ways to use it, what to install, and your first stage in a few minutes | +| [Stages](stages.md) | The contract (inputs, outputs, the 12 field types), `stage.json`, stages in C#, limits, retry and secrets | +| [Pipelines](pipelines.md) | Chaining stages, bindings, pipeline outputs and failure handling | +| [Python SDK](python-sdk.md) | Writing scripts with `psrunner`, files, errors, and testing with pytest and the CLI | +| [Persistence (PostgreSQL)](persistence.md) | Versioned catalog, run history, retention and audit | +| [Queue, worker and scheduling (RabbitMQ)](queue-and-worker.md) | Background runs, retry, cron, events and integration with other systems | +| [Dashboard](dashboard.md) | The embedded web UI, access modes and roles | +| [Hosting](hosting.md) | Container images, volumes, memory limits, Kubernetes, .NET Aspire and Docker Compose | +| [Configuration reference](configuration.md) | Every option with its default value | +| [How it works](how-it-works.md) | Process isolation, Python environments, limits, security model and observability | + +The [README](../../README.md) has an overview of everything on one page. diff --git a/guides/en/configuration.md b/guides/en/configuration.md new file mode 100644 index 0000000..d7b9007 --- /dev/null +++ b/guides/en/configuration.md @@ -0,0 +1,108 @@ +# Configuration reference + +English | [Português](../pt-br/configuration.md) + +Options are set in code, in the callback of each `Add...` method, or bound from configuration. The ready-made hosts bind +these sections, and you can do the same in your application: + +```csharp +var configuration = builder.Configuration; +builder.Services + .AddPythonScriptRunner(o => configuration.GetSection("PythonScriptRunner").Bind(o)) + .AddEntityFrameworkCore(o => + { + o.UseNpgsql(configuration.GetConnectionString("psr")!); + configuration.GetSection("PythonScriptRunner:EntityFrameworkCore").Bind(o); + }) + .AddRabbitMq(o => + { + o.ConnectionString = configuration.GetConnectionString("rabbitmq"); + configuration.GetSection("PythonScriptRunner:RabbitMQ").Bind(o); + }) + .AddWorker(o => configuration.GetSection("PythonScriptRunner:Worker").Bind(o)); +``` + +In environment variables, sections are separated by `__`: `PythonScriptRunner__Worker__MaxConcurrency=8`. Time values use +the `hh:mm:ss` format (`00:10:00`). + +## Runner (`PythonScriptRunner`) + +| Option | Default | Description | +|--------|---------|-------------| +| `WorkRoot` | `%TEMP%/psr` | Folder for the work and result folders of the attempts. | +| `EnvironmentCacheRoot` | `%LOCALAPPDATA%/psr` | Folder where Python environments and lock files are cached. | +| `UvPath` | `uv` | Path or name of the `uv` executable (searched in the PATH). | +| `DefaultPythonVersion` | `3.12` | Python version of stages that do not declare one. | +| `DefaultTimeout` | 5 minutes | Timeout of stages that do not declare one. | +| `DefaultMemoryLimitMb` | `1024` | Memory limit of stages that do not declare one. | +| `MaxFileBytes` | 50 MB | Maximum size of one file. | +| `MaxRunFilesBytes` | 200 MB | Maximum size of all files of one run. | +| `MaxLogBytes` | 1 MB | Bytes of stdout and stderr kept per attempt (the tail). | +| `MaxConcurrentProcesses` | processor count | Python processes running at the same time in this process. | +| `ThreadsPerProcess` | not set | Threads of the numeric libraries in each process. When not set: 1 on Linux when the memory limit falls back to `RLIMIT_AS`, no limit otherwise. See [Memory limits](hosting.md#memory-limits). | +| `PackageIndexUrl` | PyPI | Package index (pip `--index-url`). | +| `ExtraIndexUrls` | none | Extra package indexes (pip `--extra-index-url`). | +| `PsrunnerRequirement` | `psrunner==0.1.0` | The SDK installed in every environment: a pip requirement or the path of a local wheel. | +| `OrphanRetention` | 24 hours | Age after which leftover work folders are deleted. | + +Secrets are read from `PythonScriptRunner:Secrets:`. + +## Persistence (`PythonScriptRunner:EntityFrameworkCore`) + +| Option | Default | Description | +|--------|---------|-------------| +| `Retention:Artifacts` | 7 days | How long file contents are kept. | +| `Retention:Runs` | 90 days | How long runs, steps, attempts, logs and file metadata are kept. | +| `Retention:Audit` | 365 days | How long audit events are kept. | +| `Maintenance:Enabled` | `true` | Runs the maintenance job (partitions, retention, cleanups) in this process. | +| `Maintenance:Interval` | 1 hour | How often the maintenance job runs. | +| `Maintenance:BatchSize` | `1000` | Rows handled per batch by the maintenance job. | +| `IdempotencyWindow` | 24 hours | How long an idempotency key returns the same run. | +| `LatestVersionCacheDuration` | 30 seconds | How long "latest published version" lookups are cached in this process. | +| `UseDatabaseClock` | `true` | Uses the PostgreSQL clock for leases, retry delays, schedules and the outbox, so hosts with skewed clocks agree. | + +The connection is set in code with `o.UseNpgsql(connectionString)`. + +## RabbitMQ (`PythonScriptRunner:RabbitMQ`) + +| Option | Default | Description | +|--------|---------|-------------| +| `ConnectionString` | none | AMQP URI: `amqp://user:password@host:5672/vhost`, or `amqps://` for TLS. | +| `DeclareTopology` | `true` | Declares exchanges and queues at startup. With `false`, startup only checks that they exist (for brokers managed by an operator). | +| `RunOutboxRelay` | `true` | Publishes the outbox from this process. Several instances can run it. | +| `EnablePublicCommands` | `true` | Consumes the public `StartPipelineRun` command. | +| `InstanceId` | machine name and process id | Names the per-instance queues of this process. | +| `RelayInterval` | 250 ms | How often the relay looks for messages when the outbox is idle. | +| `RelayBatchSize` | `100` | Messages published per relay batch. | +| `DeliveryLimit` | `5` | Delivery attempts of a command before it goes to `psr.dead-letter`. | +| `StartupConnectTimeout` | 10 seconds | How long startup waits for the broker before continuing in the background. | + +## Worker (`PythonScriptRunner:Worker`) + +| Option | Default | Description | +|--------|---------|-------------| +| `WorkerId` | the RabbitMQ instance id | Identifier recorded on the attempts this worker runs. | +| `MaxConcurrency` | processor count | Steps executed at the same time by this worker. | +| `LeaseDuration` | 60 seconds | Lease of a running attempt; another worker recovers it when not renewed. | +| `LeaseRenewInterval` | 15 seconds | How often the lease of a running attempt is renewed. | +| `ReaperInterval` | 15 seconds | How often expired leases are looked for. | +| `MaxLeaseRecoveries` | `3` | Recoveries of one step after lost workers before it fails with `WorkerLost`. | +| `ShutdownGracePeriod` | 5 minutes | How long running steps may finish when the host stops. The platform must allow at least this long. | +| `WorkFolderCleanupInterval` | 1 hour | How often leftover work folders are removed. | +| `Scheduler:Enabled` | `true` | Fires schedules from this worker. Several workers never fire the same occurrence. | +| `Scheduler:PollInterval` | 5 seconds | How often due schedules are looked for (the firing precision). | +| `Scheduler:BatchSize` | `50` | Schedules fired per transaction. | + +## Dashboard fixed login (`PythonScriptRunner:Dashboard:Login`) + +| Option | Default | Description | +|--------|---------|-------------| +| `Users` | none | List of `{ Name, Role, PasswordHash }`. | +| `SlidingExpiration` | 8 hours | Session lifetime, renewed while the user is active. | +| `AbsoluteExpiration` | 24 hours | Maximum session lifetime. | +| `MaxFailedAttempts` | `5` | Failed logins in a row before the user is locked. | +| `LockoutDuration` | 15 minutes | How long a locked user waits. | +| `MaxAttemptsPerMinutePerIp` | `10` | Login attempts allowed per IP per minute. | + +The same options exist in code on `UseFixedLogin(l => ...)`, which also has `PersistKeysInDatabase` (default `true`): it +keeps the cookie keys in PostgreSQL, so sessions work on every instance and survive restarts. diff --git a/guides/en/dashboard.md b/guides/en/dashboard.md new file mode 100644 index 0000000..e4225f0 --- /dev/null +++ b/guides/en/dashboard.md @@ -0,0 +1,147 @@ +# Dashboard + +English | [Português](../pt-br/dashboard.md) + +`PythonScriptRunner.Dashboard` embeds a web UI in your ASP.NET Core application, in the style of Hangfire. It has an +internal API, real-time updates over SignalR and a Vue interface, all inside the package: there is nothing to host +separately. The dashboard needs [persistence](persistence.md) and the [queue](queue-and-worker.md), because it stores +versions and history and runs tests and runs on the workers. + +## Setup + +```bash +dotnet add package PythonScriptRunner.Dashboard +``` + +```csharp +builder.Services + .AddPythonScriptRunner() + .AddEntityFrameworkCore(o => o.UseNpgsql(builder.Configuration.GetConnectionString("psr")!)) + .AddRabbitMq(o => o.ConnectionString = builder.Configuration.GetConnectionString("rabbitmq")) + .AddDashboard(o => o.UseFixedLogin(login => login.BindConfiguration())); // section PythonScriptRunner:Dashboard:Login + +var app = builder.Build(); +app.MapPythonScriptRunnerDashboard("/psr"); +``` + +`MapPythonScriptRunnerDashboard("/psr")` serves: +- the UI at `/psr/`; +- the API at `/psr/api`; +- real-time updates at `/psr/hubs/runs`. + +Without that call there is no dashboard. To run it somewhere other than your own application, use the ready-made +`DashboardHost` image (see [Hosting](hosting.md)). + +## What you can do + +The UI is in Brazilian Portuguese, with light and dark themes. An English version is planned. + +**Edit a stage.** Write the script in a Monaco editor and build the contract with a field builder: name, type, required, +default value and description. The Dependencies tab sets the Python version and the requirements. You can test a draft +on a real worker before publishing it. + +![Stage editor](../images/dashboard-stage-editor.png) + +![Adding an output to the contract](../images/dashboard-contract.png) + +**Build a pipeline.** Drag published stages onto a canvas and connect outputs to inputs. The handles are typed, and +problems are listed while you edit. A list view offers the same editing without the canvas. + +![Visual pipeline editor](../images/dashboard-pipeline-editor.png) + +**Run and follow.** Start a run from a form generated from the contract. The flow shows each step changing status live. +The tabs show the inputs, the outputs (tables included), the logs of every attempt, the files and a timeline. Failed runs +can be retried from the step that failed, and running ones can be cancelled. + +![Run form](../images/dashboard-run-form.png) + +![A run in progress](../images/dashboard-run-live.png) + +![Outputs of a finished run](../images/dashboard-run-outputs.png) + +The UI also has: +- an overview of the last 24 hours; +- run lists with filters; +- cron schedules, with a preview of the next occurrences; +- the audit log; +- an administration page for the `Admin` role. + +## Access modes + +Choose **one** main mode: + +| Mode | Configuration | When to use | +|------|---------------|-------------| +| Fixed login | `o.UseFixedLogin(l => l.BindConfiguration())`, or `l.Users` in code | A few known users, no identity provider. | +| Host authentication | `o.UseHostAuthentication(r => { ... })` | Your application already authenticates users (cookies, OpenID Connect, Microsoft Entra ID). | +| Custom filter | `o.UseAuthorizationFilter()` | Your own rule: an `IDashboardAuthorizationFilter` returns `DashboardUser(name, role)` or `null`. | +| Local requests | `o.AllowLocalRequests = true` | Development only (startup fails outside Development). It adds to a main mode. | + +## Roles + +The four roles are cumulative: + +| Role | Can | +|------|-----| +| `Reader` | See everything: catalog, runs, logs, files, schedules. | +| `Executor` | Also start, cancel and retry runs. | +| `Author` | Also edit, test, publish, import and schedule. | +| `Admin` | Also delete, and manage retention, secrets status and the audit log. | + +## Fixed login + +Users live in configuration. The password is never stored, only its hash (PBKDF2, the ASP.NET Identity format): + +```json +"PythonScriptRunner": { + "Dashboard": { + "Login": { + "Users": [ + { "Name": "ana", "Role": "Admin", "PasswordHash": "AQAAAAIAAYagAAAAE..." }, + { "Name": "bruno", "Role": "Reader", "PasswordHash": "AQAAAAIAAYagAAAAE..." } + ] + } + } +} +``` + +Generate the hash with the `psr` tool: + +```bash +dotnet tool install -g PythonScriptRunner.Tool +psr hash-password # asks for the password twice, without echo +echo "password" | psr hash-password --stdin # for scripts +``` + +How fixed login behaves: +- **Session:** an HttpOnly `psr.session` cookie (`SameSite=Lax`), sliding for 8 hours, 24 hours at most. +- **Lockout:** 5 failed attempts in a row lock the user for 15 minutes, and one IP can try 10 times per minute. +- **Configuration changes:** they apply without a restart. A removed user or a changed password ends the session, and a + new role applies at once. +- **Cookie keys:** they are stored in the database (`psr.dashboard_keys`), so a session works on every instance and + survives restarts. + +## Host authentication + +Map each role to an authorization policy of your application, and tell the dashboard where your login page is: + +```csharp +.AddDashboard(o => o.UseHostAuthentication(r => +{ + r.Reader = p => p.RequireRole("psr-reader"); + r.Executor = p => p.RequireRole("psr-executor"); + r.Author = p => p.RequireRole("psr-author"); + r.Admin = p => p.RequireRole("psr-admin"); + r.LoginUrl = "/account/login?returnUrl={returnUrl}"; +})); +``` + +## The API + +The API is internal: it exists for the UI and can change between versions. To integrate other systems, use the .NET +library or the [RabbitMQ command and events](queue-and-worker.md#systems-written-in-other-languages). +- **Writes:** requests that change something require the header `X-PSR-Request: 1`. +- **Edits:** they use `ETag` and `If-Match`. +- **Errors:** they follow `ProblemDetails` with a `code`. + +In Development, the OpenAPI document is at `/psr/api/openapi/v1.json`. diff --git a/guides/en/getting-started.md b/guides/en/getting-started.md new file mode 100644 index 0000000..a18a525 --- /dev/null +++ b/guides/en/getting-started.md @@ -0,0 +1,125 @@ +# Getting started + +English | [Português](../pt-br/getting-started.md) + +PythonScriptRunner connects Python code to .NET applications. The calculation stays in a plain Python script, written by +whoever knows the subject, and the .NET side receives a result that was already validated against a contract. + +## Concepts + +- **Stage**: one Python script with a **contract**, the list of its inputs and outputs and their types. The script + implements a single function, `run(inputs, ctx)`; the library validates what goes in and what comes out. +- **Pipeline**: an ordered sequence of stages. The output of one step can feed the input of a later one, and the types of + every connection are checked before anything runs. +- **Run**: one execution of a stage or pipeline. Each attempt runs in its own Python process, with a timeout, a memory + limit and its own Python environment. + +## Ways to use it + +Start with the smallest setup that does the job. The code of one level keeps working at the next one. + +| Level | What you need | What you get | +|-------|---------------|--------------| +| **1. Library only** | .NET 10 and uv | Run stages and pipelines from your own code. No database, no message broker, no dashboard. | +| **2. With PostgreSQL** | PostgreSQL 16 or newer | A versioned catalog (draft, published, archived), the history of every run, files and an audit log. See [Persistence](persistence.md). | +| **3. With RabbitMQ** | PostgreSQL and RabbitMQ 3.13 or newer | Background runs on workers, automatic retry, cron schedules, events and a command for systems written in other languages. See [Queue, worker and scheduling](queue-and-worker.md). | +| **4. With the dashboard** | PostgreSQL and RabbitMQ | A web UI embedded in your application to edit, test, run and follow stages and pipelines. See [Dashboard](dashboard.md). | +| **Ready-made hosts** | Docker | Worker and dashboard container images, or the complete environment with one command (.NET Aspire). See [Hosting](hosting.md). | + +## Prerequisites + +1. **.NET 10 SDK**: +2. **uv**, which creates and caches the Python environments. You do not need to install Python: uv downloads the version + each stage asks for. + - Windows: `winget install --id=astral-sh.uv -e` + - macOS and Linux: `curl -LsSf https://astral.sh/uv/install.sh | sh` + +## Packages + +| Package | Use | +|---------|-----| +| `PythonScriptRunner.Core` | Run stages and pipelines (level 1). Brings `PythonScriptRunner.Abstractions`. | +| `PythonScriptRunner.EntityFrameworkCore` | PostgreSQL persistence (level 2) | +| `PythonScriptRunner.RabbitMQ` | Queue, outbox relay, public command and events (level 3) | +| `PythonScriptRunner.Worker` | Runs queued steps and cron schedules (level 3) | +| `PythonScriptRunner.Dashboard` | Embedded web UI (level 4) | +| `PythonScriptRunner.Tool` | The `psr` command line tool (`psr hash-password` for the dashboard login) | +| `psrunner` (PyPI) | The Python SDK. The runner installs it in every environment; install it yourself only to test scripts locally. | + +All NuGet packages share one version. See the [compatibility table](../../README.md#compatibility). + +## Your first stage + +Create a console project: + +```bash +dotnet new console -n MyCalculation +cd MyCalculation +dotnet add package PythonScriptRunner.Core +dotnet add package Microsoft.Extensions.Hosting +``` + +Create a folder `psr/stages/average` with two files. + +`psr/stages/average/stage.json`, the contract: + +```json +{ + "key": "average", + "version": 1, + "name": "Average", + "inputs": [ { "name": "values", "type": "list", "itemType": "number" } ], + "outputs": [ { "name": "average", "type": "number" } ] +} +``` + +`psr/stages/average/script.py`, the code: + +```python +from psrunner import stage + +@stage +def run(inputs, ctx): + values = inputs["values"] + ctx.log.info("averaging %d values", len(values)) + return {"average": sum(values) / len(values)} +``` + +`Program.cs`: + +```csharp +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Hosting; +using PythonScriptRunner; + +var builder = Host.CreateApplicationBuilder(args); +builder.Services + .AddPythonScriptRunner() + .AddFileCatalog("psr"); // folder with stages/ and pipelines/ + +using var host = builder.Build(); +await host.StartAsync(); // checks the configuration (uv on the PATH, the catalog) + +var runner = host.Services.GetRequiredService(); +await using var result = await runner.RunStageAsync("average", new { values = new[] { 2.0, 4.0, 9.0 } }); +result.EnsureSuccess(); +Console.WriteLine(result.GetOutput("average")); // 5 +``` + +Run it with `dotnet run`. The first run takes a few seconds longer because uv creates the Python environment; later runs +reuse it. + +What happened: +- The inputs were checked against the contract before Python started. +- The script ran in a separate process, in a clean environment created by uv. +- The output was checked against the contract and converted to `double`. + +If the script returns something outside the contract (a text where a number is expected, or `NaN`), the run fails with a +clear error instead of returning invalid data. + +## Next steps + +- Add libraries, limits and secrets to a stage, or define it in C#: [Stages](stages.md). +- Chain stages: [Pipelines](pipelines.md). +- Test scripts with pytest, without .NET: [Python SDK](python-sdk.md). +- Keep history and versions: [Persistence](persistence.md). diff --git a/guides/en/hosting.md b/guides/en/hosting.md new file mode 100644 index 0000000..067b3ac --- /dev/null +++ b/guides/en/hosting.md @@ -0,0 +1,211 @@ +# Hosting + +English | [Português](../pt-br/hosting.md) + +You do not need to write a host application. The repository ships two ready-made hosts as container images, a local +environment on .NET Aspire, and a Docker Compose file generated from it. + +## Container images + +The images are multi-arch (amd64 and arm64). Each release publishes the version tag (`0.1.0`), the minor tag (`0.1`) and +`latest`. + +| Image | Host | +|-------|------| +| `ghcr.io/scriptrunnerkit/psr-worker` | `WorkerHost`: runs queued steps and schedules | +| `ghcr.io/scriptrunnerkit/psr-dashboard` | `DashboardHost`: the dashboard with the Vue UI embedded | + +Both images share these properties: +- **Base:** `mcr.microsoft.com/dotnet/aspnet:10.0` with `uv`. +- **User:** they run as the non-root user `psr` (uid 10001). +- **Network:** they listen on port 8080, with `/health/live` and `/health/ready`. +- **SDK:** they include the `psrunner` wheel of the same commit, so environments do not download the SDK. Stage + dependencies still come from PyPI or from your package index. +- **Supply chain:** they carry an SBOM and provenance attestations. +- **Observability:** logs are JSON on stdout, and OpenTelemetry is exported when `OTEL_EXPORTER_OTLP_ENDPOINT` is set. + +```bash +docker run -d --read-only \ + --tmpfs /tmp/psr:uid=10001,gid=10001,exec \ + -v psr-envs:/var/lib/psr/envs -v psr-python:/var/lib/psr/python \ + -e ConnectionStrings__psr="Host=pg;Database=psr;Username=psr_app;Password=..." \ + -e ConnectionStrings__rabbitmq="amqp://psr:...@rabbit:5672" \ + ghcr.io/scriptrunnerkit/psr-worker:0.1.0 +``` + +### Environment variables + +| Variable | Host | Example or meaning | +|----------|------|--------------------| +| `ConnectionStrings__psr` | both | `Host=pg;Database=psr;Username=psr_app;Password=…` | +| `ConnectionStrings__rabbitmq` | both | `amqps://psr:…@rabbit:5671/psr` | +| `PSR_AUTO_MIGRATE` | both | `true` applies the migrations at startup (default `false`); concurrent hosts wait on an advisory lock | +| `PythonScriptRunner__Worker__MaxConcurrency` | worker | `8` | +| `PythonScriptRunner__Worker__ShutdownGracePeriod` | worker | `00:05:00` | +| `PythonScriptRunner__Secrets__` | worker | Secret values, from your platform's secret store | +| `PythonScriptRunner__ThreadsPerProcess` | both | Threads of the numeric libraries (see [Memory limits](#memory-limits)) | +| `PythonScriptRunner__PackageIndexUrl` | both | A private package index | +| `PythonScriptRunner__Dashboard__Login__Users__N__Name`, `__Role`, `__PasswordHash` | dashboard | Fixed login users | +| `Oidc__Authority`, `Oidc__ClientId`, `Oidc__ClientSecret` | dashboard | Turns on OpenID Connect instead of fixed login | +| `Dashboard__RoleClaim` | dashboard | Claim with the roles (default `roles`) | +| `Dashboard__Roles__Reader`, `__Executor`, `__Author`, `__Admin` | dashboard | Role values mapped to each dashboard role | +| `Dashboard__PathPrefix` | dashboard | Default `/psr` | +| `ASPNETCORE_FORWARDEDHEADERS_ENABLED` | dashboard | `true` behind a reverse proxy | +| `OTEL_EXPORTER_OTLP_ENDPOINT` | both | `http://otel-collector:4317` | +| `Serilog__MinimumLevel__Override__` | both | For example, `Information` for `Microsoft.AspNetCore` brings back request logs (default `Warning`) | + +Every option of the [configuration reference](configuration.md) can be set the same way, with `__` between sections +(`PythonScriptRunner__DefaultTimeout=00:10:00`). + +### Volumes and read-only file system + +The images run with a read-only root file system. Python writes only to these volumes: + +| Path | Contents | If it is not persistent | +|------|----------|-------------------------| +| `/var/lib/psr/envs` | Cached virtual environments (one per set of dependencies) and the uv cache | Every start recreates the environments, so the first run of each stage is slower | +| `/var/lib/psr/python` | Python interpreters installed by uv | The interpreter is downloaded again | +| `/tmp/psr` | Work folders of the runs and `TMPDIR` | Expected: it only holds temporary files | + +A tmpfs works for any of them, but it must allow execution (`exec`), because the volumes run Python; Docker mounts tmpfs +`noexec` by default. + +Without access to PyPI, environments fail with `EnvironmentSetupFailed`. In that case, point +`PythonScriptRunner__PackageIndexUrl` to a private index, or build the image with an interpreter already installed: + +```bash +docker build --build-arg PSR_PREINSTALL_PYTHON=3.12 -f hosts/PythonScriptRunner.WorkerHost/Dockerfile -t my-psr-worker . +``` + +### Memory limits + +Every Python process gets the memory limit of its stage (1024 MB by default). How it is enforced depends on the platform, +and the host logs the mode it uses at startup: + +- **Windows:** a Job Object per process. +- **Linux with a delegable cgroup v2:** + - each process runs in its own cgroup (`memory.max`, no swap); + - when the process exceeds the limit, the kernel kills it and the attempt ends with `MemoryLimitExceeded` (code + `OUT_OF_MEMORY`); + - the host needs write access to its own cgroup: a systemd service with `Delegate=yes`, or a container running as root + with a writable cgroup file system. +- **Other Linux setups, including standard containers** (they mount the cgroup file system read-only): + - an address-space limit (`RLIMIT_AS`); + - numeric libraries (OpenBLAS, MKL, OpenMP, NumExpr, vecLib) reserve address space per thread, so in this mode they get + 1 thread per process. + +`PythonScriptRunner:ThreadsPerProcess` sets the thread count in every mode. Raise it for parallel numeric work, with a +memory limit that has room for it. + +## Kubernetes + +There is no official Helm chart yet. This worker `Deployment` shows the settings that matter: + +```yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: psr-worker +spec: + replicas: 2 + selector: + matchLabels: { app: psr-worker } + template: + metadata: + labels: { app: psr-worker } + spec: + # Longer than PythonScriptRunner__Worker__ShutdownGracePeriod (default 5 min), so running steps can finish. + terminationGracePeriodSeconds: 330 + securityContext: + runAsNonRoot: true + runAsUser: 10001 + fsGroup: 10001 + containers: + - name: worker + image: ghcr.io/scriptrunnerkit/psr-worker:0.1.0 + env: + - name: ConnectionStrings__psr + valueFrom: { secretKeyRef: { name: psr, key: postgres } } + - name: ConnectionStrings__rabbitmq + valueFrom: { secretKeyRef: { name: psr, key: rabbitmq } } + ports: [ { containerPort: 8080 } ] + livenessProbe: { httpGet: { path: /health/live, port: 8080 } } + readinessProbe: { httpGet: { path: /health/ready, port: 8080 } } + securityContext: + readOnlyRootFilesystem: true + allowPrivilegeEscalation: false + resources: + limits: { memory: 4Gi } # room for MaxConcurrency processes at their stage limit + volumeMounts: + - { name: envs, mountPath: /var/lib/psr/envs } + - { name: python, mountPath: /var/lib/psr/python } + - { name: work, mountPath: /tmp/psr } + volumes: + - { name: envs, emptyDir: {} } # a PersistentVolumeClaim keeps the environments across restarts + - { name: python, emptyDir: {} } + - { name: work, emptyDir: {} } +``` + +If `terminationGracePeriodSeconds` is shorter than `ShutdownGracePeriod`, the kubelet kills attempts that are still +running. They then follow the lease rules: they resume on another worker or fail with `WorkerLost`. + +## Local environment (.NET Aspire) + +With Docker, the .NET 10 SDK, Node.js 24 and uv installed: + +```bash +git clone https://github.com/ScriptRunnerKit/PythonScriptRunner +cd PythonScriptRunner +dotnet run --project hosts/PythonScriptRunner.AppHost +``` + +The AppHost starts: +- PostgreSQL 16 and RabbitMQ 3.13 (with the management UI); +- Mailpit, a test SMTP server; +- the WorkerHost and the DashboardHost; +- the Vue UI in development mode, with hot reload. + +The Aspire dashboard (`http://localhost:15000`) shows the resources, logs and traces of every run. The **Painel** link on +the `dashboard` resource opens the UI. + +![Trace of a run in the Aspire dashboard](../images/aspire-trace.png) + +The dashboard has one development user per role, all with the password `psr-dev`: + +| User | Role | +|------|------| +| `leitor` | Reader | +| `executor` | Executor | +| `autor` | Author | +| `admin` | Admin | + +To try the sample pipelines in `tests/fixtures`: +1. Zip the contents of a folder, with `stages/` and `pipelines/` at the root. +2. Import it in the UI as `autor`. +3. Publish the stages, then the pipeline. + +## Docker Compose + +`aspire publish` turns the AppHost into a `docker-compose.yaml` that uses the published images: + +```bash +dotnet tool install --global Aspire.Cli --version 13.6.2 +aspire publish --apphost hosts/PythonScriptRunner.AppHost/PythonScriptRunner.AppHost.csproj -o deploy +``` + +Fill in `deploy/.env`, then start it with `docker compose up -d` in `deploy`: + +| Variable | Meaning | +|----------|---------| +| `POSTGRES_PASSWORD` | Password of the `postgres` user | +| `RABBITMQ_PASSWORD` | Password of the RabbitMQ `guest` user | +| `PSR_VERSION` | Image version (default `latest`) | +| `PSR_DASHBOARD_PORT` | Port of the dashboard on the host (default `8080`); the UI is at `/psr/` | + +The compose includes an Aspire dashboard for the traces. It is a starting point, and it keeps the development setup: +- the four users with the password `psr-dev`; +- Mailpit; +- the test SMTP secrets. + +Before you expose it, replace the `PasswordHash` values with hashes from `psr hash-password` (or configure OpenID +Connect), and remove what you do not use. diff --git a/guides/en/how-it-works.md b/guides/en/how-it-works.md new file mode 100644 index 0000000..dd88e2e --- /dev/null +++ b/guides/en/how-it-works.md @@ -0,0 +1,95 @@ +# How it works + +English | [Português](../pt-br/how-it-works.md) + +This page explains what happens when a stage runs, what the library guarantees, and how to observe it in production. + +## One attempt, step by step + +1. **Validate the inputs** against the contract. Invalid inputs stop here, and no Python process starts. +2. **Resolve the secrets** the stage declares. A missing secret fails the attempt with `SecretNotFound`. +3. **Prepare the environment.** uv creates (or reuses) the Python environment of the stage. Environments are keyed by a + hash of the Python version, the locked requirements and the `psrunner` version. A file lock makes concurrent runs wait + for a single creation, and an environment is used only after it was completely installed. +4. **Wait for a slot.** At most `MaxConcurrentProcesses` Python processes run at the same time in one .NET process. +5. **Start the process** in a folder of its own, with: + - a clean environment: only the variables Python needs, never the variables of your application; + - the declared secrets; + - the inputs in a file. +6. **Apply the limits:** the timeout and the memory limit (see [Memory limits](hosting.md#memory-limits)). On timeout or + cancellation, the whole process tree is stopped. +7. **Capture the output.** The end of stdout and stderr is kept (up to `MaxLogBytes`), with secret values masked as `***`. +8. **Validate the outputs** against the contract and convert them to .NET values. Output files are moved to the run, + within `MaxFileBytes` and `MaxRunFilesBytes`. + +The .NET side and `psrunner` exchange files in protocol version 1, and `psrunner` stays compatible with it for as long +as that protocol is in use. + +## Guarantees + +- **Isolation:** each attempt runs in its own process, so a crash in Python never takes the .NET application down. +- **Limits:** every attempt has a timeout and a memory limit. +- **Secrets:** they reach only the process of the stage that declares them. They are never written to the inputs file, + logs, metrics, exceptions or the database. +- **Reproducibility:** published dependencies are locked with hashes and installed with `--require-hashes`, so a + published version always runs with the same packages. +- **Typed results:** the .NET side receives values that match the contract, or a clear error. + +## Security model + +PythonScriptRunner runs code written by **trusted authors**. A stage is arbitrary Python running with the permissions of +the worker's user; it is not a sandbox for untrusted code. Anyone with the `Author` role in the dashboard can publish code +that runs on the workers, so give that role only to people you would let deploy code. + +Recommended hardening: +- Run workers as an unprivileged user with access only to what the stages need. The images already run as a non-root + user with a read-only root file system. +- Keep secrets in your platform's secret store. +- Serve the dashboard over HTTPS. +- Do not expose the development setup of the AppHost or of the generated compose. + +To report a vulnerability, see [SECURITY.md](../../SECURITY.md). + +## Observability + +**Traces.** The `ActivitySource` named `PythonScriptRunner` creates these spans: +- `psr.run.enqueue`: a run was queued; +- `psr.message.consume`: a message was handled; +- `psr.environment.ensure`: an environment was prepared; +- `psr.stage.execute`: an attempt ran. + +The `traceparent` travels with the messages, so one trace goes from the HTTP request to the worker and every Python +process. + +![Trace of a run](../images/aspire-trace.png) + +**Metrics.** The `Meter` named `PythonScriptRunner` publishes these instruments: + +| Instrument | Meaning | +|------------|---------| +| `psr.step.duration` | Duration of each attempt. | +| `psr.step.overhead` | Time spent by the runner outside the Python process. | +| `psr.step.failures` | Failed attempts. | +| `psr.environment.created` | Python environments created. | +| `psr.queue.wait` | Time from queuing a step to its start on a worker. | +| `psr.worker.active_steps` | Steps running on this worker. | +| `psr.lease.recovered` | Attempts restarted after their worker was lost. | +| `psr.schedule.fired` | Runs created by schedules. | +| `psr.outbox.publish.duration` | Time to publish one outbox batch. | +| `psr.maintenance.partitions.created`, `psr.maintenance.partitions.dropped`, `psr.maintenance.rows.deleted` | Work of the maintenance job. | + +Add the source and the meter to your OpenTelemetry setup: + +```csharp +builder.Services.AddOpenTelemetry() + .WithTracing(t => t.AddSource("PythonScriptRunner")) + .WithMetrics(m => m.AddMeter("PythonScriptRunner")); +``` + +**Health checks:** +- `AddPythonScriptRunnerDatabase()` checks PostgreSQL; +- `AddPythonScriptRunnerBroker()` checks RabbitMQ; +- `AddPythonScriptRunnerChecks()` checks the database, the broker and uv together. + +**Logs.** The library logs through `ILogger`. SQL commands are logged at `Debug`, because the outbox relay polls the +database several times per second. diff --git a/guides/en/persistence.md b/guides/en/persistence.md new file mode 100644 index 0000000..734d127 --- /dev/null +++ b/guides/en/persistence.md @@ -0,0 +1,118 @@ +# Persistence (PostgreSQL) + +English | [Português](../pt-br/persistence.md) + +With `PythonScriptRunner.EntityFrameworkCore`, stages and pipelines live in PostgreSQL with immutable versions, and every +run is recorded. This level does not need RabbitMQ: synchronous runs are recorded too. + +## Setup + +```bash +dotnet add package PythonScriptRunner.EntityFrameworkCore +``` + +```csharp +builder.Services + .AddPythonScriptRunner() + .AddEntityFrameworkCore(o => + { + o.UseNpgsql(builder.Configuration.GetConnectionString("psr")!); // PostgreSQL 16 or newer + o.Retention.Artifacts = TimeSpan.FromDays(7); // file contents + o.Retention.Runs = TimeSpan.FromDays(90); // runs, steps, attempts and logs + }); +builder.Services.AddHealthChecks().AddPythonScriptRunnerDatabase(); + +var app = builder.Build(); +await app.Services.MigratePythonScriptRunnerAsync(); +``` + +The schema is never changed implicitly. You have two options: +- **At startup:** call `MigratePythonScriptRunnerAsync()`. Concurrent instances wait on a PostgreSQL advisory lock, so + the migrations run once. +- **Through your DBA:** generate the SQL with `GeneratePythonScriptRunnerMigrationScriptAsync(idempotent: true)` and hand + it over. + +Everything is created in the `psr` schema. + +## The catalog + +With persistence, the catalog comes from the database. Each stage and pipeline has versions that move from **draft** to +**published** to **archived**. A published version never changes; editing creates a new draft. + +`ICatalogAdmin` manages the catalog. The dashboard uses the same service. + +```csharp +var admin = app.Services.GetRequiredService(); +var actor = Actor.User("ana@example.com"); + +var draft = await admin.CreateStageAsync(new NewCatalogItem("average", "Average"), actor); +var saved = await admin.SaveStageDraftAsync("average", draft.Version, draft.Data with +{ + Script = script, + Inputs = [new FieldDefinition("values", FieldType.ListOf(FieldType.Number))], + Outputs = [new FieldDefinition("average", FieldType.Number)], +}, draft.RowVersion, actor); +await admin.PublishStageAsync("average", saved.Version, saved.RowVersion, actor); +``` + +| Operation | Methods | +|-----------|---------| +| Create and edit drafts | `CreateStageAsync`, `CreateStageDraftFromAsync`, `SaveStageDraftAsync`, `DiscardStageDraftAsync` (and the `Pipeline` equivalents) | +| Validate and test a draft | `ValidateStageDraftAsync`, `TestStageDraftAsync`, `EnqueueStageDraftTestAsync` (on a worker) | +| Publish | `PublishStageAsync`, `PublishPipelineAsync` | +| Archive and delete | `ArchiveStageVersionAsync`, `DeleteStageAsync` (and the `Pipeline` equivalents) | +| Import and export | `ImportAsync(zip)` creates drafts from a zip in the folder format; `ExportStageVersionAsync` and `ExportPipelineVersionAsync` | +| Schedules | `CreateScheduleAsync`, `UpdateScheduleAsync`, `SetScheduleEnabledAsync`, `DeleteScheduleAsync` (see [Queue, worker and scheduling](queue-and-worker.md)) | + +Details of these operations: +- **Optimistic concurrency:** saves use the row version, so two people editing the same draft cannot overwrite each other + silently. +- **Publishing a stage:** + 1. checks the script syntax with the declared Python; + 2. locks the requirements with hashes; + 3. freezes the version. +- **Publishing a pipeline:** validates the types and bindings against the published stages. +- **Bringing in a folder:** to move a folder catalog into the database, zip the folder (with `stages/` and `pipelines/` at + the root), import it, then publish the stages and the pipelines. + +## Runs + +Every run is stored, whether it was synchronous (`IPythonRunner`), queued, scheduled or started from the dashboard. The +record includes the inputs, the steps, every attempt with its duration and error, the tail of stdout and stderr (up to +`MaxLogBytes`) and the files. File contents are stored in PostgreSQL (`bytea`), streamed in and out. + +`IRunManager` reads and manages them: + +| Method | Description | +|--------|-------------| +| `GetAsync(runId)` | The full run with its steps and attempts. | +| `ListAsync(query)` | Runs filtered by status, pipeline, stage, trigger or period, with keyset pagination. | +| `GetAttemptLogAsync(runId, attemptId, stream)` | The captured stdout or stderr of an attempt. | +| `CancelAsync(runId, actor)` | Cancels a run. A queued run stops on the worker that is running it. | +| `ReprocessAsync(runId, actor, mode)` | Retries a failed run with the same version and inputs. The steps that succeeded keep their outputs and the step that failed gets a new attempt. `ExecutionMode.Async` (the default) queues it for the workers and needs RabbitMQ; `ExecutionMode.Sync` runs it now in the current process. | + +**Idempotency:** `RunOptions.IdempotencyKey` returns the same run when the same key is used again within +`IdempotencyWindow` (24 hours by default). It is useful when the caller may repeat a request. + +## Retention + +History is partitioned: runs by week, file contents by day, and the audit log by month. A maintenance job, on by default +and running every hour, creates partitions ahead of time and drops the expired ones, which is cheap even with large +volumes. + +| Option | Default | +|--------|---------| +| `Retention.Artifacts` (file contents) | 7 days | +| `Retention.Runs` (runs, steps, attempts, logs, file metadata) | 90 days | +| `Retention.Audit` | 365 days | +| `Maintenance.Enabled`, `Maintenance.Interval` | `true`, 1 hour | + +A pipeline can have a shorter retention than the global one, but not a longer one. + +## Audit + +Human actions are recorded in `psr.audit_events`, with who did what, on which item and when. That covers editing, +publishing, archiving, deleting, cancelling, retrying, scheduling and changing settings. The actor of each action is an +`Actor`: +- `Actor.User("subject")` for people; +- `Actor.System("origin")` for automatic triggers, which are recorded on the run itself. diff --git a/guides/en/pipelines.md b/guides/en/pipelines.md new file mode 100644 index 0000000..24b82ea --- /dev/null +++ b/guides/en/pipelines.md @@ -0,0 +1,108 @@ +# Pipelines + +English | [Português](../pt-br/pipelines.md) + +A pipeline runs stages one after the other. Each step can use the pipeline's inputs, the outputs of any earlier step, or +fixed values, and the pipeline declares its own outputs. Types and bindings are validated when the catalog loads (with +persistence, when a version is published), so a pipeline that cannot run is rejected before it runs. + +## The pipeline file + +With the folder catalog, pipelines are files named `pipelines/.pipeline.json`: + +```json +{ + "$schema": "https://raw.githubusercontent.com/ScriptRunnerKit/PythonScriptRunner/main/schemas/pipeline.v1.json", + "key": "progression-calc", + "version": 1, + "name": "Progression calculation", + "inputs": [ + { "name": "a1", "type": "number" }, + { "name": "ratio", "type": "number" }, + { "name": "n", "type": "integer" }, + { "name": "coefficients", "type": "list", "itemType": "number" } + ], + "steps": [ + { "key": "terms", "stage": "progression-terms@1" }, + { "key": "polynomial", "stage": "polynomial@1", + "bindings": { "coefficients": "$pipeline.coefficients", "x": "$steps.terms.terms" } }, + { "key": "statistics", "stage": "statistics@1", + "bindings": { "values": "$steps.polynomial.values", "precision": { "const": 4 } } } + ], + "outputs": [ + { "name": "sum", "source": "$steps.statistics.sum" }, + { "name": "terms", "source": "$steps.terms.terms", "description": "Terms of the progression" } + ] +} +``` + +| Property | Meaning | +|----------|---------| +| `key`, `version`, `name`, `description` | Same rules as a stage. | +| `inputs` | Fields of the pipeline, with the same types and properties as stage fields. | +| `steps` | The steps, in execution order. Each has a `key` (slug), a `stage` (`stage-key@version`), an optional `displayName` and optional `bindings`. | +| `outputs` | What the pipeline returns. Each output has a `name` and a `source` (`$steps..`); its type comes from that step output. | + +## Bindings + +Each input of a step gets its value from one of three sources: + +| Binding | Example | Value | +|---------|---------|-------| +| Pipeline input | `"$pipeline.coefficients"` | The pipeline input with that name. | +| Earlier step | `"$steps.terms.terms"` | The output `terms` of the step `terms`. Any earlier step can be used, not only the previous one. | +| Constant | `{ "const": 4 }` | A fixed value. | + +An input without an explicit binding uses the **implicit binding by name**: +- in the first step, the pipeline input with the same name; +- in the other steps, the output with the same name of the previous step. + +When the types are compatible but not the same (an `integer` feeding a `decimal`, for example), the value is converted. +An incompatible binding is a validation error. + +## Running a pipeline + +```csharp +await using var result = await runner.RunPipelineAsync("progression-calc", + new { a1 = 1, ratio = 2, n = 4, coefficients = new[] { 1.0, 0.0, 1.0 } }); + +result.EnsureSuccess(); +double sum = result.GetOutput("sum"); +double[] terms = result.GetStepOutput("terms", "terms"); // any step output, useful for debugging +``` + +`result.Steps` lists every step with its status, attempts, duration and error. + +If a step fails, the following steps end as `Skipped` and the run ends as `Failed`, with the error of the step that +failed. +- **Synchronous runs:** you start a new run. +- **With persistence:** `IRunManager.ReprocessAsync` retries only from the step that failed, reusing the outputs of the + steps that succeeded. See [Persistence](persistence.md). + +## Defining a pipeline in C# + +```csharp +builder.Services.AddPythonScriptRunner() + .AddPipeline("progression-calc", p => p + .Name("Progression calculation") + .Input("a1", FieldType.Number) + .Input("ratio", FieldType.Number) + .Input("n", FieldType.Integer) + .Input("coefficients", FieldType.ListOf(FieldType.Number)) + .Step("terms", "progression-terms@1") + .Step("polynomial", "polynomial@1", s => s + .Bind("coefficients", new PipelineInputSource("coefficients")) + .Bind("x", new StepOutputSource("terms", "terms"))) + .Step("statistics", "statistics@1", s => s + .Bind("values", new StepOutputSource("polynomial", "values")) + .Bind("precision", new ConstantSource(JsonValue.Create(4)))) + .Output("sum", StepOutputSource.Step("statistics", "sum"))); +``` + +## Settings of queued runs + +With the queue (see [Queue, worker and scheduling](queue-and-worker.md)), each pipeline also has: +- **Maximum concurrent runs:** the extra runs wait as `Pending`, in order. +- **Retention:** it can only be shorter than the global retention. + +Both are changed in the dashboard or with `ICatalogAdmin.UpdatePipelineSettingsAsync` and `UpdatePipelineRetentionAsync`. diff --git a/guides/en/python-sdk.md b/guides/en/python-sdk.md new file mode 100644 index 0000000..c2caa55 --- /dev/null +++ b/guides/en/python-sdk.md @@ -0,0 +1,135 @@ +# Python SDK (psrunner) + +English | [Português](../pt-br/python-sdk.md) + +`psrunner` is the Python side of PythonScriptRunner. It reads the inputs, converts them to Python types, calls your +function, checks the outputs against the contract and sends them back to .NET. The script only contains the logic. + +The runner installs `psrunner` in every environment it creates, so a stage never lists it in `requirements.txt`. Install +it yourself only to write and test scripts locally: + +```bash +pip install psrunner # Python 3.10 or newer +pip install "psrunner[pandas]" # table inputs arrive as pandas.DataFrame +``` + +## Writing a stage + +```python +from psrunner import stage, StageError + +@stage +def run(inputs, ctx): + values = inputs["values"] + if not values: + raise StageError("At least one value is required.", code="EMPTY_INPUT") + + ctx.log.info("averaging %d values", len(values)) + return {"average": sum(values) / len(values)} +``` + +The rules: +- **One entry point:** exactly one function per script is marked with `@stage`. It receives `inputs` (a dict with the + inputs, already converted) and `ctx` (the context below). +- **Return value:** it returns a dict with the outputs. Required outputs must be present; outputs not declared in the + contract are an error. +- **Types:** values are converted as described in [Field types](stages.md#field-types). For example, `decimal` arrives as + `Decimal`, and `datetime` must carry a UTC offset. + +## The context (`ctx`) + +| Member | Description | +|--------|-------------| +| `ctx.log` | A `logging.Logger`. Its lines appear in the run's logs (dashboard and `RunResult`). `print` also works and goes to stdout. | +| `ctx.secrets` | Read-only mapping with the secrets the stage declares: `ctx.secrets["SMTP_PASSWORD"]`. | +| `ctx.new_file(name, content_type=None)` | Reserves an output file and returns a `File`; write to `file.path`. | +| `ctx.run_id`, `ctx.step_key`, `ctx.attempt` | Identify the run, the pipeline step and the attempt number. | +| `ctx.stage_key`, `ctx.stage_version` | The stage being run. | +| `ctx.is_test` | `True` for test runs of a draft. | +| `ctx.work_dir`, `ctx.out_dir`, `ctx.tmp_dir` | The attempt's own folders. Use `tmp_dir` for scratch files. | + +## Files + +A `file` input arrives as a `psrunner.File`: + +| Member | Description | +|--------|-------------| +| `path` | Local path of the file. | +| `name`, `content_type`, `size`, `sha256` | Metadata. | +| `read_bytes()`, `read_text(encoding="utf-8")`, `open(mode="rb")` | Read the content. | + +To produce a file output, reserve it with `ctx.new_file` and return the `File`: + +```python +@stage +def run(inputs, ctx): + report = ctx.new_file("report.csv", "text/csv") + report.path.write_text("x,y\n1,2\n", encoding="utf-8") + return {"report": report} +``` + +File names must be plain names, without folders. The runner enforces `MaxFileBytes` per file and `MaxRunFilesBytes` per +run (see [Configuration](configuration.md)). + +## Errors + +- **Business errors:** raise `StageError(message, code="SOME_CODE", details={...})`. The run fails with kind + `ScriptError` and that `code`, so the .NET side and the dashboard can react to it. +- **Unexpected exceptions:** any other exception also becomes `ScriptError`. The error carries the traceback of your + script, which the dashboard shows. +- **Contract violations:** a missing output, a wrong type or a non-finite number becomes `OutputContractViolation`. + +## Testing with pytest + +`psrunner.testing.run_stage` runs the function with the same conversions and validations as the real runner, without +.NET: + +```python +from psrunner.testing import run_stage + +def test_average(): + result = run_stage( + "psr/stages/average/script.py", + {"values": [2, 4, 9]}, + stage_json="psr/stages/average/stage.json", + ) + assert result.ok + assert result.outputs == {"average": 5.0} +``` + +| Parameter | Description | +|-----------|-------------| +| `script` | Path of the script, or the function itself. | +| `inputs` | Dict with the inputs. | +| `stage_json` | Path of the contract. With it, inputs and outputs are validated; without it, inputs are passed as given. | +| `secrets` | Dict with secret values. | +| `files` | Maps file fields to local paths (a list for `list` of `file`). | +| `work_dir`, `run_id` | Optional folder and run id. | + +The result has these fields: +- `ok`: `True` when the stage succeeded; +- `outputs`: the Python values; +- `wire_outputs`: the JSON sent to .NET; +- `error`: with `kind`, `code`, `message` and `traceback`; +- `logs`: the log lines; +- `work_dir`: the folder the run used. + +## Command line + +```bash +python -m psrunner validate psr/stages/average +python -m psrunner run psr/stages/average --input sample.json --secret SMTP_PASSWORD=secret --keep +``` + +| Command | What it does | +|---------|--------------| +| `validate ` | Checks `stage.json` and that the script has exactly one `@stage` function. | +| `run ` | Runs the stage locally with the real protocol, in a separate process and a temporary folder. | + +In the `sample.json` of `run`, a file input is written as `{"$path": "relative/to/sample.json"}`. `--keep` prints the work +folder so you can inspect it. The same commands are available as `psrunner validate` and `psrunner run`. + +## Versions + +`psrunner` has its own version on PyPI. It follows protocol 1, the format exchanged with the .NET runner, for as long as +that protocol is in use. See the [compatibility table](../../README.md#compatibility). diff --git a/guides/en/queue-and-worker.md b/guides/en/queue-and-worker.md new file mode 100644 index 0000000..9a196bc --- /dev/null +++ b/guides/en/queue-and-worker.md @@ -0,0 +1,142 @@ +# Queue, worker and scheduling (RabbitMQ) + +English | [Português](../pt-br/queue-and-worker.md) + +With `PythonScriptRunner.RabbitMQ` and `PythonScriptRunner.Worker`, runs happen in the background. Your application +enqueues a run and continues; workers, on one machine or many, execute the steps. This level needs +[persistence](persistence.md): PostgreSQL holds the state of every run and the outbox of messages. + +## Setup + +```bash +dotnet add package PythonScriptRunner.EntityFrameworkCore +dotnet add package PythonScriptRunner.RabbitMQ +dotnet add package PythonScriptRunner.Worker +``` + +```csharp +builder.Services + .AddPythonScriptRunner() + .AddEntityFrameworkCore(o => o.UseNpgsql(builder.Configuration.GetConnectionString("psr")!)) + .AddRabbitMq(o => o.ConnectionString = builder.Configuration.GetConnectionString("rabbitmq")) // RabbitMQ 3.13 or newer + .AddWorker(o => o.MaxConcurrency = 4); +builder.Services.AddHealthChecks().AddPythonScriptRunnerChecks(); // database, broker and uv +``` + +`AddWorker` goes only in the processes that should execute steps. A web application can enqueue runs and show their +status with `AddRabbitMq` alone, while dedicated worker processes (or the [worker image](hosting.md)) do the work. + +## Enqueuing a run + +```csharp +var queue = app.Services.GetRequiredService(); + +RunHandle run = await queue.EnqueuePipelineAsync("progression-calc", + new { a1 = 1, ratio = 2, n = 4, coefficients = new[] { 1.0, 0.0, 1.0 } }, + new RunOptions { IdempotencyKey = "order-1234" }); +``` + +`EnqueuePipelineAsync` and `EnqueueStageAsync` validate the inputs, record the run and return a handle with its id. Follow +the run with `IRunManager.GetAsync(run.RunId)`, with [events](#events), or in the [dashboard](dashboard.md). + +## How a run moves + +1. The run and the message of its first step are written in one PostgreSQL transaction (an outbox). Enqueuing works + even while RabbitMQ is down; the message is sent when the broker comes back. +2. A worker takes the step with a **lease**, runs the Python process and keeps renewing the lease while it runs. +3. When the step ends, the worker writes the result and the message of the next step in the same transaction. +4. Events about the run are published as it progresses. + +## Reliability + +- **Retry:** a failed attempt is retried according to the stage's `retry` policy (see [Stages](stages.md#retry)). The run + waits for the delay without holding a worker. +- **Lost workers:** if a worker stops renewing its lease (it crashed, or its machine went away), the step is taken back. + - A stage marked `idempotent`, or one that still has retries left, is resumed by another worker. + - Otherwise the step fails with `WorkerLost`, for a manual retry. +- **Duplicates:** a message delivered twice never runs a step twice. +- **Poison messages:** after `DeliveryLimit` failed deliveries (5), a command goes to the `psr.dead-letter` queue. +- **Graceful shutdown:** a stopping worker takes no new steps and waits up to `ShutdownGracePeriod` (5 minutes) for the + running ones. + +## Concurrency and cancellation + +- **Per worker:** `MaxConcurrency` limits how many steps a worker runs at once. The default is the processor count. +- **Per pipeline:** a maximum number of concurrent runs can be set. Extra runs wait as `Pending`, in order. +- **Cancellation:** `IRunManager.CancelAsync` stops the process tree on the worker that is running the step, wherever it + is. + +## Scheduling + +Schedules run a pipeline from a cron expression, in an IANA time zone: + +```csharp +var admin = app.Services.GetRequiredService(); +await admin.CreateScheduleAsync("progression-calc", new ScheduleData +{ + Name = "Every business day at 2 AM", + Cron = "0 2 * * 1-5", // 5 fields, or 6 with seconds + TimeZone = "America/Sao_Paulo", // the default; daylight saving time is handled + Inputs = new JsonObject { ["a1"] = 1, ["ratio"] = 2, ["n"] = 4, ["coefficients"] = new JsonArray(1, 0, 1) }, +}, Actor.User("ana@example.com")); +``` + +How schedules behave: +- **Version:** without `PinnedVersion`, each firing runs the latest published version. +- **Single firing:** several workers never fire the same occurrence. +- **Missed occurrences:** if they were missed while every worker was down, they become a single catch-up run. +- **File inputs:** a pipeline with a required `file` input cannot be scheduled. +- **Scheduler options:** the scheduler runs inside the workers, under `Scheduler` in the worker options (`Enabled`, + `PollInterval`, `BatchSize`). + +## Events + +Every run publishes events to the `psr.events` topic exchange: + +| Event | When | +|-------|------| +| `run.queued`, `run.rejected`, `run.started` | The run was accepted, refused (for example, invalid inputs) or started. | +| `step.started`, `step.succeeded`, `step.failed`, `step.retry-scheduled` | Progress of each step. | +| `run.succeeded`, `run.failed`, `run.cancelled` | The run ended. `run.succeeded` carries the outputs. | + +The routing key is `.` (or `.stage:` for single-stage runs), so you can subscribe to +exactly what you need. Inside .NET, handle them with `IRunEventHandler`: + +```csharp +builder.Services.AddPythonScriptRunner() + .AddRabbitMq(o => o.ConnectionString = builder.Configuration.GetConnectionString("rabbitmq")) + .AddRunEventHandler(s => s + .Subscription("erp") // one durable queue shared by all instances + .On("run.succeeded.progression-calc", "run.failed.*")); + +public sealed class NotifyErp : IRunEventHandler +{ + public Task HandleAsync(RunEvent runEvent, CancellationToken cancellationToken) + { + // runEvent.Type, runEvent.RunId, runEvent.Outputs, runEvent.Error, ... + return Task.CompletedTask; + } +} +``` + +Throwing from `HandleAsync` makes the broker deliver the event again, up to the delivery limit. Use `PerInstance(prefix)` +instead of `Subscription(name)` when every instance must receive every event. + +## Systems written in other languages + +Any system that talks to RabbitMQ can start pipelines and follow them, without .NET: + +- **Start a run:** + - publish a `StartPipelineRun` JSON message to the `psr.public` exchange, with routing key `pipeline.start`; + - fields: `pipelineKey` (required), `version`, `inputs`, `idempotencyKey`, `correlationId`, `requestedBy`; + - file inputs are not accepted through this command. +- **Follow it:** bind your own queue to the `psr.events` exchange with the routing keys you want, for example + `run.*.progression-calc`. The `correlationId` you sent comes back in every event. + +The message contracts are JSON Schemas in +[`schemas/messages`](../../schemas/messages): `start-pipeline-run.v1.json` and `run-event.v1.json`. + +## Observability + +The `traceparent` follows the run from the caller to the worker, so one distributed trace shows the request, the queue +and every Python process. See [How it works](how-it-works.md#observability). diff --git a/guides/en/stages.md b/guides/en/stages.md new file mode 100644 index 0000000..d396074 --- /dev/null +++ b/guides/en/stages.md @@ -0,0 +1,198 @@ +# Stages + +English | [Português](../pt-br/stages.md) + +A stage is one Python script and its contract. This page covers the contract, every option of `stage.json`, defining +stages in C#, and reading results in .NET. For the script side, see the [Python SDK](python-sdk.md). + +## Folder layout + +With the folder catalog (`AddFileCatalog("psr")`), each stage is a folder under `stages/`: + +```text +psr/ + stages/ + average/ + stage.json the contract and settings + script.py the code (one function marked with @stage) + requirements.txt optional: Python packages, one per line + pipelines/ + report.pipeline.json +``` + +When persistence is on, the catalog lives in PostgreSQL instead, and the same folder can be imported as a zip (see +[Persistence](persistence.md)). + +## stage.json + +```json +{ + "$schema": "https://raw.githubusercontent.com/ScriptRunnerKit/PythonScriptRunner/main/schemas/stage.v1.json", + "key": "polynomial", + "version": 1, + "name": "Evaluate a polynomial", + "description": "a + b·x + c·x² + … for each x", + "python": "3.12", + "timeoutSeconds": 120, + "memoryLimitMb": 2048, + "retry": { "maxAttempts": 2, "initialDelaySeconds": 10, "backoffMultiplier": 2 }, + "idempotent": true, + "secrets": ["SMTP_PASSWORD"], + "inputs": [ + { "name": "coefficients", "type": "list", "itemType": "number" }, + { "name": "x", "type": "list", "itemType": "number" } + ], + "outputs": [ + { "name": "values", "type": "list", "itemType": "number" } + ] +} +``` + +The `$schema` line gives you completion and validation in editors such as VS Code. + +| Property | Required | Meaning | +|----------|----------|---------| +| `key` | yes | Unique slug: lowercase letters and digits, words separated by single hyphens (`monthly-report`). Used to run the stage. | +| `version` | yes | Integer version. Pipelines reference `key@version`. | +| `name` | yes | Display name. | +| `description` | no | Free text shown in the dashboard. | +| `python` | no | Python version (3.10 or newer). Default: `DefaultPythonVersion`, which is 3.12. | +| `script` | no | Script file, relative to the folder. Default: `script.py`. | +| `requirements` | no | Requirements file, relative to the folder. Default: `requirements.txt`, if it exists. | +| `timeoutSeconds` | no | Time limit of one attempt. Default: `DefaultTimeout`, which is 5 minutes. | +| `memoryLimitMb` | no | Memory limit of one attempt. Default: `DefaultMemoryLimitMb`, which is 1024. | +| `retry` | no | Automatic retry after a failed attempt. See [Retry](#retry). | +| `idempotent` | no | `true` when running the stage twice has the same effect as once. Lets a worker resume it after another worker was lost. Default: `false`. | +| `secrets` | no | Names of the secrets the script receives (uppercase, digits and `_`). See [Secrets](#secrets). | +| `inputs`, `outputs` | yes | The fields of the contract. Either list can be empty. | + +## Field types + +Each input and output has a `name` and a `type`. The library converts values on both sides: + +| Type | In .NET (inputs and `GetOutput`) | In Python (`inputs[...]` and return value) | +|------|-------------------------------------|-------------------------------------------| +| `number` | `double` | `float` (`NaN` and infinity are rejected) | +| `integer` | `int`, `long` | `int` | +| `decimal` | `decimal` | `decimal.Decimal` (exact, sent as text) | +| `string` | `string` | `str` | +| `bool` | `bool` | `bool` | +| `date` | `DateOnly` | `datetime.date` | +| `datetime` | `DateTimeOffset` | `datetime.datetime` with a UTC offset (values without one are rejected) | +| `enum` | `string` from `enumValues` | `str` from `enumValues` | +| `json` | any JSON value (`JsonNode`, or your own type) | `dict`, `list`, `str`, number, `bool` or `None` | +| `table` | a list of records (your own record type, or `JsonArray`) | `list[dict]`, or `pandas.DataFrame` with the `pandas` extra | +| `file` | `PsrFile` | `psrunner.File` (input); `ctx.new_file(...)` (output) | +| `list` | arrays and lists of the item type | `list` of the item type | + +Field properties: + +| Property | Meaning | +|----------|---------| +| `name` | Field name: lowercase letters, digits and `_`, starting with a letter or `_`. | +| `type` | One of the types above. | +| `itemType` | Item type of a `list` (any type except `list`). | +| `required` | Default `true`. An optional input may be omitted; an optional output may be missing. | +| `default` | Inputs only: the value used when the input is not given. | +| `description` | Shown in the dashboard and in the generated run form. | +| `enumValues` | Allowed values of an `enum` (or of a `list` of `enum`). | +| `contentTypes` | Allowed MIME types of a `file`, for example `["application/pdf"]`. | +| `columns` | Optional schema of a `table`: `[{ "name": "x", "type": "number" }]`. Each column has a scalar type. | + +## Python packages + +Put a `requirements.txt` next to the script, with one package per line (`numpy`, `pandas>=2.2`). The runner creates one +environment per Python version and set of requirements with uv, caches it, and reuses it for every stage with the same +set. With persistence, publishing a stage locks the requirements with hashes (`uv pip compile --generate-hashes`), so a +published version always runs with the same package versions. + +The `psrunner` SDK is installed in every environment automatically. Do not list it in `requirements.txt`. + +## Defining a stage in C# + +The same contract can be declared in code, without files: + +```csharp +builder.Services.AddPythonScriptRunner() + .AddStage("average", s => s + .Script(""" + from psrunner import stage + + @stage + def run(inputs, ctx): + v = inputs["values"] + return {"average": sum(v) / len(v)} + """) + .Python("3.12") + .Requirements("numpy") + .Input("values", FieldType.ListOf(FieldType.Number)) + .Output("average", FieldType.Number) + .Timeout(TimeSpan.FromMinutes(1)) + .MemoryLimitMb(512)); +``` + +`FieldType` has `Number`, `Integer`, `Decimal`, `String`, `Bool`, `Date`, `DateTime`, `Json`, `Table`, `File`, +`Enum("a", "b")` and `ListOf(itemType)`. An invalid definition fails at startup with `PsrConfigurationException`. + +## Running a stage and reading the result + +```csharp +await using var result = await runner.RunStageAsync("average", new { values = new[] { 2.0, 4.0, 9.0 } }); + +if (result.Status == RunStatus.Succeeded) +{ + double average = result.GetOutput("average"); +} +else +{ + Console.WriteLine($"{result.Error!.Kind}: {result.Error.Message}"); +} +``` + +- **Inputs:** an anonymous object, a dictionary or any object that serializes to the expected JSON. Files are passed as + `PsrFile`: `PsrFile.FromPath(...)`, `FromStream`, `FromBytes` or `FromText`. +- **Before the run:** inputs that do not match the contract throw `InputValidationException`, and no run is created. +- **After the run:** failures are reported in `result.Status` and `result.Error`. `result.EnsureSuccess()` throws if the + run did not succeed. +- **Reading outputs:** + - `GetOutput(name)` reads a value; + - `GetFile(name)` and `GetFiles(name)` return `PsrFile` objects (`OpenReadAsync`, `SaveAsAsync`). +- **Disposing:** `RunResult` is `IAsyncDisposable`. Disposing it deletes the local output files, so save the ones you + need first (`SaveAsAsync`). +- **Options:** `RunOptions` picks a specific `Version`, sets an `IdempotencyKey` (with persistence) and records who + started the run. + +Error kinds you can get in `result.Error.Kind`: + +| Kind | Meaning | +|------|---------| +| `InputValidation` | An input did not match the contract. | +| `OutputContractViolation` | The script returned something outside the contract (missing output, wrong type, `NaN`). | +| `ScriptError` | The script raised an exception, or a `StageError` with a `code`. | +| `Timeout` | The attempt exceeded its timeout. | +| `MemoryLimitExceeded` | The attempt exceeded its memory limit. | +| `Cancelled` | The run was cancelled. | +| `WorkerLost` | The worker running the step stopped answering (queued runs). | +| `EnvironmentSetupFailed` | uv could not create the environment (package not found, no network, unknown Python version). | +| `ArtifactLimitExceeded` | Files exceeded `MaxFileBytes` or `MaxRunFilesBytes`. | +| `SecretNotFound` | A declared secret is not configured. Python does not even start. | +| `ProcessCrashed` | The Python process could not start or ended without a result. | +| `Expired` | The run reached the retention limit before finishing. | + +## Retry + +`retry` makes the runner try again after a failed attempt: + +- **`maxAttempts`:** how many extra attempts are allowed (0 to 20). The default, 0, means no automatic retry. +- **Delay before each retry:** `initialDelaySeconds × backoffMultiplier^(n-1)`, capped at 10 minutes. The defaults are 10 + seconds and 2. + +A cancelled run is never retried. Queued runs wait for the delay without holding a worker. + +## Secrets + +A stage declares the names of the secrets it needs. The values come from configuration, in the section +`PythonScriptRunner:Secrets` (for example the environment variable `PythonScriptRunner__Secrets__SMTP_PASSWORD`), or from +your own `ISecretProvider`. Values are never written to inputs, logs, metrics or the database. The process of that stage +is the only one that receives them, and they are masked (`***`) in the captured output. In Python, read them with +`ctx.secrets["SMTP_PASSWORD"]`. diff --git a/guides/images/aspire-trace.png b/guides/images/aspire-trace.png new file mode 100644 index 0000000..6e639c7 Binary files /dev/null and b/guides/images/aspire-trace.png differ diff --git a/guides/images/dashboard-contract.png b/guides/images/dashboard-contract.png new file mode 100644 index 0000000..3430f16 Binary files /dev/null and b/guides/images/dashboard-contract.png differ diff --git a/guides/images/dashboard-pipeline-editor.png b/guides/images/dashboard-pipeline-editor.png new file mode 100644 index 0000000..960f0e1 Binary files /dev/null and b/guides/images/dashboard-pipeline-editor.png differ diff --git a/guides/images/dashboard-run-form.png b/guides/images/dashboard-run-form.png new file mode 100644 index 0000000..d344a0f Binary files /dev/null and b/guides/images/dashboard-run-form.png differ diff --git a/guides/images/dashboard-run-live.png b/guides/images/dashboard-run-live.png new file mode 100644 index 0000000..c450437 Binary files /dev/null and b/guides/images/dashboard-run-live.png differ diff --git a/guides/images/dashboard-run-outputs.png b/guides/images/dashboard-run-outputs.png new file mode 100644 index 0000000..8d38857 Binary files /dev/null and b/guides/images/dashboard-run-outputs.png differ diff --git a/guides/images/dashboard-stage-editor.png b/guides/images/dashboard-stage-editor.png new file mode 100644 index 0000000..59a6e6b Binary files /dev/null and b/guides/images/dashboard-stage-editor.png differ diff --git a/guides/pt-br/README.md b/guides/pt-br/README.md new file mode 100644 index 0000000..c4057b7 --- /dev/null +++ b/guides/pt-br/README.md @@ -0,0 +1,22 @@ +# Guias do PythonScriptRunner + +[English](../en/README.md) | Português + +O PythonScriptRunner executa scripts Python a partir do .NET com um contrato tipado. Estes guias vão da configuração +mínima (só a biblioteca, sem banco de dados e sem broker de mensagens) até a completa, com fila, agendamento e painel +embutido. + +| Guia | O que cobre | +|------|-------------| +| [Primeiros passos](getting-started.md) | As formas de uso, o que instalar e a primeira Etapa em poucos minutos | +| [Etapas](stages.md) | O contrato (entradas, saídas e os 12 tipos), o `stage.json`, Etapas em C#, limites, retry e segredos | +| [Trilhas](pipelines.md) | Encadear Etapas, ligações, saídas da trilha e tratamento de falhas | +| [SDK Python](python-sdk.md) | Escrever scripts com o `psrunner`, arquivos, erros e testes com pytest e com a linha de comando | +| [Persistência (PostgreSQL)](persistence.md) | Catálogo versionado, histórico de execuções, retenção e auditoria | +| [Fila, worker e agendamento (RabbitMQ)](queue-and-worker.md) | Execução em segundo plano, retry, cron, eventos e integração com outros sistemas | +| [Painel](dashboard.md) | A interface web embutida, os modos de acesso e os papéis | +| [Hospedagem](hosting.md) | Imagens de contêiner, volumes, limites de memória, Kubernetes, .NET Aspire e Docker Compose | +| [Referência de configuração](configuration.md) | Todas as opções, com o valor padrão | +| [Como funciona](how-it-works.md) | Isolamento de processos, ambientes Python, limites, modelo de segurança e observabilidade | + +O [README](../../README.md) tem uma visão geral de tudo numa página só. diff --git a/guides/pt-br/configuration.md b/guides/pt-br/configuration.md new file mode 100644 index 0000000..54ab9d1 --- /dev/null +++ b/guides/pt-br/configuration.md @@ -0,0 +1,109 @@ +# Referência de configuração + +[English](../en/configuration.md) | Português + +As opções são definidas em código, no callback de cada método `Add...`, ou carregadas da configuração. Os hosts prontos +carregam estas seções, e você pode fazer o mesmo na sua aplicação: + +```csharp +var configuration = builder.Configuration; +builder.Services + .AddPythonScriptRunner(o => configuration.GetSection("PythonScriptRunner").Bind(o)) + .AddEntityFrameworkCore(o => + { + o.UseNpgsql(configuration.GetConnectionString("psr")!); + configuration.GetSection("PythonScriptRunner:EntityFrameworkCore").Bind(o); + }) + .AddRabbitMq(o => + { + o.ConnectionString = configuration.GetConnectionString("rabbitmq"); + configuration.GetSection("PythonScriptRunner:RabbitMQ").Bind(o); + }) + .AddWorker(o => configuration.GetSection("PythonScriptRunner:Worker").Bind(o)); +``` + +Em variáveis de ambiente, as seções são separadas por `__`: `PythonScriptRunner__Worker__MaxConcurrency=8`. Tempos usam o +formato `hh:mm:ss` (`00:10:00`). + +## Runner (`PythonScriptRunner`) + +| Opção | Padrão | Descrição | +|-------|--------|-----------| +| `WorkRoot` | `%TEMP%/psr` | Pasta das pastas de trabalho e de resultado das tentativas. | +| `EnvironmentCacheRoot` | `%LOCALAPPDATA%/psr` | Pasta onde ficam em cache os ambientes Python e os arquivos de lock. | +| `UvPath` | `uv` | Caminho ou nome do executável `uv` (procurado no PATH). | +| `DefaultPythonVersion` | `3.12` | Versão do Python das Etapas que não declaram uma. | +| `DefaultTimeout` | 5 minutos | Tempo limite das Etapas que não declaram um. | +| `DefaultMemoryLimitMb` | `1024` | Limite de memória das Etapas que não declaram um. | +| `MaxFileBytes` | 50 MB | Tamanho máximo de um arquivo. | +| `MaxRunFilesBytes` | 200 MB | Tamanho máximo de todos os arquivos de uma execução. | +| `MaxLogBytes` | 1 MB | Bytes de stdout e stderr guardados por tentativa (o final). | +| `MaxConcurrentProcesses` | número de processadores | Processos Python rodando ao mesmo tempo neste processo. | +| `ThreadsPerProcess` | não definido | Threads das bibliotecas numéricas em cada processo. Sem valor: 1 no Linux quando o limite de memória cai no `RLIMIT_AS`, sem limite nos outros casos. Veja [Limites de memória](hosting.md#limites-de-memória). | +| `PackageIndexUrl` | PyPI | Índice de pacotes (pip `--index-url`). | +| `ExtraIndexUrls` | nenhum | Índices de pacotes extras (pip `--extra-index-url`). | +| `PsrunnerRequirement` | `psrunner==0.1.0` | O SDK instalado em cada ambiente: um requisito do pip ou o caminho de um wheel local. | +| `OrphanRetention` | 24 horas | Idade a partir da qual pastas de trabalho que sobraram são apagadas. | + +Os segredos são lidos de `PythonScriptRunner:Secrets:`. + +## Persistência (`PythonScriptRunner:EntityFrameworkCore`) + +| Opção | Padrão | Descrição | +|-------|--------|-----------| +| `Retention:Artifacts` | 7 dias | Por quanto tempo o conteúdo dos arquivos é guardado. | +| `Retention:Runs` | 90 dias | Por quanto tempo execuções, passos, tentativas, logs e metadados de arquivos são guardados. | +| `Retention:Audit` | 365 dias | Por quanto tempo os eventos de auditoria são guardados. | +| `Maintenance:Enabled` | `true` | Executa o job de manutenção (partições, retenção e limpezas) neste processo. | +| `Maintenance:Interval` | 1 hora | Com que frequência o job de manutenção roda. | +| `Maintenance:BatchSize` | `1000` | Linhas tratadas por lote pelo job de manutenção. | +| `IdempotencyWindow` | 24 horas | Por quanto tempo uma chave de idempotência devolve a mesma execução. | +| `LatestVersionCacheDuration` | 30 segundos | Por quanto tempo a consulta da "última versão publicada" fica em cache neste processo. | +| `UseDatabaseClock` | `true` | Usa o relógio do PostgreSQL para leases, esperas de retry, agendamentos e outbox, para que hosts com relógios diferentes concordem. | + +A conexão é definida em código, com `o.UseNpgsql(connectionString)`. + +## RabbitMQ (`PythonScriptRunner:RabbitMQ`) + +| Opção | Padrão | Descrição | +|-------|--------|-----------| +| `ConnectionString` | nenhum | URI AMQP: `amqp://usuario:senha@host:5672/vhost`, ou `amqps://` para TLS. | +| `DeclareTopology` | `true` | Declara exchanges e filas na inicialização. Com `false`, a inicialização só confere se elas existem (para brokers administrados por um operador). | +| `RunOutboxRelay` | `true` | Publica o outbox a partir deste processo. Várias instâncias podem fazer isso. | +| `EnablePublicCommands` | `true` | Consome o comando público `StartPipelineRun`. | +| `InstanceId` | nome da máquina e id do processo | Dá nome às filas por instância deste processo. | +| `RelayInterval` | 250 ms | Com que frequência o relay procura mensagens quando o outbox está ocioso. | +| `RelayBatchSize` | `100` | Mensagens publicadas por lote do relay. | +| `DeliveryLimit` | `5` | Tentativas de entrega de um comando antes de ele ir para `psr.dead-letter`. | +| `StartupConnectTimeout` | 10 segundos | Quanto a inicialização espera pelo broker antes de continuar em segundo plano. | + +## Worker (`PythonScriptRunner:Worker`) + +| Opção | Padrão | Descrição | +|-------|--------|-----------| +| `WorkerId` | o id de instância do RabbitMQ | Identificador gravado nas tentativas que este worker executa. | +| `MaxConcurrency` | número de processadores | Passos executados ao mesmo tempo por este worker. | +| `LeaseDuration` | 60 segundos | Lease de uma tentativa em andamento; outro worker a recupera quando não é renovado. | +| `LeaseRenewInterval` | 15 segundos | Com que frequência o lease de uma tentativa em andamento é renovado. | +| `ReaperInterval` | 15 segundos | Com que frequência os leases vencidos são procurados. | +| `MaxLeaseRecoveries` | `3` | Recuperações de um passo após workers perdidos antes de ele falhar com `WorkerLost`. | +| `ShutdownGracePeriod` | 5 minutos | Quanto tempo os passos em andamento têm para terminar quando o host para. A plataforma precisa permitir pelo menos esse tempo. | +| `WorkFolderCleanupInterval` | 1 hora | Com que frequência as pastas de trabalho que sobraram são removidas. | +| `Scheduler:Enabled` | `true` | Dispara agendamentos a partir deste worker. Vários workers nunca disparam a mesma ocorrência. | +| `Scheduler:PollInterval` | 5 segundos | Com que frequência os agendamentos vencidos são procurados (a precisão do disparo). | +| `Scheduler:BatchSize` | `50` | Agendamentos disparados por transação. | + +## Login fixo do painel (`PythonScriptRunner:Dashboard:Login`) + +| Opção | Padrão | Descrição | +|-------|--------|-----------| +| `Users` | nenhum | Lista de `{ Name, Role, PasswordHash }`. | +| `SlidingExpiration` | 8 horas | Duração da sessão, renovada enquanto o usuário está ativo. | +| `AbsoluteExpiration` | 24 horas | Duração máxima da sessão. | +| `MaxFailedAttempts` | `5` | Falhas de login seguidas antes de o usuário ser bloqueado. | +| `LockoutDuration` | 15 minutos | Quanto tempo um usuário bloqueado espera. | +| `MaxAttemptsPerMinutePerIp` | `10` | Tentativas de login permitidas por IP por minuto. | + +As mesmas opções existem em código no `UseFixedLogin(l => ...)`, que também tem o `PersistKeysInDatabase` (padrão +`true`): ele guarda as chaves do cookie no PostgreSQL, para que a sessão valha em todas as instâncias e sobreviva a +reinícios. diff --git a/guides/pt-br/dashboard.md b/guides/pt-br/dashboard.md new file mode 100644 index 0000000..3b7796a --- /dev/null +++ b/guides/pt-br/dashboard.md @@ -0,0 +1,149 @@ +# Painel + +[English](../en/dashboard.md) | Português + +O `PythonScriptRunner.Dashboard` embute uma interface web na sua aplicação ASP.NET Core, no estilo do Hangfire. Ele tem +uma API interna, atualizações em tempo real por SignalR e uma interface Vue, tudo dentro do pacote: não há nada para +hospedar à parte. O painel precisa da [persistência](persistence.md) e da [fila](queue-and-worker.md), porque guarda +versões e histórico e executa testes e execuções nos workers. + +## Configuração + +```bash +dotnet add package PythonScriptRunner.Dashboard +``` + +```csharp +builder.Services + .AddPythonScriptRunner() + .AddEntityFrameworkCore(o => o.UseNpgsql(builder.Configuration.GetConnectionString("psr")!)) + .AddRabbitMq(o => o.ConnectionString = builder.Configuration.GetConnectionString("rabbitmq")) + .AddDashboard(o => o.UseFixedLogin(login => login.BindConfiguration())); // seção PythonScriptRunner:Dashboard:Login + +var app = builder.Build(); +app.MapPythonScriptRunnerDashboard("/psr"); +``` + +O `MapPythonScriptRunnerDashboard("/psr")` serve: +- a interface em `/psr/`; +- a API em `/psr/api`; +- as atualizações em tempo real em `/psr/hubs/runs`. + +Sem essa chamada não existe painel. Para rodá-lo fora da sua aplicação, use a imagem pronta do `DashboardHost` (veja +[Hospedagem](hosting.md)). + +## O que dá para fazer + +A interface está em português do Brasil, com temas claro e escuro. Uma versão em inglês está planejada. + +**Editar uma Etapa.** Escreva o script num editor Monaco e monte o contrato com um construtor de campos: nome, tipo, +obrigatório, valor padrão e descrição. A aba Dependências define a versão do Python e os requisitos. Dá para testar um +rascunho num worker de verdade antes de publicar. + +![Editor de Etapa](../images/dashboard-stage-editor.png) + +![Adicionando uma saída ao contrato](../images/dashboard-contract.png) + +**Montar uma trilha.** Arraste Etapas publicadas para um canvas e ligue saídas a entradas. Os pontos de ligação têm tipo, +e os problemas aparecem enquanto você edita. Uma visão em lista oferece a mesma edição sem o canvas. + +![Editor visual de trilhas](../images/dashboard-pipeline-editor.png) + +**Executar e acompanhar.** Dispare uma execução por um formulário gerado a partir do contrato. O fluxo mostra cada passo +mudando de status em tempo real. As abas mostram as entradas, as saídas (inclusive tabelas), os logs de cada tentativa, +os arquivos e uma linha do tempo. Execuções com falha podem ser reprocessadas a partir do passo que falhou, e as que estão +rodando podem ser canceladas. + +![Formulário de disparo](../images/dashboard-run-form.png) + +![Uma execução em andamento](../images/dashboard-run-live.png) + +![Saídas de uma execução concluída](../images/dashboard-run-outputs.png) + +A interface também tem: +- uma visão geral das últimas 24 horas; +- listas de execuções com filtros; +- agendamentos por cron, com prévia das próximas ocorrências; +- a auditoria; +- uma página de administração para o papel `Admin`. + +## Modos de acesso + +Escolha **um** modo principal: + +| Modo | Configuração | Quando usar | +|------|--------------|-------------| +| Login fixo | `o.UseFixedLogin(l => l.BindConfiguration())`, ou `l.Users` em código | Poucos usuários conhecidos, sem provedor de identidade. | +| Autenticação do host | `o.UseHostAuthentication(r => { ... })` | A sua aplicação já autentica usuários (cookies, OpenID Connect, Microsoft Entra ID). | +| Filtro próprio | `o.UseAuthorizationFilter()` | Uma regra sua: um `IDashboardAuthorizationFilter` devolve `DashboardUser(nome, papel)` ou `null`. | +| Requisições locais | `o.AllowLocalRequests = true` | Só em Development (fora dele a inicialização falha). Soma-se a um modo principal. | + +## Papéis + +Os quatro papéis são cumulativos: + +| Papel | Pode | +|-------|------| +| `Reader` | Ver tudo: catálogo, execuções, logs, arquivos e agendamentos. | +| `Executor` | Também disparar, cancelar e reprocessar execuções. | +| `Author` | Também editar, testar, publicar, importar e agendar. | +| `Admin` | Também excluir e administrar retenção, segredos e auditoria. | + +## Login fixo + +Os usuários ficam na configuração. A senha nunca é guardada, só o hash dela (PBKDF2, no formato do ASP.NET Identity): + +```json +"PythonScriptRunner": { + "Dashboard": { + "Login": { + "Users": [ + { "Name": "ana", "Role": "Admin", "PasswordHash": "AQAAAAIAAYagAAAAE..." }, + { "Name": "bruno", "Role": "Reader", "PasswordHash": "AQAAAAIAAYagAAAAE..." } + ] + } + } +} +``` + +Gere o hash com a ferramenta `psr`: + +```bash +dotnet tool install -g PythonScriptRunner.Tool +psr hash-password # pede a senha duas vezes, sem eco +echo "senha" | psr hash-password --stdin # em scripts +``` + +Como o login fixo se comporta: +- **Sessão:** um cookie `psr.session` HttpOnly (`SameSite=Lax`), renovado por 8 horas de atividade, com no máximo 24 + horas. +- **Bloqueio:** 5 falhas seguidas bloqueiam o usuário por 15 minutos, e um IP pode tentar 10 vezes por minuto. +- **Mudanças na configuração:** valem sem reiniciar. Um usuário removido ou uma senha trocada encerra a sessão, e um + papel novo vale na hora. +- **Chaves do cookie:** ficam no banco (`psr.dashboard_keys`), então a sessão vale em todas as instâncias e sobrevive a + reinícios. + +## Autenticação do host + +Associe cada papel a uma política de autorização da sua aplicação e informe ao painel onde fica a sua página de login: + +```csharp +.AddDashboard(o => o.UseHostAuthentication(r => +{ + r.Reader = p => p.RequireRole("psr-leitor"); + r.Executor = p => p.RequireRole("psr-executor"); + r.Author = p => p.RequireRole("psr-autor"); + r.Admin = p => p.RequireRole("psr-admin"); + r.LoginUrl = "/conta/entrar?returnUrl={returnUrl}"; +})); +``` + +## A API + +A API é interna: ela existe para a interface e pode mudar entre versões. Para integrar outros sistemas, use a biblioteca +.NET ou o [comando e os eventos do RabbitMQ](queue-and-worker.md#sistemas-escritos-em-outras-linguagens). +- **Alterações:** requisições que alteram algo exigem o cabeçalho `X-PSR-Request: 1`. +- **Edições:** usam `ETag` e `If-Match`. +- **Erros:** seguem o `ProblemDetails`, com um `code`. + +Em Development, o documento OpenAPI fica em `/psr/api/openapi/v1.json`. diff --git a/guides/pt-br/getting-started.md b/guides/pt-br/getting-started.md new file mode 100644 index 0000000..305fbe1 --- /dev/null +++ b/guides/pt-br/getting-started.md @@ -0,0 +1,125 @@ +# Primeiros passos + +[English](../en/getting-started.md) | Português + +O PythonScriptRunner liga código Python a aplicações .NET. O cálculo fica num script Python comum, escrito por quem +domina o assunto, e o .NET recebe um resultado já validado contra um contrato. + +## Conceitos + +- **Etapa**: um script Python com um **contrato**, a lista de entradas e saídas e seus tipos. O script implementa uma + única função, `run(inputs, ctx)`; a biblioteca valida o que entra e o que sai. +- **Trilha**: uma sequência ordenada de Etapas. A saída de um passo pode alimentar a entrada de um passo seguinte, e os + tipos de todas as ligações são conferidos antes de qualquer execução. +- **Execução**: cada vez que uma Etapa ou trilha roda. Cada tentativa roda num processo Python próprio, com timeout, + limite de memória e ambiente Python próprio. + +## Formas de uso + +Comece pela configuração mínima que resolve o seu caso. O código de um nível continua válido no seguinte. + +| Nível | O que você precisa | O que ganha | +|-------|--------------------|-------------| +| **1. Somente a biblioteca** | .NET 10 e uv | Executar Etapas e trilhas a partir do seu código. Sem banco de dados, sem broker de mensagens e sem painel. | +| **2. Com PostgreSQL** | PostgreSQL 16 ou mais novo | Catálogo versionado (rascunho, publicada e arquivada), histórico de todas as execuções, arquivos e auditoria. Veja [Persistência](persistence.md). | +| **3. Com RabbitMQ** | PostgreSQL e RabbitMQ 3.13 ou mais novo | Execução em segundo plano por workers, retry automático, agendamento por cron, eventos e um comando para sistemas escritos em outras linguagens. Veja [Fila, worker e agendamento](queue-and-worker.md). | +| **4. Com o painel** | PostgreSQL e RabbitMQ | Uma interface web embutida na sua aplicação para editar, testar, executar e acompanhar Etapas e trilhas. Veja [Painel](dashboard.md). | +| **Hosts prontos** | Docker | Imagens de contêiner do worker e do painel, ou o ambiente completo com um comando (.NET Aspire). Veja [Hospedagem](hosting.md). | + +## Pré-requisitos + +1. **SDK do .NET 10**: +2. **uv**, que cria e mantém em cache os ambientes Python. Você não precisa instalar o Python: o uv baixa a versão que + cada Etapa pede. + - Windows: `winget install --id=astral-sh.uv -e` + - macOS e Linux: `curl -LsSf https://astral.sh/uv/install.sh | sh` + +## Pacotes + +| Pacote | Uso | +|--------|-----| +| `PythonScriptRunner.Core` | Executar Etapas e trilhas (nível 1). Traz o `PythonScriptRunner.Abstractions`. | +| `PythonScriptRunner.EntityFrameworkCore` | Persistência no PostgreSQL (nível 2) | +| `PythonScriptRunner.RabbitMQ` | Fila, relay do outbox, comando público e eventos (nível 3) | +| `PythonScriptRunner.Worker` | Executa os passos enfileirados e os agendamentos (nível 3) | +| `PythonScriptRunner.Dashboard` | Interface web embutida (nível 4) | +| `PythonScriptRunner.Tool` | A ferramenta de linha de comando `psr` (`psr hash-password` para o login do painel) | +| `psrunner` (PyPI) | O SDK Python. O runner o instala em cada ambiente; instale você mesmo só para testar scripts localmente. | + +Todos os pacotes NuGet têm a mesma versão. Veja a [tabela de compatibilidade](../../README.md#compatibility). + +## Sua primeira Etapa + +Crie um projeto de console: + +```bash +dotnet new console -n MeuCalculo +cd MeuCalculo +dotnet add package PythonScriptRunner.Core +dotnet add package Microsoft.Extensions.Hosting +``` + +Crie uma pasta `psr/stages/media` com dois arquivos. + +`psr/stages/media/stage.json`, o contrato: + +```json +{ + "key": "media", + "version": 1, + "name": "Média", + "inputs": [ { "name": "valores", "type": "list", "itemType": "number" } ], + "outputs": [ { "name": "media", "type": "number" } ] +} +``` + +`psr/stages/media/script.py`, o código: + +```python +from psrunner import stage + +@stage +def run(inputs, ctx): + valores = inputs["valores"] + ctx.log.info("calculando a média de %d valores", len(valores)) + return {"media": sum(valores) / len(valores)} +``` + +`Program.cs`: + +```csharp +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Hosting; +using PythonScriptRunner; + +var builder = Host.CreateApplicationBuilder(args); +builder.Services + .AddPythonScriptRunner() + .AddFileCatalog("psr"); // pasta com stages/ e pipelines/ + +using var host = builder.Build(); +await host.StartAsync(); // confere a configuração (uv no PATH, catálogo) + +var runner = host.Services.GetRequiredService(); +await using var resultado = await runner.RunStageAsync("media", new { valores = new[] { 2.0, 4.0, 9.0 } }); +resultado.EnsureSuccess(); +Console.WriteLine(resultado.GetOutput("media")); // 5 +``` + +Rode com `dotnet run`. A primeira execução leva alguns segundos a mais porque o uv cria o ambiente Python; as seguintes +o reaproveitam. + +O que aconteceu: +- As entradas foram conferidas contra o contrato antes de o Python iniciar. +- O script rodou num processo separado, num ambiente limpo criado pelo uv. +- A saída foi conferida contra o contrato e convertida para `double`. + +Se o script devolver algo fora do contrato (um texto onde se espera um número, ou `NaN`), a execução falha com um erro +claro em vez de entregar dados inválidos. + +## Próximos passos + +- Adicionar bibliotecas, limites e segredos a uma Etapa, ou defini-la em C#: [Etapas](stages.md). +- Encadear Etapas: [Trilhas](pipelines.md). +- Testar scripts com pytest, sem .NET: [SDK Python](python-sdk.md). +- Guardar histórico e versões: [Persistência](persistence.md). diff --git a/guides/pt-br/hosting.md b/guides/pt-br/hosting.md new file mode 100644 index 0000000..ff3cf57 --- /dev/null +++ b/guides/pt-br/hosting.md @@ -0,0 +1,213 @@ +# Hospedagem + +[English](../en/hosting.md) | Português + +Você não precisa escrever uma aplicação host. O repositório traz dois hosts prontos como imagens de contêiner, um ambiente +local no .NET Aspire e um Docker Compose gerado a partir dele. + +## Imagens de contêiner + +As imagens são multi-arch (amd64 e arm64). Cada release publica a tag da versão (`0.1.0`), a tag da versão menor (`0.1`) +e `latest`. + +| Imagem | Host | +|--------|------| +| `ghcr.io/scriptrunnerkit/psr-worker` | `WorkerHost`: executa os passos enfileirados e os agendamentos | +| `ghcr.io/scriptrunnerkit/psr-dashboard` | `DashboardHost`: o painel, com a interface Vue embutida | + +As duas imagens têm estas características: +- **Base:** `mcr.microsoft.com/dotnet/aspnet:10.0` com o `uv`. +- **Usuário:** rodam com o usuário não-root `psr` (uid 10001). +- **Rede:** escutam na porta 8080, com `/health/live` e `/health/ready`. +- **SDK:** trazem o wheel do `psrunner` do mesmo commit, então os ambientes não baixam o SDK. As dependências das Etapas + continuam vindo do PyPI ou do seu índice de pacotes. +- **Cadeia de suprimentos:** trazem atestados de SBOM e proveniência. +- **Observabilidade:** os logs saem em JSON no stdout, e o OpenTelemetry é exportado quando `OTEL_EXPORTER_OTLP_ENDPOINT` + está definido. + +```bash +docker run -d --read-only \ + --tmpfs /tmp/psr:uid=10001,gid=10001,exec \ + -v psr-envs:/var/lib/psr/envs -v psr-python:/var/lib/psr/python \ + -e ConnectionStrings__psr="Host=pg;Database=psr;Username=psr_app;Password=..." \ + -e ConnectionStrings__rabbitmq="amqp://psr:...@rabbit:5672" \ + ghcr.io/scriptrunnerkit/psr-worker:0.1.0 +``` + +### Variáveis de ambiente + +| Variável | Host | Exemplo ou significado | +|----------|------|------------------------| +| `ConnectionStrings__psr` | ambos | `Host=pg;Database=psr;Username=psr_app;Password=…` | +| `ConnectionStrings__rabbitmq` | ambos | `amqps://psr:…@rabbit:5671/psr` | +| `PSR_AUTO_MIGRATE` | ambos | `true` aplica as migrações na inicialização (padrão `false`); hosts simultâneos esperam num advisory lock | +| `PythonScriptRunner__Worker__MaxConcurrency` | worker | `8` | +| `PythonScriptRunner__Worker__ShutdownGracePeriod` | worker | `00:05:00` | +| `PythonScriptRunner__Secrets__` | worker | Valores dos segredos, vindos do cofre de segredos da sua plataforma | +| `PythonScriptRunner__ThreadsPerProcess` | ambos | Threads das bibliotecas numéricas (veja [Limites de memória](#limites-de-memória)) | +| `PythonScriptRunner__PackageIndexUrl` | ambos | Um índice de pacotes privado | +| `PythonScriptRunner__Dashboard__Login__Users__N__Name`, `__Role`, `__PasswordHash` | painel | Usuários do login fixo | +| `Oidc__Authority`, `Oidc__ClientId`, `Oidc__ClientSecret` | painel | Liga o OpenID Connect no lugar do login fixo | +| `Dashboard__RoleClaim` | painel | Claim com os papéis (padrão `roles`) | +| `Dashboard__Roles__Reader`, `__Executor`, `__Author`, `__Admin` | painel | Valores associados a cada papel do painel | +| `Dashboard__PathPrefix` | painel | Padrão `/psr` | +| `ASPNETCORE_FORWARDEDHEADERS_ENABLED` | painel | `true` atrás de um proxy reverso | +| `OTEL_EXPORTER_OTLP_ENDPOINT` | ambos | `http://otel-collector:4317` | +| `Serilog__MinimumLevel__Override__` | ambos | Por exemplo, `Information` para `Microsoft.AspNetCore` traz de volta os logs de requisição (padrão `Warning`) | + +Qualquer opção da [referência de configuração](configuration.md) pode ser definida do mesmo jeito, com `__` entre as +seções (`PythonScriptRunner__DefaultTimeout=00:10:00`). + +### Volumes e sistema de arquivos somente leitura + +As imagens rodam com o sistema de arquivos raiz somente leitura. O Python grava apenas nestes volumes: + +| Caminho | Conteúdo | Se não for persistente | +|---------|----------|------------------------| +| `/var/lib/psr/envs` | Ambientes virtuais em cache (um por conjunto de dependências) e o cache do uv | Cada inicialização recria os ambientes, então a primeira execução de cada Etapa fica mais lenta | +| `/var/lib/psr/python` | Interpretadores Python instalados pelo uv | O interpretador é baixado de novo | +| `/tmp/psr` | Pastas de trabalho das execuções e o `TMPDIR` | Esperado: só guarda arquivos temporários | + +Um tmpfs funciona em qualquer um deles, mas precisa permitir execução (`exec`), porque os volumes executam Python; o +Docker monta tmpfs com `noexec` por padrão. + +Sem acesso ao PyPI, os ambientes falham com `EnvironmentSetupFailed`. Nesse caso, aponte +`PythonScriptRunner__PackageIndexUrl` para um índice privado, ou construa a imagem com um interpretador já instalado: + +```bash +docker build --build-arg PSR_PREINSTALL_PYTHON=3.12 -f hosts/PythonScriptRunner.WorkerHost/Dockerfile -t meu-psr-worker . +``` + +### Limites de memória + +Todo processo Python recebe o limite de memória da sua Etapa (1024 MB por padrão). A forma de aplicar depende da +plataforma, e o host registra no log o modo que está usando ao iniciar: + +- **Windows:** um Job Object por processo. +- **Linux com cgroup v2 delegável:** + - cada processo roda no próprio cgroup (`memory.max`, sem swap); + - quando o processo passa do limite, o kernel o encerra e a tentativa termina com `MemoryLimitExceeded` (código + `OUT_OF_MEMORY`); + - o host precisa de permissão de escrita no próprio cgroup: um serviço systemd com `Delegate=yes`, ou um contêiner + rodando como root com o sistema de arquivos de cgroup gravável. +- **Outros cenários Linux, inclusive contêineres padrão** (eles montam o sistema de arquivos de cgroup somente leitura): + - um limite de espaço de endereçamento (`RLIMIT_AS`); + - as bibliotecas numéricas (OpenBLAS, MKL, OpenMP, NumExpr, vecLib) reservam espaço de endereçamento por thread, então + nesse modo recebem 1 thread por processo. + +O `PythonScriptRunner:ThreadsPerProcess` define o número de threads em todos os modos. Aumente para trabalho numérico em +paralelo, com um limite de memória que comporte isso. + +## Kubernetes + +Ainda não há um Helm chart oficial. Este `Deployment` de worker mostra as configurações que importam: + +```yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: psr-worker +spec: + replicas: 2 + selector: + matchLabels: { app: psr-worker } + template: + metadata: + labels: { app: psr-worker } + spec: + # Maior que PythonScriptRunner__Worker__ShutdownGracePeriod (padrão 5 min), para os passos em andamento terminarem. + terminationGracePeriodSeconds: 330 + securityContext: + runAsNonRoot: true + runAsUser: 10001 + fsGroup: 10001 + containers: + - name: worker + image: ghcr.io/scriptrunnerkit/psr-worker:0.1.0 + env: + - name: ConnectionStrings__psr + valueFrom: { secretKeyRef: { name: psr, key: postgres } } + - name: ConnectionStrings__rabbitmq + valueFrom: { secretKeyRef: { name: psr, key: rabbitmq } } + ports: [ { containerPort: 8080 } ] + livenessProbe: { httpGet: { path: /health/live, port: 8080 } } + readinessProbe: { httpGet: { path: /health/ready, port: 8080 } } + securityContext: + readOnlyRootFilesystem: true + allowPrivilegeEscalation: false + resources: + limits: { memory: 4Gi } # espaço para MaxConcurrency processos no limite da Etapa + volumeMounts: + - { name: envs, mountPath: /var/lib/psr/envs } + - { name: python, mountPath: /var/lib/psr/python } + - { name: work, mountPath: /tmp/psr } + volumes: + - { name: envs, emptyDir: {} } # um PersistentVolumeClaim mantém os ambientes entre reinícios + - { name: python, emptyDir: {} } + - { name: work, emptyDir: {} } +``` + +Se o `terminationGracePeriodSeconds` for menor que o `ShutdownGracePeriod`, o kubelet encerra tentativas que ainda estão +rodando. Elas seguem então as regras do lease: são retomadas em outro worker ou falham com `WorkerLost`. + +## Ambiente local (.NET Aspire) + +Com Docker, o SDK do .NET 10, Node.js 24 e o uv instalados: + +```bash +git clone https://github.com/ScriptRunnerKit/PythonScriptRunner +cd PythonScriptRunner +dotnet run --project hosts/PythonScriptRunner.AppHost +``` + +O AppHost sobe: +- PostgreSQL 16 e RabbitMQ 3.13 (com a interface de gerenciamento); +- Mailpit, um servidor SMTP de teste; +- o WorkerHost e o DashboardHost; +- a interface Vue em modo de desenvolvimento, com hot reload. + +O dashboard do Aspire (`http://localhost:15000`) mostra os recursos, os logs e os traces de cada execução. O link +**Painel** no recurso `dashboard` abre a interface. + +![Trace de uma execução no dashboard do Aspire](../images/aspire-trace.png) + +O painel tem um usuário de desenvolvimento por papel, todos com a senha `psr-dev`: + +| Usuário | Papel | +|---------|-------| +| `leitor` | Reader | +| `executor` | Executor | +| `autor` | Author | +| `admin` | Admin | + +Para testar as trilhas de exemplo em `tests/fixtures`: +1. Compacte o conteúdo de uma pasta, com `stages/` e `pipelines/` na raiz do zip. +2. Importe no painel como `autor`. +3. Publique as Etapas e depois a trilha. + +## Docker Compose + +O `aspire publish` transforma o AppHost num `docker-compose.yaml` que usa as imagens publicadas: + +```bash +dotnet tool install --global Aspire.Cli --version 13.6.2 +aspire publish --apphost hosts/PythonScriptRunner.AppHost/PythonScriptRunner.AppHost.csproj -o deploy +``` + +Preencha o `deploy/.env` e suba com `docker compose up -d` dentro de `deploy`: + +| Variável | Significado | +|----------|-------------| +| `POSTGRES_PASSWORD` | Senha do usuário `postgres` | +| `RABBITMQ_PASSWORD` | Senha do usuário `guest` do RabbitMQ | +| `PSR_VERSION` | Versão das imagens (padrão `latest`) | +| `PSR_DASHBOARD_PORT` | Porta do painel no host (padrão `8080`); a interface fica em `/psr/` | + +O compose inclui um dashboard do Aspire para os traces. Ele é um ponto de partida e mantém a configuração de +desenvolvimento: +- os quatro usuários com a senha `psr-dev`; +- o Mailpit; +- os segredos SMTP de teste. + +Antes de expor, troque os valores de `PasswordHash` por hashes gerados com `psr hash-password` (ou configure o OpenID +Connect) e remova o que não for usar. diff --git a/guides/pt-br/how-it-works.md b/guides/pt-br/how-it-works.md new file mode 100644 index 0000000..d70f1c0 --- /dev/null +++ b/guides/pt-br/how-it-works.md @@ -0,0 +1,95 @@ +# Como funciona + +[English](../en/how-it-works.md) | Português + +Esta página explica o que acontece quando uma Etapa roda, o que a biblioteca garante e como observar tudo em produção. + +## Uma tentativa, passo a passo + +1. **Validar as entradas** contra o contrato. Entradas inválidas param aqui, e nenhum processo Python inicia. +2. **Resolver os segredos** que a Etapa declara. Um segredo faltando faz a tentativa falhar com `SecretNotFound`. +3. **Preparar o ambiente.** O uv cria (ou reaproveita) o ambiente Python da Etapa. Os ambientes são identificados por um + hash da versão do Python, dos requisitos travados e da versão do `psrunner`. Um lock em arquivo faz execuções + simultâneas esperarem uma única criação, e um ambiente só é usado depois de completamente instalado. +4. **Esperar uma vaga.** No máximo `MaxConcurrentProcesses` processos Python rodam ao mesmo tempo num processo .NET. +5. **Iniciar o processo** numa pasta só dele, com: + - um ambiente limpo: só as variáveis de que o Python precisa, nunca as variáveis da sua aplicação; + - os segredos declarados; + - as entradas num arquivo. +6. **Aplicar os limites:** o tempo limite e o limite de memória (veja [Limites de memória](hosting.md#limites-de-memória)). + No timeout ou no cancelamento, a árvore de processos inteira é encerrada. +7. **Capturar a saída.** O final do stdout e do stderr é guardado (até `MaxLogBytes`), com os valores dos segredos + mascarados como `***`. +8. **Validar as saídas** contra o contrato e convertê-las para valores .NET. Os arquivos de saída são movidos para a + execução, dentro de `MaxFileBytes` e `MaxRunFilesBytes`. + +O lado .NET e o `psrunner` trocam arquivos no protocolo versão 1, e o `psrunner` continua compatível com ele enquanto +esse protocolo estiver em uso. + +## Garantias + +- **Isolamento:** cada tentativa roda no próprio processo, então uma falha grave no Python nunca derruba a aplicação .NET. +- **Limites:** toda tentativa tem tempo limite e limite de memória. +- **Segredos:** chegam só ao processo da Etapa que os declara. Nunca são gravados no arquivo de entradas, em logs, + métricas, exceções ou no banco de dados. +- **Reprodutibilidade:** as dependências publicadas são travadas com hash e instaladas com `--require-hashes`, então uma + versão publicada sempre roda com os mesmos pacotes. +- **Resultados tipados:** o lado .NET recebe valores de acordo com o contrato, ou um erro claro. + +## Modelo de segurança + +O PythonScriptRunner executa código escrito por **autores confiáveis**. Uma Etapa é código Python arbitrário rodando com +as permissões do usuário do worker; ela não é uma sandbox para código não confiável. Quem tem o papel `Author` no painel +pode publicar código que roda nos workers, então dê esse papel só a quem você deixaria fazer deploy de código. + +Recomendações: +- Rode os workers com um usuário sem privilégios e com acesso só ao que as Etapas precisam. As imagens já rodam com um + usuário não-root e com o sistema de arquivos raiz somente leitura. +- Guarde os segredos no cofre de segredos da sua plataforma. +- Sirva o painel por HTTPS. +- Não exponha a configuração de desenvolvimento do AppHost nem do compose gerado. + +Para reportar uma vulnerabilidade, veja o [SECURITY.md](../../SECURITY.md). + +## Observabilidade + +**Traces.** O `ActivitySource` chamado `PythonScriptRunner` cria estes spans: +- `psr.run.enqueue`: uma execução foi enfileirada; +- `psr.message.consume`: uma mensagem foi tratada; +- `psr.environment.ensure`: um ambiente foi preparado; +- `psr.stage.execute`: uma tentativa rodou. + +O `traceparent` viaja com as mensagens, então um único trace vai da requisição HTTP até o worker e cada processo Python. + +![Trace de uma execução](../images/aspire-trace.png) + +**Métricas.** O `Meter` chamado `PythonScriptRunner` publica estes instrumentos: + +| Instrumento | Significado | +|-------------|-------------| +| `psr.step.duration` | Duração de cada tentativa. | +| `psr.step.overhead` | Tempo gasto pelo runner fora do processo Python. | +| `psr.step.failures` | Tentativas com falha. | +| `psr.environment.created` | Ambientes Python criados. | +| `psr.queue.wait` | Tempo entre enfileirar um passo e ele começar num worker. | +| `psr.worker.active_steps` | Passos rodando neste worker. | +| `psr.lease.recovered` | Tentativas reiniciadas depois que o worker delas foi perdido. | +| `psr.schedule.fired` | Execuções criadas por agendamentos. | +| `psr.outbox.publish.duration` | Tempo para publicar um lote do outbox. | +| `psr.maintenance.partitions.created`, `psr.maintenance.partitions.dropped`, `psr.maintenance.rows.deleted` | Trabalho do job de manutenção. | + +Inclua a fonte e o meter na sua configuração do OpenTelemetry: + +```csharp +builder.Services.AddOpenTelemetry() + .WithTracing(t => t.AddSource("PythonScriptRunner")) + .WithMetrics(m => m.AddMeter("PythonScriptRunner")); +``` + +**Health checks:** +- o `AddPythonScriptRunnerDatabase()` confere o PostgreSQL; +- o `AddPythonScriptRunnerBroker()` confere o RabbitMQ; +- o `AddPythonScriptRunnerChecks()` confere o banco, o broker e o uv juntos. + +**Logs.** A biblioteca registra pelo `ILogger`. Os comandos SQL são registrados em `Debug`, porque o relay do outbox +consulta o banco várias vezes por segundo. diff --git a/guides/pt-br/persistence.md b/guides/pt-br/persistence.md new file mode 100644 index 0000000..d989db5 --- /dev/null +++ b/guides/pt-br/persistence.md @@ -0,0 +1,117 @@ +# Persistência (PostgreSQL) + +[English](../en/persistence.md) | Português + +Com o `PythonScriptRunner.EntityFrameworkCore`, as Etapas e trilhas ficam no PostgreSQL com versões imutáveis, e toda +execução fica registrada. Este nível não precisa de RabbitMQ: as execuções síncronas também são registradas. + +## Configuração + +```bash +dotnet add package PythonScriptRunner.EntityFrameworkCore +``` + +```csharp +builder.Services + .AddPythonScriptRunner() + .AddEntityFrameworkCore(o => + { + o.UseNpgsql(builder.Configuration.GetConnectionString("psr")!); // PostgreSQL 16 ou mais novo + o.Retention.Artifacts = TimeSpan.FromDays(7); // conteúdo dos arquivos + o.Retention.Runs = TimeSpan.FromDays(90); // execuções, passos, tentativas e logs + }); +builder.Services.AddHealthChecks().AddPythonScriptRunnerDatabase(); + +var app = builder.Build(); +await app.Services.MigratePythonScriptRunnerAsync(); +``` + +O schema nunca é alterado de forma implícita. Há duas opções: +- **Na inicialização:** chame `MigratePythonScriptRunnerAsync()`. Instâncias simultâneas esperam num advisory lock do + PostgreSQL, então as migrações rodam uma única vez. +- **Pelo seu DBA:** gere o SQL com `GeneratePythonScriptRunnerMigrationScriptAsync(idempotent: true)` e entregue a ele. + +Tudo é criado no schema `psr`. + +## O catálogo + +Com persistência, o catálogo vem do banco de dados. Cada Etapa e trilha tem versões que passam de **rascunho** para +**publicada** e depois **arquivada**. Uma versão publicada nunca muda; editar cria um novo rascunho. + +O `ICatalogAdmin` administra o catálogo. O painel usa o mesmo serviço. + +```csharp +var admin = app.Services.GetRequiredService(); +var ator = Actor.User("ana@exemplo.com"); + +var rascunho = await admin.CreateStageAsync(new NewCatalogItem("media", "Média"), ator); +var salvo = await admin.SaveStageDraftAsync("media", rascunho.Version, rascunho.Data with +{ + Script = script, + Inputs = [new FieldDefinition("valores", FieldType.ListOf(FieldType.Number))], + Outputs = [new FieldDefinition("media", FieldType.Number)], +}, rascunho.RowVersion, ator); +await admin.PublishStageAsync("media", salvo.Version, salvo.RowVersion, ator); +``` + +| Operação | Métodos | +|----------|---------| +| Criar e editar rascunhos | `CreateStageAsync`, `CreateStageDraftFromAsync`, `SaveStageDraftAsync`, `DiscardStageDraftAsync` (e os equivalentes de `Pipeline`) | +| Validar e testar um rascunho | `ValidateStageDraftAsync`, `TestStageDraftAsync`, `EnqueueStageDraftTestAsync` (num worker) | +| Publicar | `PublishStageAsync`, `PublishPipelineAsync` | +| Arquivar e excluir | `ArchiveStageVersionAsync`, `DeleteStageAsync` (e os equivalentes de `Pipeline`) | +| Importar e exportar | `ImportAsync(zip)` cria rascunhos a partir de um zip no formato de pasta; `ExportStageVersionAsync` e `ExportPipelineVersionAsync` | +| Agendamentos | `CreateScheduleAsync`, `UpdateScheduleAsync`, `SetScheduleEnabledAsync`, `DeleteScheduleAsync` (veja [Fila, worker e agendamento](queue-and-worker.md)) | + +Detalhes dessas operações: +- **Concorrência otimista:** os salvamentos usam a versão da linha, então duas pessoas editando o mesmo rascunho não + sobrescrevem uma à outra sem perceber. +- **Publicar uma Etapa:** + 1. confere a sintaxe do script com o Python declarado; + 2. trava os requisitos com hash; + 3. congela a versão. +- **Publicar uma trilha:** valida os tipos e as ligações contra as Etapas publicadas. +- **Trazer uma pasta:** para levar um catálogo em pasta para o banco, compacte a pasta (com `stages/` e `pipelines/` na + raiz), importe e depois publique as Etapas e as trilhas. + +## Execuções + +Toda execução fica gravada, seja síncrona (`IPythonRunner`), enfileirada, agendada ou disparada pelo painel. O registro +inclui as entradas, os passos, cada tentativa com duração e erro, o final do stdout e do stderr (até `MaxLogBytes`) e os +arquivos. O conteúdo dos arquivos fica no PostgreSQL (`bytea`), gravado e lido em streaming. + +O `IRunManager` consulta e administra as execuções: + +| Método | Descrição | +|--------|-----------| +| `GetAsync(runId)` | A execução completa, com passos e tentativas. | +| `ListAsync(query)` | Execuções filtradas por status, trilha, Etapa, origem ou período, com paginação keyset. | +| `GetAttemptLogAsync(runId, attemptId, stream)` | O stdout ou stderr capturado de uma tentativa. | +| `CancelAsync(runId, ator)` | Cancela uma execução. Uma execução enfileirada para no worker que a executa. | +| `ReprocessAsync(runId, ator, mode)` | Repete uma execução que falhou, com a mesma versão e as mesmas entradas. Os passos que deram certo mantêm as saídas e o passo que falhou ganha uma nova tentativa. `ExecutionMode.Async` (o padrão) enfileira para os workers e precisa do RabbitMQ; `ExecutionMode.Sync` executa agora, no processo atual. | + +**Idempotência:** o `RunOptions.IdempotencyKey` devolve a mesma execução quando a mesma chave é usada de novo dentro do +`IdempotencyWindow` (24 horas por padrão). É útil quando quem chama pode repetir uma requisição. + +## Retenção + +O histórico é particionado: execuções por semana, conteúdo dos arquivos por dia e auditoria por mês. Um job de +manutenção, ligado por padrão e executado a cada hora, cria as partições com antecedência e remove as vencidas, o que é +barato mesmo com grandes volumes. + +| Opção | Padrão | +|-------|--------| +| `Retention.Artifacts` (conteúdo dos arquivos) | 7 dias | +| `Retention.Runs` (execuções, passos, tentativas, logs e metadados de arquivos) | 90 dias | +| `Retention.Audit` | 365 dias | +| `Maintenance.Enabled`, `Maintenance.Interval` | `true`, 1 hora | + +Uma trilha pode ter retenção menor que a global, mas não maior. + +## Auditoria + +As ações humanas ficam em `psr.audit_events`: quem fez o quê, em qual item e quando. Isso cobre edição, publicação, +arquivamento, exclusão, cancelamento, reprocessamento, agendamento e mudanças de configuração. O autor de cada ação é um +`Actor`: +- `Actor.User("identificador")` para pessoas; +- `Actor.System("origem")` para disparos automáticos, que ficam registrados na própria execução. diff --git a/guides/pt-br/pipelines.md b/guides/pt-br/pipelines.md new file mode 100644 index 0000000..04d6fc1 --- /dev/null +++ b/guides/pt-br/pipelines.md @@ -0,0 +1,109 @@ +# Trilhas + +[English](../en/pipelines.md) | Português + +Uma trilha executa Etapas uma depois da outra. Cada passo pode usar as entradas da trilha, as saídas de qualquer passo +anterior ou valores fixos, e a trilha declara as próprias saídas. Tipos e ligações são validados quando o catálogo é +carregado (com persistência, quando uma versão é publicada), então uma trilha que não consegue rodar é rejeitada antes da +execução. + +## O arquivo da trilha + +Com o catálogo em pasta, as trilhas são arquivos `pipelines/.pipeline.json`: + +```json +{ + "$schema": "https://raw.githubusercontent.com/ScriptRunnerKit/PythonScriptRunner/main/schemas/pipeline.v1.json", + "key": "calculo-progressao", + "version": 1, + "name": "Cálculo de progressão", + "inputs": [ + { "name": "a1", "type": "number" }, + { "name": "razao", "type": "number" }, + { "name": "n", "type": "integer" }, + { "name": "coeficientes", "type": "list", "itemType": "number" } + ], + "steps": [ + { "key": "termos", "stage": "progressao-termos@1" }, + { "key": "polinomio", "stage": "polinomio@1", + "bindings": { "coeficientes": "$pipeline.coeficientes", "x": "$steps.termos.termos" } }, + { "key": "estatisticas", "stage": "estatisticas@1", + "bindings": { "valores": "$steps.polinomio.valores", "precisao": { "const": 4 } } } + ], + "outputs": [ + { "name": "soma", "source": "$steps.estatisticas.soma" }, + { "name": "termos", "source": "$steps.termos.termos", "description": "Termos da progressão" } + ] +} +``` + +| Propriedade | Significado | +|-------------|-------------| +| `key`, `version`, `name`, `description` | Mesmas regras de uma Etapa. | +| `inputs` | Campos da trilha, com os mesmos tipos e propriedades dos campos de uma Etapa. | +| `steps` | Os passos, na ordem de execução. Cada um tem uma `key` (identificador), uma `stage` (`chave-da-etapa@versão`), um `displayName` opcional e `bindings` opcionais. | +| `outputs` | O que a trilha devolve. Cada saída tem um `name` e uma `source` (`$steps..`); o tipo vem da saída daquele passo. | + +## Ligações + +Cada entrada de um passo recebe o valor de uma de três fontes: + +| Ligação | Exemplo | Valor | +|---------|---------|-------| +| Entrada da trilha | `"$pipeline.coeficientes"` | A entrada da trilha com esse nome. | +| Passo anterior | `"$steps.termos.termos"` | A saída `termos` do passo `termos`. Vale qualquer passo anterior, não só o imediatamente anterior. | +| Constante | `{ "const": 4 }` | Um valor fixo. | + +Uma entrada sem ligação explícita usa a **ligação implícita por nome**: +- no primeiro passo, a entrada da trilha com o mesmo nome; +- nos demais passos, a saída com o mesmo nome do passo anterior. + +Quando os tipos são compatíveis mas não iguais (um `integer` alimentando um `decimal`, por exemplo), o valor é +convertido. Uma ligação incompatível é um erro de validação. + +## Executando uma trilha + +```csharp +await using var resultado = await runner.RunPipelineAsync("calculo-progressao", + new { a1 = 1, razao = 2, n = 4, coeficientes = new[] { 1.0, 0.0, 1.0 } }); + +resultado.EnsureSuccess(); +double soma = resultado.GetOutput("soma"); +double[] termos = resultado.GetStepOutput("termos", "termos"); // qualquer saída de passo, útil para depuração +``` + +O `resultado.Steps` lista cada passo com status, tentativas, duração e erro. + +Se um passo falhar, os passos seguintes terminam como `Skipped` e a execução termina como `Failed`, com o erro do passo +que falhou. +- **Execuções síncronas:** você inicia uma nova execução. +- **Com persistência:** o `IRunManager.ReprocessAsync` repete só a partir do passo que falhou, reaproveitando as saídas + dos passos que deram certo. Veja [Persistência](persistence.md). + +## Definindo uma trilha em C# + +```csharp +builder.Services.AddPythonScriptRunner() + .AddPipeline("calculo-progressao", p => p + .Name("Cálculo de progressão") + .Input("a1", FieldType.Number) + .Input("razao", FieldType.Number) + .Input("n", FieldType.Integer) + .Input("coeficientes", FieldType.ListOf(FieldType.Number)) + .Step("termos", "progressao-termos@1") + .Step("polinomio", "polinomio@1", s => s + .Bind("coeficientes", new PipelineInputSource("coeficientes")) + .Bind("x", new StepOutputSource("termos", "termos"))) + .Step("estatisticas", "estatisticas@1", s => s + .Bind("valores", new StepOutputSource("polinomio", "valores")) + .Bind("precisao", new ConstantSource(JsonValue.Create(4)))) + .Output("soma", StepOutputSource.Step("estatisticas", "soma"))); +``` + +## Configurações das execuções enfileiradas + +Com a fila (veja [Fila, worker e agendamento](queue-and-worker.md)), cada trilha também tem: +- **Máximo de execuções simultâneas:** as execuções além do limite esperam como `Pending`, em ordem. +- **Retenção:** ela só pode ser menor que a retenção global. + +As duas são alteradas no painel ou com `ICatalogAdmin.UpdatePipelineSettingsAsync` e `UpdatePipelineRetentionAsync`. diff --git a/guides/pt-br/python-sdk.md b/guides/pt-br/python-sdk.md new file mode 100644 index 0000000..d2d7c38 --- /dev/null +++ b/guides/pt-br/python-sdk.md @@ -0,0 +1,134 @@ +# SDK Python (psrunner) + +[English](../en/python-sdk.md) | Português + +O `psrunner` é o lado Python do PythonScriptRunner. Ele lê as entradas, converte para tipos Python, chama a sua função, +confere as saídas contra o contrato e as devolve ao .NET. O script contém só a lógica. + +O runner instala o `psrunner` em cada ambiente que cria, então uma Etapa nunca o coloca no `requirements.txt`. Instale +você mesmo só para escrever e testar scripts localmente: + +```bash +pip install psrunner # Python 3.10 ou mais novo +pip install "psrunner[pandas]" # entradas table chegam como pandas.DataFrame +``` + +## Escrevendo uma Etapa + +```python +from psrunner import stage, StageError + +@stage +def run(inputs, ctx): + valores = inputs["valores"] + if not valores: + raise StageError("Informe pelo menos um valor.", code="ENTRADA_VAZIA") + + ctx.log.info("calculando a média de %d valores", len(valores)) + return {"media": sum(valores) / len(valores)} +``` + +As regras: +- **Um ponto de entrada:** exatamente uma função por script é marcada com `@stage`. Ela recebe `inputs` (um dict com as + entradas, já convertidas) e `ctx` (o contexto descrito abaixo). +- **Valor de retorno:** ela devolve um dict com as saídas. Saídas obrigatórias precisam estar presentes; saídas fora do + contrato são um erro. +- **Tipos:** os valores são convertidos como descrito em [Tipos de campo](stages.md#tipos-de-campo). Por exemplo, + `decimal` chega como `Decimal`, e `datetime` precisa ter fuso UTC. + +## O contexto (`ctx`) + +| Membro | Descrição | +|--------|-----------| +| `ctx.log` | Um `logging.Logger`. As linhas aparecem nos logs da execução (painel e `RunResult`). `print` também funciona e vai para o stdout. | +| `ctx.secrets` | Mapeamento somente leitura com os segredos que a Etapa declara: `ctx.secrets["SMTP_SENHA"]`. | +| `ctx.new_file(nome, content_type=None)` | Reserva um arquivo de saída e devolve um `File`; escreva em `arquivo.path`. | +| `ctx.run_id`, `ctx.step_key`, `ctx.attempt` | Identificam a execução, o passo da trilha e o número da tentativa. | +| `ctx.stage_key`, `ctx.stage_version` | A Etapa em execução. | +| `ctx.is_test` | `True` nas execuções de teste de um rascunho. | +| `ctx.work_dir`, `ctx.out_dir`, `ctx.tmp_dir` | As pastas da própria tentativa. Use `tmp_dir` para arquivos temporários. | + +## Arquivos + +Uma entrada `file` chega como `psrunner.File`: + +| Membro | Descrição | +|--------|-----------| +| `path` | Caminho local do arquivo. | +| `name`, `content_type`, `size`, `sha256` | Metadados. | +| `read_bytes()`, `read_text(encoding="utf-8")`, `open(mode="rb")` | Leem o conteúdo. | + +Para produzir um arquivo de saída, reserve com `ctx.new_file` e devolva o `File`: + +```python +@stage +def run(inputs, ctx): + relatorio = ctx.new_file("relatorio.csv", "text/csv") + relatorio.path.write_text("x,y\n1,2\n", encoding="utf-8") + return {"relatorio": relatorio} +``` + +Os nomes de arquivo precisam ser simples, sem pastas. O runner aplica `MaxFileBytes` por arquivo e `MaxRunFilesBytes` +por execução (veja a [Configuração](configuration.md)). + +## Erros + +- **Erros de negócio:** lance `StageError(mensagem, code="ALGUM_CODIGO", details={...})`. A execução falha com o tipo + `ScriptError` e esse `code`, para que o lado .NET e o painel possam reagir. +- **Exceções inesperadas:** qualquer outra exceção também vira `ScriptError`. O erro traz o traceback do seu script, + que o painel mostra. +- **Violações de contrato:** uma saída faltando, um tipo errado ou um número não finito vira `OutputContractViolation`. + +## Testes com pytest + +O `psrunner.testing.run_stage` executa a função com as mesmas conversões e validações do runner real, sem .NET: + +```python +from psrunner.testing import run_stage + +def test_media(): + resultado = run_stage( + "psr/stages/media/script.py", + {"valores": [2, 4, 9]}, + stage_json="psr/stages/media/stage.json", + ) + assert resultado.ok + assert resultado.outputs == {"media": 5.0} +``` + +| Parâmetro | Descrição | +|-----------|-----------| +| `script` | Caminho do script, ou a própria função. | +| `inputs` | Dict com as entradas. | +| `stage_json` | Caminho do contrato. Com ele, entradas e saídas são validadas; sem ele, as entradas passam como estão. | +| `secrets` | Dict com os valores dos segredos. | +| `files` | Associa campos de arquivo a caminhos locais (uma lista para `list` de `file`). | +| `work_dir`, `run_id` | Pasta e id de execução opcionais. | + +O resultado tem estes campos: +- `ok`: `True` quando a Etapa teve sucesso; +- `outputs`: os valores Python; +- `wire_outputs`: o JSON enviado ao .NET; +- `error`: com `kind`, `code`, `message` e `traceback`; +- `logs`: as linhas de log; +- `work_dir`: a pasta usada pela execução. + +## Linha de comando + +```bash +python -m psrunner validate psr/stages/media +python -m psrunner run psr/stages/media --input exemplo.json --secret SMTP_SENHA=segredo --keep +``` + +| Comando | O que faz | +|---------|-----------| +| `validate ` | Confere o `stage.json` e se o script tem exatamente uma função `@stage`. | +| `run ` | Executa a Etapa localmente com o protocolo real, num processo separado e numa pasta temporária. | + +No `exemplo.json` do `run`, uma entrada de arquivo é escrita como `{"$path": "relativo/ao/exemplo.json"}`. O `--keep` +mostra a pasta de trabalho para você inspecionar. Os mesmos comandos existem como `psrunner validate` e `psrunner run`. + +## Versões + +O `psrunner` tem versão própria no PyPI. Ele segue o protocolo 1, o formato trocado com o runner .NET, enquanto esse +protocolo estiver em uso. Veja a [tabela de compatibilidade](../../README.md#compatibility). diff --git a/guides/pt-br/queue-and-worker.md b/guides/pt-br/queue-and-worker.md new file mode 100644 index 0000000..3c549b1 --- /dev/null +++ b/guides/pt-br/queue-and-worker.md @@ -0,0 +1,147 @@ +# Fila, worker e agendamento (RabbitMQ) + +[English](../en/queue-and-worker.md) | Português + +Com o `PythonScriptRunner.RabbitMQ` e o `PythonScriptRunner.Worker`, as execuções acontecem em segundo plano. A sua +aplicação enfileira uma execução e segue; os workers, numa máquina ou em várias, executam os passos. Este nível precisa +da [persistência](persistence.md): o PostgreSQL guarda o estado de cada execução e o outbox das mensagens. + +## Configuração + +```bash +dotnet add package PythonScriptRunner.EntityFrameworkCore +dotnet add package PythonScriptRunner.RabbitMQ +dotnet add package PythonScriptRunner.Worker +``` + +```csharp +builder.Services + .AddPythonScriptRunner() + .AddEntityFrameworkCore(o => o.UseNpgsql(builder.Configuration.GetConnectionString("psr")!)) + .AddRabbitMq(o => o.ConnectionString = builder.Configuration.GetConnectionString("rabbitmq")) // RabbitMQ 3.13 ou mais novo + .AddWorker(o => o.MaxConcurrency = 4); +builder.Services.AddHealthChecks().AddPythonScriptRunnerChecks(); // banco, broker e uv +``` + +O `AddWorker` vai só nos processos que devem executar passos. Uma aplicação web pode enfileirar execuções e mostrar o +andamento só com o `AddRabbitMq`, enquanto processos de worker dedicados (ou a [imagem do worker](hosting.md)) fazem o +trabalho. + +## Enfileirando uma execução + +```csharp +var fila = app.Services.GetRequiredService(); + +RunHandle execucao = await fila.EnqueuePipelineAsync("calculo-progressao", + new { a1 = 1, razao = 2, n = 4, coeficientes = new[] { 1.0, 0.0, 1.0 } }, + new RunOptions { IdempotencyKey = "pedido-1234" }); +``` + +O `EnqueuePipelineAsync` e o `EnqueueStageAsync` validam as entradas, registram a execução e devolvem um handle com o id +dela. Acompanhe pelo `IRunManager.GetAsync(execucao.RunId)`, pelos [eventos](#eventos) ou pelo [painel](dashboard.md). + +## O caminho de uma execução + +1. A execução e a mensagem do primeiro passo são gravadas numa única transação do PostgreSQL (um outbox). Enfileirar + funciona mesmo com o RabbitMQ fora do ar; a mensagem é enviada quando o broker volta. +2. Um worker assume o passo com um **lease**, executa o processo Python e renova o lease enquanto ele roda. +3. Quando o passo termina, o worker grava o resultado e a mensagem do próximo passo na mesma transação. +4. Os eventos da execução são publicados conforme ela avança. + +## Confiabilidade + +- **Retry:** uma tentativa com falha é repetida conforme a política `retry` da Etapa (veja [Etapas](stages.md#retry)). A + execução aguarda a espera sem ocupar um worker. +- **Workers perdidos:** se um worker para de renovar o lease (caiu, ou a máquina sumiu), o passo é recuperado. + - Uma Etapa marcada como `idempotent`, ou que ainda tem retries, é retomada por outro worker. + - Nos outros casos, o passo falha com `WorkerLost`, para reprocesso manual. +- **Duplicatas:** uma mensagem entregue duas vezes nunca executa um passo duas vezes. +- **Mensagens problemáticas:** depois de `DeliveryLimit` entregas com falha (5), um comando vai para a fila + `psr.dead-letter`. +- **Desligamento gradual:** um worker que está parando não pega passos novos e espera até `ShutdownGracePeriod` (5 + minutos) pelos que estão rodando. + +## Concorrência e cancelamento + +- **Por worker:** o `MaxConcurrency` limita quantos passos um worker executa ao mesmo tempo. O padrão é o número de + processadores. +- **Por trilha:** dá para definir um máximo de execuções simultâneas. As execuções além do limite esperam como + `Pending`, em ordem. +- **Cancelamento:** o `IRunManager.CancelAsync` encerra a árvore de processos no worker que executa o passo, onde quer + que ele esteja. + +## Agendamento + +Os agendamentos executam uma trilha a partir de uma expressão cron, num fuso IANA: + +```csharp +var admin = app.Services.GetRequiredService(); +await admin.CreateScheduleAsync("calculo-progressao", new ScheduleData +{ + Name = "Todo dia útil às 2h", + Cron = "0 2 * * 1-5", // 5 campos, ou 6 com segundos + TimeZone = "America/Sao_Paulo", // o padrão; o horário de verão é tratado + Inputs = new JsonObject { ["a1"] = 1, ["razao"] = 2, ["n"] = 4, ["coeficientes"] = new JsonArray(1, 0, 1) }, +}, Actor.User("ana@exemplo.com")); +``` + +Como os agendamentos se comportam: +- **Versão:** sem `PinnedVersion`, cada disparo executa a última versão publicada. +- **Disparo único:** vários workers nunca disparam a mesma ocorrência. +- **Ocorrências perdidas:** se foram perdidas enquanto todos os workers estavam fora do ar, viram uma única execução de + recuperação. +- **Entradas de arquivo:** uma trilha com entrada `file` obrigatória não pode ser agendada. +- **Opções do agendador:** ele roda dentro dos workers, em `Scheduler` nas opções do worker (`Enabled`, `PollInterval`, + `BatchSize`). + +## Eventos + +Toda execução publica eventos na exchange tópica `psr.events`: + +| Evento | Quando | +|--------|--------| +| `run.queued`, `run.rejected`, `run.started` | A execução foi aceita, recusada (por exemplo, entradas inválidas) ou iniciada. | +| `step.started`, `step.succeeded`, `step.failed`, `step.retry-scheduled` | O andamento de cada passo. | +| `run.succeeded`, `run.failed`, `run.cancelled` | A execução terminou. O `run.succeeded` leva as saídas. | + +A routing key é `.` (ou `.stage:` nas execuções de uma Etapa só), então +dá para assinar exatamente o que você precisa. Dentro do .NET, trate os eventos com `IRunEventHandler`: + +```csharp +builder.Services.AddPythonScriptRunner() + .AddRabbitMq(o => o.ConnectionString = builder.Configuration.GetConnectionString("rabbitmq")) + .AddRunEventHandler(s => s + .Subscription("erp") // uma fila durável compartilhada por todas as instâncias + .On("run.succeeded.calculo-progressao", "run.failed.*")); + +public sealed class NotificarErp : IRunEventHandler +{ + public Task HandleAsync(RunEvent runEvent, CancellationToken cancellationToken) + { + // runEvent.Type, runEvent.RunId, runEvent.Outputs, runEvent.Error, ... + return Task.CompletedTask; + } +} +``` + +Lançar uma exceção no `HandleAsync` faz o broker entregar o evento de novo, até o limite de entregas. Use +`PerInstance(prefixo)` no lugar de `Subscription(nome)` quando cada instância precisar receber todos os eventos. + +## Sistemas escritos em outras linguagens + +Qualquer sistema que fale com o RabbitMQ pode iniciar trilhas e acompanhá-las, sem .NET: + +- **Iniciar uma execução:** + - publique uma mensagem JSON `StartPipelineRun` na exchange `psr.public`, com a routing key `pipeline.start`; + - campos: `pipelineKey` (obrigatório), `version`, `inputs`, `idempotencyKey`, `correlationId` e `requestedBy`; + - entradas de arquivo não são aceitas por esse comando. +- **Acompanhar:** ligue uma fila sua à exchange `psr.events` com as routing keys que quiser, por exemplo + `run.*.calculo-progressao`. O `correlationId` que você enviou volta em todos os eventos. + +Os contratos das mensagens são JSON Schemas em [`schemas/messages`](../../schemas/messages): +`start-pipeline-run.v1.json` e `run-event.v1.json`. + +## Observabilidade + +O `traceparent` acompanha a execução de quem chamou até o worker, então um único trace distribuído mostra a requisição, +a fila e cada processo Python. Veja [Como funciona](how-it-works.md#observabilidade). diff --git a/guides/pt-br/stages.md b/guides/pt-br/stages.md new file mode 100644 index 0000000..c44f5bc --- /dev/null +++ b/guides/pt-br/stages.md @@ -0,0 +1,198 @@ +# Etapas + +[English](../en/stages.md) | Português + +Uma Etapa é um script Python e o seu contrato. Esta página cobre o contrato, todas as opções do `stage.json`, a definição +de Etapas em C# e a leitura dos resultados no .NET. Para o lado do script, veja o [SDK Python](python-sdk.md). + +## Estrutura de pastas + +Com o catálogo em pasta (`AddFileCatalog("psr")`), cada Etapa é uma pasta dentro de `stages/`: + +```text +psr/ + stages/ + media/ + stage.json o contrato e as configurações + script.py o código (uma função marcada com @stage) + requirements.txt opcional: pacotes Python, um por linha + pipelines/ + relatorio.pipeline.json +``` + +Com a persistência ligada, o catálogo passa a ficar no PostgreSQL, e a mesma pasta pode ser importada como zip (veja +[Persistência](persistence.md)). + +## stage.json + +```json +{ + "$schema": "https://raw.githubusercontent.com/ScriptRunnerKit/PythonScriptRunner/main/schemas/stage.v1.json", + "key": "polinomio", + "version": 1, + "name": "Avaliar um polinômio", + "description": "a + b·x + c·x² + … para cada x", + "python": "3.12", + "timeoutSeconds": 120, + "memoryLimitMb": 2048, + "retry": { "maxAttempts": 2, "initialDelaySeconds": 10, "backoffMultiplier": 2 }, + "idempotent": true, + "secrets": ["SMTP_SENHA"], + "inputs": [ + { "name": "coeficientes", "type": "list", "itemType": "number" }, + { "name": "x", "type": "list", "itemType": "number" } + ], + "outputs": [ + { "name": "valores", "type": "list", "itemType": "number" } + ] +} +``` + +A linha `$schema` dá autocompletar e validação em editores como o VS Code. + +| Propriedade | Obrigatória | Significado | +|-------------|-------------|-------------| +| `key` | sim | Identificador único: letras minúsculas e dígitos, com palavras separadas por um hífen (`relatorio-mensal`). É usado para executar a Etapa. | +| `version` | sim | Versão inteira. As trilhas referenciam `chave@versão`. | +| `name` | sim | Nome de exibição. | +| `description` | não | Texto livre exibido no painel. | +| `python` | não | Versão do Python (3.10 ou mais nova). Padrão: `DefaultPythonVersion`, que é 3.12. | +| `script` | não | Arquivo do script, relativo à pasta. Padrão: `script.py`. | +| `requirements` | não | Arquivo de requisitos, relativo à pasta. Padrão: `requirements.txt`, se existir. | +| `timeoutSeconds` | não | Tempo limite de uma tentativa. Padrão: `DefaultTimeout`, que é 5 minutos. | +| `memoryLimitMb` | não | Limite de memória de uma tentativa. Padrão: `DefaultMemoryLimitMb`, que é 1024. | +| `retry` | não | Nova tentativa automática depois de uma falha. Veja [Retry](#retry). | +| `idempotent` | não | `true` quando executar a Etapa duas vezes tem o mesmo efeito de uma. Permite que um worker a retome depois que outro worker caiu. Padrão: `false`. | +| `secrets` | não | Nomes dos segredos que o script recebe (maiúsculas, dígitos e `_`). Veja [Segredos](#segredos). | +| `inputs`, `outputs` | sim | Os campos do contrato. As listas podem ser vazias. | + +## Tipos de campo + +Cada entrada e saída tem um `name` e um `type`. A biblioteca converte os valores dos dois lados: + +| Tipo | No .NET (entradas e `GetOutput`) | No Python (`inputs[...]` e valor de retorno) | +|------|--------------------------------------|---------------------------------------------| +| `number` | `double` | `float` (`NaN` e infinito são rejeitados) | +| `integer` | `int`, `long` | `int` | +| `decimal` | `decimal` | `decimal.Decimal` (exato, enviado como texto) | +| `string` | `string` | `str` | +| `bool` | `bool` | `bool` | +| `date` | `DateOnly` | `datetime.date` | +| `datetime` | `DateTimeOffset` | `datetime.datetime` com fuso UTC (valores sem fuso são rejeitados) | +| `enum` | `string` de `enumValues` | `str` de `enumValues` | +| `json` | qualquer valor JSON (`JsonNode`, ou o seu próprio tipo) | `dict`, `list`, `str`, número, `bool` ou `None` | +| `table` | uma lista de registros (o seu próprio tipo de registro, ou `JsonArray`) | `list[dict]`, ou `pandas.DataFrame` com o extra `pandas` | +| `file` | `PsrFile` | `psrunner.File` (entrada); `ctx.new_file(...)` (saída) | +| `list` | arrays e listas do tipo do item | `list` do tipo do item | + +Propriedades de um campo: + +| Propriedade | Significado | +|-------------|-------------| +| `name` | Nome do campo: letras minúsculas, dígitos e `_`, começando por letra ou `_`. | +| `type` | Um dos tipos acima. | +| `itemType` | Tipo do item de uma `list` (qualquer tipo, exceto `list`). | +| `required` | Padrão `true`. Uma entrada opcional pode ser omitida; uma saída opcional pode faltar. | +| `default` | Só entradas: o valor usado quando a entrada não é informada. | +| `description` | Exibida no painel e no formulário de disparo gerado. | +| `enumValues` | Valores permitidos de um `enum` (ou de uma `list` de `enum`). | +| `contentTypes` | Tipos MIME permitidos de um `file`, por exemplo `["application/pdf"]`. | +| `columns` | Esquema opcional de uma `table`: `[{ "name": "x", "type": "number" }]`. Cada coluna tem um tipo escalar. | + +## Pacotes Python + +Coloque um `requirements.txt` ao lado do script, com um pacote por linha (`numpy`, `pandas>=2.2`). O runner cria, com o +uv, um ambiente por versão do Python e conjunto de requisitos, guarda em cache e o reaproveita em todas as Etapas com o +mesmo conjunto. Com a persistência, publicar uma Etapa trava os requisitos com hash (`uv pip compile --generate-hashes`), +então uma versão publicada sempre roda com as mesmas versões dos pacotes. + +O SDK `psrunner` é instalado em todos os ambientes automaticamente. Não o coloque no `requirements.txt`. + +## Definindo uma Etapa em C# + +O mesmo contrato pode ser declarado em código, sem arquivos: + +```csharp +builder.Services.AddPythonScriptRunner() + .AddStage("media", s => s + .Script(""" + from psrunner import stage + + @stage + def run(inputs, ctx): + v = inputs["valores"] + return {"media": sum(v) / len(v)} + """) + .Python("3.12") + .Requirements("numpy") + .Input("valores", FieldType.ListOf(FieldType.Number)) + .Output("media", FieldType.Number) + .Timeout(TimeSpan.FromMinutes(1)) + .MemoryLimitMb(512)); +``` + +O `FieldType` tem `Number`, `Integer`, `Decimal`, `String`, `Bool`, `Date`, `DateTime`, `Json`, `Table`, `File`, +`Enum("a", "b")` e `ListOf(tipoDoItem)`. Uma definição inválida falha na inicialização com `PsrConfigurationException`. + +## Executando uma Etapa e lendo o resultado + +```csharp +await using var resultado = await runner.RunStageAsync("media", new { valores = new[] { 2.0, 4.0, 9.0 } }); + +if (resultado.Status == RunStatus.Succeeded) +{ + double media = resultado.GetOutput("media"); +} +else +{ + Console.WriteLine($"{resultado.Error!.Kind}: {resultado.Error.Message}"); +} +``` + +- **Entradas:** um objeto anônimo, um dicionário ou qualquer objeto que serialize para o JSON esperado. Arquivos são + passados como `PsrFile`: `PsrFile.FromPath(...)`, `FromStream`, `FromBytes` ou `FromText`. +- **Antes da execução:** entradas fora do contrato lançam `InputValidationException`, e nenhuma execução é criada. +- **Depois da execução:** falhas aparecem em `resultado.Status` e `resultado.Error`. `resultado.EnsureSuccess()` lança + uma exceção se a execução não teve sucesso. +- **Leitura das saídas:** + - `GetOutput(nome)` lê um valor; + - `GetFile(nome)` e `GetFiles(nome)` devolvem objetos `PsrFile` (`OpenReadAsync`, `SaveAsAsync`). +- **Descarte:** o `RunResult` é `IAsyncDisposable`. Descartá-lo apaga os arquivos de saída locais, então salve antes os + que precisar (`SaveAsAsync`). +- **Opções:** o `RunOptions` escolhe uma `Version` específica, define uma `IdempotencyKey` (com persistência) e + registra quem iniciou a execução. + +Tipos de erro em `resultado.Error.Kind`: + +| Tipo | Significado | +|------|-------------| +| `InputValidation` | Uma entrada não correspondeu ao contrato. | +| `OutputContractViolation` | O script devolveu algo fora do contrato (saída faltando, tipo errado, `NaN`). | +| `ScriptError` | O script lançou uma exceção, ou um `StageError` com um `code`. | +| `Timeout` | A tentativa passou do tempo limite. | +| `MemoryLimitExceeded` | A tentativa passou do limite de memória. | +| `Cancelled` | A execução foi cancelada. | +| `WorkerLost` | O worker que executava o passo parou de responder (execuções enfileiradas). | +| `EnvironmentSetupFailed` | O uv não conseguiu criar o ambiente (pacote inexistente, sem rede, versão do Python desconhecida). | +| `ArtifactLimitExceeded` | Os arquivos passaram de `MaxFileBytes` ou `MaxRunFilesBytes`. | +| `SecretNotFound` | Um segredo declarado não está configurado. O Python nem chega a iniciar. | +| `ProcessCrashed` | O processo Python não iniciou ou terminou sem um resultado. | +| `Expired` | A execução atingiu o limite de retenção antes de terminar. | + +## Retry + +O `retry` faz o runner tentar de novo depois de uma tentativa com falha: + +- **`maxAttempts`:** quantas tentativas extras são permitidas (de 0 a 20). O padrão, 0, significa sem retry automático. +- **Espera antes de cada nova tentativa:** `initialDelaySeconds × backoffMultiplier^(n-1)`, com no máximo 10 minutos. Os + padrões são 10 segundos e 2. + +Uma execução cancelada nunca é repetida. As execuções enfileiradas aguardam a espera sem ocupar um worker. + +## Segredos + +A Etapa declara os nomes dos segredos de que precisa. Os valores vêm da configuração, na seção +`PythonScriptRunner:Secrets` (por exemplo, a variável de ambiente `PythonScriptRunner__Secrets__SMTP_SENHA`), ou de um +`ISecretProvider` seu. Os valores nunca são gravados nas entradas, nos logs, nas métricas nem no banco de dados. Só o +processo daquela Etapa os recebe, e eles aparecem mascarados (`***`) na saída capturada. No Python, leia com +`ctx.secrets["SMTP_SENHA"]`. diff --git a/python/psrunner/README.md b/python/psrunner/README.md index 4223301..4d3bab1 100644 --- a/python/psrunner/README.md +++ b/python/psrunner/README.md @@ -1,31 +1,155 @@ # psrunner -Python SDK for [PythonScriptRunner](https://github.com/ScriptRunnerKit/PythonScriptRunner) stages. +[![PyPI](https://img.shields.io/pypi/v/psrunner?label=PyPI)](https://pypi.org/project/psrunner/) +[![Python](https://img.shields.io/pypi/pyversions/psrunner)](https://pypi.org/project/psrunner/) +[![License: MIT](https://img.shields.io/badge/license-MIT-blue)](https://github.com/ScriptRunnerKit/PythonScriptRunner/blob/main/LICENSE) -A stage is one Python script with typed inputs and outputs. The .NET runner validates the inputs, -runs the script in an isolated process and validates the outputs. `psrunner` handles the protocol, -so the script only contains the logic: +The Python SDK of [PythonScriptRunner](https://github.com/ScriptRunnerKit/PythonScriptRunner), a .NET library that runs +Python scripts with a typed contract. + +A **stage** is one Python script with typed inputs and outputs. The .NET runner validates the inputs, runs the script in +an isolated process with a timeout and a memory limit, and validates the outputs. `psrunner` handles the protocol between +the two sides, so the script only contains the logic. + +**Documentation:** [Python SDK guide](https://github.com/ScriptRunnerKit/PythonScriptRunner/blob/main/guides/en/python-sdk.md) | +[guia em português](https://github.com/ScriptRunnerKit/PythonScriptRunner/blob/main/guides/pt-br/python-sdk.md) + +## Installation + +The .NET runner installs `psrunner` in every environment it creates, so stages never list it in their requirements. +Install it yourself to write and test scripts locally: + +```bash +pip install psrunner # Python 3.10 or newer +pip install "psrunner[pandas]" # table inputs arrive as pandas.DataFrame +``` + +## Writing a stage ```python -from psrunner import stage +from psrunner import stage, StageError @stage def run(inputs, ctx): a1, ratio, n = inputs["a1"], inputs["ratio"], inputs["n"] + if n < 1: + raise StageError("n must be at least 1.", code="INVALID_N") + terms = [a1 * ratio**i for i in range(n)] ctx.log.info("generated %d terms", n) return {"terms": terms} ``` -Test it locally with pytest, without .NET: +The contract lives in `stage.json`, next to the script: + +```json +{ + "key": "progression-terms", + "version": 1, + "name": "Progression terms", + "inputs": [ + { "name": "a1", "type": "number" }, + { "name": "ratio", "type": "number" }, + { "name": "n", "type": "integer" } + ], + "outputs": [ { "name": "terms", "type": "list", "itemType": "number" } ] +} +``` + +The rules: +- **One entry point:** exactly one function per script is marked with `@stage`. It receives the inputs, already + converted, and the context. +- **Return value:** it returns a dict with the outputs. Required outputs must be present, and outputs outside the contract + are an error. + +## Types + +| Contract type | Python value | +|---------------|--------------| +| `number` | `float` (`NaN` and infinity are rejected) | +| `integer` | `int` | +| `decimal` | `decimal.Decimal` | +| `string` | `str` | +| `bool` | `bool` | +| `date` | `datetime.date` | +| `datetime` | `datetime.datetime` with a UTC offset | +| `enum` | `str`, one of `enumValues` | +| `json` | `dict`, `list`, `str`, number, `bool` or `None` | +| `table` | `list[dict]`, or `pandas.DataFrame` with the `pandas` extra | +| `file` | `psrunner.File` | +| `list` | `list` of the item type | + +## The context + +| Member | Description | +|--------|-------------| +| `ctx.log` | A `logging.Logger`; its lines appear in the run's logs. | +| `ctx.secrets` | Read-only mapping with the secrets the stage declares. | +| `ctx.new_file(name, content_type=None)` | Reserves an output file and returns a `File`; write to `file.path` and return it. | +| `ctx.run_id`, `ctx.step_key`, `ctx.attempt` | Identify the run, the pipeline step and the attempt. | +| `ctx.stage_key`, `ctx.stage_version`, `ctx.is_test` | The stage being run, and whether this is a test of a draft. | +| `ctx.work_dir`, `ctx.out_dir`, `ctx.tmp_dir` | The attempt's own folders. | + +A `File` has `path`, `name`, `content_type`, `size` and `sha256`, and the methods `read_bytes()`, `read_text()` and +`open()`. + +```python +@stage +def run(inputs, ctx): + report = ctx.new_file("report.csv", "text/csv") + report.path.write_text("x,y\n1,2\n", encoding="utf-8") + return {"report": report} +``` + +## Errors + +- **Business errors:** `StageError(message, code="CODE", details={...})` fails the run as `ScriptError` with that code. +- **Unexpected exceptions:** any other exception also becomes `ScriptError`, with the traceback of your script. +- **Contract violations:** a missing output, a wrong type or a non-finite number becomes `OutputContractViolation`. + +## Testing with pytest + +`psrunner.testing.run_stage` runs the function with the same conversions and validations as the real runner, without +.NET: ```python from psrunner.testing import run_stage def test_terms(): - result = run_stage("stages/terms/script.py", {"a1": 1, "ratio": 2, "n": 4}, - stage_json="stages/terms/stage.json") + result = run_stage( + "stages/progression-terms/script.py", + {"a1": 1, "ratio": 2, "n": 4}, + stage_json="stages/progression-terms/stage.json", + ) + assert result.ok assert result.outputs == {"terms": [1, 2, 4, 8]} ``` -Install the `pandas` extra to receive `table` inputs as `pandas.DataFrame`: `pip install psrunner[pandas]`. +`run_stage` also accepts `secrets`, `files` (file fields mapped to local paths), `work_dir` and `run_id`. The result has +`ok`, `outputs`, `wire_outputs`, `error` (`kind`, `code`, `message`, `traceback`), `logs` and `work_dir`. + +## Command line + +```bash +psrunner validate stages/progression-terms +psrunner run stages/progression-terms --input sample.json --secret NAME=value --keep +``` + +| Command | What it does | +|---------|--------------| +| `validate ` | Checks `stage.json` and that the script has exactly one `@stage` function. | +| `run ` | Runs the stage with the real protocol, in a separate process and a temporary folder. In `sample.json`, file inputs are written as `{"$path": "relative/path"}`. | + +The same commands work as `python -m psrunner`. + +## Compatibility + +`psrunner` follows protocol 1, the format exchanged with the .NET runner, for as long as that protocol is in use. + +| psrunner | Protocol | PythonScriptRunner (NuGet) | Python | +|----------|----------|----------------------------|--------| +| 0.1.x | 1 | 0.1.x | 3.10 to 3.13 | + +## License + +[MIT](https://github.com/ScriptRunnerKit/PythonScriptRunner/blob/main/LICENSE)