Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 7 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -41,8 +41,13 @@ jobs:
- name: Build
run: dotnet build --no-restore --configuration Release

# The persistence tests start PostgreSQL 16 with Testcontainers (Docker); GitHub's Windows runners
# cannot run Linux containers, so Windows runs the Core tests only.
# weasyprint (CS-2 fixture, html-para-pdf) renders with pango.
- name: Install pango
if: runner.os == 'Linux'
run: sudo apt-get update && sudo apt-get install -y --no-install-recommends libpango-1.0-0 libpangoft2-1.0-0 libharfbuzz0b fonts-dejavu-core

# The persistence, queue and worker tests start PostgreSQL 16, RabbitMQ 3.13 and Mailpit with Testcontainers
# (Docker); GitHub's Windows runners cannot run Linux containers, so Windows runs the Core tests only.
- name: Test
if: runner.os == 'Linux'
run: dotnet test --no-build --configuration Release
Expand Down
11 changes: 11 additions & 0 deletions Directory.Packages.props
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,9 @@
<ManagePackageVersionsCentrally>true</ManagePackageVersionsCentrally>
</PropertyGroup>
<ItemGroup>
<PackageVersion Include="Cronos" Version="0.13.0" />
<PackageVersion Include="JsonSchema.Net" Version="9.4.1" />
<PackageVersion Include="Microsoft.AspNetCore.Mvc.Testing" Version="10.0.13" />
<PackageVersion Include="Microsoft.EntityFrameworkCore.Design" Version="10.0.13" />
<PackageVersion Include="Microsoft.EntityFrameworkCore.Relational" Version="10.0.13" />
<PackageVersion Include="Microsoft.Extensions.Configuration" Version="10.0.13" />
Expand All @@ -13,13 +15,22 @@
<PackageVersion Include="Microsoft.Extensions.Diagnostics" Version="10.0.13" />
<PackageVersion Include="Microsoft.Extensions.Diagnostics.HealthChecks" Version="10.0.13" />
<PackageVersion Include="Microsoft.Extensions.Diagnostics.Testing" Version="10.11.0" />
<PackageVersion Include="Microsoft.Extensions.Hosting" Version="10.0.13" />
<PackageVersion Include="Microsoft.Extensions.Hosting.Abstractions" Version="10.0.13" />
<PackageVersion Include="Microsoft.Extensions.Logging" Version="10.0.13" />
<PackageVersion Include="Microsoft.Extensions.Options" Version="10.0.13" />
<PackageVersion Include="Microsoft.Extensions.TimeProvider.Testing" Version="10.11.0" />
<PackageVersion Include="Npgsql.EntityFrameworkCore.PostgreSQL" Version="10.0.3" />
<PackageVersion Include="OpenTelemetry.Exporter.OpenTelemetryProtocol" Version="1.20.1" />
<PackageVersion Include="OpenTelemetry.Extensions.Hosting" Version="1.20.1" />
<PackageVersion Include="OpenTelemetry.Instrumentation.AspNetCore" Version="1.20.0" />
<PackageVersion Include="OpenTelemetry.Instrumentation.Runtime" Version="1.20.0" />
<PackageVersion Include="RabbitMQ.Client" Version="7.2.2" />
<PackageVersion Include="Serilog.AspNetCore" Version="10.0.0" />
<PackageVersion Include="Serilog.Formatting.Compact" Version="3.0.0" />
<PackageVersion Include="Shouldly" Version="4.3.0" />
<PackageVersion Include="Testcontainers.PostgreSql" Version="4.16.0" />
<PackageVersion Include="Testcontainers.RabbitMq" Version="4.16.0" />
<PackageVersion Include="xunit.v3" Version="4.0.2" />
</ItemGroup>
</Project>
8 changes: 8 additions & 0 deletions PythonScriptRunner.slnx
Original file line number Diff line number Diff line change
@@ -1,11 +1,19 @@
<Solution>
<Folder Name="/hosts/">
<Project Path="hosts/PythonScriptRunner.ServiceDefaults/PythonScriptRunner.ServiceDefaults.csproj" />
<Project Path="hosts/PythonScriptRunner.WorkerHost/PythonScriptRunner.WorkerHost.csproj" />
</Folder>
<Folder Name="/src/">
<Project Path="src/PythonScriptRunner.Abstractions/PythonScriptRunner.Abstractions.csproj" />
<Project Path="src/PythonScriptRunner.Core/PythonScriptRunner.Core.csproj" />
<Project Path="src/PythonScriptRunner.EntityFrameworkCore/PythonScriptRunner.EntityFrameworkCore.csproj" />
<Project Path="src/PythonScriptRunner.RabbitMQ/PythonScriptRunner.RabbitMQ.csproj" />
<Project Path="src/PythonScriptRunner.Worker/PythonScriptRunner.Worker.csproj" />
</Folder>
<Folder Name="/tests/">
<Project Path="tests/PythonScriptRunner.Core.Tests/PythonScriptRunner.Core.Tests.csproj" />
<Project Path="tests/PythonScriptRunner.EntityFrameworkCore.Tests/PythonScriptRunner.EntityFrameworkCore.Tests.csproj" />
<Project Path="tests/PythonScriptRunner.RabbitMQ.Tests/PythonScriptRunner.RabbitMQ.Tests.csproj" />
<Project Path="tests/PythonScriptRunner.Worker.Tests/PythonScriptRunner.Worker.Tests.csproj" />
</Folder>
</Solution>
48 changes: 44 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,8 @@ com contrato de entrada/saída, trilhas (**Pipelines**), execução assíncrona
cron e painel Vue opcional.

> Projeto em desenvolvimento. Esta versão entrega o **Core Runner** (execução síncrona de Stages e de
> trilhas), o SDK Python **`psrunner`** e a **persistência em PostgreSQL** (catálogo versionado e histórico).
> trilhas), o SDK Python **`psrunner`**, a **persistência em PostgreSQL** (catálogo versionado e histórico) e a
> **execução assíncrona** com RabbitMQ, worker e agendamento cron.

## Como funciona

Expand Down Expand Up @@ -120,21 +121,58 @@ await app.Services.MigratePythonScriptRunnerAsync(); // explícito; ou Generat
um job de manutenção cria partições à frente e remove as antigas. Uma trilha pode encurtar a retenção.
- **Auditoria:** ações humanas (`Actor.User(...)`) ficam em `psr.audit_events`; disparos automáticos ficam na própria execução.

## Fila, worker e agendamento (RabbitMQ)

Com `PythonScriptRunner.RabbitMQ` e `PythonScriptRunner.Worker`, execuções podem ser enfileiradas. Cada passo é uma
mensagem: o worker assume o passo com um *lease*, executa o processo Python e grava o resultado e a mensagem do próximo
passo na mesma transação (outbox no PostgreSQL). Enfileirar funciona mesmo com o broker fora do ar.

```csharp
builder.Services
.AddPythonScriptRunner()
.AddEntityFrameworkCore(o => o.UseNpgsql(builder.Configuration.GetConnectionString("psr")!))
.AddRabbitMq(o => o.ConnectionString = builder.Configuration.GetConnectionString("rabbitmq")) // RabbitMQ 3.13+
.AddWorker(o => o.MaxConcurrency = 8) // só nos processos que executam passos
.AddRunEventHandler<NotificarErp>(s => s.Subscription("erp").On("run.succeeded.calculo-progressao", "run.failed.*"));
builder.Services.AddHealthChecks().AddPythonScriptRunnerChecks(); // banco, broker e uv

// ...
RunHandle run = await queue.EnqueuePipelineAsync("calculo-progressao", new { a1 = 1, razao = 2, n = 4, coeficientes = new[] { 1, 0, 1 } });
```

- **Confiabilidade:** retry automático por Stage com atraso exponencial; worker perdido é detectado pelo *lease* e o
passo é retomado (Stage idempotente ou com retry restante) ou falha com `WorkerLost` para reprocesso manual;
mensagens duplicadas não executam o passo duas vezes.
- **Concorrência:** `MaxConcurrency` por worker e `max_concurrent_runs` por trilha (as demais esperam como `Pending`, em ordem).
- **Cancelamento:** `IRunManager.CancelAsync` encerra a árvore de processos no worker que executa o passo.
- **Agendamento:** `ICatalogAdmin.CreateScheduleAsync` com expressão cron e fuso IANA (padrão `America/Sao_Paulo`);
ocorrências perdidas viram uma execução de recuperação; vários workers nunca disparam a mesma ocorrência.
- **Sistemas externos:** o comando público `StartPipelineRun` (exchange `psr.public`) inicia trilhas e os eventos
`run.*`/`step.*` (exchange `psr.events`) informam o andamento e as saídas. Contratos em `schemas/messages`.
- **Observabilidade:** `ActivitySource` e `Meter` `PythonScriptRunner`; o `traceparent` acompanha a execução até o worker.

O host `hosts/PythonScriptRunner.WorkerHost` é um worker pronto, configurado por variáveis de ambiente
(`ConnectionStrings__psr`, `ConnectionStrings__rabbitmq`, `PythonScriptRunner__Worker__MaxConcurrency`,
`PythonScriptRunner__Secrets__<NOME>`, `PSR_AUTO_MIGRATE=true`), com `/health/live`, `/health/ready`, logs JSON e OTLP.

## Requisitos

- .NET 10 SDK
- [uv](https://docs.astral.sh/uv/) no PATH (cria e cacheia um ambiente Python por conjunto de dependências)
- PostgreSQL 16+ apenas para o pacote de persistência (os testes usam Docker/Testcontainers)
- PostgreSQL 16+ apenas para o pacote de persistência e RabbitMQ 3.13+ para a fila (os testes usam Docker/Testcontainers)

## Estrutura

| Pasta | Conteúdo |
|-------|----------|
| `src/PythonScriptRunner.Abstractions` | contratos públicos (tipos de campo, `StageDefinition`, `RunResult`, `PsrFile`, interfaces) |
| `src/PythonScriptRunner.Core` | catálogos (código e pasta), validação, ambientes uv, executor de processos, runner |
| `src/PythonScriptRunner.EntityFrameworkCore` | PostgreSQL: catálogo versionado, histórico, arquivos, idempotência, outbox, auditoria, retenção |
| `src/PythonScriptRunner.EntityFrameworkCore` | PostgreSQL: catálogo versionado, histórico, arquivos, idempotência, outbox, auditoria, retenção, agendamentos e ciclo das execuções enfileiradas |
| `src/PythonScriptRunner.RabbitMQ` | topologia, relay do outbox, `IRunQueue`, comando público, eventos e assinaturas |
| `src/PythonScriptRunner.Worker` | execução dos passos enfileirados, *lease*, retry, cancelamento, cron |
| `hosts` | `WorkerHost` e `ServiceDefaults` (Serilog, OpenTelemetry, health checks) |
| `python/psrunner` | SDK Python (`@stage`, `Context`, `File`, `StageError`, `psrunner.testing`, CLI) |
| `schemas` | JSON Schemas do `stage.json` e do `*.pipeline.json` |
| `schemas` | JSON Schemas do `stage.json`, do `*.pipeline.json` e das mensagens públicas |
| `tests` | testes .NET (unitários e de integração com Python real) e fixtures de trilha |

## Desenvolvimento
Expand All @@ -143,6 +181,8 @@ await app.Services.MigratePythonScriptRunnerAsync(); // explícito; ou Generat
dotnet build
dotnet test --project tests/PythonScriptRunner.Core.Tests
dotnet test --project tests/PythonScriptRunner.EntityFrameworkCore.Tests # precisa de Docker
dotnet test --project tests/PythonScriptRunner.RabbitMQ.Tests # precisa de Docker
dotnet test --project tests/PythonScriptRunner.Worker.Tests # precisa de Docker

cd python/psrunner
uv sync
Expand Down
11 changes: 11 additions & 0 deletions hosts/Directory.Build.props
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
<Project>
<Import Project="$([MSBuild]::GetPathOfFileAbove('Directory.Build.props', '$(MSBuildThisFileDirectory)../'))" />

<PropertyGroup>
<!-- Hosts are deployed as containers, not published as packages. -->
<IsPackable>false</IsPackable>
<!-- Required by IDE0005 (unused usings) during build; host members are not documented. -->
<GenerateDocumentationFile>true</GenerateDocumentationFile>
<NoWarn>$(NoWarn);CS1591</NoWarn>
</PropertyGroup>
</Project>
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
<Project Sdk="Microsoft.NET.Sdk">

<PropertyGroup>
<RootNamespace>PythonScriptRunner.ServiceDefaults</RootNamespace>
</PropertyGroup>

<ItemGroup>
<FrameworkReference Include="Microsoft.AspNetCore.App" />
</ItemGroup>

<ItemGroup>
<PackageReference Include="OpenTelemetry.Exporter.OpenTelemetryProtocol" />
<PackageReference Include="OpenTelemetry.Extensions.Hosting" />
<PackageReference Include="OpenTelemetry.Instrumentation.AspNetCore" />
<PackageReference Include="OpenTelemetry.Instrumentation.Runtime" />
<PackageReference Include="Serilog.AspNetCore" />
<PackageReference Include="Serilog.Formatting.Compact" />
</ItemGroup>

</Project>
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Diagnostics.HealthChecks;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Diagnostics.HealthChecks;
using OpenTelemetry;
using OpenTelemetry.Metrics;
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;
using Serilog;
using Serilog.Formatting.Compact;

namespace Microsoft.Extensions.Hosting;

/// <summary>
/// Defaults of the standalone hosts (spec 009; D-54): Serilog writing JSON to the console, OpenTelemetry traces and metrics of
/// the <c>PythonScriptRunner</c> source/meter (exported with OTLP when <c>OTEL_EXPORTER_OTLP_ENDPOINT</c> is set) and the
/// <c>/health/live</c> and <c>/health/ready</c> endpoints.
/// </summary>
public static class ServiceDefaultsExtensions
{
/// <summary>Tag of the checks behind <c>/health/live</c>.</summary>
public const string LiveTag = "live";

/// <summary>Tag of the checks behind <c>/health/ready</c>.</summary>
public const string ReadyTag = "ready";

/// <summary>Adds logging, telemetry and the liveness check.</summary>
/// <param name="builder">Host builder.</param>
/// <param name="serviceName">Service name in logs and telemetry.</param>
public static TBuilder AddServiceDefaults<TBuilder>(this TBuilder builder, string serviceName)
where TBuilder : IHostApplicationBuilder
{
ArgumentNullException.ThrowIfNull(builder);
builder.Services.AddSerilog((services, logger) => logger
.ReadFrom.Configuration(builder.Configuration)
.Enrich.FromLogContext()
.Enrich.WithProperty("service", serviceName)
.WriteTo.Console(new RenderedCompactJsonFormatter()));

var telemetry = builder.Services.AddOpenTelemetry()
.ConfigureResource(resource => resource.AddService(serviceName))
.WithTracing(tracing => tracing.AddSource("PythonScriptRunner").AddAspNetCoreInstrumentation())
.WithMetrics(metrics => metrics.AddMeter("PythonScriptRunner").AddAspNetCoreInstrumentation().AddRuntimeInstrumentation());
if (!string.IsNullOrWhiteSpace(builder.Configuration["OTEL_EXPORTER_OTLP_ENDPOINT"]))
{
telemetry.UseOtlpExporter();
}

builder.Services.AddHealthChecks().AddCheck("self", () => HealthCheckResult.Healthy(), [LiveTag]);
return builder;
}

/// <summary>
/// Maps <c>/health/live</c> (the process answers) and <c>/health/ready</c> (checks tagged <see cref="ReadyTag"/>: 200 when
/// healthy or degraded, 503 when unhealthy).
/// </summary>
/// <param name="app">Application.</param>
public static WebApplication MapDefaultEndpoints(this WebApplication app)
{
ArgumentNullException.ThrowIfNull(app);
app.MapHealthChecks("/health/live", new HealthCheckOptions { Predicate = check => check.Tags.Contains(LiveTag) });
app.MapHealthChecks("/health/ready", new HealthCheckOptions { Predicate = check => check.Tags.Contains(ReadyTag) });
return app;
}
}
36 changes: 36 additions & 0 deletions hosts/PythonScriptRunner.WorkerHost/Program.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
// Standalone worker of PythonScriptRunner (spec 009): runs queued steps, cron schedules and the maintenance job.
// Configuration comes from environment variables, e.g. ConnectionStrings__psr, ConnectionStrings__rabbitmq,
// PythonScriptRunner__Worker__MaxConcurrency, PythonScriptRunner__Secrets__<NAME> and PSR_AUTO_MIGRATE.
var builder = WebApplication.CreateBuilder(args);
builder.AddServiceDefaults("psr-worker");

var configuration = builder.Configuration;
builder.Services
.AddPythonScriptRunner(o => configuration.GetSection("PythonScriptRunner").Bind(o))
.AddEntityFrameworkCore(o =>
{
configuration.GetSection("PythonScriptRunner:EntityFrameworkCore").Bind(o);
o.UseNpgsql(configuration.GetConnectionString("psr")
?? throw new InvalidOperationException("Set the PostgreSQL connection string ConnectionStrings__psr."));
})
.AddRabbitMq(o =>
{
configuration.GetSection("PythonScriptRunner:RabbitMQ").Bind(o);
o.ConnectionString = configuration.GetConnectionString("rabbitmq")
?? throw new InvalidOperationException("Set the RabbitMQ connection string ConnectionStrings__rabbitmq.");
})
.AddWorker(o => configuration.GetSection("PythonScriptRunner:Worker").Bind(o));
builder.Services.AddHealthChecks().AddPythonScriptRunnerChecks();

var app = builder.Build();
if (configuration.GetValue<bool>("PSR_AUTO_MIGRATE"))
{
// EF Core serializes concurrent migrations with a PostgreSQL advisory lock, so replicas can start together (spec 009, AC-8).
await app.Services.MigratePythonScriptRunnerAsync();
}

app.MapDefaultEndpoints();
await app.RunAsync();

/// <summary>Entry point (public for the integration tests).</summary>
public partial class Program;
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
<Project Sdk="Microsoft.NET.Sdk.Web">

<PropertyGroup>
<RootNamespace>PythonScriptRunner.WorkerHost</RootNamespace>
</PropertyGroup>

<ItemGroup>
<ProjectReference Include="..\..\src\PythonScriptRunner.Worker\PythonScriptRunner.Worker.csproj" />
<ProjectReference Include="..\PythonScriptRunner.ServiceDefaults\PythonScriptRunner.ServiceDefaults.csproj" />
</ItemGroup>

</Project>
62 changes: 62 additions & 0 deletions schemas/messages/run-event.v1.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://raw.githubusercontent.com/net0well/PythonScriptRunner/main/schemas/messages/run-event.v1.json",
"title": "PythonScriptRunner run event (psr.run-event.v1)",
"description": "Published on the topic exchange psr.events with the routing key <type>.<pipeline key> or <type>.stage:<stage key> (AMQP type psr.run-event.v1). Fields are only added within v1; consumers must ignore unknown fields.",
"type": "object",
"required": ["eventId", "type", "occurredAt"],
"properties": {
"eventId": { "type": "string", "format": "uuid" },
"type": {
"enum": [
"run.queued", "run.rejected", "run.started", "step.started", "step.succeeded", "step.retry-scheduled", "step.failed",
"run.succeeded", "run.failed", "run.cancelled"
]
},
"occurredAt": { "type": "string", "format": "date-time" },
"runId": {
"description": "Null when a request was rejected before a run existed.",
"type": ["string", "null"],
"format": "uuid"
},
"kind": { "enum": ["Pipeline", "PipelineTest", "Stage", "StageTest", null] },
"pipelineKey": { "type": ["string", "null"] },
"pipelineVersion": { "type": ["integer", "null"] },
"stageKey": { "type": ["string", "null"] },
"stageVersion": { "type": ["integer", "null"] },
"correlationId": { "type": ["string", "null"] },
"trigger": { "enum": ["Library", "Dashboard", "Schedule", "QueueCommand", null] },
"status": { "enum": ["Pending", "Queued", "Running", "Succeeded", "Failed", "Cancelling", "Cancelled", null] },
"step": {
"description": "The step of step.* events.",
"type": ["object", "null"],
"required": ["key", "order", "attempt"],
"properties": {
"key": { "type": "string" },
"order": { "type": "integer", "minimum": 1 },
"attempt": { "type": "integer", "minimum": 0 }
}
},
"error": {
"type": ["object", "null"],
"required": ["kind", "message"],
"properties": {
"kind": { "description": "An error kind of the data mapping (e.g. ScriptError) or Rejected.", "type": "string" },
"code": { "type": ["string", "null"] },
"message": { "type": "string" },
"step": { "type": ["string", "null"] },
"reasons": { "description": "Why a request was rejected.", "type": ["array", "null"], "items": { "type": "string" } }
}
},
"outputs": {
"description": "Declared outputs of run.succeeded: values inline; files as {\"$artifact\": {id, name, contentType, size}}. Null above 256 KB.",
"type": ["object", "null"]
},
"outputsTruncated": { "type": "boolean" },
"retryAt": { "type": ["string", "null"], "format": "date-time" },
"triggerDetails": {
"description": "Details of the trigger, e.g. scheduleId, scheduledFor and missedOccurrences of a schedule.",
"type": ["object", "null"]
}
}
}
Loading
Loading