Skip to content

Commit 88ee999

Browse files
net0wellclaude
andcommitted
docs: JSON Schemas das mensagens públicas, CI e README da fila e do worker (specs 005, 006, 009)
- schemas/messages: start-pipeline-run.v1 e run-event.v1, validados contra os payloads reais. - CI: instala pango para o weasyprint (CS-2) no job Linux, que roda os testes de fila e worker. - README: fila, worker, agendamento, eventos e WorkerHost. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 parent 045178f commit 88ee999

5 files changed

Lines changed: 229 additions & 6 deletions

File tree

‎.github/workflows/ci.yml‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -41,8 +41,13 @@ jobs:
4141
- name: Build
4242
run: dotnet build --no-restore --configuration Release
4343

44-
# The persistence tests start PostgreSQL 16 with Testcontainers (Docker); GitHub's Windows runners
45-
# cannot run Linux containers, so Windows runs the Core tests only.
44+
# weasyprint (CS-2 fixture, html-para-pdf) renders with pango.
45+
- name: Install pango
46+
if: runner.os == 'Linux'
47+
run: sudo apt-get update && sudo apt-get install -y --no-install-recommends libpango-1.0-0 libpangoft2-1.0-0 libharfbuzz0b fonts-dejavu-core
48+
49+
# The persistence, queue and worker tests start PostgreSQL 16, RabbitMQ 3.13 and Mailpit with Testcontainers
50+
# (Docker); GitHub's Windows runners cannot run Linux containers, so Windows runs the Core tests only.
4651
- name: Test
4752
if: runner.os == 'Linux'
4853
run: dotnet test --no-build --configuration Release

‎README.md‎

Lines changed: 44 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,8 @@ com contrato de entrada/saída, trilhas (**Pipelines**), execução assíncrona
55
cron e painel Vue opcional.
66

77
> Projeto em desenvolvimento. Esta versão entrega o **Core Runner** (execução síncrona de Stages e de
8-
> trilhas), o SDK Python **`psrunner`** e a **persistência em PostgreSQL** (catálogo versionado e histórico).
8+
> trilhas), o SDK Python **`psrunner`**, a **persistência em PostgreSQL** (catálogo versionado e histórico) e a
9+
> **execução assíncrona** com RabbitMQ, worker e agendamento cron.
910
1011
## Como funciona
1112

@@ -120,21 +121,58 @@ await app.Services.MigratePythonScriptRunnerAsync(); // explícito; ou Generat
120121
um job de manutenção cria partições à frente e remove as antigas. Uma trilha pode encurtar a retenção.
121122
- **Auditoria:** ações humanas (`Actor.User(...)`) ficam em `psr.audit_events`; disparos automáticos ficam na própria execução.
122123

124+
## Fila, worker e agendamento (RabbitMQ)
125+
126+
Com `PythonScriptRunner.RabbitMQ` e `PythonScriptRunner.Worker`, execuções podem ser enfileiradas. Cada passo é uma
127+
mensagem: o worker assume o passo com um *lease*, executa o processo Python e grava o resultado e a mensagem do próximo
128+
passo na mesma transação (outbox no PostgreSQL). Enfileirar funciona mesmo com o broker fora do ar.
129+
130+
```csharp
131+
builder.Services
132+
.AddPythonScriptRunner()
133+
.AddEntityFrameworkCore(o => o.UseNpgsql(builder.Configuration.GetConnectionString("psr")!))
134+
.AddRabbitMq(o => o.ConnectionString = builder.Configuration.GetConnectionString("rabbitmq")) // RabbitMQ 3.13+
135+
.AddWorker(o => o.MaxConcurrency = 8) // só nos processos que executam passos
136+
.AddRunEventHandler<NotificarErp>(s => s.Subscription("erp").On("run.succeeded.calculo-progressao", "run.failed.*"));
137+
builder.Services.AddHealthChecks().AddPythonScriptRunnerChecks(); // banco, broker e uv
138+
139+
// ...
140+
RunHandle run = await queue.EnqueuePipelineAsync("calculo-progressao", new { a1 = 1, razao = 2, n = 4, coeficientes = new[] { 1, 0, 1 } });
141+
```
142+
143+
- **Confiabilidade:** retry automático por Stage com atraso exponencial; worker perdido é detectado pelo *lease* e o
144+
passo é retomado (Stage idempotente ou com retry restante) ou falha com `WorkerLost` para reprocesso manual;
145+
mensagens duplicadas não executam o passo duas vezes.
146+
- **Concorrência:** `MaxConcurrency` por worker e `max_concurrent_runs` por trilha (as demais esperam como `Pending`, em ordem).
147+
- **Cancelamento:** `IRunManager.CancelAsync` encerra a árvore de processos no worker que executa o passo.
148+
- **Agendamento:** `ICatalogAdmin.CreateScheduleAsync` com expressão cron e fuso IANA (padrão `America/Sao_Paulo`);
149+
ocorrências perdidas viram uma execução de recuperação; vários workers nunca disparam a mesma ocorrência.
150+
- **Sistemas externos:** o comando público `StartPipelineRun` (exchange `psr.public`) inicia trilhas e os eventos
151+
`run.*`/`step.*` (exchange `psr.events`) informam o andamento e as saídas. Contratos em `schemas/messages`.
152+
- **Observabilidade:** `ActivitySource` e `Meter` `PythonScriptRunner`; o `traceparent` acompanha a execução até o worker.
153+
154+
O host `hosts/PythonScriptRunner.WorkerHost` é um worker pronto, configurado por variáveis de ambiente
155+
(`ConnectionStrings__psr`, `ConnectionStrings__rabbitmq`, `PythonScriptRunner__Worker__MaxConcurrency`,
156+
`PythonScriptRunner__Secrets__<NOME>`, `PSR_AUTO_MIGRATE=true`), com `/health/live`, `/health/ready`, logs JSON e OTLP.
157+
123158
## Requisitos
124159

125160
- .NET 10 SDK
126161
- [uv](https://docs.astral.sh/uv/) no PATH (cria e cacheia um ambiente Python por conjunto de dependências)
127-
- PostgreSQL 16+ apenas para o pacote de persistência (os testes usam Docker/Testcontainers)
162+
- PostgreSQL 16+ apenas para o pacote de persistência e RabbitMQ 3.13+ para a fila (os testes usam Docker/Testcontainers)
128163

129164
## Estrutura
130165

131166
| Pasta | Conteúdo |
132167
|-------|----------|
133168
| `src/PythonScriptRunner.Abstractions` | contratos públicos (tipos de campo, `StageDefinition`, `RunResult`, `PsrFile`, interfaces) |
134169
| `src/PythonScriptRunner.Core` | catálogos (código e pasta), validação, ambientes uv, executor de processos, runner |
135-
| `src/PythonScriptRunner.EntityFrameworkCore` | PostgreSQL: catálogo versionado, histórico, arquivos, idempotência, outbox, auditoria, retenção |
170+
| `src/PythonScriptRunner.EntityFrameworkCore` | PostgreSQL: catálogo versionado, histórico, arquivos, idempotência, outbox, auditoria, retenção, agendamentos e ciclo das execuções enfileiradas |
171+
| `src/PythonScriptRunner.RabbitMQ` | topologia, relay do outbox, `IRunQueue`, comando público, eventos e assinaturas |
172+
| `src/PythonScriptRunner.Worker` | execução dos passos enfileirados, *lease*, retry, cancelamento, cron |
173+
| `hosts` | `WorkerHost` e `ServiceDefaults` (Serilog, OpenTelemetry, health checks) |
136174
| `python/psrunner` | SDK Python (`@stage`, `Context`, `File`, `StageError`, `psrunner.testing`, CLI) |
137-
| `schemas` | JSON Schemas do `stage.json` e do `*.pipeline.json` |
175+
| `schemas` | JSON Schemas do `stage.json`, do `*.pipeline.json` e das mensagens públicas |
138176
| `tests` | testes .NET (unitários e de integração com Python real) e fixtures de trilha |
139177

140178
## Desenvolvimento
@@ -143,6 +181,8 @@ await app.Services.MigratePythonScriptRunnerAsync(); // explícito; ou Generat
143181
dotnet build
144182
dotnet test --project tests/PythonScriptRunner.Core.Tests
145183
dotnet test --project tests/PythonScriptRunner.EntityFrameworkCore.Tests # precisa de Docker
184+
dotnet test --project tests/PythonScriptRunner.RabbitMQ.Tests # precisa de Docker
185+
dotnet test --project tests/PythonScriptRunner.Worker.Tests # precisa de Docker
146186

147187
cd python/psrunner
148188
uv sync

‎schemas/messages/run-event.v1.json‎

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
{
2+
"$schema": "https://json-schema.org/draft/2020-12/schema",
3+
"$id": "https://raw.githubusercontent.com/net0well/PythonScriptRunner/main/schemas/messages/run-event.v1.json",
4+
"title": "PythonScriptRunner run event (psr.run-event.v1)",
5+
"description": "Published on the topic exchange psr.events with the routing key <type>.<pipeline key> or <type>.stage:<stage key> (AMQP type psr.run-event.v1). Fields are only added within v1; consumers must ignore unknown fields.",
6+
"type": "object",
7+
"required": ["eventId", "type", "occurredAt"],
8+
"properties": {
9+
"eventId": { "type": "string", "format": "uuid" },
10+
"type": {
11+
"enum": [
12+
"run.queued", "run.rejected", "run.started", "step.started", "step.succeeded", "step.retry-scheduled", "step.failed",
13+
"run.succeeded", "run.failed", "run.cancelled"
14+
]
15+
},
16+
"occurredAt": { "type": "string", "format": "date-time" },
17+
"runId": {
18+
"description": "Null when a request was rejected before a run existed.",
19+
"type": ["string", "null"],
20+
"format": "uuid"
21+
},
22+
"kind": { "enum": ["Pipeline", "PipelineTest", "Stage", "StageTest", null] },
23+
"pipelineKey": { "type": ["string", "null"] },
24+
"pipelineVersion": { "type": ["integer", "null"] },
25+
"stageKey": { "type": ["string", "null"] },
26+
"stageVersion": { "type": ["integer", "null"] },
27+
"correlationId": { "type": ["string", "null"] },
28+
"trigger": { "enum": ["Library", "Dashboard", "Schedule", "QueueCommand", null] },
29+
"status": { "enum": ["Pending", "Queued", "Running", "Succeeded", "Failed", "Cancelling", "Cancelled", null] },
30+
"step": {
31+
"description": "The step of step.* events.",
32+
"type": ["object", "null"],
33+
"required": ["key", "order", "attempt"],
34+
"properties": {
35+
"key": { "type": "string" },
36+
"order": { "type": "integer", "minimum": 1 },
37+
"attempt": { "type": "integer", "minimum": 0 }
38+
}
39+
},
40+
"error": {
41+
"type": ["object", "null"],
42+
"required": ["kind", "message"],
43+
"properties": {
44+
"kind": { "description": "An error kind of the data mapping (e.g. ScriptError) or Rejected.", "type": "string" },
45+
"code": { "type": ["string", "null"] },
46+
"message": { "type": "string" },
47+
"step": { "type": ["string", "null"] },
48+
"reasons": { "description": "Why a request was rejected.", "type": ["array", "null"], "items": { "type": "string" } }
49+
}
50+
},
51+
"outputs": {
52+
"description": "Declared outputs of run.succeeded: values inline; files as {\"$artifact\": {id, name, contentType, size}}. Null above 256 KB.",
53+
"type": ["object", "null"]
54+
},
55+
"outputsTruncated": { "type": "boolean" },
56+
"retryAt": { "type": ["string", "null"], "format": "date-time" },
57+
"triggerDetails": {
58+
"description": "Details of the trigger, e.g. scheduleId, scheduledFor and missedOccurrences of a schedule.",
59+
"type": ["object", "null"]
60+
}
61+
}
62+
}
Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
{
2+
"$schema": "https://json-schema.org/draft/2020-12/schema",
3+
"$id": "https://raw.githubusercontent.com/net0well/PythonScriptRunner/main/schemas/messages/start-pipeline-run.v1.json",
4+
"title": "PythonScriptRunner StartPipelineRun command (psr.start-pipeline-run.v1)",
5+
"description": "Publish to the exchange psr.public with the routing key pipeline.start (AMQP type psr.start-pipeline-run.v1, content type application/json). The run is announced with run.queued, or refused with run.rejected; both carry the correlationId.",
6+
"type": "object",
7+
"required": ["pipelineKey"],
8+
"properties": {
9+
"pipelineKey": {
10+
"description": "Key of the pipeline.",
11+
"type": "string",
12+
"minLength": 3,
13+
"maxLength": 64,
14+
"pattern": "^[a-z0-9]+(-[a-z0-9]+)*$"
15+
},
16+
"version": {
17+
"description": "Version to run; null runs the latest published version.",
18+
"type": ["integer", "null"],
19+
"minimum": 1
20+
},
21+
"inputs": {
22+
"description": "Pipeline inputs in the wire format of the data mapping. Files are not accepted: pipelines with file inputs are rejected.",
23+
"type": ["object", "null"]
24+
},
25+
"idempotencyKey": {
26+
"description": "Repeated commands with the same key within 24 hours return the same run. Without it, the AMQP message_id is used.",
27+
"type": ["string", "null"],
28+
"maxLength": 190
29+
},
30+
"correlationId": {
31+
"description": "Returned in every event of the run.",
32+
"type": ["string", "null"],
33+
"maxLength": 190
34+
},
35+
"requestedBy": {
36+
"description": "Name of the calling system; the run is triggered by queue:<requestedBy>.",
37+
"type": ["string", "null"],
38+
"maxLength": 190
39+
}
40+
}
41+
}
Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
using System.Collections.Concurrent;
2+
using System.Text.Json;
3+
using System.Text.Json.Nodes;
4+
using Json.Schema;
5+
6+
namespace PythonScriptRunner.RabbitMQ.Tests;
7+
8+
/// <summary>Spec 005: the public contracts match the JSON Schemas published in <c>schemas/messages</c>.</summary>
9+
public sealed class ContractSchemaTests(PostgresFixture postgres, BrokerFixture broker, PythonFixture python)
10+
{
11+
// Schemas are registered globally by their $id, so each one is loaded once per test run.
12+
private static readonly ConcurrentDictionary<string, JsonSchema> Loaded = new();
13+
14+
private static CancellationToken Ct => TestContext.Current.CancellationToken;
15+
16+
[Fact]
17+
public async Task PublishedEventsAndTheCommandExample_MatchTheSchemas()
18+
{
19+
await using var env = await PsrEnvironment.CreateAsync(postgres, broker, python);
20+
var host = await env.StartHostAsync();
21+
await PsrEnvironment.PublishStageAsync(host, "eco", Scripts.Echo, [Scripts.Text("texto")], [Scripts.Text("texto")]);
22+
await PsrEnvironment.PublishPipelineAsync(host, "eco-contrato", Scripts.SingleStep("eco", 1, [Scripts.Text("texto")]));
23+
await PsrEnvironment.PublishStageAsync(host, "falha", Scripts.AlwaysFails, [], []);
24+
var succeeded = await host.Queue.EnqueuePipelineAsync("eco-contrato", new { texto = "ok" }, new RunOptions { CorrelationId = "contrato" }, Ct);
25+
var failed = await host.Queue.EnqueueStageAsync("falha", null, cancellationToken: Ct);
26+
await host.WaitForRunAsync(succeeded.RunId);
27+
await host.WaitForRunAsync(failed.RunId);
28+
await using (var raw = await RawBroker.ConnectAsync(env.Broker))
29+
{
30+
await raw.PublishAsync(RabbitMQ.PsrTopology.PublicExchange, RabbitMQ.PsrTopology.StartPipelineRoutingKey, """{"pipelineKey":"nao-existe"}""",
31+
PsrMessageTypes.StartPipelineRun);
32+
}
33+
34+
await host.Events.WaitForAsync(RunEventTypes.RunRejected);
35+
var eventSchema = Schema("run-event.v1.json");
36+
var payloads = await env.ColumnAsync<string>("SELECT payload::text FROM psr.outbox_messages WHERE message_type = 'psr.run-event.v1'");
37+
38+
payloads.Select(p => JsonNode.Parse(p)!["type"]!.GetValue<string>()).Distinct().ShouldBe(
39+
[
40+
RunEventTypes.RunQueued, RunEventTypes.RunStarted, RunEventTypes.StepStarted, RunEventTypes.StepSucceeded, RunEventTypes.RunSucceeded,
41+
RunEventTypes.StepFailed, RunEventTypes.RunFailed, RunEventTypes.RunRejected,
42+
],
43+
ignoreOrder: true);
44+
foreach (var payload in payloads)
45+
{
46+
Validate(eventSchema, payload);
47+
}
48+
49+
var commandSchema = Schema("start-pipeline-run.v1.json");
50+
Validate(commandSchema, """
51+
{
52+
"pipelineKey": "calculo-progressao",
53+
"version": null,
54+
"inputs": { "a1": 1, "razao": 2, "n": 4, "tipo": "geometrica", "coeficientes": [1, 0, 1] },
55+
"idempotencyKey": "pedido-123",
56+
"correlationId": "erp-77881",
57+
"requestedBy": "erp"
58+
}
59+
""");
60+
Validate(commandSchema,
61+
JsonSerializer.Serialize(new StartPipelineRunCommand { PipelineKey = "calculo-progressao", CorrelationId = "x" }, PsrMessageJson.Options));
62+
}
63+
64+
private JsonSchema Schema(string name) =>
65+
Loaded.GetOrAdd(name, file => JsonSchema.FromFile(Path.Combine(python.RepositoryRoot, "schemas", "messages", file)));
66+
67+
private static void Validate(JsonSchema schema, string json)
68+
{
69+
using var document = JsonDocument.Parse(json);
70+
var result = schema.Evaluate(document.RootElement, new EvaluationOptions { OutputFormat = OutputFormat.List });
71+
var errors = ((IEnumerable<EvaluationResults>)[result, .. result.Details ?? []])
72+
.SelectMany(d => (d.Errors?.Values ?? Enumerable.Empty<string>()).Select(e => $"{d.InstanceLocation}: {e}"));
73+
result.IsValid.ShouldBeTrue($"{json} → {string.Join("; ", errors)}");
74+
}
75+
}

0 commit comments

Comments
 (0)