diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 3affef9..d72fea1 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -41,8 +41,13 @@ jobs: - name: Build run: dotnet build --no-restore --configuration Release - # The persistence tests start PostgreSQL 16 with Testcontainers (Docker); GitHub's Windows runners - # cannot run Linux containers, so Windows runs the Core tests only. + # weasyprint (CS-2 fixture, html-para-pdf) renders with pango. + - name: Install pango + if: runner.os == 'Linux' + run: sudo apt-get update && sudo apt-get install -y --no-install-recommends libpango-1.0-0 libpangoft2-1.0-0 libharfbuzz0b fonts-dejavu-core + + # The persistence, queue and worker tests start PostgreSQL 16, RabbitMQ 3.13 and Mailpit with Testcontainers + # (Docker); GitHub's Windows runners cannot run Linux containers, so Windows runs the Core tests only. - name: Test if: runner.os == 'Linux' run: dotnet test --no-build --configuration Release diff --git a/Directory.Packages.props b/Directory.Packages.props index b891480..d1093a0 100644 --- a/Directory.Packages.props +++ b/Directory.Packages.props @@ -3,7 +3,9 @@ true + + @@ -13,13 +15,22 @@ + + + + + + + + + diff --git a/PythonScriptRunner.slnx b/PythonScriptRunner.slnx index 091dc73..0d7529d 100644 --- a/PythonScriptRunner.slnx +++ b/PythonScriptRunner.slnx @@ -1,11 +1,19 @@ + + + + + + + + diff --git a/README.md b/README.md index aeea566..b328f0d 100644 --- a/README.md +++ b/README.md @@ -5,7 +5,8 @@ com contrato de entrada/saída, trilhas (**Pipelines**), execução assíncrona cron e painel Vue opcional. > Projeto em desenvolvimento. Esta versão entrega o **Core Runner** (execução síncrona de Stages e de -> trilhas), o SDK Python **`psrunner`** e a **persistência em PostgreSQL** (catálogo versionado e histórico). +> trilhas), o SDK Python **`psrunner`**, a **persistência em PostgreSQL** (catálogo versionado e histórico) e a +> **execução assíncrona** com RabbitMQ, worker e agendamento cron. ## Como funciona @@ -120,11 +121,45 @@ await app.Services.MigratePythonScriptRunnerAsync(); // explícito; ou Generat um job de manutenção cria partições à frente e remove as antigas. Uma trilha pode encurtar a retenção. - **Auditoria:** ações humanas (`Actor.User(...)`) ficam em `psr.audit_events`; disparos automáticos ficam na própria execução. +## Fila, worker e agendamento (RabbitMQ) + +Com `PythonScriptRunner.RabbitMQ` e `PythonScriptRunner.Worker`, execuções podem ser enfileiradas. Cada passo é uma +mensagem: o worker assume o passo com um *lease*, executa o processo Python e grava o resultado e a mensagem do próximo +passo na mesma transação (outbox no PostgreSQL). Enfileirar funciona mesmo com o broker fora do ar. + +```csharp +builder.Services + .AddPythonScriptRunner() + .AddEntityFrameworkCore(o => o.UseNpgsql(builder.Configuration.GetConnectionString("psr")!)) + .AddRabbitMq(o => o.ConnectionString = builder.Configuration.GetConnectionString("rabbitmq")) // RabbitMQ 3.13+ + .AddWorker(o => o.MaxConcurrency = 8) // só nos processos que executam passos + .AddRunEventHandler(s => s.Subscription("erp").On("run.succeeded.calculo-progressao", "run.failed.*")); +builder.Services.AddHealthChecks().AddPythonScriptRunnerChecks(); // banco, broker e uv + +// ... +RunHandle run = await queue.EnqueuePipelineAsync("calculo-progressao", new { a1 = 1, razao = 2, n = 4, coeficientes = new[] { 1, 0, 1 } }); +``` + +- **Confiabilidade:** retry automático por Stage com atraso exponencial; worker perdido é detectado pelo *lease* e o + passo é retomado (Stage idempotente ou com retry restante) ou falha com `WorkerLost` para reprocesso manual; + mensagens duplicadas não executam o passo duas vezes. +- **Concorrência:** `MaxConcurrency` por worker e `max_concurrent_runs` por trilha (as demais esperam como `Pending`, em ordem). +- **Cancelamento:** `IRunManager.CancelAsync` encerra a árvore de processos no worker que executa o passo. +- **Agendamento:** `ICatalogAdmin.CreateScheduleAsync` com expressão cron e fuso IANA (padrão `America/Sao_Paulo`); + ocorrências perdidas viram uma execução de recuperação; vários workers nunca disparam a mesma ocorrência. +- **Sistemas externos:** o comando público `StartPipelineRun` (exchange `psr.public`) inicia trilhas e os eventos + `run.*`/`step.*` (exchange `psr.events`) informam o andamento e as saídas. Contratos em `schemas/messages`. +- **Observabilidade:** `ActivitySource` e `Meter` `PythonScriptRunner`; o `traceparent` acompanha a execução até o worker. + +O host `hosts/PythonScriptRunner.WorkerHost` é um worker pronto, configurado por variáveis de ambiente +(`ConnectionStrings__psr`, `ConnectionStrings__rabbitmq`, `PythonScriptRunner__Worker__MaxConcurrency`, +`PythonScriptRunner__Secrets__`, `PSR_AUTO_MIGRATE=true`), com `/health/live`, `/health/ready`, logs JSON e OTLP. + ## Requisitos - .NET 10 SDK - [uv](https://docs.astral.sh/uv/) no PATH (cria e cacheia um ambiente Python por conjunto de dependências) -- PostgreSQL 16+ apenas para o pacote de persistência (os testes usam Docker/Testcontainers) +- PostgreSQL 16+ apenas para o pacote de persistência e RabbitMQ 3.13+ para a fila (os testes usam Docker/Testcontainers) ## Estrutura @@ -132,9 +167,12 @@ await app.Services.MigratePythonScriptRunnerAsync(); // explícito; ou Generat |-------|----------| | `src/PythonScriptRunner.Abstractions` | contratos públicos (tipos de campo, `StageDefinition`, `RunResult`, `PsrFile`, interfaces) | | `src/PythonScriptRunner.Core` | catálogos (código e pasta), validação, ambientes uv, executor de processos, runner | -| `src/PythonScriptRunner.EntityFrameworkCore` | PostgreSQL: catálogo versionado, histórico, arquivos, idempotência, outbox, auditoria, retenção | +| `src/PythonScriptRunner.EntityFrameworkCore` | PostgreSQL: catálogo versionado, histórico, arquivos, idempotência, outbox, auditoria, retenção, agendamentos e ciclo das execuções enfileiradas | +| `src/PythonScriptRunner.RabbitMQ` | topologia, relay do outbox, `IRunQueue`, comando público, eventos e assinaturas | +| `src/PythonScriptRunner.Worker` | execução dos passos enfileirados, *lease*, retry, cancelamento, cron | +| `hosts` | `WorkerHost` e `ServiceDefaults` (Serilog, OpenTelemetry, health checks) | | `python/psrunner` | SDK Python (`@stage`, `Context`, `File`, `StageError`, `psrunner.testing`, CLI) | -| `schemas` | JSON Schemas do `stage.json` e do `*.pipeline.json` | +| `schemas` | JSON Schemas do `stage.json`, do `*.pipeline.json` e das mensagens públicas | | `tests` | testes .NET (unitários e de integração com Python real) e fixtures de trilha | ## Desenvolvimento @@ -143,6 +181,8 @@ await app.Services.MigratePythonScriptRunnerAsync(); // explícito; ou Generat dotnet build dotnet test --project tests/PythonScriptRunner.Core.Tests dotnet test --project tests/PythonScriptRunner.EntityFrameworkCore.Tests # precisa de Docker +dotnet test --project tests/PythonScriptRunner.RabbitMQ.Tests # precisa de Docker +dotnet test --project tests/PythonScriptRunner.Worker.Tests # precisa de Docker cd python/psrunner uv sync diff --git a/hosts/Directory.Build.props b/hosts/Directory.Build.props new file mode 100644 index 0000000..a8d6751 --- /dev/null +++ b/hosts/Directory.Build.props @@ -0,0 +1,11 @@ + + + + + + false + + true + $(NoWarn);CS1591 + + diff --git a/hosts/PythonScriptRunner.ServiceDefaults/PythonScriptRunner.ServiceDefaults.csproj b/hosts/PythonScriptRunner.ServiceDefaults/PythonScriptRunner.ServiceDefaults.csproj new file mode 100644 index 0000000..b2fe11a --- /dev/null +++ b/hosts/PythonScriptRunner.ServiceDefaults/PythonScriptRunner.ServiceDefaults.csproj @@ -0,0 +1,20 @@ + + + + PythonScriptRunner.ServiceDefaults + + + + + + + + + + + + + + + + diff --git a/hosts/PythonScriptRunner.ServiceDefaults/ServiceDefaultsExtensions.cs b/hosts/PythonScriptRunner.ServiceDefaults/ServiceDefaultsExtensions.cs new file mode 100644 index 0000000..6cf5c7a --- /dev/null +++ b/hosts/PythonScriptRunner.ServiceDefaults/ServiceDefaultsExtensions.cs @@ -0,0 +1,65 @@ +using Microsoft.AspNetCore.Builder; +using Microsoft.AspNetCore.Diagnostics.HealthChecks; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Diagnostics.HealthChecks; +using OpenTelemetry; +using OpenTelemetry.Metrics; +using OpenTelemetry.Resources; +using OpenTelemetry.Trace; +using Serilog; +using Serilog.Formatting.Compact; + +namespace Microsoft.Extensions.Hosting; + +/// +/// Defaults of the standalone hosts (spec 009; D-54): Serilog writing JSON to the console, OpenTelemetry traces and metrics of +/// the PythonScriptRunner source/meter (exported with OTLP when OTEL_EXPORTER_OTLP_ENDPOINT is set) and the +/// /health/live and /health/ready endpoints. +/// +public static class ServiceDefaultsExtensions +{ + /// Tag of the checks behind /health/live. + public const string LiveTag = "live"; + + /// Tag of the checks behind /health/ready. + public const string ReadyTag = "ready"; + + /// Adds logging, telemetry and the liveness check. + /// Host builder. + /// Service name in logs and telemetry. + public static TBuilder AddServiceDefaults(this TBuilder builder, string serviceName) + where TBuilder : IHostApplicationBuilder + { + ArgumentNullException.ThrowIfNull(builder); + builder.Services.AddSerilog((services, logger) => logger + .ReadFrom.Configuration(builder.Configuration) + .Enrich.FromLogContext() + .Enrich.WithProperty("service", serviceName) + .WriteTo.Console(new RenderedCompactJsonFormatter())); + + var telemetry = builder.Services.AddOpenTelemetry() + .ConfigureResource(resource => resource.AddService(serviceName)) + .WithTracing(tracing => tracing.AddSource("PythonScriptRunner").AddAspNetCoreInstrumentation()) + .WithMetrics(metrics => metrics.AddMeter("PythonScriptRunner").AddAspNetCoreInstrumentation().AddRuntimeInstrumentation()); + if (!string.IsNullOrWhiteSpace(builder.Configuration["OTEL_EXPORTER_OTLP_ENDPOINT"])) + { + telemetry.UseOtlpExporter(); + } + + builder.Services.AddHealthChecks().AddCheck("self", () => HealthCheckResult.Healthy(), [LiveTag]); + return builder; + } + + /// + /// Maps /health/live (the process answers) and /health/ready (checks tagged : 200 when + /// healthy or degraded, 503 when unhealthy). + /// + /// Application. + public static WebApplication MapDefaultEndpoints(this WebApplication app) + { + ArgumentNullException.ThrowIfNull(app); + app.MapHealthChecks("/health/live", new HealthCheckOptions { Predicate = check => check.Tags.Contains(LiveTag) }); + app.MapHealthChecks("/health/ready", new HealthCheckOptions { Predicate = check => check.Tags.Contains(ReadyTag) }); + return app; + } +} diff --git a/hosts/PythonScriptRunner.WorkerHost/Program.cs b/hosts/PythonScriptRunner.WorkerHost/Program.cs new file mode 100644 index 0000000..99937a6 --- /dev/null +++ b/hosts/PythonScriptRunner.WorkerHost/Program.cs @@ -0,0 +1,36 @@ +// Standalone worker of PythonScriptRunner (spec 009): runs queued steps, cron schedules and the maintenance job. +// Configuration comes from environment variables, e.g. ConnectionStrings__psr, ConnectionStrings__rabbitmq, +// PythonScriptRunner__Worker__MaxConcurrency, PythonScriptRunner__Secrets__ and PSR_AUTO_MIGRATE. +var builder = WebApplication.CreateBuilder(args); +builder.AddServiceDefaults("psr-worker"); + +var configuration = builder.Configuration; +builder.Services + .AddPythonScriptRunner(o => configuration.GetSection("PythonScriptRunner").Bind(o)) + .AddEntityFrameworkCore(o => + { + configuration.GetSection("PythonScriptRunner:EntityFrameworkCore").Bind(o); + o.UseNpgsql(configuration.GetConnectionString("psr") + ?? throw new InvalidOperationException("Set the PostgreSQL connection string ConnectionStrings__psr.")); + }) + .AddRabbitMq(o => + { + configuration.GetSection("PythonScriptRunner:RabbitMQ").Bind(o); + o.ConnectionString = configuration.GetConnectionString("rabbitmq") + ?? throw new InvalidOperationException("Set the RabbitMQ connection string ConnectionStrings__rabbitmq."); + }) + .AddWorker(o => configuration.GetSection("PythonScriptRunner:Worker").Bind(o)); +builder.Services.AddHealthChecks().AddPythonScriptRunnerChecks(); + +var app = builder.Build(); +if (configuration.GetValue("PSR_AUTO_MIGRATE")) +{ + // EF Core serializes concurrent migrations with a PostgreSQL advisory lock, so replicas can start together (spec 009, AC-8). + await app.Services.MigratePythonScriptRunnerAsync(); +} + +app.MapDefaultEndpoints(); +await app.RunAsync(); + +/// Entry point (public for the integration tests). +public partial class Program; diff --git a/hosts/PythonScriptRunner.WorkerHost/PythonScriptRunner.WorkerHost.csproj b/hosts/PythonScriptRunner.WorkerHost/PythonScriptRunner.WorkerHost.csproj new file mode 100644 index 0000000..5334549 --- /dev/null +++ b/hosts/PythonScriptRunner.WorkerHost/PythonScriptRunner.WorkerHost.csproj @@ -0,0 +1,12 @@ + + + + PythonScriptRunner.WorkerHost + + + + + + + + diff --git a/schemas/messages/run-event.v1.json b/schemas/messages/run-event.v1.json new file mode 100644 index 0000000..8655b82 --- /dev/null +++ b/schemas/messages/run-event.v1.json @@ -0,0 +1,62 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://raw.githubusercontent.com/net0well/PythonScriptRunner/main/schemas/messages/run-event.v1.json", + "title": "PythonScriptRunner run event (psr.run-event.v1)", + "description": "Published on the topic exchange psr.events with the routing key . or .stage: (AMQP type psr.run-event.v1). Fields are only added within v1; consumers must ignore unknown fields.", + "type": "object", + "required": ["eventId", "type", "occurredAt"], + "properties": { + "eventId": { "type": "string", "format": "uuid" }, + "type": { + "enum": [ + "run.queued", "run.rejected", "run.started", "step.started", "step.succeeded", "step.retry-scheduled", "step.failed", + "run.succeeded", "run.failed", "run.cancelled" + ] + }, + "occurredAt": { "type": "string", "format": "date-time" }, + "runId": { + "description": "Null when a request was rejected before a run existed.", + "type": ["string", "null"], + "format": "uuid" + }, + "kind": { "enum": ["Pipeline", "PipelineTest", "Stage", "StageTest", null] }, + "pipelineKey": { "type": ["string", "null"] }, + "pipelineVersion": { "type": ["integer", "null"] }, + "stageKey": { "type": ["string", "null"] }, + "stageVersion": { "type": ["integer", "null"] }, + "correlationId": { "type": ["string", "null"] }, + "trigger": { "enum": ["Library", "Dashboard", "Schedule", "QueueCommand", null] }, + "status": { "enum": ["Pending", "Queued", "Running", "Succeeded", "Failed", "Cancelling", "Cancelled", null] }, + "step": { + "description": "The step of step.* events.", + "type": ["object", "null"], + "required": ["key", "order", "attempt"], + "properties": { + "key": { "type": "string" }, + "order": { "type": "integer", "minimum": 1 }, + "attempt": { "type": "integer", "minimum": 0 } + } + }, + "error": { + "type": ["object", "null"], + "required": ["kind", "message"], + "properties": { + "kind": { "description": "An error kind of the data mapping (e.g. ScriptError) or Rejected.", "type": "string" }, + "code": { "type": ["string", "null"] }, + "message": { "type": "string" }, + "step": { "type": ["string", "null"] }, + "reasons": { "description": "Why a request was rejected.", "type": ["array", "null"], "items": { "type": "string" } } + } + }, + "outputs": { + "description": "Declared outputs of run.succeeded: values inline; files as {\"$artifact\": {id, name, contentType, size}}. Null above 256 KB.", + "type": ["object", "null"] + }, + "outputsTruncated": { "type": "boolean" }, + "retryAt": { "type": ["string", "null"], "format": "date-time" }, + "triggerDetails": { + "description": "Details of the trigger, e.g. scheduleId, scheduledFor and missedOccurrences of a schedule.", + "type": ["object", "null"] + } + } +} diff --git a/schemas/messages/start-pipeline-run.v1.json b/schemas/messages/start-pipeline-run.v1.json new file mode 100644 index 0000000..3d4acf3 --- /dev/null +++ b/schemas/messages/start-pipeline-run.v1.json @@ -0,0 +1,41 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://raw.githubusercontent.com/net0well/PythonScriptRunner/main/schemas/messages/start-pipeline-run.v1.json", + "title": "PythonScriptRunner StartPipelineRun command (psr.start-pipeline-run.v1)", + "description": "Publish to the exchange psr.public with the routing key pipeline.start (AMQP type psr.start-pipeline-run.v1, content type application/json). The run is announced with run.queued, or refused with run.rejected; both carry the correlationId.", + "type": "object", + "required": ["pipelineKey"], + "properties": { + "pipelineKey": { + "description": "Key of the pipeline.", + "type": "string", + "minLength": 3, + "maxLength": 64, + "pattern": "^[a-z0-9]+(-[a-z0-9]+)*$" + }, + "version": { + "description": "Version to run; null runs the latest published version.", + "type": ["integer", "null"], + "minimum": 1 + }, + "inputs": { + "description": "Pipeline inputs in the wire format of the data mapping. Files are not accepted: pipelines with file inputs are rejected.", + "type": ["object", "null"] + }, + "idempotencyKey": { + "description": "Repeated commands with the same key within 24 hours return the same run. Without it, the AMQP message_id is used.", + "type": ["string", "null"], + "maxLength": 190 + }, + "correlationId": { + "description": "Returned in every event of the run.", + "type": ["string", "null"], + "maxLength": 190 + }, + "requestedBy": { + "description": "Name of the calling system; the run is triggered by queue:.", + "type": ["string", "null"], + "maxLength": 190 + } + } +} diff --git a/src/PythonScriptRunner.Abstractions/Extensibility/Interfaces.cs b/src/PythonScriptRunner.Abstractions/Extensibility/Interfaces.cs index a4f7742..e1e4476 100644 --- a/src/PythonScriptRunner.Abstractions/Extensibility/Interfaces.cs +++ b/src/PythonScriptRunner.Abstractions/Extensibility/Interfaces.cs @@ -107,6 +107,12 @@ public sealed record RunStarting /// Deduplication key given by the caller. public string? IdempotencyKey { get; init; } + /// Correlation given by the caller. + public string? CorrelationId { get; init; } + + /// W3C trace context of the caller (traceparent). + public string? TraceParent { get; init; } + /// Validated inputs in wire format (files carry metadata only). public required JsonObject Inputs { get; init; } diff --git a/src/PythonScriptRunner.Abstractions/Ids.cs b/src/PythonScriptRunner.Abstractions/Ids.cs index e935398..8c53214 100644 --- a/src/PythonScriptRunner.Abstractions/Ids.cs +++ b/src/PythonScriptRunner.Abstractions/Ids.cs @@ -1,7 +1,10 @@ +using System.Text.Json.Serialization; + namespace PythonScriptRunner; /// Identifier of a run (time-ordered UUID v7). Its timestamp is the creation time of the run. /// The underlying GUID. +[JsonConverter(typeof(RunIdJsonConverter))] public readonly record struct RunId(Guid Value) { /// Creates a new time-ordered identifier. @@ -35,6 +38,7 @@ public readonly record struct AttemptId(Guid Value) /// Identifier of a step within a run (time-ordered UUID v7). /// The underlying GUID. +[JsonConverter(typeof(StepRunIdJsonConverter))] public readonly record struct StepRunId(Guid Value) { /// Creates a new time-ordered identifier. diff --git a/src/PythonScriptRunner.Abstractions/Management/CatalogAdmin.cs b/src/PythonScriptRunner.Abstractions/Management/CatalogAdmin.cs index 2f8d846..a2a1839 100644 --- a/src/PythonScriptRunner.Abstractions/Management/CatalogAdmin.cs +++ b/src/PythonScriptRunner.Abstractions/Management/CatalogAdmin.cs @@ -238,6 +238,52 @@ Task UpdatePipelineRetentionAsync( /// Author. /// Cancellation token. Task ImportAsync(Stream zip, Actor actor, CancellationToken cancellationToken = default); + + /// Lists schedules ordered by name. + /// Pipeline filter and page. + /// Cancellation token. + Task> ListSchedulesAsync(ScheduleQuery query, CancellationToken cancellationToken = default); + + /// Reads a schedule. + /// Schedule identifier. + /// Cancellation token. + /// The schedule, or null when it does not exist. + Task GetScheduleAsync(ScheduleId id, CancellationToken cancellationToken = default); + + /// + /// Creates a cron schedule of a pipeline. Validates the cron expression, the IANA time zone, the version (pinned or the latest + /// published) and the fixed inputs: SCHEDULE_INVALID, SCHEDULE_FILE_INPUT (pipelines with file inputs cannot be + /// scheduled) or . + /// + /// Pipeline key. + /// Schedule settings. + /// Author. + /// Cancellation token. + Task CreateScheduleAsync(string pipelineKey, ScheduleData schedule, Actor actor, CancellationToken cancellationToken = default); + + /// Changes a schedule (same validation as ); the next run is recomputed. + /// Schedule identifier. + /// New settings. + /// Row version read by the caller. + /// Author. + /// Cancellation token. + Task UpdateScheduleAsync( + ScheduleId id, ScheduleData schedule, uint expectedRowVersion, Actor actor, CancellationToken cancellationToken = default); + + /// Enables or disables a schedule; enabling recomputes the next run from now. + /// Schedule identifier. + /// Whether it fires. + /// Row version read by the caller. + /// Author. + /// Cancellation token. + Task SetScheduleEnabledAsync( + ScheduleId id, bool enabled, uint expectedRowVersion, Actor actor, CancellationToken cancellationToken = default); + + /// Deletes a schedule (its runs stay in the history). + /// Schedule identifier. + /// Author. + /// Cancellation token. + Task DeleteScheduleAsync(ScheduleId id, Actor actor, CancellationToken cancellationToken = default); } /// Search and page of a catalog list. diff --git a/src/PythonScriptRunner.Abstractions/Management/DomainErrors.cs b/src/PythonScriptRunner.Abstractions/Management/DomainErrors.cs index 6997f2d..2f1baa9 100644 --- a/src/PythonScriptRunner.Abstractions/Management/DomainErrors.cs +++ b/src/PythonScriptRunner.Abstractions/Management/DomainErrors.cs @@ -56,6 +56,18 @@ public static class PsrErrorCodes /// The imported archive is not a valid catalog folder. public const string ImportInvalid = "IMPORT_INVALID"; + + /// The schedule does not exist. + public const string ScheduleNotFound = "SCHEDULE_NOT_FOUND"; + + /// The schedule has an invalid cron expression, time zone or version. + public const string ScheduleInvalid = "SCHEDULE_INVALID"; + + /// Pipelines with file inputs cannot be scheduled nor started by the public queue command (D-89). + public const string ScheduleFileInput = "SCHEDULE_FILE_INPUT"; + + /// The pipeline has file inputs, which the public queue command cannot send (D-89). + public const string FileInputNotAllowed = "FILE_INPUT_NOT_ALLOWED"; } /// A business rule rejected the operation; is stable (see ). diff --git a/src/PythonScriptRunner.Abstractions/Management/RunManagement.cs b/src/PythonScriptRunner.Abstractions/Management/RunManagement.cs index efff25f..1e954c1 100644 --- a/src/PythonScriptRunner.Abstractions/Management/RunManagement.cs +++ b/src/PythonScriptRunner.Abstractions/Management/RunManagement.cs @@ -38,7 +38,10 @@ public interface IRunManager /// /// Run identifier. /// Who reprocesses. - /// runs it now in this process; needs a run queue. + /// + /// runs it now in this process; queues the failed step for the + /// workers (requires the RabbitMQ package). + /// /// Cancellation token (cancels a synchronous reprocess). /// The run after reprocessing (sync) or after it was queued (async). Task ReprocessAsync(RunId runId, Actor actor, ExecutionMode mode = ExecutionMode.Async, CancellationToken cancellationToken = default); @@ -141,6 +144,15 @@ public sealed record RunSnapshot /// Idempotency key given by the caller. public string? IdempotencyKey { get; init; } + /// Correlation given by the caller (returned in the events). + public string? CorrelationId { get; init; } + + /// Schedule that fired the run. + public ScheduleId? ScheduleId { get; init; } + + /// Details of the trigger, e.g. {"scheduledFor": "...", "missedOccurrences": 10} for a schedule recovery. + public JsonObject? TriggerDetails { get; init; } + /// Creation time. public required DateTimeOffset CreatedAt { get; init; } diff --git a/src/PythonScriptRunner.Abstractions/Management/Schedules.cs b/src/PythonScriptRunner.Abstractions/Management/Schedules.cs new file mode 100644 index 0000000..9b1c789 --- /dev/null +++ b/src/PythonScriptRunner.Abstractions/Management/Schedules.cs @@ -0,0 +1,65 @@ +using System.Text.Json.Nodes; + +namespace PythonScriptRunner; + +/// Settings of a cron schedule of a pipeline. +public sealed record ScheduleData +{ + /// Default IANA time zone of schedules. + public const string DefaultTimeZone = "America/Sao_Paulo"; + + /// Display name (1-200 chars). + public required string Name { get; init; } + + /// Cron expression with 5 fields (minute precision) or 6 fields (with seconds). + public required string Cron { get; init; } + + /// IANA time zone in which the expression is evaluated (daylight saving time handled). + public string TimeZone { get; init; } = DefaultTimeZone; + + /// Version to run; null runs the latest published version at each firing. + public int? PinnedVersion { get; init; } + + /// Fixed inputs of the pipeline (no files). + public JsonObject Inputs { get; init; } = []; + + /// Whether the schedule fires. + public bool IsEnabled { get; init; } = true; +} + +/// A stored schedule. +public sealed record ScheduleDetail +{ + /// Identifier. + public required ScheduleId Id { get; init; } + + /// Scheduled pipeline. + public required string PipelineKey { get; init; } + + /// Settings. + public required ScheduleData Data { get; init; } + + /// Next firing (UTC); null when disabled. + public DateTimeOffset? NextRunAt { get; init; } + + /// Last firing. + public DateTimeOffset? LastFiredAt { get; init; } + + /// Run created by the last successful firing. + public RunId? LastRunId { get; init; } + + /// Why the last firing was rejected (cleared by the next successful one). + public string? LastError { get; init; } + + /// Row version for optimistic concurrency. + public required uint RowVersion { get; init; } + + /// Who created and last changed it. + public required Authorship Authorship { get; init; } +} + +/// Filter and page of a schedule list. +/// Only schedules of this pipeline. +/// 1-based page number. +/// Items per page (1-100). +public sealed record ScheduleQuery(string? PipelineKey = null, int PageNumber = 1, int PageSize = 25); diff --git a/src/PythonScriptRunner.Abstractions/Messaging/Commands.cs b/src/PythonScriptRunner.Abstractions/Messaging/Commands.cs new file mode 100644 index 0000000..d9961cf --- /dev/null +++ b/src/PythonScriptRunner.Abstractions/Messaging/Commands.cs @@ -0,0 +1,102 @@ +using System.Text.Json; +using System.Text.Json.Nodes; +using System.Text.Json.Serialization; + +namespace PythonScriptRunner; + +/// +/// Internal command psr.execute-step.v1: execute one attempt of a step. It carries identifiers only; inputs and files +/// stay in the database. +/// +/// Run identifier. +/// Creation time of the run (its partition). +/// Step identifier. +/// Attempt number to start. +public sealed record ExecuteStepCommand(RunId RunId, DateTimeOffset RunCreatedAt, StepRunId StepRunId, int Attempt) +{ + /// Why the attempt starts. + public AttemptCause Cause { get; init; } = AttemptCause.Initial; + + /// Who started the attempt (the reprocessing user, or the run's caller). + public string? TriggeredBy { get; init; } +} + +/// +/// Public command psr.start-pipeline-run.v1 (exchange psr.public): lets systems that are not .NET start a +/// pipeline. Inputs cannot contain files. +/// +public sealed record StartPipelineRunCommand +{ + /// Pipeline key. + public required string PipelineKey { get; init; } + + /// Version to run; null runs the latest published version. + public int? Version { get; init; } + + /// Pipeline inputs (no files). + public JsonObject? Inputs { get; init; } + + /// Deduplicates repeated commands for 24 hours (default: the AMQP message_id). + public string? IdempotencyKey { get; init; } + + /// Returned in every event of the run. + public string? CorrelationId { get; init; } + + /// Name of the calling system (the run is triggered by queue:<name>). + public string? RequestedBy { get; init; } +} + +/// Control message psr.cancel-run.v1 (fanout to every instance): stop the processes of a run. +/// Run identifier. +/// Running step, when known. +/// Who cancelled. +public sealed record CancelRunCommand(RunId RunId, StepRunId? StepRunId, string RequestedBy); + +/// Versioned message types (AMQP type property). +public static class PsrMessageTypes +{ + /// . + public const string ExecuteStep = "psr.execute-step.v1"; + + /// . + public const string StartPipelineRun = "psr.start-pipeline-run.v1"; + + /// . + public const string CancelRun = "psr.cancel-run.v1"; + + /// . + public const string RunEvent = "psr.run-event.v1"; +} + +/// JSON settings of the messages: camelCase properties and enum names as strings. +public static class PsrMessageJson +{ + /// Shared options (do not modify). + public static JsonSerializerOptions Options { get; } = CreateOptions(); + + private static JsonSerializerOptions CreateOptions() + { + var options = new JsonSerializerOptions(JsonSerializerDefaults.Web) + { + Converters = { new JsonStringEnumConverter() }, + }; + options.MakeReadOnly(populateMissingResolver: true); + return options; + } +} + +/// Writes as its GUID text. +internal sealed class RunIdJsonConverter : JsonConverter +{ + public override RunId Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options) => new(reader.GetGuid()); + + public override void Write(Utf8JsonWriter writer, RunId value, JsonSerializerOptions options) => writer.WriteStringValue(value.Value); +} + +/// Writes as its GUID text. +internal sealed class StepRunIdJsonConverter : JsonConverter +{ + public override StepRunId Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options) => new(reader.GetGuid()); + + public override void Write(Utf8JsonWriter writer, StepRunId value, JsonSerializerOptions options) => writer.WriteStringValue(value.Value); +} diff --git a/src/PythonScriptRunner.Abstractions/Messaging/RunEvents.cs b/src/PythonScriptRunner.Abstractions/Messaging/RunEvents.cs new file mode 100644 index 0000000..45e07e0 --- /dev/null +++ b/src/PythonScriptRunner.Abstractions/Messaging/RunEvents.cs @@ -0,0 +1,137 @@ +using System.Text.Json.Nodes; +using System.Text.Json.Serialization; + +namespace PythonScriptRunner; + +/// +/// Event of a run (psr.run-event.v1), published on the psr.events exchange with the routing key +/// <type>.<pipeline key> or <type>.stage:<stage key>. Fields are only added within v1. +/// +public sealed record RunEvent +{ + /// Unique event identifier. + public required Guid EventId { get; init; } + + /// Event type (see ). + public required string Type { get; init; } + + /// When it happened. + public required DateTimeOffset OccurredAt { get; init; } + + /// The run (null when a request was rejected before a run existed). + public RunId? RunId { get; init; } + + /// What the run executes. + public RunKind? Kind { get; init; } + + /// Pipeline key, for pipeline runs (and rejected pipeline requests). + public string? PipelineKey { get; init; } + + /// Pipeline version, for pipeline runs. + public int? PipelineVersion { get; init; } + + /// Stage key, for single-stage runs. + public string? StageKey { get; init; } + + /// Stage version, for single-stage runs. + public int? StageVersion { get; init; } + + /// Correlation given by the caller (e.g. the correlationId of StartPipelineRun). + public string? CorrelationId { get; init; } + + /// How the run started. + public RunTrigger? Trigger { get; init; } + + /// Status of the run after the event. + public RunStatus? Status { get; init; } + + /// The step, for step.* events. + public RunEventStep? Step { get; init; } + + /// Failure (step.failed, run.failed, run.rejected). + public RunEventError? Error { get; init; } + + /// + /// Declared outputs, for run.succeeded: values inline, files as {"$artifact": {id, name, contentType, size}}. + /// null when larger than 256 KB (see ). + /// + public JsonObject? Outputs { get; init; } + + /// Whether was omitted because it was too large. + public bool OutputsTruncated { get; init; } + + /// When the next attempt starts, for step.retry-scheduled. + public DateTimeOffset? RetryAt { get; init; } + + /// Details of the trigger (schedule id, scheduled time, missed occurrences). + public JsonObject? TriggerDetails { get; init; } + + /// The target used in the routing key: the pipeline key or stage:<stage key>. + [JsonIgnore] + public string RoutingTarget => PipelineKey ?? (StageKey is { } stage ? $"stage:{stage}" : "unknown"); +} + +/// The step of a step.* event. +/// Step key. +/// 1-based position. +/// Attempt number. +public sealed record RunEventStep(string Key, int Order, int Attempt); + +/// Failure carried by an event (no traceback). +/// Error kind (an name, or Rejected). +/// Stable code. +/// Human-readable message. +/// Key of the step that failed. +public sealed record RunEventError(string Kind, string? Code, string Message, string? Step) +{ + /// Why a request was rejected (run.rejected). + public IReadOnlyList? Reasons { get; init; } + + /// Converts a run error. + /// The error. + public static RunEventError? From(RunError? error) => + error is null ? null : new(error.Kind.ToString(), error.Code, error.Message, error.StepKey); +} + +/// Types of . +public static class RunEventTypes +{ + /// A run was queued (or a duplicate request returned it). + public const string RunQueued = "run.queued"; + + /// A queue command or a schedule was rejected; no run was created. + public const string RunRejected = "run.rejected"; + + /// The first step of a run started. + public const string RunStarted = "run.started"; + + /// An attempt of a step started. + public const string StepStarted = "step.started"; + + /// A step succeeded. + public const string StepSucceeded = "step.succeeded"; + + /// A step failed and will be retried automatically. + public const string StepRetryScheduled = "step.retry-scheduled"; + + /// A step failed for good. + public const string StepFailed = "step.failed"; + + /// A run succeeded. + public const string RunSucceeded = "run.succeeded"; + + /// A run failed. + public const string RunFailed = "run.failed"; + + /// A run was cancelled. + public const string RunCancelled = "run.cancelled"; +} + +/// Receives run events from a RabbitMQ subscription (registered with AddRunEventHandler). +public interface IRunEventHandler +{ + /// Handles one event. Throwing makes the broker deliver it again (up to the delivery limit). + /// The event. + /// Cancellation token. + Task HandleAsync(RunEvent runEvent, CancellationToken cancellationToken); +} diff --git a/src/PythonScriptRunner.Abstractions/Messaging/RunQueue.cs b/src/PythonScriptRunner.Abstractions/Messaging/RunQueue.cs new file mode 100644 index 0000000..713b4e9 --- /dev/null +++ b/src/PythonScriptRunner.Abstractions/Messaging/RunQueue.cs @@ -0,0 +1,36 @@ +namespace PythonScriptRunner; + +/// +/// Queues runs for the workers (requires the persistence and RabbitMQ packages). The run, its first step and the message +/// are stored in one transaction, so a run is queued even while the broker is down. +/// +public interface IRunQueue +{ + /// Queues a published pipeline. + /// Pipeline key. + /// Pipeline inputs (same forms as ; files are stored in the database). + /// Version, idempotency key, correlation and caller identity. + /// Cancellation token. + /// The run: , or when the pipeline is at its concurrency limit. + /// The pipeline does not exist. + /// The requested version does not exist. + /// The inputs do not match the contract (no run is created). + Task EnqueuePipelineAsync(string pipelineKey, object? inputs, RunOptions? options = null, CancellationToken cancellationToken = default); + + /// Queues a published stage as a single-step run. + /// Stage key. + /// Stage inputs. + /// Version, idempotency key, correlation and caller identity. + /// Cancellation token. + /// The queued run. + /// The stage does not exist. + /// The requested version does not exist. + /// The inputs do not match the contract (no run is created). + Task EnqueueStageAsync(string stageKey, object? inputs, RunOptions? options = null, CancellationToken cancellationToken = default); +} + +/// A queued run. +/// Run identifier. +/// or (the current status when deduplicated). +/// Whether an earlier run with the same idempotency key was returned. +public sealed record RunHandle(RunId RunId, RunStatus Status, bool Deduplicated); diff --git a/src/PythonScriptRunner.Abstractions/Running/IPythonRunner.cs b/src/PythonScriptRunner.Abstractions/Running/IPythonRunner.cs index 2fc069c..c84ac5b 100644 --- a/src/PythonScriptRunner.Abstractions/Running/IPythonRunner.cs +++ b/src/PythonScriptRunner.Abstractions/Running/IPythonRunner.cs @@ -42,4 +42,7 @@ public sealed record RunOptions /// How the run was started. Default: . public RunTrigger Trigger { get; init; } = RunTrigger.Library; + + /// Correlation of the caller, stored with the run and returned in its events (requires persistence). + public string? CorrelationId { get; init; } } diff --git a/src/PythonScriptRunner.Abstractions/Running/IRunEngine.cs b/src/PythonScriptRunner.Abstractions/Running/IRunEngine.cs index 92a03e7..3d931fb 100644 --- a/src/PythonScriptRunner.Abstractions/Running/IRunEngine.cs +++ b/src/PythonScriptRunner.Abstractions/Running/IRunEngine.cs @@ -37,10 +37,123 @@ Task RunPipelineAsync( /// Cancels the run. Task ResumeAsync(ResumeRequest request, CancellationToken cancellationToken = default); - /// Cancels a run that is executing in this process. + /// Cancels a run that is executing in this process (including a step started by ). /// Run identifier. /// Whether the run was executing here. bool RequestCancellation(RunId runId); + + /// Validates inputs against declared fields and converts them to the wire format, without running anything. + /// Declared inputs (of a stage or a pipeline). + /// Inputs (same forms as ). + /// Wire values (files carry metadata only) and the files by field name. + /// The inputs do not match the contract. + BoundRunInputs BindInputs(IReadOnlyList fields, object? inputs); + + /// + /// Executes one attempt of one step of a run whose state is kept by the caller (the asynchronous worker): resolves the + /// bindings, downloads the files it needs, runs the Python process and validates the outputs. Nothing is recorded and + /// there is no automatic retry: the caller stores the result and decides what comes next. Dispose the result to delete + /// the local files. + /// + /// The step, its run and the outputs of the steps already completed. + /// Cancels the attempt (the process tree is stopped and the step ends Cancelled). + Task ExecuteStepAsync(StepExecutionRequest request, CancellationToken cancellationToken = default); + + /// Removes work folders left behind by interrupted runs (older than OrphanRetention). + /// How many folders were removed. + int RemoveOrphanWorkFolders(); +} + +/// Validated inputs. +/// Inputs in wire format; file values carry metadata only. +/// Files by field name (in list order for list<file>). +public sealed record BoundRunInputs(JsonObject Wire, IReadOnlyDictionary> Files); + +/// One attempt of one step to execute with . +public sealed record StepExecutionRequest +{ + /// Run identifier. + public required RunId RunId { get; init; } + + /// What the run executes. + public required RunKind Kind { get; init; } + + /// Exact stage version of the step (any status). + public required StageDefinition Stage { get; init; } + + /// The pipeline, for pipeline runs (its bindings give the step inputs); null for single-stage runs. + public PipelineDefinition? Pipeline { get; init; } + + /// 1-based position of the step. + public required int Order { get; init; } + + /// Validated inputs of the run in wire format. + public required JsonObject RunInputs { get; init; } + + /// Input files of the run by field name. + public IReadOnlyDictionary> RunInputFiles { get; init; } = new Dictionary>(); + + /// Earlier steps; the succeeded ones provide outputs and files to the bindings. + public IReadOnlyList CompletedSteps { get; init; } = []; + + /// Identifier of the attempt (already recorded by the caller). + public required AttemptId AttemptId { get; init; } + + /// 1-based attempt number. + public required int Attempt { get; init; } + + /// Why the attempt starts. + public required AttemptCause Cause { get; init; } + + /// Who started the attempt. + public required string TriggeredBy { get; init; } +} + +/// Result of . Dispose it to delete the local files. +/// +/// The step: holds this attempt only (empty when the bindings could not be resolved). +/// +/// Resolved step inputs with file metadata; null when the bindings failed. +/// Input files of the step by field name. +/// Declared outputs of the run, when this was its last step and it succeeded. +/// Declared file outputs of the run, under the same condition. +/// Deletes the local files. +public sealed class StepExecution( + StepResult step, + JsonObject? inputs, + IReadOnlyDictionary> inputFiles, + JsonObject? runOutputs, + IReadOnlyDictionary>? runOutputFiles, + Func? cleanup = null) : IAsyncDisposable +{ + private int _disposed; + + /// The step and its attempt. + public StepResult Step { get; } = step; + + /// The attempt, or null when no process ran because the bindings could not be resolved. + public AttemptResult? Attempt => Step.Attempts.Count > 0 ? Step.Attempts[^1] : null; + + /// Resolved step inputs with file metadata. + public JsonObject? Inputs { get; } = inputs; + + /// Input files of the step by field name. + public IReadOnlyDictionary> InputFiles { get; } = inputFiles; + + /// Declared outputs of the run, when this was its last step and it succeeded. + public JsonObject? RunOutputs { get; } = runOutputs; + + /// Declared file outputs of the run, when this was its last step and it succeeded. + public IReadOnlyDictionary>? RunOutputFiles { get; } = runOutputFiles; + + /// + public async ValueTask DisposeAsync() + { + if (Interlocked.Exchange(ref _disposed, 1) == 0 && cleanup is not null) + { + await cleanup().ConfigureAwait(false); + } + } } /// State of a failed run to resume. diff --git a/src/PythonScriptRunner.Core/PythonScriptRunner.Core.csproj b/src/PythonScriptRunner.Core/PythonScriptRunner.Core.csproj index 70f4fef..7395d79 100644 --- a/src/PythonScriptRunner.Core/PythonScriptRunner.Core.csproj +++ b/src/PythonScriptRunner.Core/PythonScriptRunner.Core.csproj @@ -18,6 +18,8 @@ + + diff --git a/src/PythonScriptRunner.Core/Running/PythonRunner.cs b/src/PythonScriptRunner.Core/Running/PythonRunner.cs index 78ec229..3dd8a0f 100644 --- a/src/PythonScriptRunner.Core/Running/PythonRunner.cs +++ b/src/PythonScriptRunner.Core/Running/PythonRunner.cs @@ -185,6 +185,134 @@ public bool RequestCancellation(RunId runId) } } + public BoundRunInputs BindInputs(IReadOnlyList fields, object? inputs) + { + ArgumentNullException.ThrowIfNull(fields); + var bound = InputBinder.Bind(fields, inputs, Settings.MaxFileBytes); + var files = bound.Files.GroupBy(f => f.Field, StringComparer.Ordinal) + .ToDictionary(g => g.Key, g => (IReadOnlyList)[.. g.OrderBy(f => f.Index).Select(f => f.File)], StringComparer.Ordinal); + return new BoundRunInputs(bound.Wire, files); + } + + public int RemoveOrphanWorkFolders() => _janitor.Clean(); + + public async Task ExecuteStepAsync(StepExecutionRequest request, CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(request); + var stage = request.Stage; + var settings = Settings; + _janitor.EnsureCleaned(); + var context = new RunContext(new RunWorkspace(settings.WorkRoot, request.RunId), settings, request.Kind, request.TriggeredBy); + var stepKey = request.Pipeline is { } definition ? definition.Steps[request.Order - 1].Key : stage.Key; + var completed = request.CompletedSteps.Where(s => s.Status == StepStatus.Succeeded).ToDictionary(s => s.StepKey, StringComparer.Ordinal); + async ValueTask Cleanup() + { + context.Workspace.DeleteRun(); + await context.Workspace.DeleteResultsAsync().ConfigureAwait(false); + } + + using var cancellation = Register(request.RunId, cancellationToken); + var token = cancellation.Token; + try + { + var resolved = request.Pipeline is { } pipeline + ? StepInputResolver.Resolve(stage, pipeline, request.Order - 1, request.RunInputs, request.RunInputFiles, completed) + : new StepInputs((JsonObject)request.RunInputs.DeepClone(), request.RunInputFiles, []); + if (resolved.Errors.Count > 0) + { + return new StepExecution(InvalidInputs(stage, stepKey, request.Order, resolved.Errors, prior: null), null, NoFiles, null, null, Cleanup); + } + + var startedAt = _time.GetUtcNow(); + context.UsedBytes = TotalBytes(request.RunInputFiles) + completed.Values.Sum(s => TotalBytes(s.Files)); + AttemptOutcome outcome; + if (context.UsedBytes > settings.MaxRunFilesBytes) + { + outcome = new AttemptOutcome( + NotStarted(request, startedAt, new RunError(ErrorKind.ArtifactLimitExceeded, "The files of the run exceed the run limit.") + { + Code = "RUN_FILES_TOO_LARGE", + }), + null, + NoFiles); + } + else + { + try + { + var local = await MaterializeStoredAsync(resolved.Files, Path.Combine(context.Workspace.RunDirectory, "step-inputs", stepKey), token) + .ConfigureAwait(false); + outcome = await RunAttemptAsync(stage, stepKey, request.Attempt, request.Cause, request.AttemptId, startedAt, resolved.Wire, + ToCopies(local), context, token).ConfigureAwait(false); + } + catch (OperationCanceledException) when (token.IsCancellationRequested) + { + outcome = new AttemptOutcome( + NotStarted(request, startedAt, new RunError(ErrorKind.Cancelled, "The run was cancelled.")) with { Status = AttemptStatus.Cancelled }, + null, + NoFiles); + } + } + + var attempt = outcome.Attempt; + var succeeded = attempt.Status == AttemptStatus.Succeeded; + var step = new StepResult + { + StepKey = stepKey, + Order = request.Order, + StageKey = stage.Key, + StageVersion = stage.Version, + Status = attempt.Status switch + { + AttemptStatus.Succeeded => StepStatus.Succeeded, + AttemptStatus.TimedOut => StepStatus.TimedOut, + AttemptStatus.Cancelled => StepStatus.Cancelled, + _ => StepStatus.Failed, + }, + Attempts = [attempt], + Outputs = succeeded ? outcome.Outputs : null, + Files = succeeded ? outcome.Files : NoFiles, + Error = attempt.Error is null ? null : attempt.Error with { StepKey = stepKey }, + }; + + JsonObject? runOutputs = null; + IReadOnlyDictionary>? runFiles = null; + if (succeeded && request.Pipeline is null) + { + (runOutputs, runFiles) = ((JsonObject)step.Outputs!.DeepClone(), step.Files); + } + else if (succeeded && request.Pipeline is { } last && request.Order == last.Steps.Count) + { + completed[stepKey] = step; + (runOutputs, runFiles) = PipelineOutputs(last, completed); + } + + return new StepExecution(step, WithFilePlaceholders(resolved.Wire, resolved.Files, stage), resolved.Files, runOutputs, runFiles, Cleanup); + } + catch + { + await Cleanup().ConfigureAwait(false); + throw; + } + finally + { + _active.TryRemove(request.RunId, out _); + } + } + + /// An attempt that failed before its process started. + private static AttemptResult NotStarted(StepExecutionRequest request, DateTimeOffset startedAt, RunError error) => new() + { + Id = request.AttemptId, + Number = request.Attempt, + Cause = request.Cause, + Status = AttemptStatus.Failed, + StartedAt = startedAt, + Duration = TimeSpan.Zero, + Overhead = TimeSpan.Zero, + Error = error, + }; + public void Dispose() { if (_concurrency.IsValueCreated) @@ -223,6 +351,8 @@ private async Task RunNewAsync( Trigger = options?.Trigger ?? RunTrigger.Library, TriggeredBy = context.TriggeredBy, IdempotencyKey = options?.IdempotencyKey, + CorrelationId = options?.CorrelationId, + TraceParent = Activity.Current?.Id, Inputs = (JsonObject)bound.Wire.DeepClone(), InputFiles = overBudget ? NoFiles : inputFiles, Steps = target.Steps, @@ -365,6 +495,14 @@ private async Task ExecutePipelineAsync( return new RunOutcome(status, steps, null, null, error); } + var (outputs, files) = PipelineOutputs(pipeline, completed); + return new RunOutcome(RunStatus.Succeeded, steps, outputs, files, null); + } + + /// The declared outputs of a pipeline, read from its completed steps. + private static (JsonObject Outputs, Dictionary> Files) PipelineOutputs( + PipelineDefinition pipeline, IReadOnlyDictionary completed) + { var outputs = new JsonObject(); var files = new Dictionary>(StringComparer.Ordinal); foreach (var output in pipeline.Outputs) @@ -381,7 +519,7 @@ private async Task ExecutePipelineAsync( } } - return new RunOutcome(RunStatus.Succeeded, steps, outputs, files, null); + return (outputs, files); } private async Task ResolveStageAsync(string key, int? version, CancellationToken cancellationToken) @@ -450,7 +588,13 @@ await _recorder.StepStartedAsync( for (var number = first; ; number++) { var cause = number == first ? firstCause : AttemptCause.AutoRetry; - var outcome = await RunAttemptAsync(stage, stepKey, order, number, cause, inputs, copies, context, cancellationToken).ConfigureAwait(false); + var attemptId = AttemptId.New(); + var startedAt = _time.GetUtcNow(); + await _recorder.AttemptStartedAsync( + new AttemptStarted(context.RunId, stepKey, order, attemptId, number, cause, context.TriggeredBy, startedAt), + CancellationToken.None).ConfigureAwait(false); + var outcome = await RunAttemptAsync(stage, stepKey, number, cause, attemptId, startedAt, inputs, copies, context, cancellationToken) + .ConfigureAwait(false); attempts.Add(outcome.Attempt); await _recorder.AttemptFinishedAsync(new AttemptFinished(context.RunId, stepKey, order, outcome.Attempt), CancellationToken.None) .ConfigureAwait(false); @@ -498,27 +642,24 @@ private async Task RecordStepAsync(RunContext context, StepResult st return step; } + /// Runs one Python process (the attempt is already reported by the caller when it is recorded). private async Task RunAttemptAsync( StageDefinition stage, string stepKey, - int order, int number, AttemptCause cause, + AttemptId attemptId, + DateTimeOffset startedAt, JsonObject inputs, IReadOnlyList files, RunContext context, CancellationToken cancellationToken) { var settings = context.Settings; - var attemptId = AttemptId.New(); - var startedAt = _time.GetUtcNow(); var started = _time.GetTimestamp(); var queued = TimeSpan.Zero; string? environmentHash = null; var workDirectory = context.Workspace.AttemptDirectory(stepKey, number); - await _recorder.AttemptStartedAsync( - new AttemptStarted(context.RunId, stepKey, order, attemptId, number, cause, context.TriggeredBy, startedAt), - CancellationToken.None).ConfigureAwait(false); using var activity = PsrDiagnostics.Source.StartActivity("psr.stage.execute"); activity?.SetTag("psr.stage.key", stage.Key); diff --git a/src/PythonScriptRunner.Core/Running/RunWorkspace.cs b/src/PythonScriptRunner.Core/Running/RunWorkspace.cs index d303ac5..a8751e4 100644 --- a/src/PythonScriptRunner.Core/Running/RunWorkspace.cs +++ b/src/PythonScriptRunner.Core/Running/RunWorkspace.cs @@ -61,13 +61,19 @@ internal sealed class WorkspaceJanitor(IOptions optio { private int _cleaned; + /// Cleans once per process (the worker also calls periodically). public void EnsureCleaned() { - if (Interlocked.Exchange(ref _cleaned, 1) == 1) + if (Interlocked.Exchange(ref _cleaned, 1) == 0) { - return; + Clean(); } + } + /// Removes leftover folders now. + /// How many folders were removed. + public int Clean() + { var settings = options.Value; var cutoff = (time.GetUtcNow() - settings.OrphanRetention).UtcDateTime; var removed = 0; @@ -93,5 +99,7 @@ public void EnsureCleaned() { Log.OrphansRemoved(logger, removed, settings.OrphanRetention); } + + return removed; } } diff --git a/src/PythonScriptRunner.EntityFrameworkCore/Catalog/EfCatalogAdmin.Pipelines.cs b/src/PythonScriptRunner.EntityFrameworkCore/Catalog/EfCatalogAdmin.Pipelines.cs index d0da7af..e14ea6c 100644 --- a/src/PythonScriptRunner.EntityFrameworkCore/Catalog/EfCatalogAdmin.Pipelines.cs +++ b/src/PythonScriptRunner.EntityFrameworkCore/Catalog/EfCatalogAdmin.Pipelines.cs @@ -235,12 +235,14 @@ public async Task PublishPipelineAsync( } var steps = await EfCatalog.LoadStepsAsync(db, row.Id, cancellationToken).ConfigureAwait(false); - var report = await pipelineValidator.ValidateAsync(CatalogMapping.ToPipeline(row.Pipeline, row, steps), cancellationToken).ConfigureAwait(false); + var definition = CatalogMapping.ToPipeline(row.Pipeline, row, steps); + var report = await pipelineValidator.ValidateAsync(definition, cancellationToken).ConfigureAwait(false); if (!report.IsValid) { throw new DefinitionInvalidException(PsrErrorCodes.PipelineInvalid, report); } + List warnings = [.. report.Warnings, .. await ScheduleWarningsAsync(db, row.PipelineId, definition, cancellationToken).ConfigureAwait(false)]; var now = time.GetUtcNow(); row.Status = VersionStatus.Published; row.PublishedAt = now; @@ -252,7 +254,7 @@ public async Task PublishPipelineAsync( AuditLog.Add(db, new AuditEntry(actor, AuditActions.PipelinePublished, "PipelineVersion", row.Id, $"{key}@{version}"), now); await SaveAsync(db, cancellationToken, conflict: Current(() => GetPipelineDraftAsync(key, CancellationToken.None))).ConfigureAwait(false); catalog.InvalidatePipeline(key); - return new PipelinePublication(CatalogMapping.ToPipelineVersionDetail(row.Pipeline, row, steps), report.Warnings); + return new PipelinePublication(CatalogMapping.ToPipelineVersionDetail(row.Pipeline, row, steps), warnings); } } diff --git a/src/PythonScriptRunner.EntityFrameworkCore/Catalog/EfCatalogAdmin.Schedules.cs b/src/PythonScriptRunner.EntityFrameworkCore/Catalog/EfCatalogAdmin.Schedules.cs new file mode 100644 index 0000000..64ec2fe --- /dev/null +++ b/src/PythonScriptRunner.EntityFrameworkCore/Catalog/EfCatalogAdmin.Schedules.cs @@ -0,0 +1,255 @@ +using System.Text.Json.Nodes; +using Microsoft.EntityFrameworkCore; +using PythonScriptRunner.EntityFrameworkCore.Entities; +using PythonScriptRunner.EntityFrameworkCore.Infrastructure; +using PythonScriptRunner.EntityFrameworkCore.Scheduling; + +namespace PythonScriptRunner.EntityFrameworkCore.Catalog; + +/// Cron schedules of pipelines (D-41, D-45, D-77, D-89). +internal sealed partial class EfCatalogAdmin +{ + public async Task> ListSchedulesAsync(ScheduleQuery query, CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(query); + var (pageNumber, pageSize) = (Math.Max(1, query.PageNumber), Math.Clamp(query.PageSize, 1, 100)); + var db = await factory.CreateDbContextAsync(cancellationToken).ConfigureAwait(false); + await using (db.ConfigureAwait(false)) + { + var rows = db.Schedules.AsNoTracking().Join(db.Pipelines, s => s.PipelineId, p => p.Id, (s, p) => new { Schedule = s, p.Key }); + if (query.PipelineKey is { } pipelineKey) + { + rows = rows.Where(r => r.Key == pipelineKey); + } + + var total = await rows.CountAsync(cancellationToken).ConfigureAwait(false); + var items = await rows.OrderBy(r => r.Schedule.Name).ThenBy(r => r.Schedule.Id) + .Skip((pageNumber - 1) * pageSize).Take(pageSize) + .ToListAsync(cancellationToken).ConfigureAwait(false); + return new Page([.. items.Select(r => ToScheduleDetail(r.Schedule, r.Key))], pageNumber, pageSize, total); + } + } + + public async Task GetScheduleAsync(ScheduleId id, CancellationToken cancellationToken = default) + { + var db = await factory.CreateDbContextAsync(cancellationToken).ConfigureAwait(false); + await using (db.ConfigureAwait(false)) + { + var row = await db.Schedules.AsNoTracking().Join(db.Pipelines, s => s.PipelineId, p => p.Id, (s, p) => new { Schedule = s, p.Key }) + .SingleOrDefaultAsync(r => r.Schedule.Id == id.Value, cancellationToken).ConfigureAwait(false); + return row is null ? null : ToScheduleDetail(row.Schedule, row.Key); + } + } + + public async Task CreateScheduleAsync(string pipelineKey, ScheduleData schedule, Actor actor, CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(schedule); + ArgumentNullException.ThrowIfNull(actor); + var db = await factory.CreateDbContextAsync(cancellationToken).ConfigureAwait(false); + await using (db.ConfigureAwait(false)) + { + var pipeline = await db.Pipelines.AsNoTracking().SingleOrDefaultAsync(p => p.Key == pipelineKey, cancellationToken).ConfigureAwait(false) + ?? throw new PipelineNotFoundException(pipelineKey); + var cron = await ValidateScheduleAsync(pipelineKey, schedule, cancellationToken).ConfigureAwait(false); + var now = time.GetUtcNow(); + var row = new ScheduleEntity + { + Id = Guid.CreateVersion7(now), + PipelineId = pipeline.Id, + Name = schedule.Name, + Cron = schedule.Cron.Trim(), + TimeZone = schedule.TimeZone, + PinnedVersion = schedule.PinnedVersion, + Inputs = schedule.Inputs.ToJsonString(), + IsEnabled = schedule.IsEnabled, + NextRunAt = schedule.IsEnabled ? cron.Next(now) : null, + CreatedAt = now, + CreatedBy = actor.Subject, + UpdatedAt = now, + UpdatedBy = actor.Subject, + }; + db.Schedules.Add(row); + AuditLog.Add(db, new AuditEntry(actor, AuditActions.ScheduleCreated, "Schedule", row.Id, ScheduleKey(pipelineKey, row.Name), Describe(row)), now); + await SaveAsync(db, cancellationToken).ConfigureAwait(false); + return ToScheduleDetail(row, pipelineKey); + } + } + + public async Task UpdateScheduleAsync( + ScheduleId id, ScheduleData schedule, uint expectedRowVersion, Actor actor, CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(schedule); + ArgumentNullException.ThrowIfNull(actor); + var db = await factory.CreateDbContextAsync(cancellationToken).ConfigureAwait(false); + await using (db.ConfigureAwait(false)) + { + var (row, pipelineKey) = await LoadScheduleAsync(db, id, expectedRowVersion, cancellationToken).ConfigureAwait(false); + var cron = await ValidateScheduleAsync(pipelineKey, schedule, cancellationToken).ConfigureAwait(false); + var before = Describe(row); + var now = time.GetUtcNow(); + row.Name = schedule.Name; + row.Cron = schedule.Cron.Trim(); + row.TimeZone = schedule.TimeZone; + row.PinnedVersion = schedule.PinnedVersion; + row.Inputs = schedule.Inputs.ToJsonString(); + row.IsEnabled = schedule.IsEnabled; + row.NextRunAt = schedule.IsEnabled ? cron.Next(now) : null; + row.UpdatedAt = now; + row.UpdatedBy = actor.Subject; + var after = Describe(row); + var changes = new JsonObject(); + foreach (var (name, value) in after) + { + if (!JsonNode.DeepEquals(before[name], value)) + { + changes[name] = new JsonArray(before[name]?.DeepClone(), value?.DeepClone()); + } + } + + AuditLog.Add(db, new AuditEntry(actor, AuditActions.ScheduleUpdated, "Schedule", row.Id, ScheduleKey(pipelineKey, row.Name), changes), now); + await SaveAsync(db, cancellationToken, conflict: Current(() => GetScheduleAsync(id, CancellationToken.None))).ConfigureAwait(false); + return ToScheduleDetail(row, pipelineKey); + } + } + + public async Task SetScheduleEnabledAsync( + ScheduleId id, bool enabled, uint expectedRowVersion, Actor actor, CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(actor); + var db = await factory.CreateDbContextAsync(cancellationToken).ConfigureAwait(false); + await using (db.ConfigureAwait(false)) + { + var (row, pipelineKey) = await LoadScheduleAsync(db, id, expectedRowVersion, cancellationToken).ConfigureAwait(false); + if (row.IsEnabled == enabled) + { + return ToScheduleDetail(row, pipelineKey); + } + + var now = time.GetUtcNow(); + row.IsEnabled = enabled; + row.NextRunAt = enabled ? CronSchedule.Parse(row.Cron, row.TimeZone).Next(now) : null; + row.UpdatedAt = now; + row.UpdatedBy = actor.Subject; + AuditLog.Add(db, new AuditEntry(actor, enabled ? AuditActions.ScheduleEnabled : AuditActions.ScheduleDisabled, "Schedule", row.Id, + ScheduleKey(pipelineKey, row.Name)), now); + await SaveAsync(db, cancellationToken, conflict: Current(() => GetScheduleAsync(id, CancellationToken.None))).ConfigureAwait(false); + return ToScheduleDetail(row, pipelineKey); + } + } + + public async Task DeleteScheduleAsync(ScheduleId id, Actor actor, CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(actor); + var db = await factory.CreateDbContextAsync(cancellationToken).ConfigureAwait(false); + await using (db.ConfigureAwait(false)) + { + var row = await db.Schedules.Join(db.Pipelines, s => s.PipelineId, p => p.Id, (s, p) => new { Schedule = s, p.Key }) + .SingleOrDefaultAsync(r => r.Schedule.Id == id.Value, cancellationToken).ConfigureAwait(false) + ?? throw ScheduleNotFound(id); + db.Schedules.Remove(row.Schedule); + AuditLog.Add(db, new AuditEntry(actor, AuditActions.ScheduleDeleted, "Schedule", row.Schedule.Id, ScheduleKey(row.Key, row.Schedule.Name), + Describe(row.Schedule)), time.GetUtcNow()); + await SaveAsync(db, cancellationToken).ConfigureAwait(false); + } + } + + /// + /// Warnings of D-77: enabled schedules that follow the latest version and do not provide a required input of the new one + /// (their next firings will be rejected). + /// + private static async Task> ScheduleWarningsAsync( + PsrDbContext db, Guid pipelineId, PipelineDefinition published, CancellationToken cancellationToken) + { + var schedules = await db.Schedules.AsNoTracking() + .Where(s => s.PipelineId == pipelineId && s.IsEnabled && s.PinnedVersion == null) + .Select(s => new { s.Name, s.Inputs }) + .ToListAsync(cancellationToken).ConfigureAwait(false); + var warnings = new List(); + foreach (var schedule in schedules) + { + var inputs = StoredJson.Object(schedule.Inputs) ?? []; + var missing = ScheduleRules.MissingInputs(published, inputs) + .Concat(published.Inputs.Where(f => f.Type.IsFile && f.Required).Select(f => f.Name)) + .Distinct(StringComparer.Ordinal) + .ToList(); + if (missing.Count > 0) + { + warnings.Add(new ValidationIssue("SCHEDULE_MISSING_INPUT", + $"schedule '{schedule.Name}' does not provide the required input(s) {string.Join(", ", missing)}; its firings will be rejected")); + } + } + + return warnings; + } + + /// Validates name, cron, time zone, version and inputs; returns the parsed expression. + private async Task ValidateScheduleAsync(string pipelineKey, ScheduleData schedule, CancellationToken cancellationToken) + { + ValidateName(schedule.Name); + ArgumentNullException.ThrowIfNull(schedule.Inputs); + var cron = CronSchedule.Parse(schedule.Cron, schedule.TimeZone); + if (schedule.PinnedVersion is < 1) + { + throw new PsrDomainException(PsrErrorCodes.ScheduleInvalid, "The pinned version must be at least 1 (or null for the latest)."); + } + + var pipeline = await ScheduleRules.ResolveAsync(catalog, pipelineKey, schedule.PinnedVersion, cancellationToken).ConfigureAwait(false); + ScheduleRules.EnsureNoFileInputs(pipeline, schedule.Inputs); + engine.BindInputs(pipeline.Inputs, schedule.Inputs); + return cron; + } + + private static async Task<(ScheduleEntity Row, string PipelineKey)> LoadScheduleAsync( + PsrDbContext db, ScheduleId id, uint expectedRowVersion, CancellationToken cancellationToken) + { + var loaded = await db.Schedules.Join(db.Pipelines, s => s.PipelineId, p => p.Id, (s, p) => new { Schedule = s, p.Key }) + .SingleOrDefaultAsync(r => r.Schedule.Id == id.Value, cancellationToken).ConfigureAwait(false) + ?? throw ScheduleNotFound(id); + if (loaded.Schedule.RowVersion != expectedRowVersion) + { + throw new ConcurrencyConflictException($"Schedule {id} was changed by someone else.", ToScheduleDetail(loaded.Schedule, loaded.Key)); + } + + db.Entry(loaded.Schedule).Property(s => s.RowVersion).OriginalValue = expectedRowVersion; + return (loaded.Schedule, loaded.Key); + } + + private static PsrDomainException ScheduleNotFound(ScheduleId id) => new(PsrErrorCodes.ScheduleNotFound, $"Schedule {id} was not found."); + + private static string ScheduleKey(string pipelineKey, string name) + { + var key = $"{pipelineKey}/{name}"; + return key.Length <= 100 ? key : key[..100]; + } + + private static JsonObject Describe(ScheduleEntity row) => new() + { + ["name"] = row.Name, + ["cron"] = row.Cron, + ["timeZone"] = row.TimeZone, + ["pinnedVersion"] = row.PinnedVersion, + ["inputs"] = StoredJson.Node(row.Inputs), + ["isEnabled"] = row.IsEnabled, + }; + + private static ScheduleDetail ToScheduleDetail(ScheduleEntity row, string pipelineKey) => new() + { + Id = new ScheduleId(row.Id), + PipelineKey = pipelineKey, + Data = new ScheduleData + { + Name = row.Name, + Cron = row.Cron, + TimeZone = row.TimeZone, + PinnedVersion = row.PinnedVersion, + Inputs = StoredJson.Object(row.Inputs) ?? [], + IsEnabled = row.IsEnabled, + }, + NextRunAt = row.NextRunAt, + LastFiredAt = row.LastFiredAt, + LastRunId = row.LastRunId is { } runId ? new RunId(runId) : null, + LastError = row.LastError, + RowVersion = row.RowVersion, + Authorship = new Authorship(row.CreatedAt, row.CreatedBy, row.UpdatedAt, row.UpdatedBy), + }; +} diff --git a/src/PythonScriptRunner.EntityFrameworkCore/Entities/RunEntities.cs b/src/PythonScriptRunner.EntityFrameworkCore/Entities/RunEntities.cs index 2da8f17..deb99a7 100644 --- a/src/PythonScriptRunner.EntityFrameworkCore/Entities/RunEntities.cs +++ b/src/PythonScriptRunner.EntityFrameworkCore/Entities/RunEntities.cs @@ -33,6 +33,9 @@ public sealed class RunEntity public Guid? ScheduleId { get; set; } + /// Details of the trigger as JSON (e.g. scheduled time and missed occurrences of a schedule recovery). + public string? TriggerDetails { get; set; } + public RunStatus Status { get; set; } /// Validated inputs as JSON (files as {"$artifact": id}). diff --git a/src/PythonScriptRunner.EntityFrameworkCore/Infrastructure/PsrClock.cs b/src/PythonScriptRunner.EntityFrameworkCore/Infrastructure/PsrClock.cs new file mode 100644 index 0000000..382d6f1 --- /dev/null +++ b/src/PythonScriptRunner.EntityFrameworkCore/Infrastructure/PsrClock.cs @@ -0,0 +1,79 @@ +using System.Diagnostics; +using Npgsql; +using PythonScriptRunner.EntityFrameworkCore.Runs; + +namespace PythonScriptRunner.EntityFrameworkCore; + +/// +/// Current time of queued runs: the host clock corrected by its offset to the PostgreSQL clock (measured every minute), so +/// lease expiry, retry delays and cron decisions agree across hosts (spec 006). With +/// off it is the registered . +/// +public sealed class PsrClock +{ + private static readonly TimeSpan SyncInterval = TimeSpan.FromMinutes(1); + + private readonly PsrDataSource _dataSource; + private readonly PsrEntityFrameworkOptions _options; + private readonly TimeProvider _time; + private readonly SemaphoreSlim _sync = new(1, 1); + private TimeSpan _offset; + private long _syncedAt; + + internal PsrClock(PsrDataSource dataSource, PsrEntityFrameworkOptions options, TimeProvider time) + { + _dataSource = dataSource; + _options = options; + _time = time; + } + + /// Current UTC time aligned to the database. + /// Cancellation token. + public async ValueTask GetUtcNowAsync(CancellationToken cancellationToken = default) + { + if (!_options.UseDatabaseClock) + { + return _time.GetUtcNow(); + } + + var syncedAt = Volatile.Read(ref _syncedAt); + if (syncedAt == 0 || Stopwatch.GetElapsedTime(syncedAt) > SyncInterval) + { + await SyncAsync(cancellationToken).ConfigureAwait(false); + } + + return _time.GetUtcNow() + _offset; + } + + private async Task SyncAsync(CancellationToken cancellationToken) + { + await _sync.WaitAsync(cancellationToken).ConfigureAwait(false); + try + { + if (_syncedAt != 0 && Stopwatch.GetElapsedTime(_syncedAt) <= SyncInterval) + { + return; + } + + var connection = await _dataSource.DataSource.OpenConnectionAsync(cancellationToken).ConfigureAwait(false); + await using (connection.ConfigureAwait(false)) + { + var command = new NpgsqlCommand("SELECT clock_timestamp()", connection); + await using (command.ConfigureAwait(false)) + { + var before = _time.GetUtcNow(); + var started = Stopwatch.GetTimestamp(); + var database = (DateTime)(await command.ExecuteScalarAsync(cancellationToken).ConfigureAwait(false))!; + var local = before + (Stopwatch.GetElapsedTime(started) / 2); + _offset = new DateTimeOffset(DateTime.SpecifyKind(database, DateTimeKind.Utc)) - local; + } + } + + Volatile.Write(ref _syncedAt, Stopwatch.GetTimestamp()); + } + finally + { + _sync.Release(); + } + } +} diff --git a/src/PythonScriptRunner.EntityFrameworkCore/Infrastructure/PsrOutbox.cs b/src/PythonScriptRunner.EntityFrameworkCore/Infrastructure/PsrOutbox.cs new file mode 100644 index 0000000..a854b9e --- /dev/null +++ b/src/PythonScriptRunner.EntityFrameworkCore/Infrastructure/PsrOutbox.cs @@ -0,0 +1,119 @@ +using System.Text.Json.Nodes; +using Microsoft.EntityFrameworkCore; + +namespace PythonScriptRunner.EntityFrameworkCore; + +/// +/// Reads psr.outbox_messages for the relay (spec 005): batches due now are locked with FOR UPDATE SKIP LOCKED, +/// so relays in several instances never publish the same row at the same time. +/// +public sealed class PsrOutbox +{ + private static readonly TimeSpan MaxBackoff = TimeSpan.FromSeconds(60); + + private readonly IDbContextFactory _factory; + private readonly PsrClock _clock; + + internal PsrOutbox(IDbContextFactory factory, PsrClock clock) + { + _factory = factory; + _clock = clock; + } + + /// + /// Locks up to due messages, hands them to and records the outcome: + /// published rows get published_at; failed ones get attempts + 1, the error and a backoff (1 s doubling up to + /// 60 s). When throws (broker down) nothing changes. + /// + /// Messages per batch. + /// Publishes the messages and returns the outcome of each one. + /// Cancellation token. + /// How many messages were locked. + public async Task DispatchAsync( + int batchSize, + Func, CancellationToken, Task>> publish, + CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(publish); + var db = await _factory.CreateDbContextAsync(cancellationToken).ConfigureAwait(false); + await using (db.ConfigureAwait(false)) + { + var transaction = await db.Database.BeginTransactionAsync(cancellationToken).ConfigureAwait(false); + await using (transaction.ConfigureAwait(false)) + { + var now = await _clock.GetUtcNowAsync(cancellationToken).ConfigureAwait(false); + var rows = await db.OutboxMessages.FromSql($""" + SELECT * FROM psr.outbox_messages + WHERE published_at IS NULL AND available_at <= {now} + ORDER BY available_at + LIMIT {batchSize} + FOR UPDATE SKIP LOCKED + """).AsNoTracking().ToListAsync(cancellationToken).ConfigureAwait(false); + if (rows.Count == 0) + { + return 0; + } + + var envelopes = rows.Select(r => new OutboxEnvelope( + r.Id, r.MessageType, r.Exchange, r.RoutingKey, r.Payload, + (JsonNode.Parse(r.Headers) as JsonObject ?? []).ToDictionary(h => h.Key, h => h.Value?.GetValue() ?? string.Empty), + r.AvailableAt, r.Attempts)).ToList(); + var deliveries = await publish(envelopes, cancellationToken).ConfigureAwait(false); + + var published = deliveries.Where(d => d.Error is null).Select(d => d.Id).ToList(); + var publishedAt = await _clock.GetUtcNowAsync(cancellationToken).ConfigureAwait(false); + await db.OutboxMessages.Where(m => published.Contains(m.Id)) + .ExecuteUpdateAsync(s => s.SetProperty(m => m.PublishedAt, publishedAt).SetProperty(m => m.LastError, (string?)null), cancellationToken) + .ConfigureAwait(false); + foreach (var failure in deliveries.Where(d => d.Error is not null)) + { + var attempts = rows.Single(r => r.Id == failure.Id).Attempts + 1; + var retryAt = publishedAt + Backoff(attempts); + var error = failure.Error!.Length <= 2000 ? failure.Error : failure.Error[..2000]; + await db.OutboxMessages.Where(m => m.Id == failure.Id) + .ExecuteUpdateAsync(s => s + .SetProperty(m => m.Attempts, attempts) + .SetProperty(m => m.LastError, error) + .SetProperty(m => m.AvailableAt, retryAt), cancellationToken).ConfigureAwait(false); + } + + await transaction.CommitAsync(cancellationToken).ConfigureAwait(false); + return rows.Count; + } + } + } + + /// Messages not published yet (gauge psr.outbox.pending). + /// Cancellation token. + public async Task CountPendingAsync(CancellationToken cancellationToken = default) + { + var db = await _factory.CreateDbContextAsync(cancellationToken).ConfigureAwait(false); + await using (db.ConfigureAwait(false)) + { + return await db.OutboxMessages.CountAsync(m => m.PublishedAt == null, cancellationToken).ConfigureAwait(false); + } + } + + /// Delay before publishing again after failures: 1 s, 2 s, 4 s... up to 60 s. + /// Failed publications so far. + public static TimeSpan Backoff(int attempts) => + attempts >= 7 ? MaxBackoff : TimeSpan.FromSeconds(Math.Min(MaxBackoff.TotalSeconds, Math.Pow(2, Math.Max(0, attempts - 1)))); +} + +/// A message to publish. +/// Outbox row id (the AMQP message_id). +/// Versioned type (AMQP type). +/// Destination exchange. +/// Routing key. +/// JSON body. +/// Headers (e.g. traceparent). +/// When it became due. +/// Failed publications so far. +public sealed record OutboxEnvelope( + Guid Id, string MessageType, string Exchange, string RoutingKey, string Payload, IReadOnlyDictionary Headers, DateTimeOffset AvailableAt, + int Attempts); + +/// Outcome of publishing one message. +/// Outbox row id. +/// null when the broker confirmed it; otherwise why it failed. +public sealed record OutboxDelivery(Guid Id, string? Error); diff --git a/src/PythonScriptRunner.EntityFrameworkCore/Migrations/20261020143224_AsyncRuns.Designer.cs b/src/PythonScriptRunner.EntityFrameworkCore/Migrations/20261020143224_AsyncRuns.Designer.cs new file mode 100644 index 0000000..81e1d25 --- /dev/null +++ b/src/PythonScriptRunner.EntityFrameworkCore/Migrations/20261020143224_AsyncRuns.Designer.cs @@ -0,0 +1,1351 @@ +// +using System; +using Microsoft.EntityFrameworkCore; +using Microsoft.EntityFrameworkCore.Infrastructure; +using Microsoft.EntityFrameworkCore.Migrations; +using Microsoft.EntityFrameworkCore.Storage.ValueConversion; +using Npgsql.EntityFrameworkCore.PostgreSQL.Metadata; +using PythonScriptRunner.EntityFrameworkCore; + +#nullable disable + +namespace PythonScriptRunner.EntityFrameworkCore.Migrations +{ + [DbContext(typeof(PsrDbContext))] + [Migration("20261020143224_AsyncRuns")] + partial class AsyncRuns + { + /// + protected override void BuildTargetModel(ModelBuilder modelBuilder) + { +#pragma warning disable 612, 618 + modelBuilder + .HasDefaultSchema("psr") + .HasAnnotation("ProductVersion", "10.0.13") + .HasAnnotation("Relational:MaxIdentifierLength", 63); + + NpgsqlModelBuilderExtensions.UseIdentityByDefaultColumns(modelBuilder); + + modelBuilder.Entity("PythonScriptRunner.EntityFrameworkCore.Entities.ArtifactContentEntity", b => + { + b.Property("ArtifactId") + .HasColumnType("uuid") + .HasColumnName("artifact_id"); + + b.Property("CreatedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("created_at"); + + b.Property("Content") + .IsRequired() + .HasColumnType("bytea") + .HasColumnName("content"); + + b.HasKey("ArtifactId", "CreatedAt") + .HasName("pk_artifact_contents"); + + b.ToTable("artifact_contents", "psr"); + }); + + modelBuilder.Entity("PythonScriptRunner.EntityFrameworkCore.Entities.ArtifactEntity", b => + { + b.Property("Id") + .HasColumnType("uuid") + .HasColumnName("id"); + + b.Property("RunCreatedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("run_created_at"); + + b.Property("ContentCreatedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("content_created_at"); + + b.Property("ContentDeletedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("content_deleted_at"); + + b.Property("ContentExpiresAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("content_expires_at"); + + b.Property("ContentType") + .IsRequired() + .HasMaxLength(255) + .HasColumnType("character varying(255)") + .HasColumnName("content_type"); + + b.Property("Direction") + .IsRequired() + .HasMaxLength(8) + .HasColumnType("character varying(8)") + .HasColumnName("direction"); + + b.Property("FieldName") + .IsRequired() + .HasMaxLength(64) + .HasColumnType("character varying(64)") + .HasColumnName("field_name"); + + b.Property("FileName") + .IsRequired() + .HasMaxLength(255) + .HasColumnType("character varying(255)") + .HasColumnName("file_name"); + + b.Property("ListIndex") + .HasColumnType("integer") + .HasColumnName("list_index"); + + b.Property("RunId") + .HasColumnType("uuid") + .HasColumnName("run_id"); + + b.Property("Sha256") + .IsRequired() + .HasMaxLength(64) + .HasColumnType("character(64)") + .HasColumnName("sha256") + .IsFixedLength(); + + b.Property("SizeBytes") + .HasColumnType("bigint") + .HasColumnName("size_bytes"); + + b.Property("StepRunId") + .HasColumnType("uuid") + .HasColumnName("step_run_id"); + + b.HasKey("Id", "RunCreatedAt") + .HasName("pk_artifacts"); + + b.HasIndex("ContentExpiresAt") + .HasDatabaseName("ix_artifacts_content_expires_at") + .HasFilter("content_deleted_at IS NULL"); + + b.HasIndex("RunId", "RunCreatedAt") + .HasDatabaseName("ix_artifacts_run_id_run_created_at"); + + b.HasIndex("StepRunId", "RunCreatedAt") + .HasDatabaseName("ix_artifacts_step_run_id_run_created_at"); + + b.ToTable("artifacts", "psr", t => + { + t.HasCheckConstraint("ck_artifacts_direction", "direction IN ('Input', 'Output')"); + }); + }); + + modelBuilder.Entity("PythonScriptRunner.EntityFrameworkCore.Entities.AttemptLogEntity", b => + { + b.Property("AttemptId") + .HasColumnType("uuid") + .HasColumnName("attempt_id"); + + b.Property("RunCreatedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("run_created_at"); + + b.Property("Stream") + .HasMaxLength(8) + .HasColumnType("character varying(8)") + .HasColumnName("stream"); + + b.Property("Content") + .IsRequired() + .HasColumnType("text") + .HasColumnName("content"); + + b.Property("OriginalBytes") + .HasColumnType("bigint") + .HasColumnName("original_bytes"); + + b.Property("Truncated") + .HasColumnType("boolean") + .HasColumnName("truncated"); + + b.HasKey("AttemptId", "RunCreatedAt", "Stream") + .HasName("pk_attempt_logs"); + + b.ToTable("attempt_logs", "psr", t => + { + t.HasCheckConstraint("ck_attempt_logs_stream", "stream IN ('stdout', 'stderr')"); + }); + }); + + modelBuilder.Entity("PythonScriptRunner.EntityFrameworkCore.Entities.AuditEventEntity", b => + { + b.Property("Id") + .HasColumnType("uuid") + .HasColumnName("id"); + + b.Property("OccurredAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("occurred_at"); + + b.Property("Action") + .IsRequired() + .HasMaxLength(60) + .HasColumnType("character varying(60)") + .HasColumnName("action"); + + b.Property("Actor") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("character varying(200)") + .HasColumnName("actor"); + + b.Property("Details") + .HasColumnType("jsonb") + .HasColumnName("details"); + + b.Property("EntityId") + .HasColumnType("uuid") + .HasColumnName("entity_id"); + + b.Property("EntityKey") + .HasMaxLength(100) + .HasColumnType("character varying(100)") + .HasColumnName("entity_key"); + + b.Property("EntityType") + .IsRequired() + .HasMaxLength(30) + .HasColumnType("character varying(30)") + .HasColumnName("entity_type"); + + b.Property("TraceId") + .HasMaxLength(32) + .HasColumnType("character varying(32)") + .HasColumnName("trace_id"); + + b.HasKey("Id", "OccurredAt") + .HasName("pk_audit_events"); + + b.HasIndex("OccurredAt", "Id") + .IsDescending() + .HasDatabaseName("ix_audit_events_occurred_at_id"); + + b.ToTable("audit_events", "psr"); + }); + + modelBuilder.Entity("PythonScriptRunner.EntityFrameworkCore.Entities.IdempotencyKeyEntity", b => + { + b.Property("Scope") + .HasMaxLength(80) + .HasColumnType("character varying(80)") + .HasColumnName("scope"); + + b.Property("Key") + .HasMaxLength(200) + .HasColumnType("character varying(200)") + .HasColumnName("key"); + + b.Property("ExpiresAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("expires_at"); + + b.Property("RunCreatedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("run_created_at"); + + b.Property("RunId") + .HasColumnType("uuid") + .HasColumnName("run_id"); + + b.HasKey("Scope", "Key") + .HasName("pk_idempotency_keys"); + + b.HasIndex("ExpiresAt") + .HasDatabaseName("ix_idempotency_keys_expires_at"); + + b.ToTable("idempotency_keys", "psr"); + }); + + modelBuilder.Entity("PythonScriptRunner.EntityFrameworkCore.Entities.OutboxMessageEntity", b => + { + b.Property("Id") + .HasColumnType("uuid") + .HasColumnName("id"); + + b.Property("Attempts") + .HasColumnType("integer") + .HasColumnName("attempts"); + + b.Property("AvailableAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("available_at"); + + b.Property("CreatedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("created_at"); + + b.Property("Exchange") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("character varying(200)") + .HasColumnName("exchange"); + + b.Property("Headers") + .IsRequired() + .HasColumnType("jsonb") + .HasColumnName("headers"); + + b.Property("LastError") + .HasColumnType("text") + .HasColumnName("last_error"); + + b.Property("MessageType") + .IsRequired() + .HasMaxLength(100) + .HasColumnType("character varying(100)") + .HasColumnName("message_type"); + + b.Property("Payload") + .IsRequired() + .HasColumnType("jsonb") + .HasColumnName("payload"); + + b.Property("PublishedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("published_at"); + + b.Property("RoutingKey") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("character varying(200)") + .HasColumnName("routing_key"); + + b.HasKey("Id") + .HasName("pk_outbox_messages"); + + b.HasIndex("AvailableAt") + .HasDatabaseName("ix_outbox_messages_available_at") + .HasFilter("published_at IS NULL"); + + b.ToTable("outbox_messages", "psr"); + }); + + modelBuilder.Entity("PythonScriptRunner.EntityFrameworkCore.Entities.PipelineEntity", b => + { + b.Property("Id") + .HasColumnType("uuid") + .HasColumnName("id"); + + b.Property("ArtifactRetentionDays") + .HasColumnType("integer") + .HasColumnName("artifact_retention_days"); + + b.Property("CreatedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("created_at"); + + b.Property("CreatedBy") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("character varying(200)") + .HasColumnName("created_by"); + + b.Property("Description") + .HasColumnType("text") + .HasColumnName("description"); + + b.Property("Key") + .IsRequired() + .HasMaxLength(64) + .HasColumnType("character varying(64)") + .HasColumnName("key"); + + b.Property("LatestPublishedVersion") + .HasColumnType("integer") + .HasColumnName("latest_published_version"); + + b.Property("MaxConcurrentRuns") + .HasColumnType("integer") + .HasColumnName("max_concurrent_runs"); + + b.Property("Name") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("character varying(200)") + .HasColumnName("name"); + + b.Property("RowVersion") + .IsConcurrencyToken() + .ValueGeneratedOnAddOrUpdate() + .HasColumnType("xid") + .HasColumnName("xmin"); + + b.Property("RunRetentionDays") + .HasColumnType("integer") + .HasColumnName("run_retention_days"); + + b.Property("UpdatedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("updated_at"); + + b.Property("UpdatedBy") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("character varying(200)") + .HasColumnName("updated_by"); + + b.HasKey("Id") + .HasName("pk_pipelines"); + + b.HasIndex("Key") + .IsUnique() + .HasDatabaseName("ux_pipelines_key"); + + b.ToTable("pipelines", "psr"); + }); + + modelBuilder.Entity("PythonScriptRunner.EntityFrameworkCore.Entities.PipelineStepEntity", b => + { + b.Property("Id") + .HasColumnType("uuid") + .HasColumnName("id"); + + b.Property("Bindings") + .IsRequired() + .HasColumnType("jsonb") + .HasColumnName("bindings"); + + b.Property("DisplayName") + .HasMaxLength(200) + .HasColumnType("character varying(200)") + .HasColumnName("display_name"); + + b.Property("Key") + .IsRequired() + .HasMaxLength(64) + .HasColumnType("character varying(64)") + .HasColumnName("key"); + + b.Property("Order") + .HasColumnType("integer") + .HasColumnName("order"); + + b.Property("PipelineVersionId") + .HasColumnType("uuid") + .HasColumnName("pipeline_version_id"); + + b.Property("StageVersionId") + .HasColumnType("uuid") + .HasColumnName("stage_version_id"); + + b.HasKey("Id") + .HasName("pk_pipeline_steps"); + + b.HasIndex("StageVersionId") + .HasDatabaseName("ix_pipeline_steps_stage_version_id"); + + b.HasIndex("PipelineVersionId", "Key") + .IsUnique() + .HasDatabaseName("ux_pipeline_steps_pipeline_version_id_key"); + + b.HasIndex("PipelineVersionId", "Order") + .IsUnique() + .HasDatabaseName("ux_pipeline_steps_pipeline_version_id_order"); + + b.ToTable("pipeline_steps", "psr"); + }); + + modelBuilder.Entity("PythonScriptRunner.EntityFrameworkCore.Entities.PipelineVersionEntity", b => + { + b.Property("Id") + .HasColumnType("uuid") + .HasColumnName("id"); + + b.Property("ArchivedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("archived_at"); + + b.Property("ArchivedBy") + .HasMaxLength(200) + .HasColumnType("character varying(200)") + .HasColumnName("archived_by"); + + b.Property("CreatedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("created_at"); + + b.Property("CreatedBy") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("character varying(200)") + .HasColumnName("created_by"); + + b.Property("InputFields") + .IsRequired() + .HasColumnType("jsonb") + .HasColumnName("input_fields"); + + b.Property("Layout") + .HasColumnType("jsonb") + .HasColumnName("layout"); + + b.Property("Outputs") + .IsRequired() + .HasColumnType("jsonb") + .HasColumnName("outputs"); + + b.Property("PipelineId") + .HasColumnType("uuid") + .HasColumnName("pipeline_id"); + + b.Property("PublishedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("published_at"); + + b.Property("PublishedBy") + .HasMaxLength(200) + .HasColumnType("character varying(200)") + .HasColumnName("published_by"); + + b.Property("RowVersion") + .IsConcurrencyToken() + .ValueGeneratedOnAddOrUpdate() + .HasColumnType("xid") + .HasColumnName("xmin"); + + b.Property("Status") + .IsRequired() + .ValueGeneratedOnAdd() + .HasMaxLength(16) + .HasColumnType("character varying(16)") + .HasDefaultValue("Draft") + .HasColumnName("status"); + + b.Property("UpdatedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("updated_at"); + + b.Property("UpdatedBy") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("character varying(200)") + .HasColumnName("updated_by"); + + b.Property("Version") + .HasColumnType("integer") + .HasColumnName("version"); + + b.HasKey("Id") + .HasName("pk_pipeline_versions"); + + b.HasIndex("PipelineId") + .IsUnique() + .HasDatabaseName("ux_pipeline_versions_draft") + .HasFilter("status = 'Draft'"); + + b.HasIndex("PipelineId", "Version") + .IsUnique() + .HasDatabaseName("ux_pipeline_versions_pipeline_id_version"); + + b.ToTable("pipeline_versions", "psr", t => + { + t.HasCheckConstraint("ck_pipeline_versions_status", "status IN ('Draft', 'Published', 'Archived')"); + }); + }); + + modelBuilder.Entity("PythonScriptRunner.EntityFrameworkCore.Entities.RunEntity", b => + { + b.Property("Id") + .HasColumnType("uuid") + .HasColumnName("id"); + + b.Property("CreatedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("created_at"); + + b.Property("CancelRequestedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("cancel_requested_at"); + + b.Property("CancelRequestedBy") + .HasMaxLength(200) + .HasColumnType("character varying(200)") + .HasColumnName("cancel_requested_by"); + + b.Property("CorrelationId") + .HasMaxLength(200) + .HasColumnType("character varying(200)") + .HasColumnName("correlation_id"); + + b.Property("Error") + .HasColumnType("jsonb") + .HasColumnName("error"); + + b.Property("ExpiresAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("expires_at"); + + b.Property("FinishedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("finished_at"); + + b.Property("IdempotencyKey") + .HasMaxLength(200) + .HasColumnType("character varying(200)") + .HasColumnName("idempotency_key"); + + b.Property("Inputs") + .IsRequired() + .HasColumnType("jsonb") + .HasColumnName("inputs"); + + b.Property("Kind") + .IsRequired() + .HasMaxLength(16) + .HasColumnType("character varying(16)") + .HasColumnName("kind"); + + b.Property("Mode") + .IsRequired() + .HasMaxLength(8) + .HasColumnType("character varying(8)") + .HasColumnName("mode"); + + b.Property("Outputs") + .HasColumnType("jsonb") + .HasColumnName("outputs"); + + b.Property("PipelineId") + .HasColumnType("uuid") + .HasColumnName("pipeline_id"); + + b.Property("PipelineKey") + .HasMaxLength(64) + .HasColumnType("character varying(64)") + .HasColumnName("pipeline_key"); + + b.Property("PipelineVersion") + .HasColumnType("integer") + .HasColumnName("pipeline_version"); + + b.Property("PipelineVersionId") + .HasColumnType("uuid") + .HasColumnName("pipeline_version_id"); + + b.Property("QueuedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("queued_at"); + + b.Property("ScheduleId") + .HasColumnType("uuid") + .HasColumnName("schedule_id"); + + b.Property("StageKey") + .HasMaxLength(64) + .HasColumnType("character varying(64)") + .HasColumnName("stage_key"); + + b.Property("StageVersion") + .HasColumnType("integer") + .HasColumnName("stage_version"); + + b.Property("StageVersionId") + .HasColumnType("uuid") + .HasColumnName("stage_version_id"); + + b.Property("StartedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("started_at"); + + b.Property("Status") + .IsRequired() + .HasMaxLength(16) + .HasColumnType("character varying(16)") + .HasColumnName("status"); + + b.Property("TraceParent") + .HasMaxLength(55) + .HasColumnType("character varying(55)") + .HasColumnName("trace_parent"); + + b.Property("Trigger") + .IsRequired() + .HasMaxLength(16) + .HasColumnType("character varying(16)") + .HasColumnName("trigger"); + + b.Property("TriggerDetails") + .HasColumnType("jsonb") + .HasColumnName("trigger_details"); + + b.Property("TriggeredBy") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("character varying(200)") + .HasColumnName("triggered_by"); + + b.HasKey("Id", "CreatedAt") + .HasName("pk_runs"); + + b.HasIndex("ExpiresAt") + .HasDatabaseName("ix_runs_expires_at") + .HasFilter("expires_at IS NOT NULL"); + + b.HasIndex("CreatedAt", "Id") + .IsDescending() + .HasDatabaseName("ix_runs_created_at_id"); + + b.HasIndex("Status", "CreatedAt") + .HasDatabaseName("ix_runs_status_created_at") + .HasFilter("status IN ('Pending', 'Queued', 'Running', 'Cancelling')"); + + b.HasIndex("PipelineId", "CreatedAt", "Id") + .IsDescending(false, true, true) + .HasDatabaseName("ix_runs_pipeline_id_created_at_id"); + + b.HasIndex("StageKey", "CreatedAt", "Id") + .IsDescending(false, true, true) + .HasDatabaseName("ix_runs_stage_key_created_at_id") + .HasFilter("stage_key IS NOT NULL"); + + b.ToTable("runs", "psr", t => + { + t.HasCheckConstraint("ck_runs_kind", "kind IN ('Pipeline', 'PipelineTest', 'Stage', 'StageTest')"); + + t.HasCheckConstraint("ck_runs_mode", "mode IN ('Sync', 'Async')"); + + t.HasCheckConstraint("ck_runs_status", "status IN ('Pending', 'Queued', 'Running', 'Succeeded', 'Failed', 'Cancelling', 'Cancelled')"); + + t.HasCheckConstraint("ck_runs_trigger", "trigger IN ('Library', 'Dashboard', 'Schedule', 'QueueCommand')"); + }); + }); + + modelBuilder.Entity("PythonScriptRunner.EntityFrameworkCore.Entities.ScheduleEntity", b => + { + b.Property("Id") + .HasColumnType("uuid") + .HasColumnName("id"); + + b.Property("CreatedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("created_at"); + + b.Property("CreatedBy") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("character varying(200)") + .HasColumnName("created_by"); + + b.Property("Cron") + .IsRequired() + .HasMaxLength(100) + .HasColumnType("character varying(100)") + .HasColumnName("cron"); + + b.Property("Inputs") + .IsRequired() + .ValueGeneratedOnAdd() + .HasColumnType("jsonb") + .HasColumnName("inputs") + .HasDefaultValueSql("'{}'::jsonb"); + + b.Property("IsEnabled") + .ValueGeneratedOnAdd() + .HasColumnType("boolean") + .HasDefaultValue(true) + .HasColumnName("is_enabled"); + + b.Property("LastError") + .HasColumnType("text") + .HasColumnName("last_error"); + + b.Property("LastFiredAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("last_fired_at"); + + b.Property("LastRunId") + .HasColumnType("uuid") + .HasColumnName("last_run_id"); + + b.Property("Name") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("character varying(200)") + .HasColumnName("name"); + + b.Property("NextRunAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("next_run_at"); + + b.Property("PinnedVersion") + .HasColumnType("integer") + .HasColumnName("pinned_version"); + + b.Property("PipelineId") + .HasColumnType("uuid") + .HasColumnName("pipeline_id"); + + b.Property("RowVersion") + .IsConcurrencyToken() + .ValueGeneratedOnAddOrUpdate() + .HasColumnType("xid") + .HasColumnName("xmin"); + + b.Property("TimeZone") + .IsRequired() + .ValueGeneratedOnAdd() + .HasMaxLength(64) + .HasColumnType("character varying(64)") + .HasDefaultValue("America/Sao_Paulo") + .HasColumnName("time_zone"); + + b.Property("UpdatedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("updated_at"); + + b.Property("UpdatedBy") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("character varying(200)") + .HasColumnName("updated_by"); + + b.HasKey("Id") + .HasName("pk_schedules"); + + b.HasIndex("NextRunAt") + .HasDatabaseName("ix_schedules_next_run_at") + .HasFilter("is_enabled"); + + b.HasIndex("PipelineId") + .HasDatabaseName("ix_schedules_pipeline_id"); + + b.ToTable("schedules", "psr"); + }); + + modelBuilder.Entity("PythonScriptRunner.EntityFrameworkCore.Entities.StageEntity", b => + { + b.Property("Id") + .HasColumnType("uuid") + .HasColumnName("id"); + + b.Property("CreatedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("created_at"); + + b.Property("CreatedBy") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("character varying(200)") + .HasColumnName("created_by"); + + b.Property("Description") + .HasColumnType("text") + .HasColumnName("description"); + + b.Property("Key") + .IsRequired() + .HasMaxLength(64) + .HasColumnType("character varying(64)") + .HasColumnName("key"); + + b.Property("LatestPublishedVersion") + .HasColumnType("integer") + .HasColumnName("latest_published_version"); + + b.Property("Name") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("character varying(200)") + .HasColumnName("name"); + + b.Property("RowVersion") + .IsConcurrencyToken() + .ValueGeneratedOnAddOrUpdate() + .HasColumnType("xid") + .HasColumnName("xmin"); + + b.Property("UpdatedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("updated_at"); + + b.Property("UpdatedBy") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("character varying(200)") + .HasColumnName("updated_by"); + + b.HasKey("Id") + .HasName("pk_stages"); + + b.HasIndex("Key") + .IsUnique() + .HasDatabaseName("ux_stages_key"); + + b.ToTable("stages", "psr"); + }); + + modelBuilder.Entity("PythonScriptRunner.EntityFrameworkCore.Entities.StageVersionEntity", b => + { + b.Property("Id") + .HasColumnType("uuid") + .HasColumnName("id"); + + b.Property("ArchivedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("archived_at"); + + b.Property("ArchivedBy") + .HasMaxLength(200) + .HasColumnType("character varying(200)") + .HasColumnName("archived_by"); + + b.Property("CreatedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("created_at"); + + b.Property("CreatedBy") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("character varying(200)") + .HasColumnName("created_by"); + + b.Property("EnvironmentHash") + .HasMaxLength(64) + .HasColumnType("character(64)") + .HasColumnName("environment_hash") + .IsFixedLength(); + + b.Property("Idempotent") + .HasColumnType("boolean") + .HasColumnName("idempotent"); + + b.Property("InputFields") + .IsRequired() + .HasColumnType("jsonb") + .HasColumnName("input_fields"); + + b.Property("LockedRequirements") + .HasColumnType("text") + .HasColumnName("locked_requirements"); + + b.Property("MemoryLimitMb") + .HasColumnType("integer") + .HasColumnName("memory_limit_mb"); + + b.Property("OutputFields") + .IsRequired() + .HasColumnType("jsonb") + .HasColumnName("output_fields"); + + b.Property("PublishedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("published_at"); + + b.Property("PublishedBy") + .HasMaxLength(200) + .HasColumnType("character varying(200)") + .HasColumnName("published_by"); + + b.Property("PythonVersion") + .IsRequired() + .HasMaxLength(10) + .HasColumnType("character varying(10)") + .HasColumnName("python_version"); + + b.PrimitiveCollection("RequiredSecrets") + .IsRequired() + .HasColumnType("text[]") + .HasColumnName("required_secrets"); + + b.Property("Requirements") + .HasColumnType("text") + .HasColumnName("requirements"); + + b.Property("RetryBackoffMultiplier") + .HasPrecision(4, 2) + .HasColumnType("numeric(4,2)") + .HasColumnName("retry_backoff_multiplier"); + + b.Property("RetryInitialDelaySeconds") + .HasColumnType("integer") + .HasColumnName("retry_initial_delay_seconds"); + + b.Property("RetryMaxAttempts") + .HasColumnType("integer") + .HasColumnName("retry_max_attempts"); + + b.Property("RowVersion") + .IsConcurrencyToken() + .ValueGeneratedOnAddOrUpdate() + .HasColumnType("xid") + .HasColumnName("xmin"); + + b.Property("Script") + .IsRequired() + .HasColumnType("text") + .HasColumnName("script"); + + b.Property("ScriptSha256") + .IsRequired() + .HasMaxLength(64) + .HasColumnType("character(64)") + .HasColumnName("script_sha256") + .IsFixedLength(); + + b.Property("StageId") + .HasColumnType("uuid") + .HasColumnName("stage_id"); + + b.Property("Status") + .IsRequired() + .ValueGeneratedOnAdd() + .HasMaxLength(16) + .HasColumnType("character varying(16)") + .HasDefaultValue("Draft") + .HasColumnName("status"); + + b.Property("TimeoutSeconds") + .HasColumnType("integer") + .HasColumnName("timeout_seconds"); + + b.Property("UpdatedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("updated_at"); + + b.Property("UpdatedBy") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("character varying(200)") + .HasColumnName("updated_by"); + + b.Property("Version") + .HasColumnType("integer") + .HasColumnName("version"); + + b.HasKey("Id") + .HasName("pk_stage_versions"); + + b.HasIndex("StageId") + .IsUnique() + .HasDatabaseName("ux_stage_versions_draft") + .HasFilter("status = 'Draft'"); + + b.HasIndex("StageId", "Version") + .IsUnique() + .HasDatabaseName("ux_stage_versions_stage_id_version"); + + b.ToTable("stage_versions", "psr", t => + { + t.HasCheckConstraint("ck_stage_versions_status", "status IN ('Draft', 'Published', 'Archived')"); + }); + }); + + modelBuilder.Entity("PythonScriptRunner.EntityFrameworkCore.Entities.StepAttemptEntity", b => + { + b.Property("Id") + .HasColumnType("uuid") + .HasColumnName("id"); + + b.Property("RunCreatedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("run_created_at"); + + b.Property("Attempt") + .HasColumnType("integer") + .HasColumnName("attempt"); + + b.Property("Cause") + .IsRequired() + .HasMaxLength(16) + .HasColumnType("character varying(16)") + .HasColumnName("cause"); + + b.Property("DurationMs") + .HasColumnType("integer") + .HasColumnName("duration_ms"); + + b.Property("EnvironmentHash") + .HasMaxLength(64) + .HasColumnType("character(64)") + .HasColumnName("environment_hash") + .IsFixedLength(); + + b.Property("Error") + .HasColumnType("jsonb") + .HasColumnName("error"); + + b.Property("ErrorKind") + .HasMaxLength(40) + .HasColumnType("character varying(40)") + .HasColumnName("error_kind"); + + b.Property("ExitCode") + .HasColumnType("integer") + .HasColumnName("exit_code"); + + b.Property("FinishedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("finished_at"); + + b.Property("LeaseExpiresAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("lease_expires_at"); + + b.Property("OverheadMs") + .HasColumnType("integer") + .HasColumnName("overhead_ms"); + + b.Property("RunId") + .HasColumnType("uuid") + .HasColumnName("run_id"); + + b.Property("StartedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("started_at"); + + b.Property("Status") + .IsRequired() + .HasMaxLength(16) + .HasColumnType("character varying(16)") + .HasColumnName("status"); + + b.Property("StepRunId") + .HasColumnType("uuid") + .HasColumnName("step_run_id"); + + b.Property("TriggeredBy") + .IsRequired() + .HasMaxLength(200) + .HasColumnType("character varying(200)") + .HasColumnName("triggered_by"); + + b.Property("WorkerId") + .HasMaxLength(200) + .HasColumnType("character varying(200)") + .HasColumnName("worker_id"); + + b.HasKey("Id", "RunCreatedAt") + .HasName("pk_step_attempts"); + + b.HasIndex("LeaseExpiresAt") + .HasDatabaseName("ix_step_attempts_lease_expires_at") + .HasFilter("status = 'Running'"); + + b.HasIndex("StepRunId", "RunCreatedAt") + .HasDatabaseName("ix_step_attempts_step_run_id_run_created_at"); + + b.ToTable("step_attempts", "psr", t => + { + t.HasCheckConstraint("ck_step_attempts_cause", "cause IN ('Initial', 'AutoRetry', 'Reprocess', 'LeaseRecovery')"); + + t.HasCheckConstraint("ck_step_attempts_status", "status IN ('Running', 'Succeeded', 'Failed', 'TimedOut', 'Cancelled', 'WorkerLost')"); + }); + }); + + modelBuilder.Entity("PythonScriptRunner.EntityFrameworkCore.Entities.StepRunEntity", b => + { + b.Property("Id") + .HasColumnType("uuid") + .HasColumnName("id"); + + b.Property("RunCreatedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("run_created_at"); + + b.Property("CurrentAttempt") + .HasColumnType("integer") + .HasColumnName("current_attempt"); + + b.Property("Error") + .HasColumnType("jsonb") + .HasColumnName("error"); + + b.Property("FinishedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("finished_at"); + + b.Property("Inputs") + .HasColumnType("jsonb") + .HasColumnName("inputs"); + + b.Property("Order") + .HasColumnType("integer") + .HasColumnName("order"); + + b.Property("Outputs") + .HasColumnType("jsonb") + .HasColumnName("outputs"); + + b.Property("PipelineStepId") + .HasColumnType("uuid") + .HasColumnName("pipeline_step_id"); + + b.Property("RunId") + .HasColumnType("uuid") + .HasColumnName("run_id"); + + b.Property("StageKey") + .IsRequired() + .HasMaxLength(64) + .HasColumnType("character varying(64)") + .HasColumnName("stage_key"); + + b.Property("StageVersion") + .HasColumnType("integer") + .HasColumnName("stage_version"); + + b.Property("StageVersionId") + .HasColumnType("uuid") + .HasColumnName("stage_version_id"); + + b.Property("StartedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("started_at"); + + b.Property("Status") + .IsRequired() + .HasMaxLength(16) + .HasColumnType("character varying(16)") + .HasColumnName("status"); + + b.Property("StepKey") + .IsRequired() + .HasMaxLength(64) + .HasColumnType("character varying(64)") + .HasColumnName("step_key"); + + b.HasKey("Id", "RunCreatedAt") + .HasName("pk_step_runs"); + + b.HasIndex("RunId", "RunCreatedAt", "Order") + .IsUnique() + .HasDatabaseName("ux_step_runs_run_id_run_created_at_order"); + + b.ToTable("step_runs", "psr", t => + { + t.HasCheckConstraint("ck_step_runs_status", "status IN ('Pending', 'Queued', 'Running', 'Succeeded', 'Failed', 'TimedOut', 'Cancelled', 'Skipped')"); + }); + }); + + modelBuilder.Entity("PythonScriptRunner.EntityFrameworkCore.Entities.ArtifactEntity", b => + { + b.HasOne("PythonScriptRunner.EntityFrameworkCore.Entities.RunEntity", null) + .WithMany() + .HasForeignKey("RunId", "RunCreatedAt") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired() + .HasConstraintName("fk_artifacts_runs_run_id_run_created_at"); + + b.HasOne("PythonScriptRunner.EntityFrameworkCore.Entities.StepRunEntity", null) + .WithMany() + .HasForeignKey("StepRunId", "RunCreatedAt") + .OnDelete(DeleteBehavior.Cascade) + .HasConstraintName("fk_artifacts_step_runs_step_run_id_run_created_at"); + }); + + modelBuilder.Entity("PythonScriptRunner.EntityFrameworkCore.Entities.AttemptLogEntity", b => + { + b.HasOne("PythonScriptRunner.EntityFrameworkCore.Entities.StepAttemptEntity", null) + .WithMany() + .HasForeignKey("AttemptId", "RunCreatedAt") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired() + .HasConstraintName("fk_attempt_logs_step_attempts_attempt_id_run_created_at"); + }); + + modelBuilder.Entity("PythonScriptRunner.EntityFrameworkCore.Entities.PipelineStepEntity", b => + { + b.HasOne("PythonScriptRunner.EntityFrameworkCore.Entities.PipelineVersionEntity", null) + .WithMany("Steps") + .HasForeignKey("PipelineVersionId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired() + .HasConstraintName("fk_pipeline_steps_pipeline_versions_pipeline_version_id"); + + b.HasOne("PythonScriptRunner.EntityFrameworkCore.Entities.StageVersionEntity", "StageVersion") + .WithMany() + .HasForeignKey("StageVersionId") + .OnDelete(DeleteBehavior.Restrict) + .IsRequired() + .HasConstraintName("fk_pipeline_steps_stage_versions_stage_version_id"); + + b.Navigation("StageVersion"); + }); + + modelBuilder.Entity("PythonScriptRunner.EntityFrameworkCore.Entities.PipelineVersionEntity", b => + { + b.HasOne("PythonScriptRunner.EntityFrameworkCore.Entities.PipelineEntity", "Pipeline") + .WithMany("Versions") + .HasForeignKey("PipelineId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired() + .HasConstraintName("fk_pipeline_versions_pipelines_pipeline_id"); + + b.Navigation("Pipeline"); + }); + + modelBuilder.Entity("PythonScriptRunner.EntityFrameworkCore.Entities.ScheduleEntity", b => + { + b.HasOne("PythonScriptRunner.EntityFrameworkCore.Entities.PipelineEntity", null) + .WithMany() + .HasForeignKey("PipelineId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired() + .HasConstraintName("fk_schedules_pipelines_pipeline_id"); + }); + + modelBuilder.Entity("PythonScriptRunner.EntityFrameworkCore.Entities.StageVersionEntity", b => + { + b.HasOne("PythonScriptRunner.EntityFrameworkCore.Entities.StageEntity", "Stage") + .WithMany("Versions") + .HasForeignKey("StageId") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired() + .HasConstraintName("fk_stage_versions_stages_stage_id"); + + b.Navigation("Stage"); + }); + + modelBuilder.Entity("PythonScriptRunner.EntityFrameworkCore.Entities.StepAttemptEntity", b => + { + b.HasOne("PythonScriptRunner.EntityFrameworkCore.Entities.StepRunEntity", null) + .WithMany("Attempts") + .HasForeignKey("StepRunId", "RunCreatedAt") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired() + .HasConstraintName("fk_step_attempts_step_runs_step_run_id_run_created_at"); + }); + + modelBuilder.Entity("PythonScriptRunner.EntityFrameworkCore.Entities.StepRunEntity", b => + { + b.HasOne("PythonScriptRunner.EntityFrameworkCore.Entities.RunEntity", null) + .WithMany("Steps") + .HasForeignKey("RunId", "RunCreatedAt") + .OnDelete(DeleteBehavior.Cascade) + .IsRequired() + .HasConstraintName("fk_step_runs_runs_run_id_run_created_at"); + }); + + modelBuilder.Entity("PythonScriptRunner.EntityFrameworkCore.Entities.PipelineEntity", b => + { + b.Navigation("Versions"); + }); + + modelBuilder.Entity("PythonScriptRunner.EntityFrameworkCore.Entities.PipelineVersionEntity", b => + { + b.Navigation("Steps"); + }); + + modelBuilder.Entity("PythonScriptRunner.EntityFrameworkCore.Entities.RunEntity", b => + { + b.Navigation("Steps"); + }); + + modelBuilder.Entity("PythonScriptRunner.EntityFrameworkCore.Entities.StageEntity", b => + { + b.Navigation("Versions"); + }); + + modelBuilder.Entity("PythonScriptRunner.EntityFrameworkCore.Entities.StepRunEntity", b => + { + b.Navigation("Attempts"); + }); +#pragma warning restore 612, 618 + } + } +} diff --git a/src/PythonScriptRunner.EntityFrameworkCore/Migrations/20261020143224_AsyncRuns.cs b/src/PythonScriptRunner.EntityFrameworkCore/Migrations/20261020143224_AsyncRuns.cs new file mode 100644 index 0000000..eab849a --- /dev/null +++ b/src/PythonScriptRunner.EntityFrameworkCore/Migrations/20261020143224_AsyncRuns.cs @@ -0,0 +1,30 @@ +using Microsoft.EntityFrameworkCore.Migrations; + +#nullable disable + +namespace PythonScriptRunner.EntityFrameworkCore.Migrations +{ + /// + public partial class AsyncRuns : Migration + { + /// + protected override void Up(MigrationBuilder migrationBuilder) + { + migrationBuilder.AddColumn( + name: "trigger_details", + schema: "psr", + table: "runs", + type: "jsonb", + nullable: true); + } + + /// + protected override void Down(MigrationBuilder migrationBuilder) + { + migrationBuilder.DropColumn( + name: "trigger_details", + schema: "psr", + table: "runs"); + } + } +} diff --git a/src/PythonScriptRunner.EntityFrameworkCore/Migrations/PsrDbContextModelSnapshot.cs b/src/PythonScriptRunner.EntityFrameworkCore/Migrations/PsrDbContextModelSnapshot.cs index 8cf4057..d03adf4 100644 --- a/src/PythonScriptRunner.EntityFrameworkCore/Migrations/PsrDbContextModelSnapshot.cs +++ b/src/PythonScriptRunner.EntityFrameworkCore/Migrations/PsrDbContextModelSnapshot.cs @@ -666,6 +666,10 @@ protected override void BuildModel(ModelBuilder modelBuilder) .HasColumnType("character varying(16)") .HasColumnName("trigger"); + b.Property("TriggerDetails") + .HasColumnType("jsonb") + .HasColumnName("trigger_details"); + b.Property("TriggeredBy") .IsRequired() .HasMaxLength(200) diff --git a/src/PythonScriptRunner.EntityFrameworkCore/PsrDbContext.cs b/src/PythonScriptRunner.EntityFrameworkCore/PsrDbContext.cs index 3d01813..3611cd2 100644 --- a/src/PythonScriptRunner.EntityFrameworkCore/PsrDbContext.cs +++ b/src/PythonScriptRunner.EntityFrameworkCore/PsrDbContext.cs @@ -177,6 +177,7 @@ private static void ConfigureRuns(ModelBuilder model) Json(e.Property(x => x.Inputs)); Json(e.Property(x => x.Outputs)); Json(e.Property(x => x.Error)); + Json(e.Property(x => x.TriggerDetails)); e.Property(x => x.IdempotencyKey).HasMaxLength(200); e.Property(x => x.CorrelationId).HasMaxLength(200); e.Property(x => x.TraceParent).HasMaxLength(55); diff --git a/src/PythonScriptRunner.EntityFrameworkCore/PsrEntityFrameworkCoreExtensions.cs b/src/PythonScriptRunner.EntityFrameworkCore/PsrEntityFrameworkCoreExtensions.cs index bf8d331..5f07ac7 100644 --- a/src/PythonScriptRunner.EntityFrameworkCore/PsrEntityFrameworkCoreExtensions.cs +++ b/src/PythonScriptRunner.EntityFrameworkCore/PsrEntityFrameworkCoreExtensions.cs @@ -44,6 +44,21 @@ public static IPythonScriptRunnerBuilder AddEntityFrameworkCore(this IPythonScri services.TryAddSingleton(); services.TryAddSingleton(); services.TryAddSingleton(); + services.TryAddSingleton(); + services.TryAddSingleton(provider => new PsrClock( + provider.GetRequiredService(), options, provider.GetRequiredService())); + services.TryAddSingleton(provider => new PsrOutbox( + provider.GetRequiredService>(), provider.GetRequiredService())); + services.TryAddSingleton(provider => new AsyncRunLifecycle( + provider.GetRequiredService>(), + provider.GetRequiredService(), + provider.GetRequiredService(), + provider.GetRequiredService(), + provider.GetRequiredService(), + provider.GetRequiredService(), + provider.GetRequiredService(), + provider.GetRequiredService(), + options)); services.Replace(ServiceDescriptor.Singleton()); services.TryAddSingleton(); services.TryAddSingleton(); diff --git a/src/PythonScriptRunner.EntityFrameworkCore/PsrEntityFrameworkOptions.cs b/src/PythonScriptRunner.EntityFrameworkCore/PsrEntityFrameworkOptions.cs index 6ba5626..c00187a 100644 --- a/src/PythonScriptRunner.EntityFrameworkCore/PsrEntityFrameworkOptions.cs +++ b/src/PythonScriptRunner.EntityFrameworkCore/PsrEntityFrameworkOptions.cs @@ -21,6 +21,12 @@ public sealed class PsrEntityFrameworkOptions /// How long "latest published version" lookups are cached in this process. Default: 30 seconds. public TimeSpan LatestVersionCacheDuration { get; set; } = TimeSpan.FromSeconds(30); + /// + /// Aligns the time used by queued runs (leases, retry delays, schedules, outbox) to the PostgreSQL clock, so hosts with + /// skewed clocks agree. Default: true. Tests that simulate time with a turn it off. + /// + public bool UseDatabaseClock { get; set; } = true; + /// Connects to PostgreSQL (16 or newer). /// Npgsql connection string. /// Extra settings of the data source (e.g. logging, password provider). diff --git a/src/PythonScriptRunner.EntityFrameworkCore/PsrMigrations.cs b/src/PythonScriptRunner.EntityFrameworkCore/PsrMigrations.cs index f64070e..c96218e 100644 --- a/src/PythonScriptRunner.EntityFrameworkCore/PsrMigrations.cs +++ b/src/PythonScriptRunner.EntityFrameworkCore/PsrMigrations.cs @@ -2,23 +2,52 @@ using Microsoft.EntityFrameworkCore.Infrastructure; using Microsoft.EntityFrameworkCore.Migrations; using PythonScriptRunner.EntityFrameworkCore; +using PythonScriptRunner.EntityFrameworkCore.Runs; namespace Microsoft.Extensions.DependencyInjection; /// Applies or scripts the migrations of the psr schema. Migrations never run implicitly. public static class PsrMigrations { - /// Creates or updates the psr schema (tables, partitions and functions). Safe to run again. + /// Advisory lock key that serializes migrations of the psr schema ("psrmigr" in ASCII). + private const long MigrationLockKey = 0x707372_6D69_6772; + + /// + /// Creates or updates the psr schema (tables, partitions and functions). Safe to run again, and from several + /// instances at once: a PostgreSQL advisory lock lets one migrate while the others wait (spec 009, AC-8). + /// /// Application services. /// Cancellation token. public static async Task MigratePythonScriptRunnerAsync(this IServiceProvider services, CancellationToken cancellationToken = default) { ArgumentNullException.ThrowIfNull(services); - var factory = services.GetRequiredService>(); - var db = await factory.CreateDbContextAsync(cancellationToken).ConfigureAwait(false); - await using (db.ConfigureAwait(false)) + var connection = await services.GetRequiredService().DataSource.OpenConnectionAsync(cancellationToken).ConfigureAwait(false); + await using (connection.ConfigureAwait(false)) + { + await LockAsync(connection, "SELECT pg_advisory_lock($1)", cancellationToken).ConfigureAwait(false); + try + { + var factory = services.GetRequiredService>(); + var db = await factory.CreateDbContextAsync(cancellationToken).ConfigureAwait(false); + await using (db.ConfigureAwait(false)) + { + await db.Database.MigrateAsync(cancellationToken).ConfigureAwait(false); + } + } + finally + { + await LockAsync(connection, "SELECT pg_advisory_unlock($1)", CancellationToken.None).ConfigureAwait(false); + } + } + } + + private static async Task LockAsync(Npgsql.NpgsqlConnection connection, string sql, CancellationToken cancellationToken) + { + var command = new Npgsql.NpgsqlCommand(sql, connection); + await using (command.ConfigureAwait(false)) { - await db.Database.MigrateAsync(cancellationToken).ConfigureAwait(false); + command.Parameters.Add(new Npgsql.NpgsqlParameter { Value = MigrationLockKey }); + await command.ExecuteScalarAsync(cancellationToken).ConfigureAwait(false); } } diff --git a/src/PythonScriptRunner.EntityFrameworkCore/PythonScriptRunner.EntityFrameworkCore.csproj b/src/PythonScriptRunner.EntityFrameworkCore/PythonScriptRunner.EntityFrameworkCore.csproj index e807626..ea98523 100644 --- a/src/PythonScriptRunner.EntityFrameworkCore/PythonScriptRunner.EntityFrameworkCore.csproj +++ b/src/PythonScriptRunner.EntityFrameworkCore/PythonScriptRunner.EntityFrameworkCore.csproj @@ -14,6 +14,7 @@ + diff --git a/src/PythonScriptRunner.EntityFrameworkCore/Runs/AsyncRunContracts.cs b/src/PythonScriptRunner.EntityFrameworkCore/Runs/AsyncRunContracts.cs new file mode 100644 index 0000000..995baa6 --- /dev/null +++ b/src/PythonScriptRunner.EntityFrameworkCore/Runs/AsyncRunContracts.cs @@ -0,0 +1,156 @@ +using System.Text.Json.Nodes; + +namespace PythonScriptRunner.EntityFrameworkCore; + +/// A run to queue with . +public sealed record NewAsyncRun +{ + /// or . + public required RunKind Kind { get; init; } + + /// The stage, for single-stage runs. + public StageDefinition? Stage { get; init; } + + /// The pipeline, for pipeline runs. + public PipelineDefinition? Pipeline { get; init; } + + /// Validated inputs (see ). + public required BoundRunInputs Inputs { get; init; } + + /// How the run started. + public RunTrigger Trigger { get; init; } = RunTrigger.Library; + + /// Who requested it (a user, system:schedule:<id>, queue:<system>...). + public required string TriggeredBy { get; init; } + + /// Deduplication key given by the caller (stored with the run). + public string? IdempotencyKey { get; init; } + + /// Deduplication key used when the caller gave none (e.g. the AMQP message_id; not stored with the run). + public string? ImplicitIdempotencyKey { get; init; } + + /// Correlation of the caller, returned in the events. + public string? CorrelationId { get; init; } + + /// W3C trace context of the caller (the worker activities become its children). + public string? TraceParent { get; init; } + + /// Schedule that fired the run. + public ScheduleId? ScheduleId { get; init; } + + /// Details of the trigger (e.g. missed occurrences of a schedule). + public JsonObject? TriggerDetails { get; init; } + + /// Publish run.queued again when an earlier run is returned for the same key (queue commands). + public bool AnnounceDuplicate { get; init; } +} + +/// A step claimed by a worker: the attempt is recorded with a lease and everything needed to execute it is loaded. +public sealed record ClaimedStep +{ + /// Run identifier. + public required RunId RunId { get; init; } + + /// What the run executes. + public required RunKind Kind { get; init; } + + /// Step identifier. + public required StepRunId StepRunId { get; init; } + + /// 1-based position. + public required int Order { get; init; } + + /// Step key. + public required string StepKey { get; init; } + + /// Attempt identifier. + public required AttemptId AttemptId { get; init; } + + /// Attempt number. + public required int Attempt { get; init; } + + /// Why the attempt started. + public required AttemptCause Cause { get; init; } + + /// Who started the attempt. + public required string TriggeredBy { get; init; } + + /// Worker that holds the lease. + public required string WorkerId { get; init; } + + /// Exact stage version of the step. + public required StageDefinition Stage { get; init; } + + /// The pipeline, for pipeline runs. + public PipelineDefinition? Pipeline { get; init; } + + /// Validated inputs of the run. + public required JsonObject RunInputs { get; init; } + + /// Input files of the run (streamed from the database). + public required IReadOnlyDictionary> RunInputFiles { get; init; } + + /// Steps of the run as stored. + public required IReadOnlyList Steps { get; init; } + + /// W3C trace context of the run. + public string? TraceParent { get; init; } + + /// The request for . + public StepExecutionRequest ToExecutionRequest() => new() + { + RunId = RunId, + Kind = Kind, + Stage = Stage, + Pipeline = Pipeline, + Order = Order, + RunInputs = RunInputs, + RunInputFiles = RunInputFiles, + CompletedSteps = Steps, + AttemptId = AttemptId, + Attempt = Attempt, + Cause = Cause, + TriggeredBy = TriggeredBy, + }; +} + +/// What happened when a step was completed. +public enum StepCompletion +{ + /// The next step was queued. + NextStepQueued, + + /// The step failed and an automatic retry was scheduled. + RetryScheduled, + + /// The run succeeded. + RunSucceeded, + + /// The run failed. + RunFailed, + + /// The run was cancelled. + RunCancelled, + + /// The lease was taken by the reaper: nothing was recorded. + LeaseLost, +} + +/// Outcome of one reaper cycle. +/// Attempts restarted (). +/// Steps failed with . +/// Cancelling runs finished. +public sealed record LeaseReaping(int Recovered, int Failed, int Cancelled) +{ + /// Attempts handled. + public int Total => Recovered + Failed + Cancelled; +} + +/// Outcome of one scheduler cycle. +/// Runs created. +/// Firings rejected (run.rejected). +public sealed record ScheduleFiring(int Fired, int Rejected) +{ + /// Schedules handled. + public int Total => Fired + Rejected; +} diff --git a/src/PythonScriptRunner.EntityFrameworkCore/Runs/AsyncRunLifecycle.cs b/src/PythonScriptRunner.EntityFrameworkCore/Runs/AsyncRunLifecycle.cs new file mode 100644 index 0000000..fcf6835 --- /dev/null +++ b/src/PythonScriptRunner.EntityFrameworkCore/Runs/AsyncRunLifecycle.cs @@ -0,0 +1,814 @@ +using System.Text.Json.Nodes; +using Microsoft.EntityFrameworkCore; +using PythonScriptRunner.EntityFrameworkCore.Catalog; +using PythonScriptRunner.EntityFrameworkCore.Entities; +using PythonScriptRunner.EntityFrameworkCore.Runs; +using PythonScriptRunner.EntityFrameworkCore.Scheduling; + +namespace PythonScriptRunner.EntityFrameworkCore; + +/// +/// Transactional lifecycle of queued runs (specs 005 and 006; D-112): creation with the outbox, claiming a step under a lease, +/// completing it (next step, retry, end of the run), the lease reaper, the per-pipeline concurrency queue and the cron +/// scheduler. Every change and its messages commit together; the queue and worker packages only move messages and processes. +/// Rows are always locked in the same order (pipeline or run first, then steps) to avoid deadlocks. +/// +public sealed class AsyncRunLifecycle +{ + private readonly IDbContextFactory _factory; + private readonly ArtifactStore _artifacts; + private readonly IIdempotencyStore _idempotency; + private readonly RunLoader _loader; + private readonly IRunMessageOutbox _messages; + private readonly PsrClock _clock; + private readonly ICatalog _catalog; + private readonly IRunEngine _engine; + private readonly PsrEntityFrameworkOptions _options; + + internal AsyncRunLifecycle( + IDbContextFactory factory, + ArtifactStore artifacts, + IIdempotencyStore idempotency, + RunLoader loader, + IRunMessageOutbox messages, + PsrClock clock, + ICatalog catalog, + IRunEngine engine, + PsrEntityFrameworkOptions options) + { + _factory = factory; + _artifacts = artifacts; + _idempotency = idempotency; + _loader = loader; + _messages = messages; + _clock = clock; + _catalog = catalog; + _engine = engine; + _options = options; + } + + /// + /// Creates a queued run in one transaction: idempotency key, run (Queued, or Pending when its pipeline is at + /// max_concurrent_runs), steps, input files, ExecuteStep of the first step and run.queued. + /// + /// The run. + /// Cancellation token. + public async Task CreateAsync(NewAsyncRun run, CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(run); + EnsureQueue(); + var db = await _factory.CreateDbContextAsync(cancellationToken).ConfigureAwait(false); + await using (db.ConfigureAwait(false)) + { + var transaction = await db.Database.BeginTransactionAsync(cancellationToken).ConfigureAwait(false); + await using (transaction.ConfigureAwait(false)) + { + var now = await _clock.GetUtcNowAsync(cancellationToken).ConfigureAwait(false); + var handle = await CreateAsync(db, run, now, cancellationToken).ConfigureAwait(false); + await transaction.CommitAsync(cancellationToken).ConfigureAwait(false); + return handle; + } + } + } + + /// Writes a run.rejected event (a queue command that created no run). + /// The event. + /// Cancellation token. + public async Task PublishRejectedAsync(RunEvent rejection, CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(rejection); + var db = await _factory.CreateDbContextAsync(cancellationToken).ConfigureAwait(false); + await using (db.ConfigureAwait(false)) + { + _messages.Publish(db, rejection, null); + await db.SaveChangesAsync(cancellationToken).ConfigureAwait(false); + } + } + + /// + /// Claims the step of an ExecuteStep command: Queued → Running only if the attempt is the expected one + /// (duplicates and stale commands return null), records the attempt with a lease and loads what the engine needs. + /// A cancelled run ends here (null). + /// + /// The command. + /// Worker that takes the lease. + /// Lease length (renewed with ). + /// Cancellation token. + public async Task ClaimStepAsync(ExecuteStepCommand command, string workerId, TimeSpan leaseDuration, CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(command); + ArgumentException.ThrowIfNullOrWhiteSpace(workerId); + var runId = command.RunId; + var createdAt = runId.CreatedAt; + var db = await _factory.CreateDbContextAsync(cancellationToken).ConfigureAwait(false); + await using (db.ConfigureAwait(false)) + { + var attemptId = AttemptId.New(); + string triggeredBy; + var transaction = await db.Database.BeginTransactionAsync(cancellationToken).ConfigureAwait(false); + await using (transaction.ConfigureAwait(false)) + { + var now = await _clock.GetUtcNowAsync(cancellationToken).ConfigureAwait(false); + var run = await LockRunAsync(db, runId, cancellationToken).ConfigureAwait(false); + var step = run?.Steps.SingleOrDefault(s => s.Id == command.StepRunId.Value); + if (run is null || step is null || step.Status != StepStatus.Queued || step.CurrentAttempt != command.Attempt - 1) + { + return null; + } + + if (run.Status is RunStatus.Cancelled or RunStatus.Cancelling) + { + // A command delivered after the cancellation (e.g. a delayed retry): the step and the run end here. + step.Status = StepStatus.Cancelled; + step.FinishedAt = now; + if (run.Status == RunStatus.Cancelling) + { + await FinishCancelledAsync(db, run, now, cancellationToken).ConfigureAwait(false); + } + + await db.SaveChangesAsync(cancellationToken).ConfigureAwait(false); + await transaction.CommitAsync(cancellationToken).ConfigureAwait(false); + return null; + } + + if (run.Status is not (RunStatus.Queued or RunStatus.Running)) + { + return null; + } + + triggeredBy = command.TriggeredBy ?? run.TriggeredBy; + step.Status = StepStatus.Running; + step.CurrentAttempt = command.Attempt; + step.StartedAt = now; + step.FinishedAt = null; + db.StepAttempts.Add(new StepAttemptEntity + { + Id = attemptId.Value, + StepRunId = step.Id, + RunId = run.Id, + RunCreatedAt = createdAt, + Attempt = command.Attempt, + Cause = command.Cause, + TriggeredBy = triggeredBy, + Status = AttemptStatus.Running, + WorkerId = workerId, + LeaseExpiresAt = now + leaseDuration, + StartedAt = now, + }); + if (run.Status == RunStatus.Queued) + { + run.Status = RunStatus.Running; + run.StartedAt ??= now; + Publish(db, run, RunEventFactory.For(run, RunEventTypes.RunStarted, now)); + } + + Publish(db, run, RunEventFactory.ForStep(run, step, command.Attempt, RunEventTypes.StepStarted, now)); + await db.SaveChangesAsync(cancellationToken).ConfigureAwait(false); + await transaction.CommitAsync(cancellationToken).ConfigureAwait(false); + } + + db.ChangeTracker.Clear(); + var loaded = await RunLoader.LoadAsync(db, runId, cancellationToken).ConfigureAwait(false) + ?? throw new RunNotFoundException(runId); + var claimed = loaded.Run.Steps.Single(s => s.Id == command.StepRunId.Value); + var (_, pipeline) = loaded.Run.Kind is RunKind.Pipeline or RunKind.PipelineTest + ? await RunDefinitions.LoadAsync(db, loaded.Run, cancellationToken).ConfigureAwait(false) + : default; + var stage = await RunDefinitions.LoadStageAsync(db, claimed, cancellationToken).ConfigureAwait(false); + var (inputs, inputFiles) = _loader.Inputs(loaded); + return new ClaimedStep + { + RunId = runId, + Kind = loaded.Run.Kind, + StepRunId = command.StepRunId, + Order = claimed.Order, + StepKey = claimed.StepKey, + AttemptId = attemptId, + Attempt = command.Attempt, + Cause = command.Cause, + TriggeredBy = triggeredBy, + WorkerId = workerId, + Stage = stage, + Pipeline = pipeline, + RunInputs = inputs, + RunInputFiles = inputFiles, + Steps = _loader.ToStepResults(loaded), + TraceParent = loaded.Run.TraceParent, + }; + } + } + + /// Extends the lease of a claimed attempt. false when the reaper took it: the caller must stop the process. + /// The claimed step. + /// New lease length from now. + /// Cancellation token. + public async Task RenewLeaseAsync(ClaimedStep step, TimeSpan leaseDuration, CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(step); + var createdAt = step.RunId.CreatedAt; + var now = await _clock.GetUtcNowAsync(cancellationToken).ConfigureAwait(false); + var db = await _factory.CreateDbContextAsync(cancellationToken).ConfigureAwait(false); + await using (db.ConfigureAwait(false)) + { + return await db.StepAttempts + .Where(a => a.Id == step.AttemptId.Value && a.RunCreatedAt == createdAt && a.Status == AttemptStatus.Running && a.WorkerId == step.WorkerId) + .ExecuteUpdateAsync(s => s.SetProperty(a => a.LeaseExpiresAt, now + leaseDuration), cancellationToken).ConfigureAwait(false) == 1; + } + } + + /// + /// Records the outcome of a claimed attempt and decides what comes next, in one transaction: next step queued, automatic + /// retry (delayed through the outbox), run succeeded, failed or cancelled. when + /// the reaper took the lease meanwhile (nothing is recorded). + /// + /// The claimed step. + /// Result of . + /// Cancellation token. + public async Task CompleteStepAsync(ClaimedStep step, StepExecution execution, CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(step); + ArgumentNullException.ThrowIfNull(execution); + var createdAt = step.RunId.CreatedAt; + var result = execution.Step; + var db = await _factory.CreateDbContextAsync(cancellationToken).ConfigureAwait(false); + await using (db.ConfigureAwait(false)) + { + var transaction = await db.Database.BeginTransactionAsync(cancellationToken).ConfigureAwait(false); + await using (transaction.ConfigureAwait(false)) + { + var now = await _clock.GetUtcNowAsync(cancellationToken).ConfigureAwait(false); + var attempt = execution.Attempt ?? new AttemptResult + { + Id = step.AttemptId, + Number = step.Attempt, + Cause = step.Cause, + Status = AttemptStatus.Failed, + StartedAt = now, + Duration = TimeSpan.Zero, + Overhead = TimeSpan.Zero, + Error = result.Error, + }; + if (!await AttemptRows.FinishAsync(db, createdAt, attempt, now, step.WorkerId, cancellationToken).ConfigureAwait(false)) + { + return StepCompletion.LeaseLost; + } + + var run = await LockRunAsync(db, step.RunId, cancellationToken).ConfigureAwait(false) ?? throw new RunNotFoundException(step.RunId); + var row = run.Steps.Single(s => s.Id == step.StepRunId.Value); + if (execution.Inputs is { } inputs) + { + row.Inputs = ArtifactRefs.ToStored(inputs, execution.InputFiles).ToJsonString(); + } + + var completion = result.Status switch + { + StepStatus.Succeeded => await SucceededAsync(db, step, execution, run, row, now, cancellationToken).ConfigureAwait(false), + StepStatus.Cancelled => await EndRunAsync(db, run, row, StepStatus.Cancelled, result.Error, now, cancellationToken).ConfigureAwait(false), + _ => await FailedAsync(db, step, execution, run, row, now, cancellationToken).ConfigureAwait(false), + }; + await db.SaveChangesAsync(cancellationToken).ConfigureAwait(false); + await transaction.CommitAsync(cancellationToken).ConfigureAwait(false); + return completion; + } + } + } + + /// + /// Handles attempts whose lease expired (worker lost; D-85): a cancelling run ends Cancelled; an idempotent stage or + /// one with retries left gets a new attempt (, at most + /// per step); otherwise the step fails with . + /// + /// Attempts handled per call (locked with SKIP LOCKED, so reapers do not collide). + /// Recoveries per step (QA-006-2: 3). + /// Cancellation token. + public async Task ReapExpiredLeasesAsync(int batchSize = 50, int maxRecoveries = 3, CancellationToken cancellationToken = default) + { + var db = await _factory.CreateDbContextAsync(cancellationToken).ConfigureAwait(false); + await using (db.ConfigureAwait(false)) + { + var transaction = await db.Database.BeginTransactionAsync(cancellationToken).ConfigureAwait(false); + await using (transaction.ConfigureAwait(false)) + { + var now = await _clock.GetUtcNowAsync(cancellationToken).ConfigureAwait(false); + var expired = await db.StepAttempts.FromSql($""" + SELECT * FROM psr.step_attempts + WHERE status = 'Running' AND lease_expires_at < {now} + ORDER BY lease_expires_at + LIMIT {batchSize} + FOR UPDATE SKIP LOCKED + """).ToListAsync(cancellationToken).ConfigureAwait(false); + int recovered = 0, failed = 0, cancelled = 0; + foreach (var attempt in expired) + { + var error = new RunError(ErrorKind.WorkerLost, $"The worker {attempt.WorkerId} stopped renewing the lease of this attempt.") + { + Code = "WORKER_LOST", + }; + attempt.Status = AttemptStatus.WorkerLost; + attempt.ErrorKind = ErrorKind.WorkerLost; + attempt.Error = StoredJson.Error(error); + attempt.FinishedAt = now; + attempt.DurationMs = (int)(now - attempt.StartedAt).TotalMilliseconds; + attempt.LeaseExpiresAt = null; + + var run = await LockRunAsync(db, new RunId(attempt.RunId), cancellationToken).ConfigureAwait(false); + var step = run?.Steps.SingleOrDefault(s => s.Id == attempt.StepRunId); + if (run is null || step is null || step.Status != StepStatus.Running || step.CurrentAttempt != attempt.Attempt) + { + continue; + } + + if (run.Status == RunStatus.Cancelling) + { + await EndRunAsync(db, run, step, StepStatus.Cancelled, new RunError(ErrorKind.Cancelled, "The run was cancelled."), now, cancellationToken) + .ConfigureAwait(false); + cancelled++; + continue; + } + + var stage = await RunDefinitions.LoadStageAsync(db, step, cancellationToken).ConfigureAwait(false); + var series = await SeriesAsync(db, step, cancellationToken).ConfigureAwait(false); + var recoveries = series.Count(a => a.Cause == AttemptCause.LeaseRecovery); + if ((stage.Idempotent || RetriesUsed(series, stage) < stage.Retry.MaxAttempts) && recoveries < maxRecoveries) + { + step.Status = StepStatus.Queued; + step.Error = StoredJson.Error(error with { StepKey = step.StepKey }); + _messages.ExecuteStep(db, new ExecuteStepCommand(new RunId(run.Id), run.CreatedAt, new StepRunId(step.Id), attempt.Attempt + 1) + { + Cause = AttemptCause.LeaseRecovery, + TriggeredBy = attempt.TriggeredBy, + }, now, run.TraceParent); + Publish(db, run, RunEventFactory.ForStep(run, step, attempt.Attempt, RunEventTypes.StepRetryScheduled, now) with + { + Error = RunEventError.From(error), + RetryAt = now, + }); + recovered++; + } + else + { + await EndRunAsync(db, run, step, StepStatus.Failed, error with { StepKey = step.StepKey }, now, cancellationToken).ConfigureAwait(false); + failed++; + } + } + + await db.SaveChangesAsync(cancellationToken).ConfigureAwait(false); + await transaction.CommitAsync(cancellationToken).ConfigureAwait(false); + return new LeaseReaping(recovered, failed, cancelled); + } + } + } + + /// + /// Fires the schedules due now (D-41, D-44, D-45, D-77): each one locked with SKIP LOCKED, so with N workers an + /// occurrence fires once. Missed occurrences become one recovery run (missedOccurrences in its trigger details); + /// an invalid firing creates no run, writes last_error and run.rejected, and the schedule stays enabled. + /// + /// Schedules handled per call. + /// Cancellation token. + public async Task FireDueSchedulesAsync(int batchSize = 50, CancellationToken cancellationToken = default) + { + EnsureQueue(); + var db = await _factory.CreateDbContextAsync(cancellationToken).ConfigureAwait(false); + await using (db.ConfigureAwait(false)) + { + var transaction = await db.Database.BeginTransactionAsync(cancellationToken).ConfigureAwait(false); + await using (transaction.ConfigureAwait(false)) + { + var now = await _clock.GetUtcNowAsync(cancellationToken).ConfigureAwait(false); + var due = await db.Schedules.FromSql($""" + SELECT *, xmin FROM psr.schedules + WHERE is_enabled AND next_run_at <= {now} + ORDER BY next_run_at + LIMIT {batchSize} + FOR UPDATE SKIP LOCKED + """).ToListAsync(cancellationToken).ConfigureAwait(false); + int fired = 0, rejected = 0; + foreach (var schedule in due) + { + var pipeline = await db.Pipelines.AsNoTracking().Where(p => p.Id == schedule.PipelineId) + .Select(p => new { p.Key, p.LatestPublishedVersion }).SingleAsync(cancellationToken).ConfigureAwait(false); + var details = new JsonObject { ["scheduleId"] = schedule.Id.ToString(), ["scheduledFor"] = schedule.NextRunAt }; + CronSchedule? cron = null; + try + { + cron = CronSchedule.Parse(schedule.Cron, schedule.TimeZone); + var missed = cron.CountOccurrences(schedule.NextRunAt!.Value, now); + if (missed > 1) + { + details["missedOccurrences"] = missed; + } + + // The latest version is read from the database (not the catalog cache): a new publication applies at once (D-77). + var definition = await ScheduleRules.ResolveAsync(_catalog, pipeline.Key, schedule.PinnedVersion ?? pipeline.LatestPublishedVersion, + cancellationToken).ConfigureAwait(false); + var inputs = StoredJson.Object(schedule.Inputs) ?? []; + ScheduleRules.EnsureNoFileInputs(definition, inputs); + var handle = await CreateAsync(db, new NewAsyncRun + { + Kind = RunKind.Pipeline, + Pipeline = definition, + Inputs = _engine.BindInputs(definition.Inputs, inputs), + Trigger = RunTrigger.Schedule, + TriggeredBy = $"system:schedule:{schedule.Id}", + ScheduleId = new ScheduleId(schedule.Id), + TriggerDetails = details, + }, now, cancellationToken).ConfigureAwait(false); + schedule.LastRunId = handle.RunId.Value; + schedule.LastError = null; + fired++; + } + catch (PsrException ex) + { + schedule.LastError = ex.Message.Length <= 2000 ? ex.Message : ex.Message[..2000]; + _messages.Publish(db, new RunEvent + { + EventId = Guid.CreateVersion7(now), + Type = RunEventTypes.RunRejected, + OccurredAt = now, + Kind = RunKind.Pipeline, + PipelineKey = pipeline.Key, + Trigger = RunTrigger.Schedule, + TriggerDetails = details, + Error = Rejection(ex), + }, null); + rejected++; + } + + schedule.LastFiredAt = now; + schedule.NextRunAt = cron?.Next(now); + } + + await db.SaveChangesAsync(cancellationToken).ConfigureAwait(false); + await transaction.CommitAsync(cancellationToken).ConfigureAwait(false); + return new ScheduleFiring(fired, rejected); + } + } + } + + /// Runs waiting for a free slot of their pipeline (gauge psr.runs.pending). + /// Cancellation token. + public async Task CountPendingRunsAsync(CancellationToken cancellationToken = default) + { + var db = await _factory.CreateDbContextAsync(cancellationToken).ConfigureAwait(false); + await using (db.ConfigureAwait(false)) + { + return await db.Runs.CountAsync(r => r.Status == RunStatus.Pending, cancellationToken).ConfigureAwait(false); + } + } + + /// Reasons of a rejected request. + /// Why it was rejected. + public static RunEventError Rejection(Exception exception) + { + ArgumentNullException.ThrowIfNull(exception); + var (code, reasons) = exception switch + { + InputValidationException invalid => ("INPUT_INVALID", invalid.Errors.Select(e => $"{e.Field}: {e.Message}").ToList()), + PsrDomainException domain => (domain.Code, [domain.Message]), + PipelineNotFoundException => ("PIPELINE_NOT_FOUND", [exception.Message]), + VersionNotFoundException or VersionNotRunnableException => ("VERSION_NOT_RUNNABLE", [exception.Message]), + _ => ("REJECTED", (List)[exception.Message]), + }; + return new RunEventError("Rejected", code, exception.Message, null) { Reasons = reasons }; + } + + /// Creates the run inside the caller's transaction. + internal async Task CreateAsync(PsrDbContext db, NewAsyncRun run, DateTimeOffset now, CancellationToken cancellationToken) + { + var (targetKey, targetVersion, steps) = run switch + { + { Pipeline: { } pipeline } => (pipeline.Key, pipeline.Version, + (IReadOnlyList)[.. pipeline.Steps.Select((s, i) => new PlannedStep(s.Key, i + 1, s.StageKey, s.StageVersion))]), + { Stage: { } stage } => (stage.Key, stage.Version, [new PlannedStep(stage.Key, 1, stage.Key, stage.Version)]), + _ => throw new ArgumentException("Set the stage or the pipeline of the run.", nameof(run)), + }; + var runId = RunId.New(now); + if ((run.IdempotencyKey ?? run.ImplicitIdempotencyKey) is { } key + && await _idempotency.ReserveAsync(db, IdempotencyScopes.For(run.Kind, targetKey), key, runId, cancellationToken).ConfigureAwait(false) is { } existingId + && await db.Runs.SingleOrDefaultAsync(r => r.Id == existingId.Value && r.CreatedAt == existingId.CreatedAt, cancellationToken).ConfigureAwait(false) + is { } existing) + { + if (run.AnnounceDuplicate) + { + Publish(db, existing, RunEventFactory.For(existing, RunEventTypes.RunQueued, now)); + await db.SaveChangesAsync(cancellationToken).ConfigureAwait(false); + } + + return new RunHandle(existingId, existing.Status, Deduplicated: true); + } + + var (row, retention) = await RunRows.NewAsync(db, runId, run.Kind, targetKey, targetVersion, steps, cancellationToken).ConfigureAwait(false); + var status = RunStatus.Queued; + if (row.PipelineId is { } pipelineId && await LockPipelineAsync(db, pipelineId, cancellationToken).ConfigureAwait(false) is { } limit + && await CountActiveAsync(db, pipelineId, cancellationToken).ConfigureAwait(false) >= limit) + { + status = RunStatus.Pending; + } + + row.Mode = ExecutionMode.Async; + row.Status = status; + row.Trigger = run.Trigger; + row.TriggeredBy = run.TriggeredBy; + row.Inputs = run.Inputs.Wire.ToJsonString(); + row.IdempotencyKey = run.IdempotencyKey; + row.CorrelationId = run.CorrelationId; + row.TraceParent = run.TraceParent; + row.ScheduleId = run.ScheduleId?.Value; + row.TriggerDetails = run.TriggerDetails?.ToJsonString(); + row.QueuedAt = now; + db.Runs.Add(row); + await db.SaveChangesAsync(cancellationToken).ConfigureAwait(false); + + await RunRows.StoreFilesAsync(db, _artifacts, runId, null, ArtifactDirection.Input, run.Inputs.Wire, run.Inputs.Files, + retention ?? _options.Retention.Artifacts, cancellationToken).ConfigureAwait(false); + row.Inputs = ArtifactRefs.ToStored(run.Inputs.Wire, run.Inputs.Files).ToJsonString(); + if (status == RunStatus.Queued) + { + QueueStep(db, row, row.Steps.Single(s => s.Order == 1), now, AttemptCause.Initial, row.TriggeredBy); + } + + Publish(db, row, RunEventFactory.For(row, RunEventTypes.RunQueued, now)); + await db.SaveChangesAsync(cancellationToken).ConfigureAwait(false); + return new RunHandle(runId, status, Deduplicated: false); + } + + /// + /// Cancels a run inside the caller's transaction (its row is locked here): waiting runs end now; running ones move to + /// Cancelling, every instance is told to stop the process, and a queued run between steps ends now. + /// + /// The status after the call, or null when the run does not exist. + internal async Task CancelAsync(PsrDbContext db, RunId runId, Actor actor, DateTimeOffset now, CancellationToken cancellationToken) + { + var run = await LockRunAsync(db, runId, cancellationToken).ConfigureAwait(false); + if (run is null) + { + return null; + } + + switch (run.Status) + { + case RunStatus.Pending or RunStatus.Queued: + run.CancelRequestedAt = now; + run.CancelRequestedBy = actor.Subject; + await FinishCancelledAsync(db, run, now, cancellationToken).ConfigureAwait(false); + break; + case RunStatus.Running or RunStatus.Cancelling: + run.Status = RunStatus.Cancelling; + run.CancelRequestedAt ??= now; + run.CancelRequestedBy ??= actor.Subject; + var running = run.Steps.FirstOrDefault(s => s.Status == StepStatus.Running); + _messages.CancelRun(db, new CancelRunCommand(runId, running is null ? null : new StepRunId(running.Id), actor.Subject), now); + if (run.Mode == ExecutionMode.Async && running is null) + { + // Between steps (e.g. waiting for a retry): nothing is executing, so the run ends now. + await FinishCancelledAsync(db, run, now, cancellationToken).ConfigureAwait(false); + } + + break; + default: + throw new PsrDomainException(PsrErrorCodes.RunNotCancellable, $"Run {runId} already finished ({run.Status})."); + } + + return run.Status; + } + + /// + /// Queues a failed run again inside the caller's transaction (D-76, QA-003-3): successful steps keep their outputs and the + /// first unsuccessful step gets a new attempt (). + /// + internal async Task QueueReprocessAsync(PsrDbContext db, RunId runId, Actor actor, DateTimeOffset now, CancellationToken cancellationToken) + { + EnsureQueue(); + var run = await LockRunAsync(db, runId, cancellationToken).ConfigureAwait(false); + if (run is not { Status: RunStatus.Failed }) + { + return false; + } + + run.Status = RunStatus.Running; + run.Mode = ExecutionMode.Async; + run.Outputs = null; + run.Error = null; + run.FinishedAt = null; + foreach (var skipped in run.Steps.Where(s => s.Status == StepStatus.Skipped)) + { + skipped.Status = StepStatus.Pending; + } + + var step = run.Steps.OrderBy(s => s.Order).First(s => s.Status != StepStatus.Succeeded); + QueueStep(db, run, step, now, step.CurrentAttempt == 0 ? AttemptCause.Initial : AttemptCause.Reprocess, actor.Subject); + Publish(db, run, RunEventFactory.For(run, RunEventTypes.RunStarted, now)); + return true; + } + + private async Task SucceededAsync( + PsrDbContext db, ClaimedStep claimed, StepExecution execution, RunEntity run, StepRunEntity step, DateTimeOffset now, CancellationToken cancellationToken) + { + var result = execution.Step; + var retention = await RunRows.ArtifactRetentionAsync(db, claimed.RunId, cancellationToken).ConfigureAwait(false) ?? _options.Retention.Artifacts; + await RunRows.StoreFilesAsync(db, _artifacts, claimed.RunId, claimed.StepRunId, ArtifactDirection.Output, result.Outputs!, result.Files, retention, + cancellationToken).ConfigureAwait(false); + step.Outputs = ArtifactRefs.ToStored(result.Outputs!, result.Files).ToJsonString(); + step.Status = StepStatus.Succeeded; + step.Error = null; + step.FinishedAt = now; + Publish(db, run, RunEventFactory.ForStep(run, step, claimed.Attempt, RunEventTypes.StepSucceeded, now)); + if (run.Status == RunStatus.Cancelling) + { + await FinishCancelledAsync(db, run, now, cancellationToken).ConfigureAwait(false); + return StepCompletion.RunCancelled; + } + + if (run.Steps.SingleOrDefault(s => s.Order == step.Order + 1) is { } next) + { + QueueStep(db, run, next, now, next.CurrentAttempt == 0 ? AttemptCause.Initial : AttemptCause.Reprocess, claimed.TriggeredBy); + return StepCompletion.NextStepQueued; + } + + run.Outputs = execution.RunOutputs is { } outputs ? ArtifactRefs.ToStored(outputs, execution.RunOutputFiles).ToJsonString() : "{}"; + run.Status = RunStatus.Succeeded; + run.Error = null; + run.FinishedAt = now; + await db.SaveChangesAsync(cancellationToken).ConfigureAwait(false); + var artifacts = await db.Artifacts.AsNoTracking().Where(a => a.RunId == run.Id && a.RunCreatedAt == run.CreatedAt) + .ToDictionaryAsync(a => a.Id, cancellationToken).ConfigureAwait(false); + Publish(db, run, RunEventFactory.Succeeded(run, artifacts, now)); + await PromotePendingAsync(db, run, now, cancellationToken).ConfigureAwait(false); + return StepCompletion.RunSucceeded; + } + + private async Task FailedAsync( + PsrDbContext db, ClaimedStep claimed, StepExecution execution, RunEntity run, StepRunEntity step, DateTimeOffset now, CancellationToken cancellationToken) + { + var result = execution.Step; + var error = result.Error ?? new RunError(ErrorKind.ProcessCrashed, "The step failed.") { StepKey = step.StepKey }; + if (run.Status != RunStatus.Cancelling && execution.Attempt is not null) + { + var series = await SeriesAsync(db, step, cancellationToken).ConfigureAwait(false); + var used = RetriesUsed(series, claimed.Stage); + if (used < claimed.Stage.Retry.MaxAttempts) + { + var retryAt = now + claimed.Stage.Retry.GetDelay(used + 1); + step.Status = StepStatus.Queued; + step.Error = StoredJson.Error(error); + QueueStep(db, run, step, retryAt, AttemptCause.AutoRetry, claimed.TriggeredBy); + Publish(db, run, RunEventFactory.ForStep(run, step, claimed.Attempt, RunEventTypes.StepRetryScheduled, now) with + { + Error = RunEventError.From(error), + RetryAt = retryAt, + }); + return StepCompletion.RetryScheduled; + } + } + + var status = result.Status == StepStatus.TimedOut ? StepStatus.TimedOut : StepStatus.Failed; + return await EndRunAsync(db, run, step, status, error, now, cancellationToken).ConfigureAwait(false); + } + + /// + /// Ends the run at : the step gets its final status, the following steps are skipped, the run fails + /// (or is cancelled) and its pipeline slot is released. + /// + private async Task EndRunAsync( + PsrDbContext db, RunEntity run, StepRunEntity step, StepStatus status, RunError? error, DateTimeOffset now, CancellationToken cancellationToken) + { + step.Status = status; + step.Error = StoredJson.Error(error); + step.FinishedAt = now; + if (status == StepStatus.Cancelled || run.Status == RunStatus.Cancelling) + { + await FinishCancelledAsync(db, run, now, cancellationToken).ConfigureAwait(false); + return StepCompletion.RunCancelled; + } + + Publish(db, run, RunEventFactory.ForStep(run, step, step.CurrentAttempt, RunEventTypes.StepFailed, now) with { Error = RunEventError.From(error) }); + foreach (var later in run.Steps.Where(s => s.Order > step.Order && s.Status is StepStatus.Pending or StepStatus.Queued)) + { + later.Status = StepStatus.Skipped; + } + + run.Status = RunStatus.Failed; + run.Error = StoredJson.Error(error); + run.FinishedAt = now; + Publish(db, run, RunEventFactory.For(run, RunEventTypes.RunFailed, now) with { Error = RunEventError.From(error) }); + await db.SaveChangesAsync(cancellationToken).ConfigureAwait(false); + await PromotePendingAsync(db, run, now, cancellationToken).ConfigureAwait(false); + return StepCompletion.RunFailed; + } + + /// Ends a run as cancelled: open steps are cancelled, pending ones skipped, run.cancelled, slot released. + private async Task FinishCancelledAsync(PsrDbContext db, RunEntity run, DateTimeOffset now, CancellationToken cancellationToken) + { + foreach (var step in run.Steps) + { + if (step.Status is StepStatus.Running or StepStatus.Queued) + { + step.Status = StepStatus.Cancelled; + step.FinishedAt = now; + } + else if (step.Status == StepStatus.Pending) + { + step.Status = StepStatus.Skipped; + } + } + + run.Status = RunStatus.Cancelled; + run.FinishedAt = now; + run.Error ??= StoredJson.Error(new RunError(ErrorKind.Cancelled, $"Cancelled by {run.CancelRequestedBy ?? "the system"}.")); + Publish(db, run, RunEventFactory.For(run, RunEventTypes.RunCancelled, now)); + await db.SaveChangesAsync(cancellationToken).ConfigureAwait(false); + await PromotePendingAsync(db, run, now, cancellationToken).ConfigureAwait(false); + } + + /// A finished queued run frees a slot of its pipeline: the oldest Pending runs become Queued (FIFO; D-39). + private async Task PromotePendingAsync(PsrDbContext db, RunEntity finished, DateTimeOffset now, CancellationToken cancellationToken) + { + if (finished.Mode != ExecutionMode.Async || finished.PipelineId is not { } pipelineId) + { + return; + } + + var limit = await LockPipelineAsync(db, pipelineId, cancellationToken).ConfigureAwait(false); + var active = await CountActiveAsync(db, pipelineId, cancellationToken).ConfigureAwait(false); + while (limit is null || active < limit) + { + var waiting = await db.Runs.FromSql($""" + SELECT * FROM psr.runs + WHERE pipeline_id = {pipelineId} AND mode = 'Async' AND status = 'Pending' + ORDER BY created_at, id + LIMIT 1 + FOR UPDATE SKIP LOCKED + """).ToListAsync(cancellationToken).ConfigureAwait(false); + if (waiting.Count == 0) + { + break; + } + + var run = waiting[0]; + var first = await db.StepRuns.SingleAsync(s => s.RunId == run.Id && s.RunCreatedAt == run.CreatedAt && s.Order == 1, cancellationToken) + .ConfigureAwait(false); + run.Status = RunStatus.Queued; + QueueStep(db, run, first, now, AttemptCause.Initial, run.TriggeredBy); + await db.SaveChangesAsync(cancellationToken).ConfigureAwait(false); + active++; + } + } + + /// Marks the step Queued and writes the ExecuteStep of its next attempt. + private void QueueStep(PsrDbContext db, RunEntity run, StepRunEntity step, DateTimeOffset availableAt, AttemptCause cause, string triggeredBy) + { + step.Status = StepStatus.Queued; + step.FinishedAt = null; + _messages.ExecuteStep(db, new ExecuteStepCommand(new RunId(run.Id), run.CreatedAt, new StepRunId(step.Id), step.CurrentAttempt + 1) + { + Cause = cause, + TriggeredBy = triggeredBy, + }, availableAt, run.TraceParent); + } + + private void Publish(PsrDbContext db, RunEntity run, RunEvent runEvent) => _messages.Publish(db, runEvent, run.TraceParent); + + private void EnsureQueue() + { + if (!_messages.IsEnabled) + { + throw new PsrConfigurationException("Queued runs need a run queue: register the RabbitMQ package (AddRabbitMq)."); + } + } + + /// Locks the run row (FOR UPDATE) and loads it with its steps for change. + private static async Task LockRunAsync(PsrDbContext db, RunId runId, CancellationToken cancellationToken) + { + var createdAt = runId.CreatedAt; + var locked = await IdempotencyStore.ScalarAsync(db, "SELECT 1 FROM psr.runs WHERE id = $1 AND created_at = $2 FOR UPDATE", cancellationToken, + runId.Value, createdAt).ConfigureAwait(false); + return locked is null + ? null + : await db.Runs.Include(r => r.Steps).SingleAsync(r => r.Id == runId.Value && r.CreatedAt == createdAt, cancellationToken).ConfigureAwait(false); + } + + /// Locks the pipeline row and returns its max_concurrent_runs (creations and releases of a pipeline queue up here). + private static Task LockPipelineAsync(PsrDbContext db, Guid pipelineId, CancellationToken cancellationToken) => + IdempotencyStore.ScalarAsync(db, "SELECT max_concurrent_runs FROM psr.pipelines WHERE id = $1 FOR UPDATE", cancellationToken, pipelineId); + + /// Queued runs of the pipeline that hold a slot (synchronous runs do not count; QA-006-1). + private static Task CountActiveAsync(PsrDbContext db, Guid pipelineId, CancellationToken cancellationToken) => + db.Runs.CountAsync(r => r.PipelineId == pipelineId && r.Mode == ExecutionMode.Async + && (r.Status == RunStatus.Queued || r.Status == RunStatus.Running || r.Status == RunStatus.Cancelling), cancellationToken); + + /// Attempts of the current series of a step: since its last Initial or Reprocess attempt. + private static async Task> SeriesAsync(PsrDbContext db, StepRunEntity step, CancellationToken cancellationToken) + { + var attempts = await db.StepAttempts.AsNoTracking() + .Where(a => a.StepRunId == step.Id && a.RunCreatedAt == step.RunCreatedAt) + .OrderBy(a => a.Attempt) + .ToListAsync(cancellationToken).ConfigureAwait(false); + var start = attempts.FindLastIndex(a => a.Cause is AttemptCause.Initial or AttemptCause.Reprocess); + return start < 0 ? attempts : attempts[start..]; + } + + /// Automatic retries used by a series; a recovery of a non-idempotent stage uses a retry (D-85). + private static int RetriesUsed(IEnumerable series, StageDefinition stage) => + series.Count(a => a.Cause == AttemptCause.AutoRetry || (a.Cause == AttemptCause.LeaseRecovery && !stage.Idempotent)); +} diff --git a/src/PythonScriptRunner.EntityFrameworkCore/Runs/AttemptRows.cs b/src/PythonScriptRunner.EntityFrameworkCore/Runs/AttemptRows.cs new file mode 100644 index 0000000..75d9b60 --- /dev/null +++ b/src/PythonScriptRunner.EntityFrameworkCore/Runs/AttemptRows.cs @@ -0,0 +1,59 @@ +using Microsoft.EntityFrameworkCore; +using PythonScriptRunner.EntityFrameworkCore.Catalog; +using PythonScriptRunner.EntityFrameworkCore.Entities; + +namespace PythonScriptRunner.EntityFrameworkCore.Runs; + +/// Writes the outcome of an attempt (shared by the synchronous recorder and the queued runs). +internal static class AttemptRows +{ + /// + /// Finishes the attempt row and adds its log tails to the context (saved by the caller). With , + /// only an attempt still running under that worker's lease is finished. + /// + /// Whether the attempt was finished (false: the lease was lost). + public static async Task FinishAsync( + PsrDbContext db, DateTimeOffset runCreatedAt, AttemptResult result, DateTimeOffset finishedAt, string? workerId, CancellationToken cancellationToken) + { + var error = StoredJson.Error(result.Error); + var attempts = db.StepAttempts.Where(a => a.Id == result.Id.Value && a.RunCreatedAt == runCreatedAt); + if (workerId is not null) + { + attempts = attempts.Where(a => a.Status == AttemptStatus.Running && a.WorkerId == workerId); + } + + var finished = await attempts + .ExecuteUpdateAsync(s => s + .SetProperty(a => a.Status, result.Status) + .SetProperty(a => a.ExitCode, result.ExitCode) + .SetProperty(a => a.ErrorKind, result.Error == null ? null : result.Error.Kind) + .SetProperty(a => a.Error, error) + .SetProperty(a => a.EnvironmentHash, result.EnvironmentHash) + .SetProperty(a => a.FinishedAt, finishedAt) + .SetProperty(a => a.DurationMs, (int)result.Duration.TotalMilliseconds) + .SetProperty(a => a.OverheadMs, (int)result.Overhead.TotalMilliseconds) + .SetProperty(a => a.LeaseExpiresAt, (DateTimeOffset?)null), cancellationToken).ConfigureAwait(false); + if (finished == 0) + { + return false; + } + + foreach (var (stream, tail) in (ReadOnlySpan<(LogStream, LogTail)>)[(LogStream.Stdout, result.Stdout), (LogStream.Stderr, result.Stderr)]) + { + if (tail.OriginalBytes > 0 || tail.Content.Length > 0) + { + db.AttemptLogs.Add(new AttemptLogEntity + { + AttemptId = result.Id.Value, + RunCreatedAt = runCreatedAt, + Stream = stream, + Content = tail.Content, + Truncated = tail.Truncated, + OriginalBytes = tail.OriginalBytes, + }); + } + } + + return true; + } +} diff --git a/src/PythonScriptRunner.EntityFrameworkCore/Runs/EfRunManager.cs b/src/PythonScriptRunner.EntityFrameworkCore/Runs/EfRunManager.cs index 21e0acf..fb37eaa 100644 --- a/src/PythonScriptRunner.EntityFrameworkCore/Runs/EfRunManager.cs +++ b/src/PythonScriptRunner.EntityFrameworkCore/Runs/EfRunManager.cs @@ -1,7 +1,6 @@ using System.Text.Json.Nodes; using Microsoft.EntityFrameworkCore; using PythonScriptRunner.EntityFrameworkCore.Catalog; -using PythonScriptRunner.EntityFrameworkCore.Entities; using PythonScriptRunner.EntityFrameworkCore.Infrastructure; namespace PythonScriptRunner.EntityFrameworkCore.Runs; @@ -11,6 +10,9 @@ internal sealed class EfRunManager( IDbContextFactory factory, RunLoader loader, IRunEngine engine, + AsyncRunLifecycle lifecycle, + IRunMessageOutbox messages, + PsrClock clock, TimeProvider time) : IRunManager { public async Task GetAsync(RunId runId, CancellationToken cancellationToken = default) @@ -107,51 +109,28 @@ public async Task> ListAsync(RunQuery query, Cancellation public async Task CancelAsync(RunId runId, Actor actor, CancellationToken cancellationToken = default) { ArgumentNullException.ThrowIfNull(actor); - var createdAt = runId.CreatedAt; + RunStatus? status; var db = await factory.CreateDbContextAsync(cancellationToken).ConfigureAwait(false); await using (db.ConfigureAwait(false)) { - var runs = db.Runs.Where(r => r.Id == runId.Value && r.CreatedAt == createdAt); - var steps = db.StepRuns.Where(s => s.RunId == runId.Value && s.RunCreatedAt == createdAt); - var now = time.GetUtcNow(); - var error = StoredJson.Error(new RunError(ErrorKind.Cancelled, $"Cancelled by {actor.Subject}.")); - - // Transitions are conditional on the current status, so a run that finishes meanwhile is never reopened. - var waiting = await runs.Where(r => r.Status == RunStatus.Pending || r.Status == RunStatus.Queued) - .ExecuteUpdateAsync(s => s - .SetProperty(r => r.Status, RunStatus.Cancelled) - .SetProperty(r => r.FinishedAt, now) - .SetProperty(r => r.Error, error) - .SetProperty(r => r.CancelRequestedAt, now) - .SetProperty(r => r.CancelRequestedBy, actor.Subject), cancellationToken).ConfigureAwait(false); - if (waiting > 0) + var transaction = await db.Database.BeginTransactionAsync(cancellationToken).ConfigureAwait(false); + await using (transaction.ConfigureAwait(false)) { - await steps.Where(s => s.Status == StepStatus.Queued) - .ExecuteUpdateAsync(s => s.SetProperty(x => x.Status, StepStatus.Cancelled), cancellationToken).ConfigureAwait(false); - await steps.Where(s => s.Status == StepStatus.Pending) - .ExecuteUpdateAsync(s => s.SetProperty(x => x.Status, StepStatus.Skipped), cancellationToken).ConfigureAwait(false); - } - else - { - var running = await runs.Where(r => r.Status == RunStatus.Running || r.Status == RunStatus.Cancelling) - .ExecuteUpdateAsync(s => s - .SetProperty(r => r.Status, RunStatus.Cancelling) - .SetProperty(r => r.CancelRequestedAt, r => r.CancelRequestedAt ?? now) - .SetProperty(r => r.CancelRequestedBy, r => r.CancelRequestedBy ?? actor.Subject), cancellationToken).ConfigureAwait(false); - if (running == 0) - { - var status = await runs.Select(r => (RunStatus?)r.Status).SingleOrDefaultAsync(cancellationToken).ConfigureAwait(false) - ?? throw new RunNotFoundException(runId); - throw new PsrDomainException(PsrErrorCodes.RunNotCancellable, $"Run {runId} already finished ({status})."); - } - - // Runs executing in this process stop now; the queue package stops the others (control messages). - engine.RequestCancellation(runId); + var now = await clock.GetUtcNowAsync(cancellationToken).ConfigureAwait(false); + status = await lifecycle.CancelAsync(db, runId, actor, now, cancellationToken).ConfigureAwait(false) ?? throw new RunNotFoundException(runId); + var createdAt = runId.CreatedAt; + var target = await db.Runs.Where(r => r.Id == runId.Value && r.CreatedAt == createdAt).Select(r => r.PipelineKey ?? r.StageKey) + .SingleAsync(cancellationToken).ConfigureAwait(false); + AuditLog.Add(db, new AuditEntry(actor, AuditActions.RunCancelled, "Run", runId.Value, target), now); + await db.SaveChangesAsync(cancellationToken).ConfigureAwait(false); + await transaction.CommitAsync(cancellationToken).ConfigureAwait(false); } + } - var target = await runs.Select(r => r.PipelineKey ?? r.StageKey).SingleAsync(cancellationToken).ConfigureAwait(false); - AuditLog.Add(db, new AuditEntry(actor, AuditActions.RunCancelled, "Run", runId.Value, target), now); - await db.SaveChangesAsync(cancellationToken).ConfigureAwait(false); + if (status == RunStatus.Cancelling) + { + // Runs executing in this process stop now; the queue stops the others (CancelRun to every instance). + engine.RequestCancellation(runId); } } @@ -160,8 +139,7 @@ public async Task ReprocessAsync(RunId runId, Actor actor, Executio ArgumentNullException.ThrowIfNull(actor); if (mode == ExecutionMode.Async) { - throw new PsrConfigurationException( - "Asynchronous reprocessing needs a run queue (PythonScriptRunner.RabbitMQ). Use ExecutionMode.Sync to reprocess in this process."); + return await QueueReprocessAsync(runId, actor, cancellationToken).ConfigureAwait(false); } var request = await PrepareResumeAsync(runId, actor, cancellationToken).ConfigureAwait(false); @@ -191,6 +169,59 @@ await db.Runs.Where(r => r.Id == runId.Value && r.CreatedAt == createdAt && r.St } } + /// Queues the failed step for the workers (the run keeps its id, version and inputs). + private async Task QueueReprocessAsync(RunId runId, Actor actor, CancellationToken cancellationToken) + { + if (!messages.IsEnabled) + { + throw new PsrConfigurationException( + "Asynchronous reprocessing needs a run queue (PythonScriptRunner.RabbitMQ). Use ExecutionMode.Sync to reprocess in this process."); + } + + var db = await factory.CreateDbContextAsync(cancellationToken).ConfigureAwait(false); + await using (db.ConfigureAwait(false)) + { + var loaded = await RunLoader.LoadAsync(db, runId, cancellationToken).ConfigureAwait(false) ?? throw new RunNotFoundException(runId); + var now = await clock.GetUtcNowAsync(cancellationToken).ConfigureAwait(false); + EnsureReprocessable(loaded, runId, now); + var transaction = await db.Database.BeginTransactionAsync(cancellationToken).ConfigureAwait(false); + await using (transaction.ConfigureAwait(false)) + { + if (!await lifecycle.QueueReprocessAsync(db, runId, actor, now, cancellationToken).ConfigureAwait(false)) + { + throw new PsrDomainException(PsrErrorCodes.RunNotFailed, $"Run {runId} is already being reprocessed."); + } + + AuditLog.Add(db, new AuditEntry(actor, AuditActions.RunReprocessed, "Run", runId.Value, RunLoader.TargetKey(loaded.Run), + new JsonObject { ["mode"] = nameof(ExecutionMode.Async) }), now); + await db.SaveChangesAsync(cancellationToken).ConfigureAwait(false); + await transaction.CommitAsync(cancellationToken).ConfigureAwait(false); + } + } + + return await GetAsync(runId, CancellationToken.None).ConfigureAwait(false) ?? throw new RunNotFoundException(runId); + } + + /// Only failed runs whose files are still retained can be reprocessed. + private static void EnsureReprocessable(LoadedRun loaded, RunId runId, DateTimeOffset now) + { + var run = loaded.Run; + if (run.Status != RunStatus.Failed) + { + throw new PsrDomainException(PsrErrorCodes.RunNotFailed, $"Only failed runs can be reprocessed; run {runId} is {run.Status}."); + } + + var needed = ArtifactRefs.Referenced(StoredJson.Object(run.Inputs)) + .Concat(run.Steps.Where(s => s.Status == StepStatus.Succeeded).SelectMany(s => ArtifactRefs.Referenced(StoredJson.Object(s.Outputs)))) + .ToHashSet(); + var expired = needed.Where(id => !loaded.Artifacts.TryGetValue(id, out var a) || a.ContentDeletedAt is not null || a.ContentExpiresAt <= now).ToList(); + if (expired.Count > 0) + { + throw new PsrDomainException(PsrErrorCodes.ArtifactsExpired, + $"Run {runId} cannot be reprocessed: {expired.Count} file(s) it needs are beyond the retention."); + } + } + /// Checks the run can be reprocessed, claims it (Failed → Running) and loads the state to resume. private async Task PrepareResumeAsync(RunId runId, Actor actor, CancellationToken cancellationToken) { @@ -200,23 +231,9 @@ private async Task PrepareResumeAsync(RunId runId, Actor actor, C { var loaded = await RunLoader.LoadAsync(db, runId, cancellationToken).ConfigureAwait(false) ?? throw new RunNotFoundException(runId); var run = loaded.Run; - if (run.Status != RunStatus.Failed) - { - throw new PsrDomainException(PsrErrorCodes.RunNotFailed, $"Only failed runs can be reprocessed; run {runId} is {run.Status}."); - } - var now = time.GetUtcNow(); - var needed = ArtifactRefs.Referenced(StoredJson.Object(run.Inputs)) - .Concat(run.Steps.Where(s => s.Status == StepStatus.Succeeded).SelectMany(s => ArtifactRefs.Referenced(StoredJson.Object(s.Outputs)))) - .ToHashSet(); - var expired = needed.Where(id => !loaded.Artifacts.TryGetValue(id, out var a) || a.ContentDeletedAt is not null || a.ContentExpiresAt <= now).ToList(); - if (expired.Count > 0) - { - throw new PsrDomainException(PsrErrorCodes.ArtifactsExpired, - $"Run {runId} cannot be reprocessed: {expired.Count} file(s) it needs are beyond the retention."); - } - - var (stage, pipeline) = await LoadDefinitionAsync(db, run, cancellationToken).ConfigureAwait(false); + EnsureReprocessable(loaded, runId, now); + var (stage, pipeline) = await RunDefinitions.LoadAsync(db, run, cancellationToken).ConfigureAwait(false); var claimed = await db.Runs.Where(r => r.Id == run.Id && r.CreatedAt == createdAt && r.Status == RunStatus.Failed) .ExecuteUpdateAsync(s => s.SetProperty(r => r.Status, RunStatus.Running), cancellationToken).ConfigureAwait(false); if (claimed == 0) @@ -242,23 +259,4 @@ private async Task PrepareResumeAsync(RunId runId, Actor actor, C }; } } - - /// The exact version the run executed (any status: archived versions can still be reprocessed). - private static async Task<(StageDefinition? Stage, PipelineDefinition? Pipeline)> LoadDefinitionAsync( - PsrDbContext db, RunEntity run, CancellationToken cancellationToken) - { - if (run.Kind is RunKind.Pipeline or RunKind.PipelineTest) - { - var version = await db.PipelineVersions.AsNoTracking().Include(v => v.Pipeline) - .SingleOrDefaultAsync(v => v.Id == run.PipelineVersionId, cancellationToken).ConfigureAwait(false) - ?? throw new VersionNotFoundException(run.PipelineKey ?? string.Empty, run.PipelineVersion ?? 0); - var steps = await EfCatalog.LoadStepsAsync(db, version.Id, cancellationToken).ConfigureAwait(false); - return (null, CatalogMapping.ToPipeline(version.Pipeline, version, steps)); - } - - var stageVersion = await db.StageVersions.AsNoTracking().Include(v => v.Stage) - .SingleOrDefaultAsync(v => v.Id == run.StageVersionId, cancellationToken).ConfigureAwait(false) - ?? throw new VersionNotFoundException(run.StageKey ?? string.Empty, run.StageVersion ?? 0); - return (CatalogMapping.ToStage(stageVersion.Stage, stageVersion), null); - } } diff --git a/src/PythonScriptRunner.EntityFrameworkCore/Runs/EfRunRecorder.cs b/src/PythonScriptRunner.EntityFrameworkCore/Runs/EfRunRecorder.cs index b853553..bf635bd 100644 --- a/src/PythonScriptRunner.EntityFrameworkCore/Runs/EfRunRecorder.cs +++ b/src/PythonScriptRunner.EntityFrameworkCore/Runs/EfRunRecorder.cs @@ -1,5 +1,5 @@ -using System.Text.Json.Nodes; using Microsoft.EntityFrameworkCore; +using Microsoft.EntityFrameworkCore.Storage; using PythonScriptRunner.EntityFrameworkCore.Catalog; using PythonScriptRunner.EntityFrameworkCore.Entities; @@ -7,13 +7,15 @@ namespace PythonScriptRunner.EntityFrameworkCore.Runs; /// /// Persists every run executed by the engine in this process (synchronous runs, draft tests and reprocessing): run, -/// steps, attempts, log tails and files. Rows are addressed by run id + creation time (the partition) + step order. +/// steps, attempts, log tails and files. Rows are addressed by run id + creation time (the partition) + step order. With a +/// queue registered, every transition also writes its event to the outbox in the same transaction (D-105). /// internal sealed class EfRunRecorder( IDbContextFactory factory, ArtifactStore artifacts, IIdempotencyStore idempotency, RunLoader loader, + IRunMessageOutbox messages, PsrEntityFrameworkOptions options) : IRunRecorder { private static readonly RunStatus[] Terminal = [RunStatus.Succeeded, RunStatus.Failed, RunStatus.Cancelled]; @@ -38,12 +40,24 @@ internal sealed class EfRunRecorder( } } - var (row, retention) = await NewRunAsync(db, run, cancellationToken).ConfigureAwait(false); + var (row, retention) = await RunRows.NewAsync(db, run.RunId, run.Kind, run.TargetKey, run.TargetVersion, run.Steps, cancellationToken) + .ConfigureAwait(false); + row.Mode = ExecutionMode.Sync; + row.Trigger = run.Trigger; + row.TriggeredBy = run.TriggeredBy; + row.Status = RunStatus.Running; + row.Inputs = run.Inputs.ToJsonString(); + row.IdempotencyKey = run.IdempotencyKey; + row.CorrelationId = run.CorrelationId; + row.TraceParent = run.TraceParent; + row.StartedAt = run.StartedAt; db.Runs.Add(row); await db.SaveChangesAsync(cancellationToken).ConfigureAwait(false); - await StoreFilesAsync(db, run.RunId, null, ArtifactDirection.Input, run.Inputs, run.InputFiles, retention, cancellationToken).ConfigureAwait(false); + await RunRows.StoreFilesAsync(db, artifacts, run.RunId, null, ArtifactDirection.Input, run.Inputs, run.InputFiles, + retention ?? options.Retention.Artifacts, cancellationToken).ConfigureAwait(false); row.Inputs = ArtifactRefs.ToStored(run.Inputs, run.InputFiles).ToJsonString(); + messages.Publish(db, RunEventFactory.For(row, RunEventTypes.RunStarted, run.StartedAt), row.TraceParent); await db.SaveChangesAsync(cancellationToken).ConfigureAwait(false); await transaction.CommitAsync(cancellationToken).ConfigureAwait(false); return null; @@ -55,8 +69,7 @@ public async ValueTask RunResumingAsync(RunResuming run, CancellationToken cance { ArgumentNullException.ThrowIfNull(run); var createdAt = run.RunId.CreatedAt; - var db = await factory.CreateDbContextAsync(cancellationToken).ConfigureAwait(false); - await using (db.ConfigureAwait(false)) + await WriteAsync(async db => { await db.Runs.Where(r => r.Id == run.RunId.Value && r.CreatedAt == createdAt) .ExecuteUpdateAsync(s => s @@ -66,7 +79,8 @@ await db.Runs.Where(r => r.Id == run.RunId.Value && r.CreatedAt == createdAt) .SetProperty(r => r.FinishedAt, (DateTimeOffset?)null), cancellationToken).ConfigureAwait(false); await db.StepRuns.Where(s => s.RunId == run.RunId.Value && s.RunCreatedAt == createdAt && s.Status == StepStatus.Skipped) .ExecuteUpdateAsync(s => s.SetProperty(x => x.Status, StepStatus.Pending), cancellationToken).ConfigureAwait(false); - } + await PublishAsync(db, run.RunId, row => RunEventFactory.For(row, RunEventTypes.RunStarted, run.ResumedAt), cancellationToken).ConfigureAwait(false); + }, cancellationToken).ConfigureAwait(false); } public async ValueTask StepStartedAsync(StepStarted step, CancellationToken cancellationToken) @@ -91,16 +105,15 @@ public async ValueTask AttemptStartedAsync(AttemptStarted attempt, CancellationT { ArgumentNullException.ThrowIfNull(attempt); var createdAt = attempt.RunId.CreatedAt; - var db = await factory.CreateDbContextAsync(cancellationToken).ConfigureAwait(false); - await using (db.ConfigureAwait(false)) + await WriteAsync(async db => { - var stepRunId = await db.StepRuns - .Where(s => s.RunId == attempt.RunId.Value && s.RunCreatedAt == createdAt && s.Order == attempt.Order) - .Select(s => s.Id).SingleAsync(cancellationToken).ConfigureAwait(false); + var step = await db.StepRuns.AsNoTracking() + .SingleAsync(s => s.RunId == attempt.RunId.Value && s.RunCreatedAt == createdAt && s.Order == attempt.Order, cancellationToken) + .ConfigureAwait(false); db.StepAttempts.Add(new StepAttemptEntity { Id = attempt.AttemptId.Value, - StepRunId = stepRunId, + StepRunId = step.Id, RunId = attempt.RunId.Value, RunCreatedAt = createdAt, Attempt = attempt.Number, @@ -109,47 +122,23 @@ public async ValueTask AttemptStartedAsync(AttemptStarted attempt, CancellationT Status = AttemptStatus.Running, StartedAt = attempt.StartedAt, }); + await PublishAsync(db, attempt.RunId, row => RunEventFactory.ForStep(row, step, attempt.Number, RunEventTypes.StepStarted, attempt.StartedAt), + cancellationToken).ConfigureAwait(false); await db.SaveChangesAsync(cancellationToken).ConfigureAwait(false); - await db.StepRuns.Where(s => s.Id == stepRunId && s.RunCreatedAt == createdAt) + await db.StepRuns.Where(s => s.Id == step.Id && s.RunCreatedAt == createdAt) .ExecuteUpdateAsync(s => s.SetProperty(x => x.CurrentAttempt, attempt.Number), cancellationToken).ConfigureAwait(false); - } + }, cancellationToken).ConfigureAwait(false); } public async ValueTask AttemptFinishedAsync(AttemptFinished attempt, CancellationToken cancellationToken) { ArgumentNullException.ThrowIfNull(attempt); var createdAt = attempt.RunId.CreatedAt; - var result = attempt.Attempt; - var error = StoredJson.Error(result.Error); var db = await factory.CreateDbContextAsync(cancellationToken).ConfigureAwait(false); await using (db.ConfigureAwait(false)) { - await db.StepAttempts.Where(a => a.Id == result.Id.Value && a.RunCreatedAt == createdAt) - .ExecuteUpdateAsync(s => s - .SetProperty(a => a.Status, result.Status) - .SetProperty(a => a.ExitCode, result.ExitCode) - .SetProperty(a => a.ErrorKind, result.Error == null ? null : result.Error.Kind) - .SetProperty(a => a.Error, error) - .SetProperty(a => a.EnvironmentHash, result.EnvironmentHash) - .SetProperty(a => a.FinishedAt, result.StartedAt + result.Duration) - .SetProperty(a => a.DurationMs, (int)result.Duration.TotalMilliseconds) - .SetProperty(a => a.OverheadMs, (int)result.Overhead.TotalMilliseconds), cancellationToken).ConfigureAwait(false); - foreach (var (stream, tail) in (ReadOnlySpan<(LogStream, LogTail)>)[(LogStream.Stdout, result.Stdout), (LogStream.Stderr, result.Stderr)]) - { - if (tail.OriginalBytes > 0 || tail.Content.Length > 0) - { - db.AttemptLogs.Add(new AttemptLogEntity - { - AttemptId = result.Id.Value, - RunCreatedAt = createdAt, - Stream = stream, - Content = tail.Content, - Truncated = tail.Truncated, - OriginalBytes = tail.OriginalBytes, - }); - } - } - + await AttemptRows.FinishAsync(db, createdAt, attempt.Attempt, attempt.Attempt.StartedAt + attempt.Attempt.Duration, workerId: null, cancellationToken) + .ConfigureAwait(false); await db.SaveChangesAsync(cancellationToken).ConfigureAwait(false); } } @@ -169,9 +158,9 @@ public async ValueTask StepFinishedAsync(StepFinished step, CancellationToken ca .ConfigureAwait(false); if (result.Status == StepStatus.Succeeded && result.Outputs is { } outputs) { - var retention = await ArtifactRetentionAsync(db, step.RunId, cancellationToken).ConfigureAwait(false); - await StoreFilesAsync(db, step.RunId, new StepRunId(row.Id), ArtifactDirection.Output, outputs, result.Files, retention, cancellationToken) - .ConfigureAwait(false); + var retention = await RunRows.ArtifactRetentionAsync(db, step.RunId, cancellationToken).ConfigureAwait(false); + await RunRows.StoreFilesAsync(db, artifacts, step.RunId, new StepRunId(row.Id), ArtifactDirection.Output, outputs, result.Files, + retention ?? options.Retention.Artifacts, cancellationToken).ConfigureAwait(false); row.Outputs = ArtifactRefs.ToStored(outputs, result.Files).ToJsonString(); } @@ -179,6 +168,20 @@ public async ValueTask StepFinishedAsync(StepFinished step, CancellationToken ca row.Error = StoredJson.Error(result.Error); row.FinishedAt = result.Status == StepStatus.Skipped ? null : step.FinishedAt; row.CurrentAttempt = result.Attempts.Count == 0 ? row.CurrentAttempt : result.Attempts[^1].Number; + var type = result.Status switch + { + StepStatus.Succeeded => RunEventTypes.StepSucceeded, + StepStatus.Failed or StepStatus.TimedOut => RunEventTypes.StepFailed, + _ => null, + }; + if (type is not null) + { + await PublishAsync(db, step.RunId, run => RunEventFactory.ForStep(run, row, row.CurrentAttempt, type, step.FinishedAt) with + { + Error = RunEventError.From(result.Error), + }, cancellationToken).ConfigureAwait(false); + } + await db.SaveChangesAsync(cancellationToken).ConfigureAwait(false); await transaction.CommitAsync(cancellationToken).ConfigureAwait(false); } @@ -191,8 +194,7 @@ public async ValueTask RunFinishedAsync(RunFinished run, CancellationToken cance var createdAt = run.RunId.CreatedAt; var outputs = run.Outputs is null ? null : ArtifactRefs.ToStored(run.Outputs, run.OutputFiles).ToJsonString(); var error = StoredJson.Error(run.Error); - var db = await factory.CreateDbContextAsync(cancellationToken).ConfigureAwait(false); - await using (db.ConfigureAwait(false)) + await WriteAsync(async db => { await db.Runs.Where(r => r.Id == run.RunId.Value && r.CreatedAt == createdAt) .ExecuteUpdateAsync(s => s @@ -214,134 +216,68 @@ await db.StepAttempts.Where(a => a.RunId == run.RunId.Value && a.RunCreatedAt == .SetProperty(a => a.Status, run.Status == RunStatus.Cancelled ? AttemptStatus.Cancelled : AttemptStatus.Failed) .SetProperty(a => a.FinishedAt, run.FinishedAt), cancellationToken).ConfigureAwait(false); } - } - } - - /// Builds the run row and its pending steps; resolves catalog ids and the retention of the pipeline. - private async Task<(RunEntity Run, TimeSpan? ArtifactRetention)> NewRunAsync(PsrDbContext db, RunStarting run, CancellationToken cancellationToken) - { - var createdAt = run.RunId.CreatedAt; - var row = new RunEntity - { - Id = run.RunId.Value, - CreatedAt = createdAt, - Kind = run.Kind, - Mode = ExecutionMode.Sync, - Trigger = run.Trigger, - TriggeredBy = run.TriggeredBy, - Status = RunStatus.Running, - Inputs = run.Inputs.ToJsonString(), - IdempotencyKey = run.IdempotencyKey, - StartedAt = run.StartedAt, - }; - TimeSpan? artifactRetention = null; - var pipelineSteps = new Dictionary(); - if (run.Kind is RunKind.Pipeline or RunKind.PipelineTest) - { - var pipeline = await db.PipelineVersions.AsNoTracking() - .Where(v => v.Pipeline.Key == run.TargetKey && v.Version == run.TargetVersion) - .Select(v => new { VersionId = v.Id, v.PipelineId, v.Pipeline.ArtifactRetentionDays, v.Pipeline.RunRetentionDays }) - .SingleOrDefaultAsync(cancellationToken).ConfigureAwait(false); - row.PipelineKey = run.TargetKey; - row.PipelineVersion = run.TargetVersion; - if (pipeline is not null) + if (messages.IsEnabled && await RunHeaderAsync(db, run.RunId, cancellationToken).ConfigureAwait(false) is { } row) { - row.PipelineId = pipeline.PipelineId; - row.PipelineVersionId = pipeline.VersionId; - row.ExpiresAt = pipeline.RunRetentionDays is { } days ? createdAt.AddDays(days) : null; - artifactRetention = pipeline.ArtifactRetentionDays is { } artifactDays ? TimeSpan.FromDays(artifactDays) : null; - pipelineSteps = await db.PipelineSteps.AsNoTracking().Where(s => s.PipelineVersionId == pipeline.VersionId) - .ToDictionaryAsync(s => s.Order, s => s.Id, cancellationToken).ConfigureAwait(false); + RunEvent? runEvent = run.Status switch + { + RunStatus.Succeeded => RunEventFactory.Succeeded(row, + await db.Artifacts.AsNoTracking().Where(a => a.RunId == row.Id && a.RunCreatedAt == createdAt) + .ToDictionaryAsync(a => a.Id, cancellationToken).ConfigureAwait(false), + run.FinishedAt), + RunStatus.Failed => RunEventFactory.For(row, RunEventTypes.RunFailed, run.FinishedAt) with { Error = RunEventError.From(run.Error) }, + RunStatus.Cancelled => RunEventFactory.For(row, RunEventTypes.RunCancelled, run.FinishedAt), + _ => null, + }; + if (runEvent is not null) + { + messages.Publish(db, runEvent, row.TraceParent); + await db.SaveChangesAsync(cancellationToken).ConfigureAwait(false); + } } - } - else - { - row.StageKey = run.TargetKey; - row.StageVersion = run.TargetVersion; - } - - var stageKeys = run.Steps.Select(s => s.StageKey).Distinct().ToList(); - var stageVersions = await db.StageVersions.AsNoTracking() - .Where(v => stageKeys.Contains(v.Stage.Key)) - .Select(v => new { v.Id, v.Stage.Key, v.Version }) - .ToListAsync(cancellationToken).ConfigureAwait(false); - Guid? StageVersionId(string key, int version) => stageVersions.FirstOrDefault(v => v.Key == key && v.Version == version)?.Id; - - if (row.Kind is RunKind.Stage or RunKind.StageTest) - { - row.StageVersionId = StageVersionId(run.TargetKey, run.TargetVersion); - } - - foreach (var step in run.Steps) - { - row.Steps.Add(new StepRunEntity - { - Id = Guid.CreateVersion7(createdAt), - RunId = row.Id, - RunCreatedAt = createdAt, - PipelineStepId = pipelineSteps.TryGetValue(step.Order, out var pipelineStepId) ? pipelineStepId : null, - StepKey = step.StepKey, - Order = step.Order, - StageVersionId = StageVersionId(step.StageKey, step.StageVersion), - StageKey = step.StageKey, - StageVersion = step.StageVersion, - Status = StepStatus.Pending, - }); - } - - return (row, artifactRetention); - } - - private static async Task ArtifactRetentionAsync(PsrDbContext db, RunId runId, CancellationToken cancellationToken) - { - var createdAt = runId.CreatedAt; - var days = await db.Runs.AsNoTracking() - .Where(r => r.Id == runId.Value && r.CreatedAt == createdAt && r.PipelineId != null) - .Join(db.Pipelines, r => r.PipelineId, p => p.Id, (r, p) => p.ArtifactRetentionDays) - .SingleOrDefaultAsync(cancellationToken).ConfigureAwait(false); - return days is { } value ? TimeSpan.FromDays(value) : null; + }, cancellationToken).ConfigureAwait(false); } - /// Stores the files that are not stored yet (a file passed between steps keeps its artifact). - private async Task StoreFilesAsync( - PsrDbContext db, - RunId runId, - StepRunId? stepRunId, - ArtifactDirection direction, - JsonObject wire, - IReadOnlyDictionary> files, - TimeSpan? retention, - CancellationToken cancellationToken) + /// Runs the changes in one transaction when events are written (so a change and its event commit together). + private async Task WriteAsync(Func change, CancellationToken cancellationToken) { - foreach (var (field, list) in files) + var db = await factory.CreateDbContextAsync(cancellationToken).ConfigureAwait(false); + await using (db.ConfigureAwait(false)) { - var isList = wire[field] is JsonArray; - for (var i = 0; i < list.Count; i++) + IDbContextTransaction? transaction = messages.IsEnabled + ? await db.Database.BeginTransactionAsync(cancellationToken).ConfigureAwait(false) + : null; + try { - var file = list[i]; - if (file.ArtifactId is not null) + await change(db).ConfigureAwait(false); + await db.SaveChangesAsync(cancellationToken).ConfigureAwait(false); + if (transaction is not null) { - continue; + await transaction.CommitAsync(cancellationToken).ConfigureAwait(false); } - - var content = await file.OpenReadAsync(cancellationToken).ConfigureAwait(false); - await using (content.ConfigureAwait(false)) + } + finally + { + if (transaction is not null) { - var stored = await artifacts.SaveAsync(db, new NewArtifact - { - RunId = runId, - StepRunId = stepRunId, - Direction = direction, - FieldName = field, - ListIndex = isList ? i : null, - FileName = file.FileName, - ContentType = file.ContentType, - Retention = retention ?? options.Retention.Artifacts, - }, content, cancellationToken).ConfigureAwait(false); - file.AttachArtifact(new ArtifactId(stored.Id)); + await transaction.DisposeAsync().ConfigureAwait(false); } } } } + + /// Adds an event of the run to the outbox (saved by the caller); nothing without a queue. + private async Task PublishAsync(PsrDbContext db, RunId runId, Func build, CancellationToken cancellationToken) + { + if (messages.IsEnabled && await RunHeaderAsync(db, runId, cancellationToken).ConfigureAwait(false) is { } row) + { + messages.Publish(db, build(row), row.TraceParent); + } + } + + private static Task RunHeaderAsync(PsrDbContext db, RunId runId, CancellationToken cancellationToken) + { + var createdAt = runId.CreatedAt; + return db.Runs.AsNoTracking().SingleOrDefaultAsync(r => r.Id == runId.Value && r.CreatedAt == createdAt, cancellationToken); + } } diff --git a/src/PythonScriptRunner.EntityFrameworkCore/Runs/RunDefinitions.cs b/src/PythonScriptRunner.EntityFrameworkCore/Runs/RunDefinitions.cs new file mode 100644 index 0000000..01221a3 --- /dev/null +++ b/src/PythonScriptRunner.EntityFrameworkCore/Runs/RunDefinitions.cs @@ -0,0 +1,38 @@ +using Microsoft.EntityFrameworkCore; +using PythonScriptRunner.EntityFrameworkCore.Catalog; +using PythonScriptRunner.EntityFrameworkCore.Entities; + +namespace PythonScriptRunner.EntityFrameworkCore.Runs; + +/// Loads the exact versions a run executes (any status: archived versions still run and can be reprocessed). +internal static class RunDefinitions +{ + public static async Task<(StageDefinition? Stage, PipelineDefinition? Pipeline)> LoadAsync(PsrDbContext db, RunEntity run, CancellationToken cancellationToken) + { + if (run.Kind is RunKind.Pipeline or RunKind.PipelineTest) + { + var version = await db.PipelineVersions.AsNoTracking().Include(v => v.Pipeline) + .SingleOrDefaultAsync(v => v.Id == run.PipelineVersionId, cancellationToken).ConfigureAwait(false) + ?? throw new VersionNotFoundException(run.PipelineKey ?? string.Empty, run.PipelineVersion ?? 0); + var steps = await EfCatalog.LoadStepsAsync(db, version.Id, cancellationToken).ConfigureAwait(false); + return (null, CatalogMapping.ToPipeline(version.Pipeline, version, steps)); + } + + var stageVersion = await db.StageVersions.AsNoTracking().Include(v => v.Stage) + .SingleOrDefaultAsync(v => v.Id == run.StageVersionId, cancellationToken).ConfigureAwait(false) + ?? throw new VersionNotFoundException(run.StageKey ?? string.Empty, run.StageVersion ?? 0); + return (CatalogMapping.ToStage(stageVersion.Stage, stageVersion), null); + } + + /// The stage version executed by a step. + public static async Task LoadStageAsync(PsrDbContext db, StepRunEntity step, CancellationToken cancellationToken) + { + var versions = db.StageVersions.AsNoTracking().Include(v => v.Stage); + var version = await (step.StageVersionId is { } id + ? versions.Where(v => v.Id == id) + : versions.Where(v => v.Stage.Key == step.StageKey && v.Version == step.StageVersion)) + .SingleOrDefaultAsync(cancellationToken).ConfigureAwait(false) + ?? throw new VersionNotFoundException(step.StageKey, step.StageVersion); + return CatalogMapping.ToStage(version.Stage, version); + } +} diff --git a/src/PythonScriptRunner.EntityFrameworkCore/Runs/RunLoader.cs b/src/PythonScriptRunner.EntityFrameworkCore/Runs/RunLoader.cs index 88d1d10..02e315d 100644 --- a/src/PythonScriptRunner.EntityFrameworkCore/Runs/RunLoader.cs +++ b/src/PythonScriptRunner.EntityFrameworkCore/Runs/RunLoader.cs @@ -51,6 +51,9 @@ public static RunSnapshot ToSnapshot(LoadedRun loaded) Outputs = StoredJson.Object(run.Outputs), Error = StoredJson.Error(run.Error), IdempotencyKey = run.IdempotencyKey, + CorrelationId = run.CorrelationId, + ScheduleId = run.ScheduleId is { } scheduleId ? new ScheduleId(scheduleId) : null, + TriggerDetails = StoredJson.Object(run.TriggerDetails), CreatedAt = run.CreatedAt, QueuedAt = run.QueuedAt, StartedAt = run.StartedAt, diff --git a/src/PythonScriptRunner.EntityFrameworkCore/Runs/RunMessages.cs b/src/PythonScriptRunner.EntityFrameworkCore/Runs/RunMessages.cs new file mode 100644 index 0000000..73fc4e4 --- /dev/null +++ b/src/PythonScriptRunner.EntityFrameworkCore/Runs/RunMessages.cs @@ -0,0 +1,115 @@ +using System.Text; +using System.Text.Json.Nodes; +using PythonScriptRunner.EntityFrameworkCore.Catalog; +using PythonScriptRunner.EntityFrameworkCore.Entities; +using PythonScriptRunner.EntityFrameworkCore.Runs; + +namespace PythonScriptRunner.EntityFrameworkCore; + +/// +/// Writes the messages of run transitions (commands and events) to the outbox, in the transaction of the change. The default +/// writes nothing (no queue); the RabbitMQ package replaces it and its relay publishes the rows. +/// +public interface IRunMessageOutbox +{ + /// Whether messages are delivered (a queue is registered). Asynchronous runs require it. + bool IsEnabled { get; } + + /// Asks a worker to execute one attempt of a step. + /// Context of the change. + /// The command. + /// Not delivered before this instant (retry delays). + /// W3C trace context of the run. + void ExecuteStep(PsrDbContext db, ExecuteStepCommand command, DateTimeOffset availableAt, string? traceParent); + + /// Asks every instance to stop the processes of a run. + /// Context of the change. + /// The command. + /// Current time. + void CancelRun(PsrDbContext db, CancelRunCommand command, DateTimeOffset now); + + /// Publishes a run event. + /// Context of the change. + /// The event. + /// W3C trace context of the run. + void Publish(PsrDbContext db, RunEvent runEvent, string? traceParent); +} + +/// No queue: nothing is written. +internal sealed class NullRunMessageOutbox : IRunMessageOutbox +{ + public bool IsEnabled => false; + + public void ExecuteStep(PsrDbContext db, ExecuteStepCommand command, DateTimeOffset availableAt, string? traceParent) + { + } + + public void CancelRun(PsrDbContext db, CancelRunCommand command, DateTimeOffset now) + { + } + + public void Publish(PsrDbContext db, RunEvent runEvent, string? traceParent) + { + } +} + +/// Builds run events from persisted rows. +internal static class RunEventFactory +{ + /// Largest outputs carried by run.succeeded (QA-005-1). + public const int MaxOutputBytes = 256 * 1024; + + public static RunEvent For(RunEntity run, string type, DateTimeOffset at) => new() + { + EventId = Guid.CreateVersion7(at), + Type = type, + OccurredAt = at, + RunId = new RunId(run.Id), + Kind = run.Kind, + PipelineKey = run.PipelineKey, + PipelineVersion = run.PipelineVersion, + StageKey = run.StageKey, + StageVersion = run.StageVersion, + CorrelationId = run.CorrelationId, + Trigger = run.Trigger, + Status = run.Status, + TriggerDetails = StoredJson.Object(run.TriggerDetails), + }; + + public static RunEvent ForStep(RunEntity run, StepRunEntity step, int attempt, string type, DateTimeOffset at) => + For(run, type, at) with { Step = new RunEventStep(step.StepKey, step.Order, attempt) }; + + /// run.succeeded with the outputs: file references become metadata; above 256 KB they are omitted. + public static RunEvent Succeeded(RunEntity run, IReadOnlyDictionary artifacts, DateTimeOffset at) + { + var outputs = StoredJson.Object(run.Outputs) ?? []; + foreach (var (name, value) in outputs.ToList()) + { + outputs[name] = value switch + { + JsonArray array when array.All(IsReference) => new JsonArray([.. array.Select(item => Describe(item!, artifacts))]), + JsonObject reference when IsReference(reference) => Describe(reference, artifacts), + _ => value?.DeepClone(), + }; + } + + var truncated = Encoding.UTF8.GetByteCount(outputs.ToJsonString()) > MaxOutputBytes; + return For(run, RunEventTypes.RunSucceeded, at) with { Outputs = truncated ? null : outputs, OutputsTruncated = truncated }; + } + + private static bool IsReference(JsonNode? node) => node is JsonObject reference && reference[ArtifactRefs.Marker] is JsonValue; + + private static JsonNode Describe(JsonNode reference, IReadOnlyDictionary artifacts) + { + var id = Guid.Parse(reference[ArtifactRefs.Marker]!.GetValue()); + var metadata = new JsonObject { ["id"] = id.ToString() }; + if (artifacts.TryGetValue(id, out var artifact)) + { + metadata["name"] = artifact.FileName; + metadata["contentType"] = artifact.ContentType; + metadata["size"] = artifact.SizeBytes; + } + + return new JsonObject { [ArtifactRefs.Marker] = metadata }; + } +} diff --git a/src/PythonScriptRunner.EntityFrameworkCore/Runs/RunRows.cs b/src/PythonScriptRunner.EntityFrameworkCore/Runs/RunRows.cs new file mode 100644 index 0000000..dd91858 --- /dev/null +++ b/src/PythonScriptRunner.EntityFrameworkCore/Runs/RunRows.cs @@ -0,0 +1,138 @@ +using System.Text.Json.Nodes; +using Microsoft.EntityFrameworkCore; +using PythonScriptRunner.EntityFrameworkCore.Entities; + +namespace PythonScriptRunner.EntityFrameworkCore.Runs; + +/// Builds run rows and stores run files (shared by the synchronous recorder and the queued runs). +internal static class RunRows +{ + /// + /// Builds the run row with its pending steps; resolves catalog ids and the retention of the pipeline. The caller sets the + /// mode, status, trigger and inputs. + /// + public static async Task<(RunEntity Run, TimeSpan? ArtifactRetention)> NewAsync( + PsrDbContext db, RunId runId, RunKind kind, string targetKey, int targetVersion, IReadOnlyList steps, CancellationToken cancellationToken) + { + var createdAt = runId.CreatedAt; + var row = new RunEntity + { + Id = runId.Value, + CreatedAt = createdAt, + Kind = kind, + TriggeredBy = string.Empty, + Inputs = "{}", + }; + + TimeSpan? artifactRetention = null; + var pipelineSteps = new Dictionary(); + if (kind is RunKind.Pipeline or RunKind.PipelineTest) + { + var pipeline = await db.PipelineVersions.AsNoTracking() + .Where(v => v.Pipeline.Key == targetKey && v.Version == targetVersion) + .Select(v => new { VersionId = v.Id, v.PipelineId, v.Pipeline.ArtifactRetentionDays, v.Pipeline.RunRetentionDays }) + .SingleOrDefaultAsync(cancellationToken).ConfigureAwait(false); + row.PipelineKey = targetKey; + row.PipelineVersion = targetVersion; + if (pipeline is not null) + { + row.PipelineId = pipeline.PipelineId; + row.PipelineVersionId = pipeline.VersionId; + row.ExpiresAt = pipeline.RunRetentionDays is { } days ? createdAt.AddDays(days) : null; + artifactRetention = pipeline.ArtifactRetentionDays is { } artifactDays ? TimeSpan.FromDays(artifactDays) : null; + pipelineSteps = await db.PipelineSteps.AsNoTracking().Where(s => s.PipelineVersionId == pipeline.VersionId) + .ToDictionaryAsync(s => s.Order, s => s.Id, cancellationToken).ConfigureAwait(false); + } + } + else + { + row.StageKey = targetKey; + row.StageVersion = targetVersion; + } + + var stageKeys = steps.Select(s => s.StageKey).Distinct().ToList(); + var stageVersions = await db.StageVersions.AsNoTracking() + .Where(v => stageKeys.Contains(v.Stage.Key)) + .Select(v => new { v.Id, v.Stage.Key, v.Version }) + .ToListAsync(cancellationToken).ConfigureAwait(false); + Guid? StageVersionId(string key, int version) => stageVersions.FirstOrDefault(v => v.Key == key && v.Version == version)?.Id; + + if (kind is RunKind.Stage or RunKind.StageTest) + { + row.StageVersionId = StageVersionId(targetKey, targetVersion); + } + + foreach (var step in steps) + { + row.Steps.Add(new StepRunEntity + { + Id = Guid.CreateVersion7(createdAt), + RunId = row.Id, + RunCreatedAt = createdAt, + PipelineStepId = pipelineSteps.TryGetValue(step.Order, out var pipelineStepId) ? pipelineStepId : null, + StepKey = step.StepKey, + Order = step.Order, + StageVersionId = StageVersionId(step.StageKey, step.StageVersion), + StageKey = step.StageKey, + StageVersion = step.StageVersion, + Status = StepStatus.Pending, + }); + } + + return (row, artifactRetention); + } + + /// Retention of the files of a run (shortened by its pipeline), or null for the global one. + public static async Task ArtifactRetentionAsync(PsrDbContext db, RunId runId, CancellationToken cancellationToken) + { + var createdAt = runId.CreatedAt; + var days = await db.Runs.AsNoTracking() + .Where(r => r.Id == runId.Value && r.CreatedAt == createdAt && r.PipelineId != null) + .Join(db.Pipelines, r => r.PipelineId, p => p.Id, (r, p) => p.ArtifactRetentionDays) + .SingleOrDefaultAsync(cancellationToken).ConfigureAwait(false); + return days is { } value ? TimeSpan.FromDays(value) : null; + } + + /// Stores the files that are not stored yet (a file passed between steps keeps its artifact). + public static async Task StoreFilesAsync( + PsrDbContext db, + ArtifactStore artifacts, + RunId runId, + StepRunId? stepRunId, + ArtifactDirection direction, + JsonObject wire, + IReadOnlyDictionary> files, + TimeSpan retention, + CancellationToken cancellationToken) + { + foreach (var (field, list) in files) + { + var isList = wire[field] is JsonArray; + for (var i = 0; i < list.Count; i++) + { + var file = list[i]; + if (file.ArtifactId is not null) + { + continue; + } + + var content = await file.OpenReadAsync(cancellationToken).ConfigureAwait(false); + await using (content.ConfigureAwait(false)) + { + var stored = await artifacts.SaveAsync(db, new NewArtifact + { + RunId = runId, + StepRunId = stepRunId, + Direction = direction, + FieldName = field, + ListIndex = isList ? i : null, + FileName = file.FileName, + ContentType = file.ContentType, + Retention = retention, + }, content, cancellationToken).ConfigureAwait(false); + file.AttachArtifact(new ArtifactId(stored.Id)); + } + } + } + } +} diff --git a/src/PythonScriptRunner.EntityFrameworkCore/Scheduling/CronSchedule.cs b/src/PythonScriptRunner.EntityFrameworkCore/Scheduling/CronSchedule.cs new file mode 100644 index 0000000..45c15b0 --- /dev/null +++ b/src/PythonScriptRunner.EntityFrameworkCore/Scheduling/CronSchedule.cs @@ -0,0 +1,61 @@ +using Cronos; + +namespace PythonScriptRunner.EntityFrameworkCore.Scheduling; + +/// A parsed cron expression in an IANA time zone (Cronos; daylight saving time handled). +internal sealed class CronSchedule +{ + private readonly CronExpression _expression; + private readonly TimeZoneInfo _zone; + + private CronSchedule(CronExpression expression, TimeZoneInfo zone) + { + _expression = expression; + _zone = zone; + } + + /// Parses a 5-field (minutes) or 6-field (seconds) expression. SCHEDULE_INVALID when the expression or the zone is invalid. + /// Cron expression. + /// IANA time zone. + public static CronSchedule Parse(string cron, string timeZone) + { + if (string.IsNullOrWhiteSpace(cron) || cron.Length > 100) + { + throw Invalid("The cron expression is required (up to 100 chars)."); + } + + var fields = cron.Split(' ', StringSplitOptions.RemoveEmptyEntries).Length; + CronExpression expression; + try + { + expression = CronExpression.Parse(cron.Trim(), fields == 6 ? CronFormat.IncludeSeconds : CronFormat.Standard); + } + catch (CronFormatException ex) + { + throw Invalid($"Invalid cron expression '{cron}': {ex.Message}"); + } + + try + { + return new CronSchedule(expression, TimeZoneInfo.FindSystemTimeZoneById(timeZone)); + } + catch (Exception ex) when (ex is TimeZoneNotFoundException or InvalidTimeZoneException or ArgumentException) + { + throw Invalid($"Unknown IANA time zone '{timeZone}'."); + } + } + + /// The first occurrence strictly after (UTC), or null when there is none. + /// Reference instant. + public DateTimeOffset? Next(DateTimeOffset after) => + _expression.GetNextOccurrence(after.ToUniversalTime(), _zone) is { } next ? next.ToUniversalTime() : null; + + /// How many occurrences fall in [from, to], counting up to . + /// First instant (inclusive). + /// Last instant (inclusive). + /// Stops counting here (a schedule every second missed for days has millions). + public int CountOccurrences(DateTimeOffset from, DateTimeOffset to, int max = 100_000) => + to < from ? 0 : _expression.GetOccurrences(from.ToUniversalTime(), to.ToUniversalTime(), _zone, fromInclusive: true, toInclusive: true).Take(max).Count(); + + private static PsrDomainException Invalid(string message) => new(PsrErrorCodes.ScheduleInvalid, message); +} diff --git a/src/PythonScriptRunner.EntityFrameworkCore/Scheduling/ScheduleRules.cs b/src/PythonScriptRunner.EntityFrameworkCore/Scheduling/ScheduleRules.cs new file mode 100644 index 0000000..38104de --- /dev/null +++ b/src/PythonScriptRunner.EntityFrameworkCore/Scheduling/ScheduleRules.cs @@ -0,0 +1,38 @@ +using System.Text.Json.Nodes; + +namespace PythonScriptRunner.EntityFrameworkCore.Scheduling; + +/// Rules shared by the schedule editor and the scheduler (D-77, D-89). +internal static class ScheduleRules +{ + /// The version a schedule runs: the pinned one or the latest published. SCHEDULE_INVALID when it cannot run. + public static async Task ResolveAsync(ICatalog catalog, string pipelineKey, int? pinnedVersion, CancellationToken cancellationToken) + { + try + { + return await catalog.FindPipelineAsync(pipelineKey, pinnedVersion, cancellationToken).ConfigureAwait(false) + ?? throw new PsrDomainException(PsrErrorCodes.ScheduleInvalid, pinnedVersion is { } version + ? $"Version {version} of pipeline '{pipelineKey}' does not exist." + : $"Pipeline '{pipelineKey}' has no published version."); + } + catch (VersionNotRunnableException ex) + { + throw new PsrDomainException(PsrErrorCodes.ScheduleInvalid, ex.Message, ex); + } + } + + /// Schedules cannot provide files: a required file input, or a file given as fixed input, is SCHEDULE_FILE_INPUT. + public static void EnsureNoFileInputs(PipelineDefinition pipeline, JsonObject inputs) + { + var files = pipeline.Inputs.Where(f => f.Type.IsFile && (f.Required || inputs.ContainsKey(f.Name))).Select(f => f.Name).ToList(); + if (files.Count > 0) + { + throw new PsrDomainException(PsrErrorCodes.ScheduleFileInput, + $"Pipeline '{pipeline.Key}@{pipeline.Version}' has file inputs ({string.Join(", ", files)}); schedules cannot provide files."); + } + } + + /// Required inputs of that the fixed inputs do not provide (they have no default). + public static IEnumerable MissingInputs(PipelineDefinition pipeline, JsonObject inputs) => + pipeline.Inputs.Where(f => f.Required && f.Default is null && inputs[f.Name] is null).Select(f => f.Name); +} diff --git a/src/PythonScriptRunner.RabbitMQ/Consumers.cs b/src/PythonScriptRunner.RabbitMQ/Consumers.cs new file mode 100644 index 0000000..60b8e7d --- /dev/null +++ b/src/PythonScriptRunner.RabbitMQ/Consumers.cs @@ -0,0 +1,190 @@ +using System.Diagnostics; +using System.Text.Json; +using System.Text.Json.Nodes; +using Microsoft.Extensions.Hosting; +using Microsoft.Extensions.Logging; +using PythonScriptRunner.EntityFrameworkCore; + +namespace PythonScriptRunner.RabbitMQ; + +/// Runs one for the lifetime of the host. +internal abstract class ConsumerService(PsrBroker broker, ILogger logger) : IHostedService +{ + // Never disposed: a stop can still be running when the host disposes its services (stop called twice); a plain + // CancellationTokenSource holds no unmanaged resources. + private readonly CancellationTokenSource _stopping = new(); + private QueueConsumer? _consumer; + private Task? _running; + + protected PsrBroker Broker { get; } = broker; + + protected ILogger Logger { get; } = logger; + + /// How long deliveries in progress may finish when the host stops. + protected virtual TimeSpan GracePeriod => TimeSpan.FromSeconds(30); + + public Task StartAsync(CancellationToken cancellationToken) + { + // The task captures the consumer: a stop right after the start may already have cleared the field. + var consumer = new QueueConsumer(Broker, Settings(), HandleAsync, Logger); + _consumer = consumer; + _running = Task.Run(() => consumer.RunAsync(_stopping.Token), CancellationToken.None); + return Task.CompletedTask; + } + + public async Task StopAsync(CancellationToken cancellationToken) + { + // A host can be stopped more than once (e.g. stop, then dispose). + if (Interlocked.Exchange(ref _consumer, null) is not { } consumer || _running is not { } running) + { + return; + } + + await consumer.StopAsync(GracePeriod, cancellationToken).ConfigureAwait(false); + await _stopping.CancelAsync().ConfigureAwait(false); + try + { + await running.WaitAsync(cancellationToken).ConfigureAwait(false); + } + catch (OperationCanceledException) + { + // Host shutdown timeout. + } + + await consumer.DisposeAsync().ConfigureAwait(false); + } + + protected abstract QueueConsumerSettings Settings(); + + protected abstract Task HandleAsync(PsrDelivery delivery, CancellationToken cancellationToken); + + /// Reads the JSON body; null (logged) when it is not valid. + protected T? Read(PsrDelivery delivery, string expectedType) + where T : class + { + if (delivery.Type is { Length: > 0 } type && type != expectedType) + { + Log.InvalidMessage(Logger, delivery.MessageId, delivery.Queue, $"unknown message type '{type}'"); + return null; + } + + try + { + return JsonSerializer.Deserialize(delivery.Body.Span, PsrMessageJson.Options) + ?? throw new JsonException("The body is null."); + } + catch (JsonException ex) + { + Log.InvalidMessage(Logger, delivery.MessageId, delivery.Queue, ex.Message); + return null; + } + } +} + +/// +/// Consumes the public StartPipelineRun command (D-72): valid commands create a run (run.queued); unknown +/// pipelines, invalid inputs and file inputs produce run.rejected with the reasons. Both carry the caller's +/// correlationId. +/// +internal sealed class StartPipelineRunService( + PsrBroker broker, ICatalog catalog, IRunEngine engine, AsyncRunLifecycle lifecycle, ILogger logger) + : ConsumerService(broker, logger) +{ + private const int MaxTextLength = 200; + + protected override QueueConsumerSettings Settings() => new() { Queue = PsrTopology.StartPipelineQueue, Prefetch = 8 }; + + protected override async Task HandleAsync(PsrDelivery delivery, CancellationToken cancellationToken) + { + var command = Read(delivery, PsrMessageTypes.StartPipelineRun); + if (command is null || string.IsNullOrWhiteSpace(command.PipelineKey)) + { + return DeliveryOutcome.DeadLetter; + } + + using var activity = RabbitMqDiagnostics.StartFrom("psr.message.consume", ActivityKind.Consumer, delivery.TraceParent); + activity?.SetTag("messaging.destination.name", delivery.Queue); + activity?.SetTag("psr.pipeline.key", command.PipelineKey); + try + { + if (new[] { command.RequestedBy, command.CorrelationId, command.IdempotencyKey }.Any(text => text?.Length > MaxTextLength - 10)) + { + throw new PsrDomainException("TEXT_TOO_LONG", $"requestedBy, correlationId and idempotencyKey must have up to {MaxTextLength - 10} chars."); + } + + var pipeline = await catalog.FindPipelineAsync(command.PipelineKey, command.Version, cancellationToken).ConfigureAwait(false) + ?? throw (command.Version is { } version ? new VersionNotFoundException(command.PipelineKey, version) : new PipelineNotFoundException(command.PipelineKey)); + var inputs = command.Inputs ?? []; + var files = pipeline.Inputs.Where(f => f.Type.IsFile && (f.Required || inputs.ContainsKey(f.Name))).Select(f => f.Name).ToList(); + if (files.Count > 0) + { + throw new PsrDomainException(PsrErrorCodes.FileInputNotAllowed, + $"Pipeline '{pipeline.Key}' has file inputs ({string.Join(", ", files)}); the queue command cannot send files."); + } + + await lifecycle.CreateAsync(new NewAsyncRun + { + Kind = RunKind.Pipeline, + Pipeline = pipeline, + Inputs = engine.BindInputs(pipeline.Inputs, (JsonObject)inputs.DeepClone()), + Trigger = RunTrigger.QueueCommand, + TriggeredBy = command.RequestedBy is { Length: > 0 } requestedBy ? $"queue:{requestedBy}" : "system:queue", + IdempotencyKey = command.IdempotencyKey, + ImplicitIdempotencyKey = delivery.MessageId is { Length: > 0 } messageId ? $"message:{messageId}" : null, + CorrelationId = command.CorrelationId, + TraceParent = Activity.Current?.Id ?? delivery.TraceParent, + AnnounceDuplicate = true, + }, cancellationToken).ConfigureAwait(false); + } + catch (Exception ex) when (ex is PsrException and not PsrConfigurationException) + { + Log.CommandRejected(Logger, command.PipelineKey, ex.Message); + await lifecycle.PublishRejectedAsync(new RunEvent + { + EventId = Guid.CreateVersion7(), + Type = RunEventTypes.RunRejected, + OccurredAt = DateTimeOffset.UtcNow, + Kind = RunKind.Pipeline, + PipelineKey = command.PipelineKey, + CorrelationId = command.CorrelationId, + Trigger = RunTrigger.QueueCommand, + Error = AsyncRunLifecycle.Rejection(ex), + }, cancellationToken).ConfigureAwait(false); + } + + return DeliveryOutcome.Ack; + } +} + +/// Consumes the control queue of this instance: CancelRun stops the processes of the run if it executes here. +internal sealed class ControlService(PsrBroker broker, IRunEngine engine, ILogger logger) : ConsumerService(broker, logger) +{ + protected override TimeSpan GracePeriod => TimeSpan.FromSeconds(5); + + protected override QueueConsumerSettings Settings() + { + var queue = PsrTopology.ControlQueue(Broker.InstanceId); + return new() + { + Queue = queue, + Prefetch = 4, + Declare = async (channel, cancellationToken) => + { + await channel.QueueDeclareAsync(queue, durable: false, exclusive: true, autoDelete: true, cancellationToken: cancellationToken) + .ConfigureAwait(false); + await channel.QueueBindAsync(queue, PsrTopology.ControlExchange, string.Empty, cancellationToken: cancellationToken).ConfigureAwait(false); + }, + }; + } + + protected override Task HandleAsync(PsrDelivery delivery, CancellationToken cancellationToken) + { + if (Read(delivery, PsrMessageTypes.CancelRun) is not { } command) + { + return Task.FromResult(DeliveryOutcome.DeadLetter); + } + + engine.RequestCancellation(command.RunId); + return Task.FromResult(DeliveryOutcome.Ack); + } +} diff --git a/src/PythonScriptRunner.RabbitMQ/Diagnostics.cs b/src/PythonScriptRunner.RabbitMQ/Diagnostics.cs new file mode 100644 index 0000000..d9ee157 --- /dev/null +++ b/src/PythonScriptRunner.RabbitMQ/Diagnostics.cs @@ -0,0 +1,67 @@ +using System.Diagnostics; +using System.Diagnostics.Metrics; +using Microsoft.Extensions.Logging; + +namespace PythonScriptRunner.RabbitMQ; + +/// Tracing of the transport (source PythonScriptRunner, shared by every package). +internal static class RabbitMqDiagnostics +{ + public static readonly ActivitySource Source = new("PythonScriptRunner"); + + /// Starts an activity whose parent is the W3C traceparent carried by a message (the trace continues across processes). + public static Activity? StartFrom(string name, ActivityKind kind, string? traceParent) => + ActivityContext.TryParse(traceParent, null, out var parent) + ? Source.StartActivity(name, kind, parent) + : Source.StartActivity(name, kind); +} + +/// Metrics of the transport (meter PythonScriptRunner). +internal sealed class RabbitMqMetrics +{ + private readonly Histogram _publishDuration; + private int _pending; + + public RabbitMqMetrics(IMeterFactory meterFactory) + { + var meter = meterFactory.Create("PythonScriptRunner"); + _publishDuration = meter.CreateHistogram("psr.outbox.publish.duration", "ms", "Time to publish one outbox batch with confirms."); + meter.CreateObservableGauge("psr.outbox.pending", () => Volatile.Read(ref _pending), description: "Outbox messages not published yet."); + } + + public void BatchPublished(TimeSpan elapsed, int count) => + _publishDuration.Record(elapsed.TotalMilliseconds, new KeyValuePair("psr.messages", count)); + + public void Pending(int count) => Volatile.Write(ref _pending, count); +} + +/// Source-generated log messages. +internal static partial class Log +{ + [LoggerMessage(100, LogLevel.Information, "Connected to RabbitMQ at {Endpoint} (declare topology: {DeclareTopology})")] + public static partial void Connected(ILogger logger, string endpoint, bool declareTopology); + + [LoggerMessage(101, LogLevel.Warning, "RabbitMQ is unreachable ({Reason}); queued runs stay in the outbox until it comes back")] + public static partial void BrokerUnreachable(ILogger logger, string reason); + + [LoggerMessage(102, LogLevel.Critical, "The RabbitMQ topology is invalid; the consumer of {Queue} stopped")] + public static partial void TopologyInvalid(ILogger logger, string queue, Exception exception); + + [LoggerMessage(103, LogLevel.Warning, "Outbox message {MessageId} ({MessageType}) was not published: {Error}")] + public static partial void PublishFailed(ILogger logger, Guid messageId, string messageType, string error); + + [LoggerMessage(104, LogLevel.Error, "Message {MessageId} on {Queue} is invalid and goes to the dead-letter queue: {Reason}")] + public static partial void InvalidMessage(ILogger logger, string? messageId, string queue, string reason); + + [LoggerMessage(105, LogLevel.Warning, "Handling message {MessageId} on {Queue} failed; it will be delivered again")] + public static partial void HandlerFailed(ILogger logger, string? messageId, string queue, Exception exception); + + [LoggerMessage(106, LogLevel.Warning, "Consumer of {Queue} stopped ({Reason}); restarting")] + public static partial void ConsumerRestarting(ILogger logger, string queue, string reason); + + [LoggerMessage(107, LogLevel.Information, "StartPipelineRun for '{PipelineKey}' rejected: {Reason}")] + public static partial void CommandRejected(ILogger logger, string pipelineKey, string reason); + + [LoggerMessage(108, LogLevel.Error, "The outbox relay failed")] + public static partial void RelayFailed(ILogger logger, Exception exception); +} diff --git a/src/PythonScriptRunner.RabbitMQ/OutboxRelay.cs b/src/PythonScriptRunner.RabbitMQ/OutboxRelay.cs new file mode 100644 index 0000000..daab470 --- /dev/null +++ b/src/PythonScriptRunner.RabbitMQ/OutboxRelay.cs @@ -0,0 +1,187 @@ +using System.Diagnostics; +using System.Text; +using Microsoft.Extensions.Hosting; +using Microsoft.Extensions.Logging; +using PythonScriptRunner.EntityFrameworkCore; +using RabbitMQ.Client; +using RabbitMQ.Client.Exceptions; + +namespace PythonScriptRunner.RabbitMQ; + +/// +/// Publishes the outbox (spec 005): every it locks the due rows +/// (SKIP LOCKED), publishes them with publisher confirms and marks them published. A message lost between the confirm +/// and the commit is published again; consumers are idempotent. +/// +internal sealed class OutboxRelay(PsrBroker broker, PsrOutbox outbox, RabbitMqMetrics metrics, ILogger logger) : BackgroundService +{ + private static readonly TimeSpan PendingRefresh = TimeSpan.FromSeconds(5); + + private IChannel? _channel; + + protected override async Task ExecuteAsync(CancellationToken stoppingToken) + { + var options = broker.Options; + var lastPendingCount = Stopwatch.GetTimestamp() - (long)(PendingRefresh.TotalSeconds * Stopwatch.Frequency); + var unreachableLogged = false; + while (!stoppingToken.IsCancellationRequested) + { + var delay = options.RelayInterval; + try + { + if (Stopwatch.GetElapsedTime(lastPendingCount) >= PendingRefresh) + { + metrics.Pending(await outbox.CountPendingAsync(stoppingToken).ConfigureAwait(false)); + lastPendingCount = Stopwatch.GetTimestamp(); + } + + var count = await outbox.DispatchAsync(options.RelayBatchSize, PublishAsync, stoppingToken).ConfigureAwait(false); + unreachableLogged = false; + if (count == options.RelayBatchSize) + { + delay = TimeSpan.Zero; + } + } + catch (OperationCanceledException) when (stoppingToken.IsCancellationRequested) + { + break; + } + catch (Exception ex) when (ex is BrokerUnreachableException or OperationInterruptedException or AlreadyClosedException or IOException + or TimeoutException) + { + if (!unreachableLogged) + { + Log.BrokerUnreachable(logger, ex.Message); + unreachableLogged = true; + } + + await ResetChannelAsync().ConfigureAwait(false); + delay = TimeSpan.FromSeconds(2); + } + catch (PsrConfigurationException) + { + throw; + } + catch (Exception ex) + { + Log.RelayFailed(logger, ex); + delay = TimeSpan.FromSeconds(2); + } + + if (delay > TimeSpan.Zero) + { + try + { + await Task.Delay(delay, stoppingToken).ConfigureAwait(false); + } + catch (OperationCanceledException) + { + break; + } + } + } + + await ResetChannelAsync().ConfigureAwait(false); + } + + /// Publishes a batch concurrently on one confirm channel; a missing channel or connection fails the whole batch. + private async Task> PublishAsync(IReadOnlyList messages, CancellationToken cancellationToken) + { + var channel = await ChannelAsync(cancellationToken).ConfigureAwait(false); + var started = Stopwatch.GetTimestamp(); + var pending = new List<(OutboxEnvelope Message, Task Publish, Activity? Activity)>(messages.Count); + foreach (var message in messages) + { + message.Headers.TryGetValue("traceparent", out var traceParent); + var activity = RabbitMqDiagnostics.StartFrom("psr.outbox.publish", ActivityKind.Producer, traceParent); + activity?.SetTag("messaging.system", "rabbitmq"); + activity?.SetTag("messaging.destination.name", message.Exchange); + activity?.SetTag("messaging.rabbitmq.destination.routing_key", message.RoutingKey); + activity?.SetTag("messaging.message.id", message.Id.ToString()); + var headers = message.Headers.ToDictionary(h => h.Key, h => (object?)h.Value); + headers["psr-available-at"] = message.AvailableAt.ToUnixTimeMilliseconds(); + var properties = new BasicProperties + { + MessageId = message.Id.ToString(), + Type = message.MessageType, + ContentType = "application/json", + DeliveryMode = DeliveryModes.Persistent, + Timestamp = new AmqpTimestamp(message.AvailableAt.ToUnixTimeSeconds()), + Headers = headers, + }; + pending.Add((message, channel.BasicPublishAsync(message.Exchange, message.RoutingKey, mandatory: false, properties, + Encoding.UTF8.GetBytes(message.Payload), cancellationToken).AsTask(), activity)); + } + + try + { + await Task.WhenAll(pending.Select(p => p.Publish)).ConfigureAwait(false); + } + catch (Exception ex) when (ex is PublishException or OperationInterruptedException or AlreadyClosedException or IOException or TimeoutException + or OperationCanceledException) + { + // Inspected per message below. + } + + var results = new List(messages.Count); + Exception? broken = null; + foreach (var (message, publish, activity) in pending) + { + if (publish.IsCompletedSuccessfully) + { + results.Add(new OutboxDelivery(message.Id, null)); + } + else if (publish.Exception?.InnerException is PublishException rejected) + { + activity?.SetStatus(ActivityStatusCode.Error, rejected.Message); + Log.PublishFailed(logger, message.Id, message.MessageType, rejected.Message); + results.Add(new OutboxDelivery(message.Id, $"The broker did not confirm the message: {rejected.Message}")); + } + else + { + broken ??= publish.Exception?.InnerException ?? new OperationCanceledException(cancellationToken); + } + + activity?.Dispose(); + } + + if (broken is not null) + { + // The channel or the connection failed: nothing is recorded and the batch is published again later. + System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw(broken); + } + + metrics.BatchPublished(Stopwatch.GetElapsedTime(started), messages.Count); + return results; + } + + private async Task ChannelAsync(CancellationToken cancellationToken) + { + if (_channel is { IsOpen: true } open) + { + return open; + } + + await ResetChannelAsync().ConfigureAwait(false); + var connection = await broker.ConnectAsync(cancellationToken).ConfigureAwait(false); + _channel = await connection.CreateChannelAsync( + new CreateChannelOptions(publisherConfirmationsEnabled: true, publisherConfirmationTrackingEnabled: true), cancellationToken).ConfigureAwait(false); + return _channel; + } + + private async Task ResetChannelAsync() + { + if (_channel is { } channel) + { + _channel = null; + try + { + await channel.DisposeAsync().ConfigureAwait(false); + } + catch (Exception ex) when (ex is OperationInterruptedException or AlreadyClosedException or IOException or TimeoutException) + { + // Already closed. + } + } + } +} diff --git a/src/PythonScriptRunner.RabbitMQ/PsrBroker.cs b/src/PythonScriptRunner.RabbitMQ/PsrBroker.cs new file mode 100644 index 0000000..0b4b7ec --- /dev/null +++ b/src/PythonScriptRunner.RabbitMQ/PsrBroker.cs @@ -0,0 +1,100 @@ +using Microsoft.Extensions.Logging; +using RabbitMQ.Client; + +namespace PythonScriptRunner.RabbitMQ; + +/// +/// The RabbitMQ connection of this process. It connects on first use (the broker may be down when the host starts: queued +/// runs still go to the outbox) and makes sure the topology exists; afterwards RabbitMQ.Client recovers the connection, +/// channels and consumers automatically. +/// +public sealed class PsrBroker : IAsyncDisposable +{ + private readonly RabbitMqOptions _options; + private readonly ILogger _logger; + private readonly SemaphoreSlim _lock = new(1, 1); + private IConnection? _connection; + + internal PsrBroker(RabbitMqOptions options, ILogger logger) + { + _options = options; + _logger = logger; + } + + /// Identifier of this instance. + public string InstanceId => _options.InstanceId; + + /// Transport settings. + public RabbitMqOptions Options => _options; + + /// Whether the connection is open now (it is closed while the client recovers it). + public bool IsConnected => _connection?.IsOpen == true; + + /// The connection, connecting first when needed (the topology is declared or checked once). + /// Cancellation token. + /// The broker cannot be reached (try again later). + /// The topology is missing and must not be declared. + public async Task ConnectAsync(CancellationToken cancellationToken = default) + { + if (_connection is { } connection) + { + return connection; + } + + await _lock.WaitAsync(cancellationToken).ConfigureAwait(false); + try + { + if (_connection is not null) + { + return _connection; + } + + var factory = new ConnectionFactory + { + Uri = new Uri(_options.ConnectionString!), + AutomaticRecoveryEnabled = true, + TopologyRecoveryEnabled = true, + NetworkRecoveryInterval = TimeSpan.FromSeconds(2), + ClientProvidedName = $"psr:{_options.InstanceId}", + }; + var created = await factory.CreateConnectionAsync(cancellationToken).ConfigureAwait(false); + try + { + await PsrTopology.EnsureAsync(created, _options, cancellationToken).ConfigureAwait(false); + } + catch + { + await created.DisposeAsync().ConfigureAwait(false); + throw; + } + + Log.Connected(_logger, created.Endpoint.ToString(), _options.DeclareTopology); + _connection = created; + return created; + } + finally + { + _lock.Release(); + } + } + + /// + public async ValueTask DisposeAsync() + { + if (_connection is { } connection) + { + try + { + await connection.CloseAsync().ConfigureAwait(false); + } + catch (Exception ex) when (ex is IOException or global::RabbitMQ.Client.Exceptions.AlreadyClosedException or OperationCanceledException) + { + // Already gone. + } + + await connection.DisposeAsync().ConfigureAwait(false); + } + + _lock.Dispose(); + } +} diff --git a/src/PythonScriptRunner.RabbitMQ/PsrTopology.cs b/src/PythonScriptRunner.RabbitMQ/PsrTopology.cs new file mode 100644 index 0000000..ccc3c1b --- /dev/null +++ b/src/PythonScriptRunner.RabbitMQ/PsrTopology.cs @@ -0,0 +1,147 @@ +using RabbitMQ.Client; +using RabbitMQ.Client.Exceptions; + +namespace PythonScriptRunner.RabbitMQ; + +/// Exchanges, queues and routing keys of PythonScriptRunner (spec 005). +public static class PsrTopology +{ + /// Internal commands (direct). + public const string CommandsExchange = "psr.commands"; + + /// Public commands (direct); external systems only need write permission here. + public const string PublicExchange = "psr.public"; + + /// Run events (topic): <event>.<pipeline key> or <event>.stage:<stage key>. + public const string EventsExchange = "psr.events"; + + /// Control messages to every instance (fanout). + public const string ControlExchange = "psr.control"; + + /// Dead letters (fanout). + public const string DeadLetterExchange = "psr.dlx"; + + /// One message per step to execute. + public const string ExecuteStepQueue = "psr.step.execute"; + + /// Routing key of . + public const string ExecuteStepRoutingKey = "step.execute"; + + /// Public StartPipelineRun commands. + public const string StartPipelineQueue = "psr.pipeline.start"; + + /// Routing key of . + public const string StartPipelineRoutingKey = "pipeline.start"; + + /// Messages that could not be processed (manual inspection). + public const string DeadLetterQueue = "psr.dead-letter"; + + /// Control queue of an instance (exclusive, auto-delete). + /// Instance identifier. + public static string ControlQueue(string instanceId) => $"psr.control.{instanceId}"; + + /// Durable queue of a named event subscription. + /// Subscription name. + public static string SubscriberQueue(string name) => $"psr.subscriber.{name}"; + + /// Exclusive event queue of one instance (e.g. psr.dashboard.<instance>). + /// Queue prefix. + /// Instance identifier. + public static string InstanceQueue(string prefix, string instanceId) => $"psr.{prefix}.{instanceId}"; + + /// Arguments of a quorum command queue with a delivery limit and dead-lettering. + /// Deliveries before dead-lettering. + internal static Dictionary QuorumArguments(int deliveryLimit) => new() + { + ["x-queue-type"] = "quorum", + ["x-delivery-limit"] = deliveryLimit, + ["x-dead-letter-exchange"] = DeadLetterExchange, + }; + + /// Declares (idempotently) or checks the shared topology. + /// Open connection. + /// Transport settings. + /// Cancellation token. + /// Checking only and something is missing. + internal static async Task EnsureAsync(IConnection connection, RabbitMqOptions options, CancellationToken cancellationToken) + { + (string Name, string Type)[] exchanges = + [ + (CommandsExchange, ExchangeType.Direct), + (PublicExchange, ExchangeType.Direct), + (EventsExchange, ExchangeType.Topic), + (ControlExchange, ExchangeType.Fanout), + (DeadLetterExchange, ExchangeType.Fanout), + ]; + (string Name, IDictionary Arguments, string Exchange, string RoutingKey)[] queues = + [ + (ExecuteStepQueue, QuorumArguments(options.DeliveryLimit), CommandsExchange, ExecuteStepRoutingKey), + (StartPipelineQueue, QuorumArguments(options.DeliveryLimit), PublicExchange, StartPipelineRoutingKey), + (DeadLetterQueue, new Dictionary { ["x-queue-type"] = "quorum" }, DeadLetterExchange, string.Empty), + ]; + + if (options.DeclareTopology) + { + var channel = await connection.CreateChannelAsync(cancellationToken: cancellationToken).ConfigureAwait(false); + await using (channel.ConfigureAwait(false)) + { + foreach (var (name, type) in exchanges) + { + await channel.ExchangeDeclareAsync(name, type, durable: true, autoDelete: false, cancellationToken: cancellationToken).ConfigureAwait(false); + } + + foreach (var (name, arguments, exchange, routingKey) in queues) + { + await channel.QueueDeclareAsync(name, durable: true, exclusive: false, autoDelete: false, arguments, cancellationToken: cancellationToken) + .ConfigureAwait(false); + await channel.QueueBindAsync(name, exchange, routingKey, cancellationToken: cancellationToken).ConfigureAwait(false); + } + } + + return; + } + + var missing = new List(); + foreach (var (name, _) in exchanges) + { + if (!await ExistsAsync(connection, channel => channel.ExchangeDeclarePassiveAsync(name, cancellationToken), cancellationToken).ConfigureAwait(false)) + { + missing.Add($"exchange '{name}'"); + } + } + + foreach (var (name, _, _, _) in queues) + { + if (!await ExistsAsync(connection, async channel => await channel.QueueDeclarePassiveAsync(name, cancellationToken).ConfigureAwait(false), cancellationToken) + .ConfigureAwait(false)) + { + missing.Add($"queue '{name}'"); + } + } + + if (missing.Count > 0) + { + throw new PsrConfigurationException( + $"The RabbitMQ topology is incomplete and DeclareTopology is false: missing {string.Join(", ", missing)}. " + + "Create them (see the documentation of spec 005) or set DeclareTopology = true."); + } + } + + /// A passive declaration closes its channel when the entity is missing, so each check uses its own channel. + internal static async Task ExistsAsync(IConnection connection, Func check, CancellationToken cancellationToken) + { + var channel = await connection.CreateChannelAsync(cancellationToken: cancellationToken).ConfigureAwait(false); + await using (channel.ConfigureAwait(false)) + { + try + { + await check(channel).ConfigureAwait(false); + return true; + } + catch (OperationInterruptedException ex) when (ex.ShutdownReason?.ReplyCode == Constants.NotFound) + { + return false; + } + } + } +} diff --git a/src/PythonScriptRunner.RabbitMQ/PythonScriptRunner.RabbitMQ.csproj b/src/PythonScriptRunner.RabbitMQ/PythonScriptRunner.RabbitMQ.csproj new file mode 100644 index 0000000..d4f1e52 --- /dev/null +++ b/src/PythonScriptRunner.RabbitMQ/PythonScriptRunner.RabbitMQ.csproj @@ -0,0 +1,23 @@ + + + + PythonScriptRunner.RabbitMQ + RabbitMQ transport for PythonScriptRunner: outbox relay with publisher confirms, queued runs, public StartPipelineRun command, run events and subscriptions. + + + + + + + + + + + + + + + + + + diff --git a/src/PythonScriptRunner.RabbitMQ/QueueConsumer.cs b/src/PythonScriptRunner.RabbitMQ/QueueConsumer.cs new file mode 100644 index 0000000..e6261c4 --- /dev/null +++ b/src/PythonScriptRunner.RabbitMQ/QueueConsumer.cs @@ -0,0 +1,311 @@ +using System.Globalization; +using System.Text; +using Microsoft.Extensions.Logging; +using RabbitMQ.Client; +using RabbitMQ.Client.Events; +using RabbitMQ.Client.Exceptions; + +namespace PythonScriptRunner.RabbitMQ; + +/// What to do with a delivery after handling it. +public enum DeliveryOutcome +{ + /// Done (basic.ack). + Ack, + + /// Deliver it again later (basic.nack with requeue; quorum queues count it toward the delivery limit). + Requeue, + + /// Send it to psr.dead-letter (basic.nack without requeue). + DeadLetter, + + /// Leave it unacknowledged: the broker delivers it again when the channel closes (shutdown past the grace period). + Abandon, +} + +/// A message received from a queue. +/// Queue it came from. +/// AMQP message_id. +/// AMQP type (versioned message type). +/// Routing key it was published with. +/// Whether the broker delivered it before. +/// Headers as text. +/// Body (JSON). +public sealed record PsrDelivery( + string Queue, string? MessageId, string? Type, string RoutingKey, bool Redelivered, IReadOnlyDictionary Headers, ReadOnlyMemory Body) +{ + /// W3C trace context of the publisher. + public string? TraceParent => Headers.GetValueOrDefault("traceparent"); +} + +/// How to consume a queue. +public sealed record QueueConsumerSettings +{ + /// Queue name. + public required string Queue { get; init; } + + /// Unacknowledged deliveries at a time; also how many are handled concurrently. Default: 1. + public ushort Prefetch { get; init; } = 1; + + /// Declares the queue and its bindings before consuming (queues owned by this instance or subscription). + public Func? Declare { get; init; } +} + +/// +/// Consumes a queue with manual acknowledgements, up to deliveries in parallel. +/// The channel is recreated when it closes for good; connection losses are recovered by the client. A failing handler makes +/// the message be delivered again; stops receiving and waits for the deliveries in progress. +/// +public sealed class QueueConsumer : IAsyncDisposable +{ + private static readonly TimeSpan RequeueDelay = TimeSpan.FromSeconds(1); + + private readonly PsrBroker _broker; + private readonly QueueConsumerSettings _settings; + private readonly Func> _handler; + private readonly ILogger _logger; + private readonly CancellationTokenSource _handlers = new(); + private IChannel? _channel; + private string? _consumerTag; + private int _inFlight; + private volatile bool _stopping; + + /// Creates the consumer (nothing is consumed before ). + /// Connection. + /// Queue and concurrency. + /// Handles one delivery; its token is cancelled only when a shutdown exceeds the grace period. + /// Logger. + public QueueConsumer(PsrBroker broker, QueueConsumerSettings settings, Func> handler, ILogger logger) + { + ArgumentNullException.ThrowIfNull(broker); + ArgumentNullException.ThrowIfNull(settings); + ArgumentNullException.ThrowIfNull(handler); + ArgumentNullException.ThrowIfNull(logger); + _broker = broker; + _settings = settings; + _handler = handler; + _logger = logger; + } + + /// Deliveries being handled now. + public int InFlight => Volatile.Read(ref _inFlight); + + /// Whether the consumer is registered on an open channel. + public bool IsConsuming => _channel is { IsOpen: true } && _consumerTag is not null && !_stopping; + + /// Consumes until is cancelled or is called. + /// Stops the loop. + /// The topology is invalid. + public async Task RunAsync(CancellationToken stoppingToken) + { + var backoff = TimeSpan.FromSeconds(1); + while (!stoppingToken.IsCancellationRequested && !_stopping) + { + try + { + var connection = await _broker.ConnectAsync(stoppingToken).ConfigureAwait(false); + var channel = await connection.CreateChannelAsync( + new CreateChannelOptions(publisherConfirmationsEnabled: false, publisherConfirmationTrackingEnabled: false, + consumerDispatchConcurrency: _settings.Prefetch), + stoppingToken).ConfigureAwait(false); + _channel = channel; + if (_settings.Declare is { } declare) + { + await declare(channel, stoppingToken).ConfigureAwait(false); + } + + await channel.BasicQosAsync(0, _settings.Prefetch, false, stoppingToken).ConfigureAwait(false); + var consumer = new AsyncEventingBasicConsumer(channel); + consumer.ReceivedAsync += (_, args) => OnReceivedAsync(channel, args); + _consumerTag = await channel.BasicConsumeAsync(_settings.Queue, autoAck: false, consumer, stoppingToken).ConfigureAwait(false); + backoff = TimeSpan.FromSeconds(1); + + // The client recovers channels after connection losses; recreate only a channel closed while the connection is up. + while (!_stopping && !(channel.IsClosed && connection.IsOpen)) + { + await Task.Delay(TimeSpan.FromSeconds(1), stoppingToken).ConfigureAwait(false); + } + + if (!_stopping) + { + Log.ConsumerRestarting(_logger, _settings.Queue, channel.CloseReason?.ReplyText ?? "channel closed"); + await channel.DisposeAsync().ConfigureAwait(false); + } + } + catch (OperationCanceledException) when (stoppingToken.IsCancellationRequested) + { + return; + } + catch (PsrConfigurationException ex) + { + Log.TopologyInvalid(_logger, _settings.Queue, ex); + throw; + } + catch (Exception ex) when (ex is BrokerUnreachableException or OperationInterruptedException or AlreadyClosedException or IOException + or TimeoutException) + { + Log.ConsumerRestarting(_logger, _settings.Queue, ex.Message); + } + + if (_stopping) + { + return; + } + + try + { + await Task.Delay(backoff, stoppingToken).ConfigureAwait(false); + } + catch (OperationCanceledException) + { + return; + } + + backoff = TimeSpan.FromSeconds(Math.Min(30, backoff.TotalSeconds * 2)); + } + } + + /// + /// Stops receiving (basic.cancel), waits up to for the deliveries in progress, then + /// cancels their handlers (which abandon them) and closes the channel. + /// + /// How long deliveries in progress may run. + /// Ends the wait early (host shutdown timeout). + public async Task StopAsync(TimeSpan gracePeriod, CancellationToken cancellationToken = default) + { + _stopping = true; + if (_channel is { IsOpen: true } channel && _consumerTag is { } tag) + { + try + { + await channel.BasicCancelAsync(tag, noWait: false, CancellationToken.None).ConfigureAwait(false); + } + catch (Exception ex) when (ex is OperationInterruptedException or AlreadyClosedException or IOException or TimeoutException) + { + // The channel is going away anyway. + } + } + + await WaitIdleAsync(gracePeriod, cancellationToken).ConfigureAwait(false); + await _handlers.CancelAsync().ConfigureAwait(false); + await WaitIdleAsync(TimeSpan.FromSeconds(10), CancellationToken.None).ConfigureAwait(false); + if (_channel is { } open) + { + try + { + await open.CloseAsync(CancellationToken.None).ConfigureAwait(false); + } + catch (Exception ex) when (ex is OperationInterruptedException or AlreadyClosedException or IOException or TimeoutException) + { + // Already closed. + } + } + } + + /// + public async ValueTask DisposeAsync() + { + _stopping = true; + await _handlers.CancelAsync().ConfigureAwait(false); + if (_channel is { } channel) + { + await channel.DisposeAsync().ConfigureAwait(false); + } + + _handlers.Dispose(); + } + + private async Task WaitIdleAsync(TimeSpan timeout, CancellationToken cancellationToken) + { + using var deadline = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); + deadline.CancelAfter(timeout); + try + { + while (InFlight > 0) + { + await Task.Delay(50, deadline.Token).ConfigureAwait(false); + } + } + catch (OperationCanceledException) + { + // Grace period over. + } + } + + private async Task OnReceivedAsync(IChannel channel, BasicDeliverEventArgs args) + { + Interlocked.Increment(ref _inFlight); + try + { + var properties = args.BasicProperties; + var delivery = new PsrDelivery( + _settings.Queue, properties.MessageId, properties.Type, args.RoutingKey, args.Redelivered, Headers(properties.Headers), args.Body.ToArray()); + DeliveryOutcome outcome; + try + { + outcome = await _handler(delivery, _handlers.Token).ConfigureAwait(false); + } + catch (OperationCanceledException) when (_handlers.IsCancellationRequested) + { + outcome = DeliveryOutcome.Abandon; + } + catch (Exception ex) + { + Log.HandlerFailed(_logger, delivery.MessageId, _settings.Queue, ex); + outcome = DeliveryOutcome.Requeue; + } + + if (outcome == DeliveryOutcome.Requeue && !_stopping) + { + // Spaces out redeliveries of a failing message (e.g. while the database is down). + await Task.Delay(RequeueDelay, CancellationToken.None).ConfigureAwait(false); + } + + await SettleAsync(channel, args.DeliveryTag, outcome).ConfigureAwait(false); + } + finally + { + Interlocked.Decrement(ref _inFlight); + } + } + + private static async Task SettleAsync(IChannel channel, ulong deliveryTag, DeliveryOutcome outcome) + { + try + { + switch (outcome) + { + case DeliveryOutcome.Ack: + await channel.BasicAckAsync(deliveryTag, multiple: false, CancellationToken.None).ConfigureAwait(false); + break; + case DeliveryOutcome.Requeue: + await channel.BasicNackAsync(deliveryTag, multiple: false, requeue: true, CancellationToken.None).ConfigureAwait(false); + break; + case DeliveryOutcome.DeadLetter: + await channel.BasicNackAsync(deliveryTag, multiple: false, requeue: false, CancellationToken.None).ConfigureAwait(false); + break; + } + } + catch (Exception ex) when (ex is OperationInterruptedException or AlreadyClosedException or IOException or TimeoutException) + { + // The channel closed: the broker delivers the message again and the handlers are idempotent. + } + } + + private static Dictionary Headers(IDictionary? headers) + { + var result = new Dictionary(StringComparer.Ordinal); + foreach (var (key, value) in headers ?? new Dictionary()) + { + result[key] = value switch + { + null => string.Empty, + byte[] bytes => Encoding.UTF8.GetString(bytes), + string text => text, + _ => Convert.ToString(value, CultureInfo.InvariantCulture) ?? string.Empty, + }; + } + + return result; + } +} diff --git a/src/PythonScriptRunner.RabbitMQ/RabbitMqOptions.cs b/src/PythonScriptRunner.RabbitMQ/RabbitMqOptions.cs new file mode 100644 index 0000000..311cff6 --- /dev/null +++ b/src/PythonScriptRunner.RabbitMQ/RabbitMqOptions.cs @@ -0,0 +1,53 @@ +namespace PythonScriptRunner.RabbitMQ; + +/// Settings of the RabbitMQ transport. +public sealed class RabbitMqOptions +{ + /// AMQP URI (amqp://user:password@host:5672/vhost; amqps:// for TLS). + public string? ConnectionString { get; set; } + + /// + /// Declares exchanges and queues at startup (idempotent). With false, startup checks that they exist and fails + /// with a clear message when something is missing (for brokers managed by an operator). Default: true. + /// + public bool DeclareTopology { get; set; } = true; + + /// Publishes the outbox from this process (several instances can run it). Default: true. + public bool RunOutboxRelay { get; set; } = true; + + /// Consumes the public StartPipelineRun command (psr.pipeline.start; D-72). Default: true. + public bool EnablePublicCommands { get; set; } = true; + + /// Identifier of this instance (names its control and per-instance event queues). Default: machine name + process id. + public string InstanceId { get; set; } = $"{Environment.MachineName}-{Environment.ProcessId}".ToLowerInvariant(); + + /// How often the relay looks for due messages when the outbox is idle. Default: 250 ms (D-58). + public TimeSpan RelayInterval { get; set; } = TimeSpan.FromMilliseconds(250); + + /// Messages published per relay batch. Default: 100. + public int RelayBatchSize { get; set; } = 100; + + /// Delivery attempts of a command before it goes to psr.dead-letter. Default: 5. + public int DeliveryLimit { get; set; } = 5; + + /// How long startup waits for the broker before continuing in the background. Default: 10 seconds. + public TimeSpan StartupConnectTimeout { get; set; } = TimeSpan.FromSeconds(10); + + internal void Validate() + { + if (string.IsNullOrWhiteSpace(ConnectionString) || !Uri.TryCreate(ConnectionString, UriKind.Absolute, out var uri) || uri.Scheme is not ("amqp" or "amqps")) + { + throw new PsrConfigurationException("AddRabbitMq requires o.ConnectionString with an amqp:// or amqps:// URI."); + } + + if (string.IsNullOrWhiteSpace(InstanceId) || InstanceId.Length > 100) + { + throw new PsrConfigurationException("RabbitMQ InstanceId must have 1-100 chars."); + } + + if (RelayInterval <= TimeSpan.Zero || RelayBatchSize < 1 || DeliveryLimit < 1) + { + throw new PsrConfigurationException("RabbitMQ relay interval, relay batch size and delivery limit must be positive."); + } + } +} diff --git a/src/PythonScriptRunner.RabbitMQ/RabbitMqRunMessageOutbox.cs b/src/PythonScriptRunner.RabbitMQ/RabbitMqRunMessageOutbox.cs new file mode 100644 index 0000000..55c0c78 --- /dev/null +++ b/src/PythonScriptRunner.RabbitMQ/RabbitMqRunMessageOutbox.cs @@ -0,0 +1,48 @@ +using System.Text.Json; +using PythonScriptRunner.EntityFrameworkCore; + +namespace PythonScriptRunner.RabbitMQ; + +/// Maps run messages to outbox rows with their RabbitMQ destination (published by ). +internal sealed class RabbitMqRunMessageOutbox(IOutboxWriter outbox) : IRunMessageOutbox +{ + /// Version of the message contracts (header psr-schema-version). + public const string SchemaVersion = "1"; + + public bool IsEnabled => true; + + public void ExecuteStep(PsrDbContext db, ExecuteStepCommand command, DateTimeOffset availableAt, string? traceParent) => + outbox.Enqueue(db, new OutboxMessage(PsrMessageTypes.ExecuteStep, PsrTopology.CommandsExchange, PsrTopology.ExecuteStepRoutingKey, Json(command)) + { + AvailableAt = availableAt, + Headers = Headers(traceParent), + }); + + public void CancelRun(PsrDbContext db, CancelRunCommand command, DateTimeOffset now) => + outbox.Enqueue(db, new OutboxMessage(PsrMessageTypes.CancelRun, PsrTopology.ControlExchange, string.Empty, Json(command)) + { + AvailableAt = now, + Headers = Headers(null), + }); + + public void Publish(PsrDbContext db, RunEvent runEvent, string? traceParent) => + outbox.Enqueue(db, new OutboxMessage(PsrMessageTypes.RunEvent, PsrTopology.EventsExchange, $"{runEvent.Type}.{runEvent.RoutingTarget}", Json(runEvent)) + { + Id = runEvent.EventId, + AvailableAt = runEvent.OccurredAt, + Headers = Headers(traceParent), + }); + + private static System.Text.Json.Nodes.JsonNode Json(T message) => JsonSerializer.SerializeToNode(message, PsrMessageJson.Options)!; + + private static Dictionary Headers(string? traceParent) + { + var headers = new Dictionary { ["psr-schema-version"] = SchemaVersion }; + if (traceParent is not null) + { + headers["traceparent"] = traceParent; + } + + return headers; + } +} diff --git a/src/PythonScriptRunner.RabbitMQ/RabbitMqRunQueue.cs b/src/PythonScriptRunner.RabbitMQ/RabbitMqRunQueue.cs new file mode 100644 index 0000000..b7ed9ae --- /dev/null +++ b/src/PythonScriptRunner.RabbitMQ/RabbitMqRunQueue.cs @@ -0,0 +1,60 @@ +using System.Diagnostics; +using PythonScriptRunner.EntityFrameworkCore; + +namespace PythonScriptRunner.RabbitMQ; + +/// Queues runs: validates them like a synchronous run, then stores run + first step + message in one transaction. +internal sealed class RabbitMqRunQueue(ICatalog catalog, IRunEngine engine, AsyncRunLifecycle lifecycle) : IRunQueue +{ + public async Task EnqueuePipelineAsync(string pipelineKey, object? inputs, RunOptions? options = null, CancellationToken cancellationToken = default) + { + ArgumentException.ThrowIfNullOrWhiteSpace(pipelineKey); + var pipeline = await catalog.FindPipelineAsync(pipelineKey, options?.Version, cancellationToken).ConfigureAwait(false) + ?? throw (options?.Version is { } version && await catalog.FindPipelineAsync(pipelineKey, null, cancellationToken).ConfigureAwait(false) is not null + ? new VersionNotFoundException(pipelineKey, version) + : new PipelineNotFoundException(pipelineKey)); + var bound = engine.BindInputs(pipeline.Inputs, inputs); + return await EnqueueAsync(new NewAsyncRun + { + Kind = RunKind.Pipeline, + Pipeline = pipeline, + Inputs = bound, + TriggeredBy = string.Empty, + }, pipeline.Key, options, cancellationToken).ConfigureAwait(false); + } + + public async Task EnqueueStageAsync(string stageKey, object? inputs, RunOptions? options = null, CancellationToken cancellationToken = default) + { + ArgumentException.ThrowIfNullOrWhiteSpace(stageKey); + var stage = await catalog.FindStageAsync(stageKey, options?.Version, cancellationToken).ConfigureAwait(false) + ?? throw (options?.Version is { } version && await catalog.FindStageAsync(stageKey, null, cancellationToken).ConfigureAwait(false) is not null + ? new VersionNotFoundException(stageKey, version) + : new StageNotFoundException(stageKey)); + var bound = engine.BindInputs(stage.Inputs, inputs); + return await EnqueueAsync(new NewAsyncRun + { + Kind = RunKind.Stage, + Stage = stage, + Inputs = bound, + TriggeredBy = string.Empty, + }, stage.Key, options, cancellationToken).ConfigureAwait(false); + } + + private async Task EnqueueAsync(NewAsyncRun run, string targetKey, RunOptions? options, CancellationToken cancellationToken) + { + // The worker activities of every step become children of this one (the trace context travels with the run). + using var activity = RabbitMqDiagnostics.Source.StartActivity("psr.run.enqueue", ActivityKind.Producer); + activity?.SetTag("psr.run.kind", run.Kind.ToString()); + activity?.SetTag("psr.target.key", targetKey); + var handle = await lifecycle.CreateAsync(run with + { + Trigger = options?.Trigger ?? RunTrigger.Library, + TriggeredBy = options?.TriggeredBy ?? Actor.Library.Subject, + IdempotencyKey = options?.IdempotencyKey, + CorrelationId = options?.CorrelationId, + TraceParent = Activity.Current?.Id, + }, cancellationToken).ConfigureAwait(false); + activity?.SetTag("psr.run.id", handle.RunId.ToString()); + return handle; + } +} diff --git a/src/PythonScriptRunner.RabbitMQ/RabbitMqServiceCollectionExtensions.cs b/src/PythonScriptRunner.RabbitMQ/RabbitMqServiceCollectionExtensions.cs new file mode 100644 index 0000000..23c172d --- /dev/null +++ b/src/PythonScriptRunner.RabbitMQ/RabbitMqServiceCollectionExtensions.cs @@ -0,0 +1,87 @@ +using Microsoft.Extensions.DependencyInjection.Extensions; +using Microsoft.Extensions.Diagnostics.HealthChecks; +using Microsoft.Extensions.Hosting; +using Microsoft.Extensions.Logging; +using PythonScriptRunner; +using PythonScriptRunner.EntityFrameworkCore; +using PythonScriptRunner.RabbitMQ; + +namespace Microsoft.Extensions.DependencyInjection; + +/// Registers the RabbitMQ transport of PythonScriptRunner. +public static class RabbitMqServiceCollectionExtensions +{ + /// + /// Adds queued runs over RabbitMQ (requires AddEntityFrameworkCore): , the outbox relay, the + /// public StartPipelineRun command, run events (also for synchronous runs) and the cancellation of runs in any instance. + /// Executing queued steps needs a worker (AddWorker). + /// + /// Builder returned by AddPythonScriptRunner. + /// Connection and behaviour. + public static IPythonScriptRunnerBuilder AddRabbitMq(this IPythonScriptRunnerBuilder builder, Action configure) + { + ArgumentNullException.ThrowIfNull(builder); + ArgumentNullException.ThrowIfNull(configure); + var services = builder.Services; + if (!services.Any(d => d.ServiceType == typeof(AsyncRunLifecycle))) + { + throw new PsrConfigurationException("AddRabbitMq requires the persistence package: call AddEntityFrameworkCore(...) first (queued runs live in the database)."); + } + + var options = new RabbitMqOptions(); + configure(options); + options.Validate(); + + services.AddSingleton(options); + services.TryAddSingleton(provider => new PsrBroker(options, provider.GetRequiredService>())); + services.TryAddSingleton(); + services.Replace(ServiceDescriptor.Singleton()); + services.TryAddSingleton(); + services.TryAddEnumerable(ServiceDescriptor.Singleton()); + services.TryAddEnumerable(ServiceDescriptor.Singleton()); + services.TryAddEnumerable(ServiceDescriptor.Singleton()); + if (options.RunOutboxRelay) + { + services.TryAddEnumerable(ServiceDescriptor.Singleton()); + } + + if (options.EnablePublicCommands) + { + services.TryAddEnumerable(ServiceDescriptor.Singleton()); + } + + return builder; + } + + /// + /// Handles run events in (resolved in a scope per event), e.g. + /// .AddRunEventHandler<NotifyErp>(s => s.Subscription("erp").On("run.succeeded.calculo-progressao", "run.failed.*")). + /// + /// Handler type. + /// Builder. + /// Queue (durable or per instance) and routing key patterns. + public static IPythonScriptRunnerBuilder AddRunEventHandler(this IPythonScriptRunnerBuilder builder, Action configure) + where THandler : class, IRunEventHandler + { + ArgumentNullException.ThrowIfNull(builder); + ArgumentNullException.ThrowIfNull(configure); + var subscription = new RunEventSubscriptionBuilder(); + configure(subscription); + builder.Services.TryAddScoped(); + builder.Services.AddSingleton(subscription.Build(typeof(THandler))); + return builder; + } + + /// Health of the broker connection: when unreachable (runs wait in the outbox). + /// Health checks builder. + /// Check name. + /// Status reported while disconnected. + /// Tags (e.g. ready). + public static IHealthChecksBuilder AddPythonScriptRunnerBroker( + this IHealthChecksBuilder builder, string name = "psr-broker", HealthStatus failureStatus = HealthStatus.Degraded, IEnumerable? tags = null) + { + ArgumentNullException.ThrowIfNull(builder); + builder.Services.TryAddSingleton(); + return builder.Add(new HealthCheckRegistration(name, provider => provider.GetRequiredService(), failureStatus, tags)); + } +} diff --git a/src/PythonScriptRunner.RabbitMQ/RabbitMqStartup.cs b/src/PythonScriptRunner.RabbitMQ/RabbitMqStartup.cs new file mode 100644 index 0000000..7945196 --- /dev/null +++ b/src/PythonScriptRunner.RabbitMQ/RabbitMqStartup.cs @@ -0,0 +1,42 @@ +using Microsoft.Extensions.Diagnostics.HealthChecks; +using Microsoft.Extensions.Hosting; +using Microsoft.Extensions.Logging; +using RabbitMQ.Client.Exceptions; + +namespace PythonScriptRunner.RabbitMQ; + +/// +/// Connects when the host starts: an invalid topology fails the start with a clear message (AC-12); an unreachable broker +/// does not (queued runs wait in the outbox, AC-2) and the consumers keep retrying. +/// +internal sealed class RabbitMqStartup(PsrBroker broker, ILogger logger) : IHostedService +{ + public async Task StartAsync(CancellationToken cancellationToken) + { + using var timeout = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); + timeout.CancelAfter(broker.Options.StartupConnectTimeout); + try + { + await broker.ConnectAsync(timeout.Token).ConfigureAwait(false); + } + catch (Exception ex) when (ex is BrokerUnreachableException or IOException or TimeoutException or OperationInterruptedException + || (ex is OperationCanceledException && !cancellationToken.IsCancellationRequested)) + { + Log.BrokerUnreachable(logger, ex.Message); + } + } + + public Task StopAsync(CancellationToken cancellationToken) => Task.CompletedTask; +} + +/// Healthy while connected; otherwise the configured failure status (default : the outbox keeps the runs). +internal sealed class RabbitMqHealthCheck(PsrBroker broker) : IHealthCheck +{ + public Task CheckHealthAsync(HealthCheckContext context, CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(context); + return Task.FromResult(broker.IsConnected + ? HealthCheckResult.Healthy("Connected to RabbitMQ.") + : new HealthCheckResult(context.Registration.FailureStatus, "RabbitMQ is unreachable; queued runs wait in the outbox.")); + } +} diff --git a/src/PythonScriptRunner.RabbitMQ/RunEventSubscriptions.cs b/src/PythonScriptRunner.RabbitMQ/RunEventSubscriptions.cs new file mode 100644 index 0000000..16b3dd9 --- /dev/null +++ b/src/PythonScriptRunner.RabbitMQ/RunEventSubscriptions.cs @@ -0,0 +1,166 @@ +using System.Diagnostics; +using System.Text.RegularExpressions; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Hosting; +using Microsoft.Extensions.Logging; + +namespace PythonScriptRunner.RabbitMQ; + +/// Configures a run event subscription (AddRunEventHandler). +public sealed partial class RunEventSubscriptionBuilder +{ + private readonly List _patterns = []; + private string? _name; + private string? _instancePrefix; + private ushort _concurrency = 1; + + [GeneratedRegex("^[a-z0-9]+([.-][a-z0-9]+)*$")] + private static partial Regex NamePattern(); + + /// + /// Durable subscription: queue psr.subscriber.<name> shared by every instance (each event is handled once) and + /// kept while the application is down. + /// + /// Subscription name (lowercase letters, digits, '-' and '.'). + public RunEventSubscriptionBuilder Subscription(string name) + { + _name = Validate(name); + _instancePrefix = null; + return this; + } + + /// + /// Per-instance subscription: exclusive, auto-deleted queue psr.<prefix>.<instance>, so every instance + /// receives every event while it runs (e.g. the dashboard pushing updates to its browsers; D-110). + /// + /// Queue prefix (e.g. dashboard). + public RunEventSubscriptionBuilder PerInstance(string prefix) + { + _instancePrefix = Validate(prefix); + _name = null; + return this; + } + + /// + /// Routing key patterns of the events (topic syntax: * one word, # zero or more), e.g. + /// run.succeeded.calculo-progressao or run.failed.*. Default: every event (#). + /// + /// Patterns. + public RunEventSubscriptionBuilder On(params string[] patterns) + { + ArgumentNullException.ThrowIfNull(patterns); + _patterns.AddRange(patterns.Where(p => !string.IsNullOrWhiteSpace(p))); + return this; + } + + /// Events handled in parallel (default: 1, which keeps the order of delivery). + /// 1-256. + public RunEventSubscriptionBuilder WithConcurrency(int concurrency) + { + ArgumentOutOfRangeException.ThrowIfLessThan(concurrency, 1); + ArgumentOutOfRangeException.ThrowIfGreaterThan(concurrency, 256); + _concurrency = (ushort)concurrency; + return this; + } + + internal RunEventSubscription Build(Type handlerType) + { + if (_name is null && _instancePrefix is null) + { + throw new PsrConfigurationException($"The subscription of {handlerType.Name} needs Subscription(name) or PerInstance(prefix)."); + } + + return new RunEventSubscription(handlerType, _name, _instancePrefix, _patterns.Count > 0 ? [.. _patterns.Distinct()] : ["#"], _concurrency); + } + + private static string Validate(string name) + { + ArgumentException.ThrowIfNullOrWhiteSpace(name); + return name.Length <= 64 && NamePattern().IsMatch(name) + ? name + : throw new PsrConfigurationException($"'{name}' is not a valid subscription name (lowercase letters, digits, '-' and '.'; up to 64 chars)."); + } +} + +/// A registered subscription. +internal sealed record RunEventSubscription(Type HandlerType, string? Name, string? InstancePrefix, IReadOnlyList Patterns, ushort Concurrency) +{ + public string Queue(string instanceId) => Name is not null ? PsrTopology.SubscriberQueue(Name) : PsrTopology.InstanceQueue(InstancePrefix!, instanceId); +} + +/// Consumes the queue of one subscription and calls its (resolved in a scope per event). +internal sealed class RunEventSubscriptionService( + PsrBroker broker, RunEventSubscription subscription, IServiceScopeFactory scopes, ILogger logger) + : ConsumerService(broker, logger) +{ + protected override QueueConsumerSettings Settings() + { + var queue = subscription.Queue(Broker.InstanceId); + var durable = subscription.Name is not null; + return new() + { + Queue = queue, + Prefetch = subscription.Concurrency, + Declare = async (channel, cancellationToken) => + { + if (durable && !Broker.Options.DeclareTopology) + { + if (!await PsrTopology.ExistsAsync(await Broker.ConnectAsync(cancellationToken).ConfigureAwait(false), + async check => await check.QueueDeclarePassiveAsync(queue, cancellationToken).ConfigureAwait(false), cancellationToken) + .ConfigureAwait(false)) + { + throw new PsrConfigurationException($"The queue '{queue}' of subscription {subscription.HandlerType.Name} is missing and DeclareTopology is false."); + } + + return; + } + + await channel.QueueDeclareAsync(queue, durable, exclusive: !durable, autoDelete: !durable, + durable ? PsrTopology.QuorumArguments(Broker.Options.DeliveryLimit) : null, cancellationToken: cancellationToken).ConfigureAwait(false); + foreach (var pattern in subscription.Patterns) + { + await channel.QueueBindAsync(queue, PsrTopology.EventsExchange, pattern, cancellationToken: cancellationToken).ConfigureAwait(false); + } + }, + }; + } + + protected override async Task HandleAsync(PsrDelivery delivery, CancellationToken cancellationToken) + { + if (Read(delivery, PsrMessageTypes.RunEvent) is not { } runEvent) + { + return DeliveryOutcome.DeadLetter; + } + + using var activity = RabbitMqDiagnostics.StartFrom("psr.message.consume", ActivityKind.Consumer, delivery.TraceParent); + activity?.SetTag("messaging.destination.name", delivery.Queue); + activity?.SetTag("psr.event.type", runEvent.Type); + var scope = scopes.CreateAsyncScope(); + await using (scope.ConfigureAwait(false)) + { + var handler = (IRunEventHandler)scope.ServiceProvider.GetRequiredService(subscription.HandlerType); + await handler.HandleAsync(runEvent, cancellationToken).ConfigureAwait(false); + } + + return DeliveryOutcome.Ack; + } +} + +/// Starts one consumer per registered subscription. +internal sealed class RunEventSubscriptionHost(IEnumerable subscriptions, IServiceProvider services) : IHostedService +{ + private readonly List _consumers = []; + + public async Task StartAsync(CancellationToken cancellationToken) + { + foreach (var subscription in subscriptions) + { + var consumer = ActivatorUtilities.CreateInstance(services, subscription); + _consumers.Add(consumer); + await consumer.StartAsync(cancellationToken).ConfigureAwait(false); + } + } + + public async Task StopAsync(CancellationToken cancellationToken) => + await Task.WhenAll(_consumers.Select(c => c.StopAsync(cancellationToken))).ConfigureAwait(false); +} diff --git a/src/PythonScriptRunner.Worker/PythonScriptRunner.Worker.csproj b/src/PythonScriptRunner.Worker/PythonScriptRunner.Worker.csproj new file mode 100644 index 0000000..0a1c457 --- /dev/null +++ b/src/PythonScriptRunner.Worker/PythonScriptRunner.Worker.csproj @@ -0,0 +1,22 @@ + + + + PythonScriptRunner.Worker + Worker for PythonScriptRunner: executes queued steps with leases and retries, cron schedules, cancellation, graceful shutdown and health checks. + + + + + + + + + + + + + + + + + diff --git a/src/PythonScriptRunner.Worker/StepWorker.cs b/src/PythonScriptRunner.Worker/StepWorker.cs new file mode 100644 index 0000000..cf2b8a6 --- /dev/null +++ b/src/PythonScriptRunner.Worker/StepWorker.cs @@ -0,0 +1,254 @@ +using System.Data.Common; +using System.Diagnostics; +using System.Globalization; +using System.Text.Json; +using Microsoft.Extensions.Hosting; +using Microsoft.Extensions.Logging; +using PythonScriptRunner.EntityFrameworkCore; +using PythonScriptRunner.RabbitMQ; + +namespace PythonScriptRunner.Worker; + +/// +/// Executes queued steps, one ExecuteStep message per step (D-34; spec 006): claims the step with a lease, runs the +/// Python process while renewing the lease, records the outcome and the next message in one transaction, then acknowledges. +/// Duplicate or stale messages are acknowledged without effect. On shutdown it stops receiving and lets steps in progress +/// finish within ; past it, processes are stopped and their attempts are left +/// to the reaper. +/// +internal sealed class StepWorker( + PsrBroker broker, + AsyncRunLifecycle lifecycle, + IRunEngine engine, + WorkerOptions options, + WorkerMetrics metrics, + TimeProvider time, + ILogger logger) : IHostedService +{ + // Never disposed: a stop can still be running when the host disposes its services (stop called twice); a plain + // CancellationTokenSource holds no unmanaged resources. + private readonly CancellationTokenSource _stopping = new(); + + // The prefetch limits deliveries per channel; after a connection recovery the new channel receives more while the old + // deliveries still run, so the limit of processes is enforced here too (AC-8). + private readonly SemaphoreSlim _slots = new(options.MaxConcurrency, options.MaxConcurrency); + private QueueConsumer? _consumer; + private Task? _running; + + /// Identifier recorded on the attempts. + public string WorkerId { get; } = options.WorkerId ?? broker.InstanceId; + + /// The consumer (for tests and health). + public QueueConsumer? Consumer => _consumer; + + public Task StartAsync(CancellationToken cancellationToken) + { + // The task captures the consumer: a stop right after the start may already have cleared the field. + var consumer = new QueueConsumer(broker, new QueueConsumerSettings + { + Queue = PsrTopology.ExecuteStepQueue, + Prefetch = (ushort)options.MaxConcurrency, + }, HandleAsync, logger); + _consumer = consumer; + _running = Task.Run(() => consumer.RunAsync(_stopping.Token), CancellationToken.None); + Log.Started(logger, WorkerId, PsrTopology.ExecuteStepQueue, options.MaxConcurrency); + return Task.CompletedTask; + } + + public async Task StopAsync(CancellationToken cancellationToken) + { + // A host can be stopped more than once (e.g. stop, then dispose). + if (Interlocked.Exchange(ref _consumer, null) is not { } consumer || _running is not { } running) + { + return; + } + + await consumer.StopAsync(options.ShutdownGracePeriod, cancellationToken).ConfigureAwait(false); + await _stopping.CancelAsync().ConfigureAwait(false); + try + { + await running.WaitAsync(cancellationToken).ConfigureAwait(false); + } + catch (OperationCanceledException) + { + // Host shutdown timeout. + } + + await consumer.DisposeAsync().ConfigureAwait(false); + } + + /// The message. + /// Cancelled when the shutdown grace period is over. + internal async Task HandleAsync(PsrDelivery delivery, CancellationToken abandon) + { + ExecuteStepCommand command; + try + { + if (delivery.Type is { Length: > 0 } type && type != PsrMessageTypes.ExecuteStep) + { + throw new JsonException($"unknown message type '{type}'"); + } + + command = JsonSerializer.Deserialize(delivery.Body.Span, PsrMessageJson.Options) ?? throw new JsonException("The body is null."); + if (command.RunId.Value == Guid.Empty || command.StepRunId.Value == Guid.Empty || command.Attempt < 1) + { + throw new JsonException("runId, stepRunId and attempt (>= 1) are required."); + } + } + catch (JsonException ex) + { + Log.InvalidCommand(logger, delivery.MessageId, ex.Message); + return DeliveryOutcome.DeadLetter; + } + + using var activity = StartActivity(delivery, command); + await _slots.WaitAsync(abandon).ConfigureAwait(false); + try + { + var claimed = await RetryAsync("claim step", () => lifecycle.ClaimStepAsync(command, WorkerId, options.LeaseDuration, abandon), abandon) + .ConfigureAwait(false); + if (claimed is null) + { + // Duplicate, stale (already executed or recovered elsewhere) or cancelled: nothing to do. + return DeliveryOutcome.Ack; + } + + RecordQueueWait(delivery); + activity?.SetTag("psr.step.key", claimed.StepKey); + activity?.SetTag("psr.stage.key", claimed.Stage.Key); + metrics.StepStarted(); + try + { + return await ExecuteAsync(claimed, abandon).ConfigureAwait(false); + } + finally + { + metrics.StepEnded(); + } + } + finally + { + _slots.Release(); + } + } + + private async Task ExecuteAsync(ClaimedStep claimed, CancellationToken abandon) + { + using var execution = CancellationTokenSource.CreateLinkedTokenSource(abandon); + using var renewing = new CancellationTokenSource(); + var leaseLost = false; + var renewal = Task.Run(async () => + { + using var timer = new PeriodicTimer(options.LeaseRenewInterval, time); + try + { + while (await timer.WaitForNextTickAsync(renewing.Token).ConfigureAwait(false)) + { + try + { + if (!await lifecycle.RenewLeaseAsync(claimed, options.LeaseDuration, renewing.Token).ConfigureAwait(false)) + { + // The reaper took the attempt: stop the process and record nothing. + leaseLost = true; + await execution.CancelAsync().ConfigureAwait(false); + return; + } + } + catch (Exception ex) when (IsTransient(ex)) + { + Log.DatabaseRetry(logger, "renew lease", options.LeaseRenewInterval, ex); + } + } + } + catch (OperationCanceledException) + { + // Step finished. + } + }, CancellationToken.None); + + StepExecution result; + try + { + result = await engine.ExecuteStepAsync(claimed.ToExecutionRequest(), execution.Token).ConfigureAwait(false); + } + finally + { + await renewing.CancelAsync().ConfigureAwait(false); + await renewal.ConfigureAwait(false); + } + + await using (result.ConfigureAwait(false)) + { + if (leaseLost) + { + Log.LeaseLost(logger, claimed.StepKey, claimed.RunId); + return DeliveryOutcome.Ack; + } + + if (abandon.IsCancellationRequested) + { + Log.Abandoned(logger, claimed.StepKey, claimed.RunId); + return DeliveryOutcome.Abandon; + } + + // Retried while the lease lasts (database blips); afterwards the reaper decides (D-85). + var completion = await RetryAsync("complete step", () => lifecycle.CompleteStepAsync(claimed, result, CancellationToken.None), + CancellationToken.None, options.LeaseDuration).ConfigureAwait(false); + if (completion == StepCompletion.LeaseLost) + { + Log.LeaseLost(logger, claimed.StepKey, claimed.RunId); + } + else + { + Log.StepCompleted(logger, claimed.StepKey, claimed.Attempt, claimed.RunId, completion.ToString()); + } + + return DeliveryOutcome.Ack; + } + } + + /// Retries transient database failures with backoff (up to ; default 30 seconds). + private async Task RetryAsync(string operation, Func> action, CancellationToken cancellationToken, TimeSpan? limit = null) + { + var started = Stopwatch.GetTimestamp(); + var delay = TimeSpan.FromMilliseconds(200); + while (true) + { + try + { + return await action().ConfigureAwait(false); + } + catch (Exception ex) when (IsTransient(ex) && Stopwatch.GetElapsedTime(started) + delay < (limit ?? TimeSpan.FromSeconds(30))) + { + Log.DatabaseRetry(logger, operation, delay, ex); + await Task.Delay(delay, cancellationToken).ConfigureAwait(false); + delay = TimeSpan.FromMilliseconds(Math.Min(5000, delay.TotalMilliseconds * 2)); + } + } + } + + private static bool IsTransient(Exception exception) => + exception is DbException or TimeoutException || exception.InnerException is DbException or TimeoutException; + + private static Activity? StartActivity(PsrDelivery delivery, ExecuteStepCommand command) + { + var activity = ActivityContext.TryParse(delivery.TraceParent, null, out var parent) + ? WorkerDiagnostics.Source.StartActivity("psr.message.consume", ActivityKind.Consumer, parent) + : WorkerDiagnostics.Source.StartActivity("psr.message.consume", ActivityKind.Consumer); + activity?.SetTag("messaging.system", "rabbitmq"); + activity?.SetTag("messaging.destination.name", delivery.Queue); + activity?.SetTag("psr.run.id", command.RunId.ToString()); + activity?.SetTag("psr.attempt", command.Attempt); + return activity; + } + + /// psr.queue.wait: from the moment the message became due (header set by the relay) to the claim. + private void RecordQueueWait(PsrDelivery delivery) + { + if (delivery.Headers.TryGetValue("psr-available-at", out var text) + && long.TryParse(text, NumberStyles.Integer, CultureInfo.InvariantCulture, out var milliseconds)) + { + metrics.QueueWait(time.GetUtcNow() - DateTimeOffset.FromUnixTimeMilliseconds(milliseconds)); + } + } +} diff --git a/src/PythonScriptRunner.Worker/WorkerDiagnostics.cs b/src/PythonScriptRunner.Worker/WorkerDiagnostics.cs new file mode 100644 index 0000000..9d8eea4 --- /dev/null +++ b/src/PythonScriptRunner.Worker/WorkerDiagnostics.cs @@ -0,0 +1,80 @@ +using System.Diagnostics; +using System.Diagnostics.Metrics; +using Microsoft.Extensions.Logging; + +namespace PythonScriptRunner.Worker; + +/// Tracing of the worker (source PythonScriptRunner). +internal static class WorkerDiagnostics +{ + public static readonly ActivitySource Source = new("PythonScriptRunner"); +} + +/// Metrics of the worker (meter PythonScriptRunner; spec 006). +internal sealed class WorkerMetrics +{ + private readonly UpDownCounter _activeSteps; + private readonly Histogram _queueWait; + private readonly Counter _leaseRecovered; + private readonly Counter _scheduleFired; + private readonly Counter _scheduleRejected; + private int _pendingRuns; + + public WorkerMetrics(IMeterFactory meterFactory) + { + var meter = meterFactory.Create("PythonScriptRunner"); + _activeSteps = meter.CreateUpDownCounter("psr.worker.active_steps", description: "Steps executing in this worker."); + _queueWait = meter.CreateHistogram("psr.queue.wait", "ms", "Time from queuing a step to its start in a worker."); + _leaseRecovered = meter.CreateCounter("psr.lease.recovered", description: "Attempts restarted after their worker was lost."); + _scheduleFired = meter.CreateCounter("psr.schedule.fired", description: "Runs created by schedules."); + _scheduleRejected = meter.CreateCounter("psr.schedule.rejected", description: "Schedule firings rejected."); + meter.CreateObservableGauge("psr.runs.pending", () => Volatile.Read(ref _pendingRuns), description: "Runs waiting for a free slot of their pipeline."); + } + + public void StepStarted() => _activeSteps.Add(1); + + public void StepEnded() => _activeSteps.Add(-1); + + public void QueueWait(TimeSpan wait) => _queueWait.Record(Math.Max(0, wait.TotalMilliseconds)); + + public void LeasesRecovered(int count) => _leaseRecovered.Add(count); + + public void Schedules(int fired, int rejected) + { + _scheduleFired.Add(fired); + _scheduleRejected.Add(rejected); + } + + public void PendingRuns(int count) => Volatile.Write(ref _pendingRuns, count); +} + +/// Source-generated log messages. +internal static partial class Log +{ + [LoggerMessage(200, LogLevel.Information, "Worker {WorkerId} consuming {Queue} with {MaxConcurrency} concurrent steps")] + public static partial void Started(ILogger logger, string workerId, string queue, int maxConcurrency); + + [LoggerMessage(201, LogLevel.Information, "Step {StepKey} (attempt {Attempt}) of run {RunId}: {Completion}")] + public static partial void StepCompleted(ILogger logger, string stepKey, int attempt, RunId runId, string completion); + + [LoggerMessage(202, LogLevel.Warning, "The lease of step {StepKey} of run {RunId} was taken by another worker; the process was stopped")] + public static partial void LeaseLost(ILogger logger, string stepKey, RunId runId); + + [LoggerMessage(203, LogLevel.Warning, "Shutdown grace period over: step {StepKey} of run {RunId} was stopped and will be recovered by another worker")] + public static partial void Abandoned(ILogger logger, string stepKey, RunId runId); + + [LoggerMessage(204, LogLevel.Warning, "Database operation failed ({Operation}); retrying in {Delay}")] + public static partial void DatabaseRetry(ILogger logger, string operation, TimeSpan delay, Exception exception); + + [LoggerMessage(205, LogLevel.Information, "Lease reaper: {Recovered} recovered, {Failed} failed (worker lost), {Cancelled} cancelled")] + public static partial void Reaped(ILogger logger, int recovered, int failed, int cancelled); + + [LoggerMessage(206, LogLevel.Information, "Scheduler: {Fired} runs created, {Rejected} firings rejected")] + public static partial void SchedulesFired(ILogger logger, int fired, int rejected); + + [LoggerMessage(207, LogLevel.Error, "{Loop} failed")] + public static partial void LoopFailed(ILogger logger, string loop, Exception exception); + + [LoggerMessage(208, LogLevel.Error, "ExecuteStep message {MessageId} is invalid and goes to the dead-letter queue: {Reason}")] + public static partial void InvalidCommand(ILogger logger, string? messageId, string reason); +} diff --git a/src/PythonScriptRunner.Worker/WorkerLoops.cs b/src/PythonScriptRunner.Worker/WorkerLoops.cs new file mode 100644 index 0000000..0505720 --- /dev/null +++ b/src/PythonScriptRunner.Worker/WorkerLoops.cs @@ -0,0 +1,115 @@ +using Microsoft.Extensions.Hosting; +using Microsoft.Extensions.Logging; +using PythonScriptRunner.EntityFrameworkCore; + +namespace PythonScriptRunner.Worker; + +/// A loop that runs a cycle every interval (on the host's ) and survives failures. +internal abstract class PeriodicLoop(TimeProvider time, ILogger logger) : BackgroundService +{ + protected abstract TimeSpan Interval { get; } + + protected abstract string Name { get; } + + protected override async Task ExecuteAsync(CancellationToken stoppingToken) + { + using var timer = new PeriodicTimer(Interval, time); + do + { + try + { + await RunCycleAsync(stoppingToken).ConfigureAwait(false); + } + catch (OperationCanceledException) when (stoppingToken.IsCancellationRequested) + { + return; + } + catch (Exception ex) + { + Log.LoopFailed(logger, Name, ex); + } + } + while (await WaitAsync(timer, stoppingToken).ConfigureAwait(false)); + } + + protected abstract Task RunCycleAsync(CancellationToken cancellationToken); + + private static async Task WaitAsync(PeriodicTimer timer, CancellationToken stoppingToken) + { + try + { + return await timer.WaitForNextTickAsync(stoppingToken).ConfigureAwait(false); + } + catch (OperationCanceledException) + { + return false; + } + } +} + +/// Recovers or fails attempts whose worker stopped renewing the lease (D-85), and refreshes psr.runs.pending. +internal sealed class LeaseReaper(AsyncRunLifecycle lifecycle, WorkerOptions options, WorkerMetrics metrics, TimeProvider time, ILogger logger) + : PeriodicLoop(time, logger) +{ + private const int BatchSize = 50; + + protected override TimeSpan Interval => options.ReaperInterval; + + protected override string Name => "Lease reaper"; + + protected override async Task RunCycleAsync(CancellationToken cancellationToken) + { + LeaseReaping reaping; + do + { + reaping = await lifecycle.ReapExpiredLeasesAsync(BatchSize, options.MaxLeaseRecoveries, cancellationToken).ConfigureAwait(false); + if (reaping.Total > 0) + { + metrics.LeasesRecovered(reaping.Recovered); + Log.Reaped(logger, reaping.Recovered, reaping.Failed, reaping.Cancelled); + } + } + while (reaping.Total == BatchSize); + + metrics.PendingRuns(await lifecycle.CountPendingRunsAsync(cancellationToken).ConfigureAwait(false)); + } +} + +/// Fires due cron schedules (D-41, D-44); with several workers each occurrence fires once. +internal sealed class ScheduleFirer(AsyncRunLifecycle lifecycle, WorkerOptions options, WorkerMetrics metrics, TimeProvider time, ILogger logger) + : PeriodicLoop(time, logger) +{ + protected override TimeSpan Interval => options.Scheduler.PollInterval; + + protected override string Name => "Scheduler"; + + protected override async Task RunCycleAsync(CancellationToken cancellationToken) + { + ScheduleFiring firing; + do + { + firing = await lifecycle.FireDueSchedulesAsync(options.Scheduler.BatchSize, cancellationToken).ConfigureAwait(false); + if (firing.Total > 0) + { + metrics.Schedules(firing.Fired, firing.Rejected); + Log.SchedulesFired(logger, firing.Fired, firing.Rejected); + } + } + while (firing.Total == options.Scheduler.BatchSize); + } +} + +/// Removes work folders left by interrupted runs, at startup and then every . +internal sealed class WorkFolderJanitor(IRunEngine engine, WorkerOptions options, TimeProvider time, ILogger logger) + : PeriodicLoop(time, logger) +{ + protected override TimeSpan Interval => options.WorkFolderCleanupInterval; + + protected override string Name => "Work folder cleanup"; + + protected override Task RunCycleAsync(CancellationToken cancellationToken) + { + engine.RemoveOrphanWorkFolders(); + return Task.CompletedTask; + } +} diff --git a/src/PythonScriptRunner.Worker/WorkerOptions.cs b/src/PythonScriptRunner.Worker/WorkerOptions.cs new file mode 100644 index 0000000..514be3f --- /dev/null +++ b/src/PythonScriptRunner.Worker/WorkerOptions.cs @@ -0,0 +1,71 @@ +namespace PythonScriptRunner.Worker; + +/// Settings of the worker (spec 006). +public sealed class WorkerOptions +{ + /// Identifier recorded on the attempts this worker runs. Default: the RabbitMQ instance id (machine name + process id). + public string? WorkerId { get; set; } + + /// Steps executed at the same time by this worker (the queue prefetch; D-39). Default: number of CPUs. + public int MaxConcurrency { get; set; } = Environment.ProcessorCount; + + /// Lease of a running attempt; another worker recovers it when not renewed (QA-006-2). Default: 60 seconds. + public TimeSpan LeaseDuration { get; set; } = TimeSpan.FromSeconds(60); + + /// How often the lease of a running attempt is renewed. Default: 15 seconds. + public TimeSpan LeaseRenewInterval { get; set; } = TimeSpan.FromSeconds(15); + + /// How often expired leases are looked for. Default: 15 seconds. + public TimeSpan ReaperInterval { get; set; } = TimeSpan.FromSeconds(15); + + /// Recoveries of one step after lost workers before it fails with WorkerLost (QA-006-2). Default: 3. + public int MaxLeaseRecoveries { get; set; } = 3; + + /// + /// How long steps in progress may finish when the host stops (QA-006-3). Default: 5 minutes (the default timeout). The + /// platform must grant at least this time before killing the process. + /// + public TimeSpan ShutdownGracePeriod { get; set; } = TimeSpan.FromMinutes(5); + + /// How often leftover work folders (older than OrphanRetention) are removed. Default: 1 hour. + public TimeSpan WorkFolderCleanupInterval { get; set; } = TimeSpan.FromHours(1); + + /// Cron scheduler. + public WorkerSchedulerOptions Scheduler { get; } = new(); + + internal void Validate() + { + if (MaxConcurrency is < 1 or > ushort.MaxValue) + { + throw new PsrConfigurationException("Worker MaxConcurrency must be between 1 and 65535."); + } + + if (LeaseDuration <= TimeSpan.Zero || LeaseRenewInterval <= TimeSpan.Zero || LeaseRenewInterval >= LeaseDuration) + { + throw new PsrConfigurationException("Worker LeaseRenewInterval must be positive and shorter than LeaseDuration."); + } + + if (ReaperInterval <= TimeSpan.Zero || ShutdownGracePeriod < TimeSpan.Zero || WorkFolderCleanupInterval <= TimeSpan.Zero || MaxLeaseRecoveries < 0) + { + throw new PsrConfigurationException("Worker intervals must be positive and MaxLeaseRecoveries cannot be negative."); + } + + if (Scheduler.PollInterval <= TimeSpan.Zero || Scheduler.BatchSize < 1) + { + throw new PsrConfigurationException("Scheduler PollInterval and BatchSize must be positive."); + } + } +} + +/// Settings of the cron scheduler. +public sealed class WorkerSchedulerOptions +{ + /// Fires schedules from this worker (several workers never fire the same occurrence twice). Default: true. + public bool Enabled { get; set; } = true; + + /// How often due schedules are looked for (the firing precision). Default: 5 seconds. + public TimeSpan PollInterval { get; set; } = TimeSpan.FromSeconds(5); + + /// Schedules fired per transaction. Default: 50. + public int BatchSize { get; set; } = 50; +} diff --git a/src/PythonScriptRunner.Worker/WorkerServiceCollectionExtensions.cs b/src/PythonScriptRunner.Worker/WorkerServiceCollectionExtensions.cs new file mode 100644 index 0000000..15af8c0 --- /dev/null +++ b/src/PythonScriptRunner.Worker/WorkerServiceCollectionExtensions.cs @@ -0,0 +1,99 @@ +using Microsoft.Extensions.DependencyInjection.Extensions; +using Microsoft.Extensions.Diagnostics.HealthChecks; +using Microsoft.Extensions.Hosting; +using Microsoft.Extensions.Options; +using PythonScriptRunner; +using PythonScriptRunner.RabbitMQ; +using PythonScriptRunner.Worker; + +namespace Microsoft.Extensions.DependencyInjection; + +/// Registers the PythonScriptRunner worker. +public static class WorkerServiceCollectionExtensions +{ + /// + /// Runs queued steps in this process (requires AddEntityFrameworkCore and AddRabbitMq): one message per step, + /// leases with recovery, automatic retries, per-worker and per-pipeline concurrency, cancellation, the cron scheduler and the + /// cleanup of work folders. The host shutdown timeout is raised to cover . + /// + /// Builder returned by AddPythonScriptRunner. + /// Worker settings. + public static IPythonScriptRunnerBuilder AddWorker(this IPythonScriptRunnerBuilder builder, Action? configure = null) + { + ArgumentNullException.ThrowIfNull(builder); + var services = builder.Services; + if (!services.Any(d => d.ServiceType == typeof(PsrBroker))) + { + throw new PsrConfigurationException("AddWorker requires AddEntityFrameworkCore(...) and AddRabbitMq(...) first."); + } + + var options = new WorkerOptions(); + configure?.Invoke(options); + options.Validate(); + + services.AddSingleton(options); + services.TryAddSingleton(); + services.TryAddSingleton(); + services.Configure(host => + { + var needed = options.ShutdownGracePeriod + TimeSpan.FromSeconds(30); + if (host.ShutdownTimeout < needed) + { + host.ShutdownTimeout = needed; + } + }); + services.AddHostedService(); + services.AddHostedService(); + if (options.Scheduler.Enabled) + { + services.AddHostedService(); + } + + // Registered last so it stops first: steps in progress finish while the relay still publishes their messages. + services.AddHostedService(provider => provider.GetRequiredService()); + return builder; + } + + /// + /// Health of a PythonScriptRunner worker (spec 006, AC-16): the database (unhealthy when the schema is pending or no + /// partitions are created ahead), the broker (degraded while unreachable: runs wait in the outbox) and uv (unhealthy when + /// missing). All checks are tagged ready. + /// + /// Health checks builder. + public static IHealthChecksBuilder AddPythonScriptRunnerChecks(this IHealthChecksBuilder builder) + { + ArgumentNullException.ThrowIfNull(builder); + string[] ready = ["ready"]; + builder.AddPythonScriptRunnerDatabase(tags: ready); + builder.AddPythonScriptRunnerBroker(tags: ready); + builder.Services.TryAddSingleton(); + return builder.Add(new HealthCheckRegistration("psr-uv", provider => provider.GetRequiredService(), HealthStatus.Unhealthy, ready)); + } +} + +/// uv must be available to create the Python environments. +internal sealed class UvHealthCheck(IOptions options) : IHealthCheck +{ + public Task CheckHealthAsync(HealthCheckContext context, CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(context); + var uv = Find(options.Value.UvPath); + return Task.FromResult(uv is not null + ? HealthCheckResult.Healthy($"uv found at {uv}.") + : new HealthCheckResult(context.Registration.FailureStatus, $"uv was not found ('{options.Value.UvPath}'); Python environments cannot be created.")); + } + + private static string? Find(string nameOrPath) + { + string[] names = OperatingSystem.IsWindows() && !Path.HasExtension(nameOrPath) ? [nameOrPath, nameOrPath + ".exe"] : [nameOrPath]; + if (Path.IsPathRooted(nameOrPath) || nameOrPath.Contains('/', StringComparison.Ordinal) || nameOrPath.Contains('\\', StringComparison.Ordinal)) + { + return names.Select(Path.GetFullPath).FirstOrDefault(File.Exists); + } + + return (Environment.GetEnvironmentVariable("PATH") ?? string.Empty) + .Split(Path.PathSeparator, StringSplitOptions.RemoveEmptyEntries | StringSplitOptions.TrimEntries) + .SelectMany(directory => names.Select(name => Path.Combine(directory, name))) + .FirstOrDefault(File.Exists); + } +} diff --git a/tests/PythonScriptRunner.EntityFrameworkCore.Tests/SchemaTests.cs b/tests/PythonScriptRunner.EntityFrameworkCore.Tests/SchemaTests.cs index 62ffb7b..8201395 100644 --- a/tests/PythonScriptRunner.EntityFrameworkCore.Tests/SchemaTests.cs +++ b/tests/PythonScriptRunner.EntityFrameworkCore.Tests/SchemaTests.cs @@ -1,3 +1,4 @@ +using Microsoft.EntityFrameworkCore; using Microsoft.Extensions.Diagnostics.HealthChecks; using Microsoft.Extensions.Hosting; using Npgsql; @@ -46,7 +47,24 @@ public async Task Spec004_AC01_MigrateCreatesSchemaPartitionsAndIsRepeatable() await ExecuteAsync(scripted.ConnectionString, script); (await scripted.ScalarAsync(Signature)).ShouldBe(signature); - (await scripted.ScalarAsync("SELECT count(*) FROM psr.__ef_migrations_history")).ShouldBe(1); + await using var db = await migrated.DbAsync(); + (await scripted.ScalarAsync("SELECT count(*) FROM psr.__ef_migrations_history")).ShouldBe(db.Database.GetMigrations().Count()); + } + + [Fact] + public async Task Spec009_AC08_ConcurrentMigrations_AreAppliedOnce() + { + await using var first = await PsrTestHost.CreateAsync(postgres, python, migrate: false); + var services = new ServiceCollection(); + services.AddPythonScriptRunner().AddEntityFrameworkCore(o => o.UseNpgsql(first.ConnectionString)); + await using var second = services.BuildServiceProvider(); + + await Task.WhenAll( + first.Services.MigratePythonScriptRunnerAsync(TestContext.Current.CancellationToken), + second.MigratePythonScriptRunnerAsync(TestContext.Current.CancellationToken)); + + await using var db = await first.DbAsync(); + (await first.ScalarAsync("SELECT count(*) FROM psr.__ef_migrations_history")).ShouldBe(db.Database.GetMigrations().Count()); } [Fact] diff --git a/tests/PythonScriptRunner.RabbitMQ.Tests/ContractSchemaTests.cs b/tests/PythonScriptRunner.RabbitMQ.Tests/ContractSchemaTests.cs new file mode 100644 index 0000000..a12d5ff --- /dev/null +++ b/tests/PythonScriptRunner.RabbitMQ.Tests/ContractSchemaTests.cs @@ -0,0 +1,75 @@ +using System.Collections.Concurrent; +using System.Text.Json; +using System.Text.Json.Nodes; +using Json.Schema; + +namespace PythonScriptRunner.RabbitMQ.Tests; + +/// Spec 005: the public contracts match the JSON Schemas published in schemas/messages. +public sealed class ContractSchemaTests(PostgresFixture postgres, BrokerFixture broker, PythonFixture python) +{ + // Schemas are registered globally by their $id, so each one is loaded once per test run. + private static readonly ConcurrentDictionary Loaded = new(); + + private static CancellationToken Ct => TestContext.Current.CancellationToken; + + [Fact] + public async Task PublishedEventsAndTheCommandExample_MatchTheSchemas() + { + await using var env = await PsrEnvironment.CreateAsync(postgres, broker, python); + var host = await env.StartHostAsync(); + await PsrEnvironment.PublishStageAsync(host, "eco", Scripts.Echo, [Scripts.Text("texto")], [Scripts.Text("texto")]); + await PsrEnvironment.PublishPipelineAsync(host, "eco-contrato", Scripts.SingleStep("eco", 1, [Scripts.Text("texto")])); + await PsrEnvironment.PublishStageAsync(host, "falha", Scripts.AlwaysFails, [], []); + var succeeded = await host.Queue.EnqueuePipelineAsync("eco-contrato", new { texto = "ok" }, new RunOptions { CorrelationId = "contrato" }, Ct); + var failed = await host.Queue.EnqueueStageAsync("falha", null, cancellationToken: Ct); + await host.WaitForRunAsync(succeeded.RunId); + await host.WaitForRunAsync(failed.RunId); + await using (var raw = await RawBroker.ConnectAsync(env.Broker)) + { + await raw.PublishAsync(RabbitMQ.PsrTopology.PublicExchange, RabbitMQ.PsrTopology.StartPipelineRoutingKey, """{"pipelineKey":"nao-existe"}""", + PsrMessageTypes.StartPipelineRun); + } + + await host.Events.WaitForAsync(RunEventTypes.RunRejected); + var eventSchema = Schema("run-event.v1.json"); + var payloads = await env.ColumnAsync("SELECT payload::text FROM psr.outbox_messages WHERE message_type = 'psr.run-event.v1'"); + + payloads.Select(p => JsonNode.Parse(p)!["type"]!.GetValue()).Distinct().ShouldBe( + [ + RunEventTypes.RunQueued, RunEventTypes.RunStarted, RunEventTypes.StepStarted, RunEventTypes.StepSucceeded, RunEventTypes.RunSucceeded, + RunEventTypes.StepFailed, RunEventTypes.RunFailed, RunEventTypes.RunRejected, + ], + ignoreOrder: true); + foreach (var payload in payloads) + { + Validate(eventSchema, payload); + } + + var commandSchema = Schema("start-pipeline-run.v1.json"); + Validate(commandSchema, """ + { + "pipelineKey": "calculo-progressao", + "version": null, + "inputs": { "a1": 1, "razao": 2, "n": 4, "tipo": "geometrica", "coeficientes": [1, 0, 1] }, + "idempotencyKey": "pedido-123", + "correlationId": "erp-77881", + "requestedBy": "erp" + } + """); + Validate(commandSchema, + JsonSerializer.Serialize(new StartPipelineRunCommand { PipelineKey = "calculo-progressao", CorrelationId = "x" }, PsrMessageJson.Options)); + } + + private JsonSchema Schema(string name) => + Loaded.GetOrAdd(name, file => JsonSchema.FromFile(Path.Combine(python.RepositoryRoot, "schemas", "messages", file))); + + private static void Validate(JsonSchema schema, string json) + { + using var document = JsonDocument.Parse(json); + var result = schema.Evaluate(document.RootElement, new EvaluationOptions { OutputFormat = OutputFormat.List }); + var errors = ((IEnumerable)[result, .. result.Details ?? []]) + .SelectMany(d => (d.Errors?.Values ?? Enumerable.Empty()).Select(e => $"{d.InstanceLocation}: {e}")); + result.IsValid.ShouldBeTrue($"{json} → {string.Join("; ", errors)}"); + } +} diff --git a/tests/PythonScriptRunner.RabbitMQ.Tests/Infrastructure/RawBroker.cs b/tests/PythonScriptRunner.RabbitMQ.Tests/Infrastructure/RawBroker.cs new file mode 100644 index 0000000..33da6c8 --- /dev/null +++ b/tests/PythonScriptRunner.RabbitMQ.Tests/Infrastructure/RawBroker.cs @@ -0,0 +1,50 @@ +using System.Text; +using RabbitMQ.Client; + +namespace PythonScriptRunner.Testing; + +/// Talks to the broker directly, like a system that is not .NET (or a misbehaving one) would. +public sealed class RawBroker : IAsyncDisposable +{ + private readonly IConnection _connection; + private readonly IChannel _channel; + + private RawBroker(IConnection connection, IChannel channel) + { + _connection = connection; + _channel = channel; + } + + public static async Task ConnectAsync(string uri) + { + var connection = await new ConnectionFactory { Uri = new Uri(uri) }.CreateConnectionAsync(TestContext.Current.CancellationToken); + var channel = await connection.CreateChannelAsync( + new CreateChannelOptions(publisherConfirmationsEnabled: true, publisherConfirmationTrackingEnabled: true), TestContext.Current.CancellationToken); + return new RawBroker(connection, channel); + } + + public async Task PublishAsync(string exchange, string routingKey, string body, string? type, string? messageId = null) + { + var properties = new BasicProperties + { + MessageId = messageId ?? Guid.NewGuid().ToString(), + Type = type, + ContentType = "application/json", + DeliveryMode = DeliveryModes.Persistent, + }; + await _channel.BasicPublishAsync(exchange, routingKey, mandatory: true, properties, Encoding.UTF8.GetBytes(body), TestContext.Current.CancellationToken); + } + + /// Reads one message of a queue (without acknowledging others). + public async Task GetAsync(string queue) + { + var result = await _channel.BasicGetAsync(queue, autoAck: true, TestContext.Current.CancellationToken); + return result is null ? null : Encoding.UTF8.GetString(result.Body.Span); + } + + public async ValueTask DisposeAsync() + { + await _channel.DisposeAsync(); + await _connection.DisposeAsync(); + } +} diff --git a/tests/PythonScriptRunner.RabbitMQ.Tests/PublicContractTests.cs b/tests/PythonScriptRunner.RabbitMQ.Tests/PublicContractTests.cs new file mode 100644 index 0000000..f971bd1 --- /dev/null +++ b/tests/PythonScriptRunner.RabbitMQ.Tests/PublicContractTests.cs @@ -0,0 +1,199 @@ +using System.Collections.Concurrent; +using System.Text.Json.Nodes; + +namespace PythonScriptRunner.RabbitMQ.Tests; + +/// Spec 005: the public StartPipelineRun command and run event subscriptions (D-72, D-35, D-110). +public sealed class PublicContractTests(PostgresFixture postgres, BrokerFixture broker, PythonFixture python) +{ + private static CancellationToken Ct => TestContext.Current.CancellationToken; + + [Fact] + public async Task Spec005_AC05_StartPipelineRun_CreatesAQueueCommandRunWithTheCorrelation() + { + await using var env = await PsrEnvironment.CreateAsync(postgres, broker, python); + var host = await env.StartHostAsync(); + await PublishEchoPipelineAsync(host); + + await SendAsync(env, new JsonObject + { + ["pipelineKey"] = "eco-publica", + ["inputs"] = new JsonObject { ["texto"] = "de fora" }, + ["correlationId"] = "erp-77881", + ["requestedBy"] = "erp", + }); + + var queued = await host.Events.WaitForAsync(RunEventTypes.RunQueued, e => e.CorrelationId == "erp-77881"); + var run = await host.WaitForRunAsync(queued.RunId!.Value); + run.Status.ShouldBe(RunStatus.Succeeded); + run.Trigger.ShouldBe(RunTrigger.QueueCommand); + run.TriggeredBy.ShouldBe("queue:erp"); + run.CorrelationId.ShouldBe("erp-77881"); + var succeeded = await host.Events.WaitForAsync(RunEventTypes.RunSucceeded, e => e.RunId == run.Id); + succeeded.CorrelationId.ShouldBe("erp-77881"); + succeeded.Outputs!["texto"]!.GetValue().ShouldBe("de fora"); + } + + [Fact] + public async Task Spec005_AC06_InvalidStartPipelineRun_IsRejectedWithReasonsAndCreatesNoRun() + { + await using var env = await PsrEnvironment.CreateAsync(postgres, broker, python); + var host = await env.StartHostAsync(); + await PublishEchoPipelineAsync(host); + await PsrEnvironment.PublishStageAsync(host, "le-arquivo", "from psrunner import stage\n\n\n@stage\ndef run(inputs, ctx):\n return {}\n", + [Scripts.File("arquivo")], []); + await PsrEnvironment.PublishPipelineAsync(host, "com-arquivo", new PipelineDraftData + { + Inputs = [Scripts.File("arquivo")], + Steps = [new PipelineStepDefinition("le", "le-arquivo", 1)], + }); + + await SendAsync(env, new JsonObject { ["pipelineKey"] = "nao-existe", ["correlationId"] = "c-1" }); + await SendAsync(env, new JsonObject { ["pipelineKey"] = "eco-publica", ["inputs"] = new JsonObject { ["texto"] = 42 }, ["correlationId"] = "c-2" }); + await SendAsync(env, new JsonObject { ["pipelineKey"] = "com-arquivo", ["correlationId"] = "c-3" }); + + var notFound = await host.Events.WaitForAsync(RunEventTypes.RunRejected, e => e.CorrelationId == "c-1"); + var invalid = await host.Events.WaitForAsync(RunEventTypes.RunRejected, e => e.CorrelationId == "c-2"); + var withFile = await host.Events.WaitForAsync(RunEventTypes.RunRejected, e => e.CorrelationId == "c-3"); + notFound.Error!.Code.ShouldBe("PIPELINE_NOT_FOUND"); + notFound.RunId.ShouldBeNull(); + invalid.Error!.Code.ShouldBe("INPUT_INVALID"); + invalid.Error.Reasons!.ShouldContain(r => r.StartsWith("texto", StringComparison.Ordinal)); + withFile.Error!.Code.ShouldBe(PsrErrorCodes.FileInputNotAllowed); + withFile.Error.Reasons!.Single().ShouldContain("arquivo"); + (await env.ScalarAsync("SELECT count(*) FROM psr.runs")).ShouldBe(0); + } + + [Fact] + public async Task Spec005_AC07_SameIdempotencyKey_CreatesOneRun() + { + await using var env = await PsrEnvironment.CreateAsync(postgres, broker, python); + var host = await env.StartHostAsync(); + await PublishEchoPipelineAsync(host); + var command = new JsonObject + { + ["pipelineKey"] = "eco-publica", + ["inputs"] = new JsonObject { ["texto"] = "uma vez" }, + ["idempotencyKey"] = "pedido-123", + ["correlationId"] = "dup", + }; + + await SendAsync(env, command); + await SendAsync(env, command); + + await Eventually.WaitAsync(() => Task.FromResult(host.Events.All.Count(e => e.Type == RunEventTypes.RunQueued && e.CorrelationId == "dup") == 2), + TimeSpan.FromSeconds(30)); + host.Events.All.Where(e => e.Type == RunEventTypes.RunQueued && e.CorrelationId == "dup").Select(e => e.RunId).Distinct().Count().ShouldBe(1); + (await env.ScalarAsync("SELECT count(*) FROM psr.runs")).ShouldBe(1); + } + + [Fact] + public async Task StartPipelineRun_RedeliveredWithoutIdempotencyKey_IsDeduplicatedByMessageId() + { + await using var env = await PsrEnvironment.CreateAsync(postgres, broker, python); + var host = await env.StartHostAsync(); + await PublishEchoPipelineAsync(host); + var command = new JsonObject { ["pipelineKey"] = "eco-publica", ["inputs"] = new JsonObject { ["texto"] = "x" }, ["correlationId"] = "msg" }; + var messageId = Guid.NewGuid().ToString(); + + await SendAsync(env, command, messageId); + await SendAsync(env, command, messageId); + + await Eventually.WaitAsync(() => Task.FromResult(host.Events.All.Count(e => e.Type == RunEventTypes.RunQueued && e.CorrelationId == "msg") == 2), + TimeSpan.FromSeconds(30)); + (await env.ScalarAsync("SELECT count(*) FROM psr.runs")).ShouldBe(1); + } + + [Fact] + public async Task Spec005_AC08_Subscriber_ReceivesOnlyTheSuccessesOfItsPipelineWithOutputs() + { + await using var env = await PsrEnvironment.CreateAsync(postgres, broker, python); + var host = await env.StartHostAsync(s => + { + s.Services = services => services.AddSingleton(); + s.Builder = psr => psr.AddRunEventHandler(sub => sub.Subscription("calculo").On("run.succeeded.calculo-progressao")); + }); + await env.ImportFixtureAsync(host, "calculo-progressao"); + await PublishEchoPipelineAsync(host); + await PsrEnvironment.PublishStageAsync(host, "falha", Scripts.AlwaysFails, [], []); + var inputs = new { a1 = 1, razao = 2, n = 4, tipo = "geometrica", coeficientes = new[] { 1, 0, 1 } }; + + var runs = new[] + { + await host.Queue.EnqueuePipelineAsync("calculo-progressao", inputs, cancellationToken: Ct), + await host.Queue.EnqueuePipelineAsync("calculo-progressao", inputs, cancellationToken: Ct), + await host.Queue.EnqueuePipelineAsync("eco-publica", new { texto = "outra" }, cancellationToken: Ct), + await host.Queue.EnqueueStageAsync("falha", null, cancellationToken: Ct), + }; + foreach (var run in runs) + { + await host.WaitForRunAsync(run.RunId, timeout: TimeSpan.FromMinutes(3)); + } + + await Task.Delay(1500, Ct); + var received = host.Services.GetRequiredService().Events; + received.Select(e => e.RunId!.Value.Value).ShouldBe([runs[0].RunId.Value, runs[1].RunId.Value], ignoreOrder: true); + received.ShouldAllBe(e => e.Type == RunEventTypes.RunSucceeded && e.PipelineKey == "calculo-progressao"); + received.ShouldAllBe(e => e.Outputs!["soma"]!.GetValue() == 89 && !e.OutputsTruncated); + } + + [Fact] + public async Task Spec005_AC11_EventHandler_ReceivesTheConfiguredPatterns() + { + await using var env = await PsrEnvironment.CreateAsync(postgres, broker, python); + var host = await env.StartHostAsync(s => + { + s.Services = services => services.AddSingleton(); + s.Builder = psr => psr.AddRunEventHandler(sub => sub.Subscription("padroes").On("run.*.eco-publica", "step.succeeded.#")); + }); + await PublishEchoPipelineAsync(host); + + var handle = await host.Queue.EnqueuePipelineAsync("eco-publica", new { texto = "padrões" }, cancellationToken: Ct); + await host.WaitForRunAsync(handle.RunId); + await Task.Delay(1500, Ct); + + host.Services.GetRequiredService().Events.Select(e => e.Type).ShouldBe( + [RunEventTypes.RunQueued, RunEventTypes.RunStarted, RunEventTypes.StepSucceeded, RunEventTypes.RunSucceeded], ignoreOrder: true); + } + + [Fact] + public async Task PerInstanceSubscription_EveryInstanceReceivesEveryEvent() + { + await using var env = await PsrEnvironment.CreateAsync(postgres, broker, python); + var first = await env.StartHostAsync(s => s.Worker = false); + var second = await env.StartHostAsync(); + await PublishEchoPipelineAsync(first); + + var handle = await first.Queue.EnqueuePipelineAsync("eco-publica", new { texto = "todos" }, cancellationToken: Ct); + + await first.Events.WaitForAsync(RunEventTypes.RunSucceeded, e => e.RunId == handle.RunId); + await second.Events.WaitForAsync(RunEventTypes.RunSucceeded, e => e.RunId == handle.RunId); + } + + private static async Task PublishEchoPipelineAsync(PsrHost host) + { + await PsrEnvironment.PublishStageAsync(host, "eco", Scripts.Echo, [Scripts.Text("texto")], [Scripts.Text("texto")]); + await PsrEnvironment.PublishPipelineAsync(host, "eco-publica", Scripts.SingleStep("eco", 1, [Scripts.Text("texto")])); + } + + private static async Task SendAsync(PsrEnvironment env, JsonObject command, string? messageId = null) + { + await using var raw = await RawBroker.ConnectAsync(env.Broker); + await raw.PublishAsync(PsrTopology.PublicExchange, PsrTopology.StartPipelineRoutingKey, command.ToJsonString(), PsrMessageTypes.StartPipelineRun, + messageId); + } + + /// Events of a durable subscription. + public sealed class SubscriptionRecorder : IRunEventHandler + { + private readonly ConcurrentQueue _events = new(); + + public IReadOnlyList Events => [.. _events]; + + public Task HandleAsync(RunEvent runEvent, CancellationToken cancellationToken) + { + _events.Enqueue(runEvent); + return Task.CompletedTask; + } + } +} diff --git a/tests/PythonScriptRunner.RabbitMQ.Tests/PythonScriptRunner.RabbitMQ.Tests.csproj b/tests/PythonScriptRunner.RabbitMQ.Tests/PythonScriptRunner.RabbitMQ.Tests.csproj new file mode 100644 index 0000000..79f0787 --- /dev/null +++ b/tests/PythonScriptRunner.RabbitMQ.Tests/PythonScriptRunner.RabbitMQ.Tests.csproj @@ -0,0 +1,36 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/tests/PythonScriptRunner.RabbitMQ.Tests/QueueTests.cs b/tests/PythonScriptRunner.RabbitMQ.Tests/QueueTests.cs new file mode 100644 index 0000000..7267cce --- /dev/null +++ b/tests/PythonScriptRunner.RabbitMQ.Tests/QueueTests.cs @@ -0,0 +1,256 @@ +using System.Diagnostics; +using System.Net; +using System.Net.Sockets; +using PythonScriptRunner.EntityFrameworkCore; +using Testcontainers.RabbitMq; + +namespace PythonScriptRunner.RabbitMQ.Tests; + +/// Spec 005: queued runs, the outbox relay and delivery guarantees. +public sealed class QueueTests(PostgresFixture postgres, BrokerFixture broker, PythonFixture python) +{ + private static CancellationToken Ct => TestContext.Current.CancellationToken; + + [Fact] + public async Task Spec005_AC01_Enqueue_StoresRunStepAndMessageTogetherAndStartsQuickly() + { + await using var env = await PsrEnvironment.CreateAsync(postgres, broker, python); + var app = await env.StartHostAsync(s => + { + s.Worker = false; + s.Relay = false; + }); + await PsrEnvironment.PublishStageAsync(app, "eco", Scripts.Echo, [Scripts.Text("texto")], [Scripts.Text("texto")]); + + var queued = await app.Queue.EnqueueStageAsync("eco", new { texto = "fila" }, cancellationToken: Ct); + + var run = (await app.Runs.GetAsync(queued.RunId, Ct))!; + run.Status.ShouldBe(RunStatus.Queued); + run.Mode.ShouldBe(ExecutionMode.Async); + run.Steps.Single().Status.ShouldBe(StepStatus.Queued); + (await env.ScalarAsync( + $"SELECT count(*) FROM psr.outbox_messages WHERE message_type = 'psr.execute-step.v1' AND payload->>'runId' = '{queued.RunId}' AND published_at IS NULL")) + .ShouldBe(1); + + // With an idle worker, enqueue → start p95 ≤ 2 s over 100 runs (D-58). + var worker = await env.StartHostAsync(s => s.MaxConcurrency = 8); + var waits = new List(); + for (var i = 0; i < 100; i++) + { + var handle = await worker.Queue.EnqueueStageAsync("eco", new { texto = $"t{i}" }, cancellationToken: Ct); + var started = await worker.WaitForRunAsync(handle.RunId, r => r.Steps[0].Attempts.Count > 0); + waits.Add(started.Steps[0].Attempts[0].StartedAt - started.QueuedAt!.Value); + } + + waits.Sort(); + waits[94].ShouldBeLessThanOrEqualTo(TimeSpan.FromSeconds(2)); + } + + [Fact] + public async Task Spec005_AC02_BrokerDown_EnqueueSucceedsAndRunsWhenItComesBack() + { + var port = FreePort(); + await using var dedicated = new RabbitMqBuilder(BrokerFixture.Image).WithPortBinding(port, RabbitMqBuilder.RabbitMqPort).Build(); + await dedicated.StartAsync(Ct); + await using var env = await PsrEnvironment.CreateAsync(postgres, broker, python); + var uri = new UriBuilder(dedicated.GetConnectionString()) { Port = port }.Uri.ToString(); + var host = await env.StartHostAsync(s => s.BrokerConnectionString = uri); + await PsrEnvironment.PublishStageAsync(host, "eco", Scripts.Echo, [Scripts.Text("texto")], [Scripts.Text("texto")]); + await Eventually.WaitAsync(() => Task.FromResult(host.Broker.IsConnected), TimeSpan.FromSeconds(10)); + + await dedicated.StopAsync(Ct); + var handle = await host.Queue.EnqueueStageAsync("eco", new { texto = "sem broker" }, cancellationToken: Ct); + handle.Status.ShouldBe(RunStatus.Queued); + await Task.Delay(1000, Ct); + (await host.Runs.GetAsync(handle.RunId, Ct))!.Status.ShouldBe(RunStatus.Queued); + + await dedicated.StartAsync(Ct); + var run = await host.WaitForRunAsync(handle.RunId, timeout: TimeSpan.FromMinutes(2)); + run.Status.ShouldBe(RunStatus.Succeeded); + } + + [Fact] + public async Task Spec005_AC03_DuplicateExecuteStep_RunsTheStepOnce() + { + await using var env = await PsrEnvironment.CreateAsync(postgres, broker, python); + var app = await env.StartHostAsync(s => + { + s.Worker = false; + s.Relay = false; + }); + var marker = await PublishCounterAsync(env, app); + var handle = await app.Queue.EnqueueStageAsync("conta", new { marcador = marker }, cancellationToken: Ct); + var payload = await env.ScalarAsync( + $"SELECT payload::text FROM psr.outbox_messages WHERE message_type = 'psr.execute-step.v1' AND payload->>'runId' = '{handle.RunId}'"); + + await using (var raw = await RawBroker.ConnectAsync(env.Broker)) + { + await raw.PublishAsync(PsrTopology.CommandsExchange, PsrTopology.ExecuteStepRoutingKey, payload, PsrMessageTypes.ExecuteStep); + await raw.PublishAsync(PsrTopology.CommandsExchange, PsrTopology.ExecuteStepRoutingKey, payload, PsrMessageTypes.ExecuteStep); + } + + var worker = await env.StartHostAsync(s => s.Relay = false); + var run = await worker.WaitForRunAsync(handle.RunId); + await Task.Delay(1000, Ct); + + run.Status.ShouldBe(RunStatus.Succeeded); + run.Steps.Single().Attempts.Count.ShouldBe(1); + (await File.ReadAllLinesAsync(marker, Ct)).Length.ShouldBe(1); + } + + [Fact] + public async Task Spec005_AC04_InvalidMessage_GoesToTheDeadLetterQueueAndOthersContinue() + { + await using var env = await PsrEnvironment.CreateAsync(postgres, broker, python); + var host = await env.StartHostAsync(); + await PsrEnvironment.PublishStageAsync(host, "eco", Scripts.Echo, [Scripts.Text("texto")], [Scripts.Text("texto")]); + await using var raw = await RawBroker.ConnectAsync(env.Broker); + + await raw.PublishAsync(PsrTopology.CommandsExchange, PsrTopology.ExecuteStepRoutingKey, "{ isto não é json", PsrMessageTypes.ExecuteStep); + var handle = await host.Queue.EnqueueStageAsync("eco", new { texto = "depois" }, cancellationToken: Ct); + + (await host.WaitForRunAsync(handle.RunId)).Status.ShouldBe(RunStatus.Succeeded); + string? dead = null; + await Eventually.WaitAsync(async () => (dead = await raw.GetAsync(PsrTopology.DeadLetterQueue)) is not null, TimeSpan.FromSeconds(10)); + dead.ShouldBe("{ isto não é json"); + } + + [Fact] + public async Task Spec005_AC10_LostConfirm_IsPublishedAgainAndDeduplicated() + { + await using var env = await PsrEnvironment.CreateAsync(postgres, broker, python); + var app = await env.StartHostAsync(s => + { + s.Worker = false; + s.Relay = false; + }); + var marker = await PublishCounterAsync(env, app); + var handle = await app.Queue.EnqueueStageAsync("conta", new { marcador = marker }, cancellationToken: Ct); + + // The broker receives the message, but the relay "crashes" before recording the confirm. + await using (var raw = await RawBroker.ConnectAsync(env.Broker)) + { + await Should.ThrowAsync(() => app.Services.GetRequiredService().DispatchAsync(100, async (messages, _) => + { + foreach (var message in messages) + { + await raw.PublishAsync(message.Exchange, message.RoutingKey, message.Payload, message.MessageType, message.Id.ToString()); + } + + throw new IOException("simulated crash before the confirm was recorded"); + }, Ct)); + } + + var worker = await env.StartHostAsync(); + var run = await worker.WaitForRunAsync(handle.RunId); + await Eventually.WaitAsync(async () => await env.ScalarAsync( + $"SELECT count(*) FROM psr.outbox_messages WHERE payload->>'runId' = '{handle.RunId}' AND message_type = 'psr.execute-step.v1' AND published_at IS NULL") == 0, + TimeSpan.FromSeconds(10)); + await Task.Delay(1000, Ct); + + run.Status.ShouldBe(RunStatus.Succeeded); + run.Steps.Single().Attempts.Count.ShouldBe(1); + (await File.ReadAllLinesAsync(marker, Ct)).Length.ShouldBe(1); + } + + [Fact] + public async Task Spec005_AC09_TraceParent_WorkerActivityIsInTheCallerTrace() + { + using var source = new ActivitySource("tests"); + var captured = new List(); + using var listener = new ActivityListener + { + ShouldListenTo = s => s.Name is "PythonScriptRunner" or "tests", + Sample = (ref ActivityCreationOptions _) => ActivitySamplingResult.AllDataAndRecorded, + ActivityStopped = activity => + { + lock (captured) + { + captured.Add(activity); + } + }, + }; + ActivitySource.AddActivityListener(listener); + await using var env = await PsrEnvironment.CreateAsync(postgres, broker, python); + var host = await env.StartHostAsync(); + await PsrEnvironment.PublishStageAsync(host, "eco", Scripts.Echo, [Scripts.Text("texto")], [Scripts.Text("texto")]); + + ActivityTraceId traceId; + RunHandle handle; + using (var parent = source.StartActivity("chamador")!) + { + traceId = parent.TraceId; + handle = await host.Queue.EnqueueStageAsync("eco", new { texto = "rastreado" }, cancellationToken: Ct); + } + + (await host.WaitForRunAsync(handle.RunId)).Status.ShouldBe(RunStatus.Succeeded); + await Eventually.WaitAsync(() => Task.FromResult(Snapshot().Any(a => a.OperationName == "psr.stage.execute" && a.TraceId == traceId)), + TimeSpan.FromSeconds(10)); + var consume = Snapshot().Single(a => a.OperationName == "psr.message.consume" && a.TraceId == traceId && (string?)a.GetTagItem("psr.run.id") == handle.RunId.ToString()); + Snapshot().Single(a => a.OperationName == "psr.stage.execute" && a.TraceId == traceId).ParentSpanId.ShouldBe(consume.SpanId); + + List Snapshot() + { + lock (captured) + { + return [.. captured]; + } + } + } + + [Fact] + public async Task Spec005_AC12_Topology_IsDeclaredIdempotentlyOrValidated() + { + await using var env = await PsrEnvironment.CreateAsync(postgres, broker, python); + + var notDeclared = env.CreateHost(s => + { + s.Worker = false; + s.DeclareTopology = false; + }); + var error = await Should.ThrowAsync(() => notDeclared.Host.StartAsync(Ct)); + error.Message.ShouldContain("psr.step.execute"); + error.Message.ShouldContain("DeclareTopology"); + + await env.StartHostAsync(s => s.Worker = false); + await env.StartHostAsync(s => s.Worker = false); + var validated = await env.StartHostAsync(s => + { + s.Worker = false; + s.DeclareTopology = false; + }); + await Eventually.WaitAsync(() => Task.FromResult(validated.Broker.IsConnected), TimeSpan.FromSeconds(10)); + } + + [Fact] + public async Task Sync_RunWithTheQueueRegistered_PublishesItsEvents() + { + await using var env = await PsrEnvironment.CreateAsync(postgres, broker, python); + var host = await env.StartHostAsync(s => s.Worker = false); + await PsrEnvironment.PublishStageAsync(host, "eco", Scripts.Echo, [Scripts.Text("texto")], [Scripts.Text("texto")]); + + await using var result = await host.Runner.RunStageAsync("eco", new { texto = "síncrono" }, new RunOptions { CorrelationId = "sync-1" }, Ct); + + var succeeded = await host.Events.WaitForAsync(RunEventTypes.RunSucceeded, e => e.RunId == result.Id); + succeeded.CorrelationId.ShouldBe("sync-1"); + succeeded.Outputs!["texto"]!.GetValue().ShouldBe("síncrono"); + host.Events.For(result.Id).Select(e => e.Type).ShouldBe( + [RunEventTypes.RunStarted, RunEventTypes.StepStarted, RunEventTypes.StepSucceeded, RunEventTypes.RunSucceeded], ignoreOrder: true); + (await host.Runs.GetAsync(result.Id, Ct))!.Mode.ShouldBe(ExecutionMode.Sync); + } + + private static async Task PublishCounterAsync(PsrEnvironment env, PsrHost host) + { + await PsrEnvironment.PublishStageAsync(host, "conta", Scripts.Count, [Scripts.Text("marcador")], [Scripts.Integer("execucoes")]); + var folder = Path.Combine(env.WorkRoot, "markers"); + Directory.CreateDirectory(folder); + return Path.Combine(folder, $"{Guid.NewGuid():N}.txt"); + } + + private static int FreePort() + { + using var listener = new TcpListener(IPAddress.Loopback, 0); + listener.Start(); + return ((IPEndPoint)listener.LocalEndpoint).Port; + } +} diff --git a/tests/PythonScriptRunner.Worker.Tests/HealthTests.cs b/tests/PythonScriptRunner.Worker.Tests/HealthTests.cs new file mode 100644 index 0000000..68f9266 --- /dev/null +++ b/tests/PythonScriptRunner.Worker.Tests/HealthTests.cs @@ -0,0 +1,38 @@ +using Microsoft.Extensions.Diagnostics.HealthChecks; +using Microsoft.Extensions.Time.Testing; + +namespace PythonScriptRunner.Worker.Tests; + +/// Spec 006, AC-16: readiness of a worker. +public sealed class HealthTests(PostgresFixture postgres, BrokerFixture broker, PythonFixture python) +{ + [Fact] + public async Task Spec006_AC16_Ready_IsHealthyDegradedWithoutBrokerAndUnhealthyWithoutPartitions() + { + await using var env = await PsrEnvironment.CreateAsync(postgres, broker, python); + var healthy = await env.StartHostAsync(WithChecks); + await Eventually.WaitAsync(() => Task.FromResult(healthy.Broker.IsConnected), TimeSpan.FromSeconds(10)); + (await ReadyAsync(healthy)).Status.ShouldBe(HealthStatus.Healthy); + + var withoutBroker = await env.StartHostAsync(s => + { + WithChecks(s); + s.BrokerConnectionString = "amqp://guest:guest@127.0.0.1:1/"; + }); + var degraded = await ReadyAsync(withoutBroker); + degraded.Status.ShouldBe(HealthStatus.Degraded); + degraded.Entries["psr-broker"].Status.ShouldBe(HealthStatus.Degraded); + + // 120 days ahead no partition has been created for "now". + await using var future = await PsrEnvironment.CreateAsync(postgres, broker, python, new FakeTimeProvider(DateTimeOffset.UtcNow.AddDays(120))); + var stale = await future.StartHostAsync(WithChecks); + var unhealthy = await ReadyAsync(stale); + unhealthy.Status.ShouldBe(HealthStatus.Unhealthy); + unhealthy.Entries["psr-database"].Status.ShouldBe(HealthStatus.Unhealthy); + } + + private static void WithChecks(HostSettings settings) => settings.Services = services => services.AddHealthChecks().AddPythonScriptRunnerChecks(); + + private static Task ReadyAsync(PsrHost host) => + host.Services.GetRequiredService().CheckHealthAsync(r => r.Tags.Contains("ready"), TestContext.Current.CancellationToken); +} diff --git a/tests/PythonScriptRunner.Worker.Tests/HtmlToEmailTests.cs b/tests/PythonScriptRunner.Worker.Tests/HtmlToEmailTests.cs new file mode 100644 index 0000000..7e62779 --- /dev/null +++ b/tests/PythonScriptRunner.Worker.Tests/HtmlToEmailTests.cs @@ -0,0 +1,71 @@ +using System.Net.Http.Json; +using System.Text.Json.Nodes; +using DotNet.Testcontainers.Builders; + +namespace PythonScriptRunner.Worker.Tests; + +/// Spec 006, AC-2 (CS-2): HTML → PDF → e-mail, queued by the library, with Mailpit as the SMTP server. +public sealed class HtmlToEmailTests(PostgresFixture postgres, BrokerFixture broker, PythonFixture python) +{ + private const ushort SmtpPort = 1025; + private const ushort ApiPort = 8025; + + private static CancellationToken Ct => TestContext.Current.CancellationToken; + + [Fact] + public async Task Spec006_AC02_HtmlToEmail_SendsThePdfAndKeepsItDownloadableForSevenDays() + { + // weasyprint needs pango (GTK) at runtime: the CI runs this test on Linux (D-106). + Assert.SkipWhen(OperatingSystem.IsWindows(), "weasyprint needs GTK/pango; CS-2 runs on Linux (D-106)."); + + await using var mailpit = new ContainerBuilder("axllent/mailpit:v1.27.0") + .WithEnvironment("MP_SMTP_AUTH_ACCEPT_ANY", "1") + .WithEnvironment("MP_SMTP_AUTH_ALLOW_INSECURE", "1") + .WithPortBinding(SmtpPort, true) + .WithPortBinding(ApiPort, true) + .WithWaitStrategy(Wait.ForUnixContainer().UntilHttpRequestIsSucceeded(r => r.ForPort(ApiPort).ForPath("/livez"))) + .Build(); + await mailpit.StartAsync(Ct); + await using var env = await PsrEnvironment.CreateAsync(postgres, broker, python); + var host = await env.StartHostAsync(s => + { + s.Configuration["PythonScriptRunner:Secrets:SMTP_USUARIO"] = "psr"; + s.Configuration["PythonScriptRunner:Secrets:SMTP_SENHA"] = "segredo-de-teste"; + }); + await env.ImportFixtureAsync(host, "html-para-email", (path, content) => path.EndsWith(".pipeline.json", StringComparison.Ordinal) + ? content.Replace("smtp.exemplo.com", mailpit.Hostname, StringComparison.Ordinal) + .Replace("\"const\": 465", $"\"const\": {mailpit.GetMappedPublicPort(SmtpPort)}", StringComparison.Ordinal) + : content); + var html = PsrFile.FromText("

Relatório

Gerado pelo PythonScriptRunner.

", "pagina.html", "text/html"); + + var handle = await host.Queue.EnqueuePipelineAsync("html-para-email", new + { + html, + destinatarios = new[] { "financeiro@exemplo.com" }, + assunto = "Relatório mensal", + }, cancellationToken: Ct); + var run = await host.WaitForRunAsync(handle.RunId, timeout: TimeSpan.FromMinutes(5)); + + run.Status.ShouldBe(RunStatus.Succeeded, run.Error?.Message); + run.Outputs!["paginas"]!.GetValue().ShouldBeGreaterThanOrEqualTo(1); + var pdf = run.Artifacts.Single(a => a.Direction == ArtifactDirection.Output && a.FieldName == "pdf"); + pdf.ContentType.ShouldBe("application/pdf"); + (pdf.ContentExpiresAt - run.CreatedAt).ShouldBe(TimeSpan.FromDays(7), TimeSpan.FromMinutes(5)); + await using (var content = await host.Artifacts.OpenReadAsync(pdf.Id, Ct)) + { + var header = new byte[4]; + await content.ReadExactlyAsync(header, Ct); + System.Text.Encoding.ASCII.GetString(header).ShouldBe("%PDF"); + } + + using var http = new HttpClient { BaseAddress = new Uri($"http://{mailpit.Hostname}:{mailpit.GetMappedPublicPort(ApiPort)}") }; + var messages = (await http.GetFromJsonAsync("/api/v1/messages", Ct))!["messages"]!.AsArray(); + var message = messages.Single()!; + message["Subject"]!.GetValue().ShouldBe("Relatório mensal"); + var detail = (await http.GetFromJsonAsync($"/api/v1/message/{message["ID"]!.GetValue()}", Ct))!; + var attachment = detail["Attachments"]!.AsArray().Single()!; + attachment["FileName"]!.GetValue().ShouldBe("relatorio.pdf"); + attachment["ContentType"]!.GetValue().ShouldBe("application/pdf"); + run.Outputs!["message_id"]!.GetValue().ShouldBe($"<{message["MessageID"]!.GetValue()}>"); + } +} diff --git a/tests/PythonScriptRunner.Worker.Tests/Infrastructure/BrokerFixture.cs b/tests/PythonScriptRunner.Worker.Tests/Infrastructure/BrokerFixture.cs new file mode 100644 index 0000000..ff6880a --- /dev/null +++ b/tests/PythonScriptRunner.Worker.Tests/Infrastructure/BrokerFixture.cs @@ -0,0 +1,36 @@ +using Testcontainers.RabbitMq; + +[assembly: AssemblyFixture(typeof(PythonScriptRunner.Testing.BrokerFixture))] + +namespace PythonScriptRunner.Testing; + +/// One RabbitMQ 3.13 container (the minimum supported version; D-111) for the whole test run; each test gets its own virtual host. +public sealed class BrokerFixture : IAsyncLifetime +{ + public const string Image = "rabbitmq:3.13-alpine"; + + private readonly RabbitMqContainer _container = new RabbitMqBuilder(Image).Build(); + + public async ValueTask InitializeAsync() => await _container.StartAsync(); + + public async ValueTask DisposeAsync() => await _container.DisposeAsync(); + + /// Creates an empty virtual host and returns the AMQP URI to it. + public async Task CreateVirtualHostAsync() + { + var name = $"psr-{Guid.NewGuid():N}"; + await RunAsync("rabbitmqctl", "add_vhost", name); + var user = new Uri(_container.GetConnectionString()).UserInfo.Split(':')[0]; + await RunAsync("rabbitmqctl", "set_permissions", "-p", name, user, ".*", ".*", ".*"); + return new UriBuilder(_container.GetConnectionString()) { Path = "/" + name }.Uri.ToString(); + } + + private async Task RunAsync(params string[] command) + { + var result = await _container.ExecAsync(command); + if (result.ExitCode != 0) + { + throw new InvalidOperationException($"{string.Join(' ', command)} failed: {result.Stderr}"); + } + } +} diff --git a/tests/PythonScriptRunner.Worker.Tests/Infrastructure/PsrEnvironment.cs b/tests/PythonScriptRunner.Worker.Tests/Infrastructure/PsrEnvironment.cs new file mode 100644 index 0000000..d7fcc5a --- /dev/null +++ b/tests/PythonScriptRunner.Worker.Tests/Infrastructure/PsrEnvironment.cs @@ -0,0 +1,369 @@ +using System.IO.Compression; +using Microsoft.Extensions.Configuration; +using Microsoft.Extensions.Hosting; +using Microsoft.Extensions.Logging; +using Microsoft.Extensions.Time.Testing; +using Npgsql; +using PythonScriptRunner.EntityFrameworkCore; +using PythonScriptRunner.RabbitMQ; + +namespace PythonScriptRunner.Testing; + +/// +/// One isolated deployment for a test: a fresh migrated database and a fresh RabbitMQ virtual host. Hosts (application, +/// workers) started from it share both, like separate processes would. +/// +public sealed class PsrEnvironment : IAsyncDisposable +{ + private readonly List _hosts = []; + + private PsrEnvironment(string database, string broker, PythonFixture python, FakeTimeProvider? time) + { + Database = database; + Broker = broker; + Python = python; + Time = time; + } + + public static Actor Author { get; } = Actor.User("autora@teste"); + + public static CancellationToken Ct => TestContext.Current.CancellationToken; + + public string Database { get; } + + public string Broker { get; } + + public PythonFixture Python { get; } + + /// Simulated time shared by every host (the database clock is then not used). + public FakeTimeProvider? Time { get; } + + public string WorkRoot { get; } = Path.Combine(Path.GetTempPath(), "psr-tests", "work", Guid.NewGuid().ToString("N")); + + public static async Task CreateAsync( + PostgresFixture postgres, BrokerFixture broker, PythonFixture python, FakeTimeProvider? time = null, bool migrate = true) + { + var environment = new PsrEnvironment(await postgres.CreateDatabaseAsync(), await broker.CreateVirtualHostAsync(), python, time); + if (migrate) + { + var services = new ServiceCollection(); + services.AddPythonScriptRunner().AddEntityFrameworkCore(o => o.UseNpgsql(environment.Database)); + await using var provider = services.BuildServiceProvider(); + await provider.MigratePythonScriptRunnerAsync(Ct); + } + + return environment; + } + + /// Starts a host (application and/or worker) on this deployment. + public async Task StartHostAsync(Action? configure = null) + { + var host = CreateHost(configure); + await host.Host.StartAsync(Ct); + return host; + } + + /// Builds a host without starting it. + public PsrHost CreateHost(Action? configure = null) + { + var settings = new HostSettings(); + configure?.Invoke(settings); + var builder = Host.CreateApplicationBuilder(new HostApplicationBuilderSettings { DisableDefaults = true, EnvironmentName = "Test" }); + builder.Configuration.AddInMemoryCollection(settings.Configuration); + if (Environment.GetEnvironmentVariable("PSR_TEST_LOG") is { Length: > 0 } logFile) + { + builder.Logging.AddFilter(level => level >= LogLevel.Debug); + builder.Logging.AddProvider(new TestFileLoggerProvider(logFile, settings.InstanceId)); + } + else + { + builder.Logging.AddFilter(level => level >= LogLevel.Warning); + } + if (Time is not null) + { + builder.Services.AddSingleton(Time); + } + + builder.Services.AddSingleton(); + var psr = builder.Services.AddPythonScriptRunner(o => + { + o.WorkRoot = Path.Combine(WorkRoot, settings.InstanceId); + o.EnvironmentCacheRoot = Python.EnvironmentCacheRoot; + o.UvPath = Python.UvPath; + o.PsrunnerRequirement = Python.WheelPath; + }) + .AddEntityFrameworkCore(o => + { + o.UseNpgsql(Database); + o.Maintenance.Enabled = false; + o.LatestVersionCacheDuration = TimeSpan.Zero; + o.UseDatabaseClock = Time is null; + }); + if (settings.Broker) + { + psr.AddRabbitMq(o => + { + o.ConnectionString = settings.BrokerConnectionString ?? Broker; + o.InstanceId = settings.InstanceId; + o.DeclareTopology = settings.DeclareTopology; + o.RunOutboxRelay = settings.Relay; + o.EnablePublicCommands = settings.PublicCommands; + o.RelayInterval = TimeSpan.FromMilliseconds(100); + o.StartupConnectTimeout = TimeSpan.FromSeconds(3); + }); + if (settings.RecordEvents) + { + psr.AddRunEventHandler(s => s.PerInstance("tests").On(settings.EventPatterns)); + } + } + + if (settings.Worker) + { + psr.AddWorker(o => + { + o.WorkerId = settings.InstanceId; + o.MaxConcurrency = settings.MaxConcurrency; + o.LeaseDuration = settings.LeaseDuration; + o.LeaseRenewInterval = settings.LeaseRenewInterval; + o.ReaperInterval = settings.ReaperInterval; + o.ShutdownGracePeriod = settings.ShutdownGracePeriod; + o.Scheduler.Enabled = settings.Scheduler; + o.Scheduler.PollInterval = settings.SchedulerPollInterval; + }); + } + + settings.Builder?.Invoke(psr); + settings.Services?.Invoke(builder.Services); + var host = new PsrHost(builder.Build(), settings.InstanceId); + _hosts.Add(host); + return host; + } + + /// Creates, saves and publishes a stage. + public static async Task PublishStageAsync( + PsrHost host, string key, string script, IReadOnlyList inputs, IReadOnlyList outputs, + RetryPolicy? retry = null, bool idempotent = false, string? requirements = null) + { + var existing = await host.Admin.GetStageAsync(key, Ct); + var draft = existing is null + ? await host.Admin.CreateStageAsync(new NewCatalogItem(key, key), Author, Ct) + : await host.Admin.CreateStageDraftFromAsync(key, existing.Versions[0].Version, Author, Ct); + var saved = await host.Admin.SaveStageDraftAsync(key, draft.Version, draft.Data with + { + Script = script, + Inputs = inputs, + Outputs = outputs, + Requirements = requirements, + Retry = retry ?? RetryPolicy.None, + Idempotent = idempotent, + }, draft.RowVersion, Author, Ct); + return await host.Admin.PublishStageAsync(key, saved.Version, saved.RowVersion, Author, Ct); + } + + /// Creates (or drafts a new version of) and publishes a pipeline. + public static async Task PublishPipelineAsync(PsrHost host, string key, PipelineDraftData data) + { + var existing = await host.Admin.GetPipelineAsync(key, Ct); + var draft = existing is null + ? await host.Admin.CreatePipelineAsync(new NewCatalogItem(key, key), Author, Ct) + : await host.Admin.CreatePipelineDraftFromAsync(key, existing.Versions[0].Version, Author, Ct); + var saved = await host.Admin.SavePipelineDraftAsync(key, draft.Version, data, draft.RowVersion, Author, Ct); + return await host.Admin.PublishPipelineAsync(key, saved.Version, saved.RowVersion, Author, Ct); + } + + /// + /// Imports a fixture folder (tests/fixtures/<name>) and publishes its stages and pipelines. + /// may rewrite file contents (e.g. point a constant at a test container). + /// + public async Task ImportFixtureAsync(PsrHost host, string name, Func? transform = null) + { + var root = Path.Combine(Python.RepositoryRoot, "tests", "fixtures", name); + using var zip = new MemoryStream(); + using (var archive = new ZipArchive(zip, ZipArchiveMode.Create, leaveOpen: true)) + { + foreach (var file in Directory.EnumerateFiles(root, "*", SearchOption.AllDirectories) + .Where(f => !f.Contains("__pycache__", StringComparison.Ordinal))) + { + var relative = Path.GetRelativePath(root, file).Replace('\\', '/'); + var content = await File.ReadAllTextAsync(file, Ct); + await using var writer = new StreamWriter(archive.CreateEntry(relative).Open()); + await writer.WriteAsync(transform is null ? content : transform(relative, content)); + } + } + + zip.Position = 0; + var report = await host.Admin.ImportAsync(zip, Author, Ct); + foreach (var stage in report.Items.Where(i => i.Kind == "Stage")) + { + var draft = (await host.Admin.GetStageDraftAsync(stage.Key, Ct))!; + await host.Admin.PublishStageAsync(stage.Key, draft.Version, draft.RowVersion, Author, Ct); + } + + foreach (var pipeline in report.Items.Where(i => i.Kind == "Pipeline")) + { + var draft = (await host.Admin.GetPipelineDraftAsync(pipeline.Key, Ct))!; + await host.Admin.PublishPipelineAsync(pipeline.Key, draft.Version, draft.RowVersion, Author, Ct); + } + } + + /// Runs a scalar query on the database. + public async Task ScalarAsync(string sql) + { + await using var connection = new NpgsqlConnection(Database); + await connection.OpenAsync(Ct); + await using var command = new NpgsqlCommand(sql, connection); + return (T)(await command.ExecuteScalarAsync(Ct))!; + } + + /// Runs a query and returns the first column of every row. + public async Task> ColumnAsync(string sql) + { + await using var connection = new NpgsqlConnection(Database); + await connection.OpenAsync(Ct); + await using var command = new NpgsqlCommand(sql, connection); + await using var reader = await command.ExecuteReaderAsync(Ct); + var values = new List(); + while (await reader.ReadAsync(Ct)) + { + values.Add(reader.GetFieldValue(0)); + } + + return values; + } + + public async ValueTask DisposeAsync() + { + foreach (var host in Enumerable.Reverse(_hosts)) + { + await host.DisposeAsync(); + } + + try + { + if (Directory.Exists(WorkRoot)) + { + Directory.Delete(WorkRoot, recursive: true); + } + } + catch (IOException) + { + // Best effort. + } + catch (UnauthorizedAccessException) + { + // Best effort. + } + } +} + +/// What a test host runs. +public sealed class HostSettings +{ + public string InstanceId { get; set; } = $"test-{Guid.NewGuid().ToString("N")[..8]}"; + + public bool Broker { get; set; } = true; + + public string? BrokerConnectionString { get; set; } + + public bool Worker { get; set; } = true; + + public bool Relay { get; set; } = true; + + public bool PublicCommands { get; set; } = true; + + public bool DeclareTopology { get; set; } = true; + + public bool RecordEvents { get; set; } = true; + + public string[] EventPatterns { get; set; } = ["#"]; + + public int MaxConcurrency { get; set; } = 4; + + public TimeSpan LeaseDuration { get; set; } = TimeSpan.FromSeconds(60); + + public TimeSpan LeaseRenewInterval { get; set; } = TimeSpan.FromSeconds(15); + + public TimeSpan ReaperInterval { get; set; } = TimeSpan.FromSeconds(1); + + public TimeSpan ShutdownGracePeriod { get; set; } = TimeSpan.FromSeconds(30); + + public bool Scheduler { get; set; } + + public TimeSpan SchedulerPollInterval { get; set; } = TimeSpan.FromSeconds(1); + + public Dictionary Configuration { get; } = []; + + public Action? Builder { get; set; } + + public Action? Services { get; set; } +} + +/// A started host and its services. +public sealed class PsrHost(IHost host, string instanceId) : IAsyncDisposable +{ + private bool _disposed; + + public IHost Host { get; } = host; + + public string InstanceId { get; } = instanceId; + + public IServiceProvider Services => Host.Services; + + public IRunQueue Queue => Services.GetRequiredService(); + + public IPythonRunner Runner => Services.GetRequiredService(); + + public ICatalogAdmin Admin => Services.GetRequiredService(); + + public IRunManager Runs => Services.GetRequiredService(); + + public IArtifactStore Artifacts => Services.GetRequiredService(); + + public AsyncRunLifecycle Lifecycle => Services.GetRequiredService(); + + public PsrBroker Broker => Services.GetRequiredService(); + + public EventRecorder Events => Services.GetRequiredService(); + + /// Waits until the run satisfies (default: a final status). + public async Task WaitForRunAsync(RunId runId, Func? condition = null, TimeSpan? timeout = null) + { + condition ??= r => r.Status is RunStatus.Succeeded or RunStatus.Failed or RunStatus.Cancelled; + RunSnapshot? last = null; + await Eventually.WaitAsync(async () => + { + last = await Runs.GetAsync(runId, PsrEnvironment.Ct); + return last is not null && condition(last); + }, timeout ?? TimeSpan.FromSeconds(60), () => $"run {runId} is {last?.Status}"); + return last!; + } + + /// Stops the host gracefully. + public async Task StopAsync(CancellationToken cancellationToken = default) + { + if (!_disposed) + { + await Host.StopAsync(cancellationToken); + } + } + + public async ValueTask DisposeAsync() + { + if (_disposed) + { + return; + } + + _disposed = true; + try + { + await Host.StopAsync(CancellationToken.None); + } + catch (ObjectDisposedException) + { + // Already stopped. + } + + Host.Dispose(); + } +} diff --git a/tests/PythonScriptRunner.Worker.Tests/Infrastructure/TestSupport.cs b/tests/PythonScriptRunner.Worker.Tests/Infrastructure/TestSupport.cs new file mode 100644 index 0000000..a7dd0dc --- /dev/null +++ b/tests/PythonScriptRunner.Worker.Tests/Infrastructure/TestSupport.cs @@ -0,0 +1,161 @@ +using System.Collections.Concurrent; +using Microsoft.Extensions.Logging; + +namespace PythonScriptRunner.Testing; + +/// Collects the run events delivered to a host (registered as a per-instance subscription). +public sealed class EventRecorder : IRunEventHandler +{ + private readonly ConcurrentQueue _events = new(); + + public IReadOnlyList All => [.. _events]; + + public Task HandleAsync(RunEvent runEvent, CancellationToken cancellationToken) + { + _events.Enqueue(runEvent); + return Task.CompletedTask; + } + + public IReadOnlyList For(RunId runId) => [.. _events.Where(e => e.RunId == runId)]; + + /// Waits for an event of matching . + public async Task WaitForAsync(string type, Func? match = null, TimeSpan? timeout = null) + { + RunEvent? found = null; + await Eventually.WaitAsync(() => + { + found = _events.FirstOrDefault(e => e.Type == type && (match is null || match(e))); + return Task.FromResult(found is not null); + }, timeout ?? TimeSpan.FromSeconds(30), () => $"no {type} event; received: {string.Join(", ", _events.Select(e => e.Type))}"); + return found!; + } +} + +/// Polls a condition. +public static class Eventually +{ + public static async Task WaitAsync(Func> condition, TimeSpan timeout, Func? describe = null) + { + var deadline = DateTime.UtcNow + timeout; + while (true) + { + if (await condition()) + { + return; + } + + if (DateTime.UtcNow > deadline) + { + throw new TimeoutException($"Condition not met in {timeout}: {describe?.Invoke()}"); + } + + await Task.Delay(100, TestContext.Current.CancellationToken); + } + } +} + +/// Scripts and fields shared by the queue and worker tests. +public static class Scripts +{ + public const string Echo = """ + from psrunner import stage + + + @stage + def run(inputs, ctx): + return {"texto": inputs["texto"]} + """; + + /// Sleeps segundos and reports when it ran. + public const string Sleep = """ + import time + + from psrunner import stage + + + @stage + def run(inputs, ctx): + time.sleep(inputs["segundos"]) + return {"texto": inputs["texto"]} + """; + + /// Appends one line to marcador per execution and returns the count. + public const string Count = """ + from psrunner import stage + + + @stage + def run(inputs, ctx): + with open(inputs["marcador"], "a", encoding="utf-8") as marker: + marker.write("x\n") + with open(inputs["marcador"], encoding="utf-8") as marker: + return {"execucoes": len(marker.readlines())} + """; + + /// Sleeps a long time on the first attempt only (the worker is killed meanwhile). + public const string SlowFirstAttempt = """ + import time + + from psrunner import stage + + + @stage + def run(inputs, ctx): + if ctx.attempt == 1: + time.sleep(120) + return {"tentativa": ctx.attempt} + """; + + public const string AlwaysFails = """ + from psrunner import StageError, stage + + + @stage + def run(inputs, ctx): + raise StageError("sempre falha", code="SEMPRE") + """; + + public static FieldDefinition Text(string name) => new(name, FieldType.String); + + public static FieldDefinition Number(string name) => new(name, FieldType.Number); + + public static FieldDefinition Integer(string name) => new(name, FieldType.Integer); + + public static FieldDefinition File(string name) => new(name, FieldType.File); + + /// A pipeline with one step running @ (inputs and outputs by name). + public static PipelineDraftData SingleStep(string stageKey, int stageVersion, IReadOnlyList inputs, string output = "texto") => new() + { + Inputs = inputs, + Steps = [new PipelineStepDefinition("passo", stageKey, stageVersion)], + Outputs = [new PipelineOutputDefinition(output, new StepOutputSource("passo", output))], + }; +} + +/// Appends host logs to the file named by PSR_TEST_LOG (diagnostics of failing tests). +public sealed class TestFileLoggerProvider(string path, string instance) : ILoggerProvider +{ + private static readonly Lock Gate = new(); + + public ILogger CreateLogger(string categoryName) => new Logger(path, instance, categoryName); + + public void Dispose() + { + } + + private sealed class Logger(string path, string instance, string category) : ILogger + { + public IDisposable? BeginScope(TState state) + where TState : notnull => null; + + public bool IsEnabled(LogLevel logLevel) => logLevel != LogLevel.None; + + public void Log(LogLevel logLevel, EventId eventId, TState state, Exception? exception, Func formatter) + { + lock (Gate) + { + File.AppendAllText(path, $"[{DateTime.UtcNow:HH:mm:ss.fff} {instance} {logLevel} {category}] {formatter(state, exception)} {exception}{Environment.NewLine}"); + } + } + } +} diff --git a/tests/PythonScriptRunner.Worker.Tests/PythonScriptRunner.Worker.Tests.csproj b/tests/PythonScriptRunner.Worker.Tests/PythonScriptRunner.Worker.Tests.csproj new file mode 100644 index 0000000..1e24f8a --- /dev/null +++ b/tests/PythonScriptRunner.Worker.Tests/PythonScriptRunner.Worker.Tests.csproj @@ -0,0 +1,37 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/tests/PythonScriptRunner.Worker.Tests/RecoveryTests.cs b/tests/PythonScriptRunner.Worker.Tests/RecoveryTests.cs new file mode 100644 index 0000000..a702501 --- /dev/null +++ b/tests/PythonScriptRunner.Worker.Tests/RecoveryTests.cs @@ -0,0 +1,191 @@ +using System.Diagnostics; + +namespace PythonScriptRunner.Worker.Tests; + +/// Spec 006: lost workers (leases), cancellation, graceful shutdown and queued reprocessing. +public sealed class RecoveryTests(PostgresFixture postgres, BrokerFixture broker, PythonFixture python) +{ + private static readonly TimeSpan ShortLease = TimeSpan.FromSeconds(4); + + private static CancellationToken Ct => TestContext.Current.CancellationToken; + + [Fact] + public async Task Spec006_AC05_LostWorker_IdempotentStepIsRecoveredByAnother() + { + await using var env = await PsrEnvironment.CreateAsync(postgres, broker, python); + var (doomed, survivor, runId) = await StartAndLoseAWorkerAsync(env, idempotent: true); + + var lostAt = Stopwatch.GetTimestamp(); + var run = await survivor.WaitForRunAsync(runId, timeout: TimeSpan.FromSeconds(60)); + + run.Status.ShouldBe(RunStatus.Succeeded); + var attempts = run.Steps.Single().Attempts; + attempts.Select(a => (a.Status, a.Cause, a.WorkerId)).ShouldBe( + [ + (AttemptStatus.WorkerLost, AttemptCause.Initial, doomed.InstanceId), + (AttemptStatus.Succeeded, AttemptCause.LeaseRecovery, survivor.InstanceId), + ]); + + // Lease + reaper interval (the spec bound is LeaseDuration + 15 s with the default 60 s lease). + Stopwatch.GetElapsedTime(lostAt).ShouldBeLessThan(ShortLease + TimeSpan.FromSeconds(15)); + } + + [Fact] + public async Task Spec006_AC06_LostWorker_NonIdempotentStepFailsAndReprocessCompletesIt() + { + await using var env = await PsrEnvironment.CreateAsync(postgres, broker, python); + var (_, survivor, runId) = await StartAndLoseAWorkerAsync(env, idempotent: false); + + var failed = await survivor.WaitForRunAsync(runId, timeout: TimeSpan.FromSeconds(60)); + failed.Status.ShouldBe(RunStatus.Failed); + failed.Steps.Single().Error!.Kind.ShouldBe(ErrorKind.WorkerLost); + failed.Steps.Single().Attempts.Single().Status.ShouldBe(AttemptStatus.WorkerLost); + + await survivor.Runs.ReprocessAsync(runId, PsrEnvironment.Author, ExecutionMode.Async, Ct); + var run = await survivor.WaitForRunAsync(runId, r => r.Status != RunStatus.Running); + + run.Status.ShouldBe(RunStatus.Succeeded); + run.Steps.Single().Attempts.Select(a => a.Cause).ShouldBe([AttemptCause.Initial, AttemptCause.Reprocess]); + } + + [Fact] + public async Task Spec006_AC09_Cancel_StopsTheProcessInTheWorker() + { + await using var env = await PsrEnvironment.CreateAsync(postgres, broker, python); + var worker = await env.StartHostAsync(); + var app = await env.StartHostAsync(s => s.Worker = false); + await PsrEnvironment.PublishStageAsync(app, "dorme", Scripts.Sleep, [Scripts.Number("segundos"), Scripts.Text("texto")], [Scripts.Text("texto")]); + var handle = await app.Queue.EnqueueStageAsync("dorme", new { segundos = 120, texto = "longo" }, cancellationToken: Ct); + await app.WaitForRunAsync(handle.RunId, r => r.Steps[0].Attempts.Count == 1); + await Task.Delay(500, Ct); // the Python process is running + + var started = Stopwatch.GetTimestamp(); + await app.Runs.CancelAsync(handle.RunId, PsrEnvironment.Author, Ct); + var run = await app.WaitForRunAsync(handle.RunId, timeout: TimeSpan.FromSeconds(10)); + + Stopwatch.GetElapsedTime(started).ShouldBeLessThan(TimeSpan.FromSeconds(3)); + run.Status.ShouldBe(RunStatus.Cancelled); + run.Steps.Single().Status.ShouldBe(StepStatus.Cancelled); + run.Steps.Single().Attempts.Single().Status.ShouldBe(AttemptStatus.Cancelled); + await worker.Events.WaitForAsync(RunEventTypes.RunCancelled, e => e.RunId == handle.RunId); + } + + [Fact] + public async Task Cancel_QueuedRun_EndsAtOnceAndFreesThePipelineSlot() + { + await using var env = await PsrEnvironment.CreateAsync(postgres, broker, python); + var app = await env.StartHostAsync(s => s.Worker = false); + await PsrEnvironment.PublishStageAsync(app, "eco", Scripts.Echo, [Scripts.Text("texto")], [Scripts.Text("texto")]); + await PsrEnvironment.PublishPipelineAsync(app, "unica", Scripts.SingleStep("eco", 1, [Scripts.Text("texto")])); + var pipeline = (await app.Admin.GetPipelineAsync("unica", Ct))!; + await app.Admin.UpdatePipelineSettingsAsync("unica", new PipelineSettings(1), pipeline.RowVersion, PsrEnvironment.Author, Ct); + var first = await app.Queue.EnqueuePipelineAsync("unica", new { texto = "a" }, cancellationToken: Ct); + var second = await app.Queue.EnqueuePipelineAsync("unica", new { texto = "b" }, cancellationToken: Ct); + second.Status.ShouldBe(RunStatus.Pending); + + await app.Runs.CancelAsync(first.RunId, PsrEnvironment.Author, Ct); + + (await app.Runs.GetAsync(first.RunId, Ct))!.Status.ShouldBe(RunStatus.Cancelled); + (await app.Runs.GetAsync(second.RunId, Ct))!.Status.ShouldBe(RunStatus.Queued); + await env.StartHostAsync(); + (await app.WaitForRunAsync(second.RunId)).Status.ShouldBe(RunStatus.Succeeded); + await Should.ThrowAsync(() => app.Runs.CancelAsync(first.RunId, PsrEnvironment.Author, Ct)); + } + + [Fact] + public async Task Spec006_AC15_Shutdown_FinishesTheStepInProgressAndStopsConsuming() + { + await using var env = await PsrEnvironment.CreateAsync(postgres, broker, python); + var worker = await env.StartHostAsync(s => s.ShutdownGracePeriod = TimeSpan.FromSeconds(60)); + var app = await env.StartHostAsync(s => s.Worker = false); + await PsrEnvironment.PublishStageAsync(app, "dorme", Scripts.Sleep, [Scripts.Number("segundos"), Scripts.Text("texto")], [Scripts.Text("texto")]); + var running = await app.Queue.EnqueueStageAsync("dorme", new { segundos = 5, texto = "em andamento" }, cancellationToken: Ct); + await app.WaitForRunAsync(running.RunId, r => r.Steps[0].Attempts.Count == 1); + + var stopping = worker.StopAsync(Ct); + await Task.Delay(500, Ct); + var later = await app.Queue.EnqueueStageAsync("dorme", new { segundos = 0, texto = "depois" }, cancellationToken: Ct); + await stopping; + + var finished = (await app.Runs.GetAsync(running.RunId, Ct))!; + finished.Status.ShouldBe(RunStatus.Succeeded); + finished.Steps.Single().Attempts.Single().Status.ShouldBe(AttemptStatus.Succeeded); + (await env.ScalarAsync("SELECT count(*) FROM psr.step_attempts WHERE status = 'Running'")).ShouldBe(0); + await Task.Delay(1000, Ct); + (await app.Runs.GetAsync(later.RunId, Ct))!.Status.ShouldBe(RunStatus.Queued); + } + + [Fact] + public async Task Spec003_AC10_ReprocessAsync_ResumesFromTheFailedStep() + { + await using var env = await PsrEnvironment.CreateAsync(postgres, broker, python); + var host = await env.StartHostAsync(); + await PsrEnvironment.PublishStageAsync(host, "conta", Scripts.Count, [Scripts.Text("marcador")], [Scripts.Integer("execucoes")]); + await PsrEnvironment.PublishStageAsync(host, "falha-uma-vez", FailOnce, [Scripts.Text("marcador")], [Scripts.Text("texto")]); + await PsrEnvironment.PublishPipelineAsync(host, "retoma", new PipelineDraftData + { + Inputs = [Scripts.Text("contador"), Scripts.Text("marcador")], + Steps = + [ + new PipelineStepDefinition("conta", "conta", 1) { Bindings = [new InputBinding("marcador", new PipelineInputSource("contador"))] }, + new PipelineStepDefinition("falha", "falha-uma-vez", 1) { Bindings = [new InputBinding("marcador", new PipelineInputSource("marcador"))] }, + ], + Outputs = [new PipelineOutputDefinition("texto", new StepOutputSource("falha", "texto"))], + }); + var folder = Path.Combine(env.WorkRoot, "markers"); + Directory.CreateDirectory(folder); + var inputs = new { contador = Path.Combine(folder, "conta.txt"), marcador = Path.Combine(folder, "falha.txt") }; + + var handle = await host.Queue.EnqueuePipelineAsync("retoma", inputs, cancellationToken: Ct); + (await host.WaitForRunAsync(handle.RunId)).Status.ShouldBe(RunStatus.Failed); + var queued = await host.Runs.ReprocessAsync(handle.RunId, PsrEnvironment.Author, ExecutionMode.Async, Ct); + queued.Status.ShouldBe(RunStatus.Running); + var run = await host.WaitForRunAsync(handle.RunId, r => r.Status != RunStatus.Running); + + run.Status.ShouldBe(RunStatus.Succeeded); + run.Id.ShouldBe(handle.RunId); + run.Steps[0].Attempts.Count.ShouldBe(1); + run.Steps[0].Outputs!["execucoes"]!.GetValue().ShouldBe(1); + run.Steps[1].Attempts.Select(a => (a.Cause, a.Status)).ShouldBe([(AttemptCause.Initial, AttemptStatus.Failed), (AttemptCause.Reprocess, AttemptStatus.Succeeded)]); + run.Steps[1].Attempts[1].TriggeredBy.ShouldBe(PsrEnvironment.Author.Subject); + } + + private const string FailOnce = """ + import os + + from psrunner import StageError, stage + + + @stage + def run(inputs, ctx): + if not os.path.exists(inputs["marcador"]): + open(inputs["marcador"], "w").close() + raise StageError("primeira tentativa falha", code="PRIMEIRA") + return {"texto": "ok"} + """; + + /// A worker takes the first attempt of a long step and is "killed" (no completion, no ack; D-109); a second worker keeps running. + private static async Task<(PsrHost Doomed, PsrHost Survivor, RunId RunId)> StartAndLoseAWorkerAsync(PsrEnvironment env, bool idempotent) + { + void Leases(HostSettings s) + { + s.LeaseDuration = ShortLease; + s.LeaseRenewInterval = TimeSpan.FromSeconds(1); + s.ReaperInterval = TimeSpan.FromSeconds(1); + } + + var doomed = await env.StartHostAsync(s => + { + Leases(s); + s.ShutdownGracePeriod = TimeSpan.Zero; + }); + await PsrEnvironment.PublishStageAsync(doomed, "lenta", Scripts.SlowFirstAttempt, [], [Scripts.Integer("tentativa")], idempotent: idempotent); + var handle = await doomed.Queue.EnqueueStageAsync("lenta", null, cancellationToken: Ct); + await doomed.WaitForRunAsync(handle.RunId, r => r.Steps[0].Attempts.Count == 1); + await Task.Delay(500, Ct); + + var survivor = await env.StartHostAsync(Leases); + await doomed.StopAsync(Ct); + return (doomed, survivor, handle.RunId); + } +} diff --git a/tests/PythonScriptRunner.Worker.Tests/SchedulerTests.cs b/tests/PythonScriptRunner.Worker.Tests/SchedulerTests.cs new file mode 100644 index 0000000..dee75ee --- /dev/null +++ b/tests/PythonScriptRunner.Worker.Tests/SchedulerTests.cs @@ -0,0 +1,193 @@ +using System.Text.Json.Nodes; +using Microsoft.Extensions.Time.Testing; + +namespace PythonScriptRunner.Worker.Tests; + +/// Spec 006: cron schedules (D-41, D-44, D-45, D-77, D-89) and the schedule editor (D-101). +public sealed class SchedulerTests(PostgresFixture postgres, BrokerFixture broker, PythonFixture python) +{ + private static CancellationToken Ct => TestContext.Current.CancellationToken; + + [Fact] + public async Task Spec006_AC10_TwoSchedulers_FireEachOccurrenceOnce() + { + var time = new FakeTimeProvider(HalfPastTheMinute()); + await using var env = await PsrEnvironment.CreateAsync(postgres, broker, python, time); + var first = await env.StartHostAsync(s => s.Scheduler = true); + await env.StartHostAsync(s => s.Scheduler = true); + var schedule = await CreateEchoScheduleAsync(first, "*/1 * * * *"); + + for (var minute = 1; minute <= 5; minute++) + { + time.Advance(TimeSpan.FromMinutes(1)); + var expected = minute; + await Eventually.WaitAsync(async () => await CountRunsAsync(env, schedule) >= expected, TimeSpan.FromSeconds(20)); + } + + time.Advance(TimeSpan.FromSeconds(20)); + await Task.Delay(2000, Ct); + (await CountRunsAsync(env, schedule)).ShouldBe(5); + (await first.Admin.GetScheduleAsync(schedule.Id, Ct))!.LastRunId.ShouldNotBeNull(); + } + + [Fact] + public async Task Spec006_AC11_MissedOccurrences_BecomeOneRecoveryRun() + { + var time = new FakeTimeProvider(DateTimeOffset.UtcNow); + await using var env = await PsrEnvironment.CreateAsync(postgres, broker, python, time); + var app = await env.StartHostAsync(s => s.Worker = false); + var schedule = await CreateEchoScheduleAsync(app, "*/1 * * * *"); + + // Every worker is down for 10 occurrences. + time.SetUtcNow(schedule.NextRunAt!.Value + TimeSpan.FromMinutes(9) + TimeSpan.FromSeconds(30)); + await env.StartHostAsync(s => s.Scheduler = true); + await Eventually.WaitAsync(async () => await CountRunsAsync(env, schedule) >= 1, TimeSpan.FromSeconds(20)); + await Task.Delay(1500, Ct); + + (await CountRunsAsync(env, schedule)).ShouldBe(1); + var runId = (await app.Admin.GetScheduleAsync(schedule.Id, Ct))!.LastRunId!.Value; + var run = (await app.Runs.GetAsync(runId, Ct))!; + run.Trigger.ShouldBe(RunTrigger.Schedule); + run.TriggeredBy.ShouldBe($"system:schedule:{schedule.Id}"); + run.ScheduleId.ShouldBe(schedule.Id); + run.TriggerDetails!["missedOccurrences"]!.GetValue().ShouldBe(10); + } + + [Fact] + public async Task Spec006_AC12_LatestVersion_FollowsPublicationsAndPinnedVersionDoesNot() + { + var time = new FakeTimeProvider(HalfPastTheMinute()); + await using var env = await PsrEnvironment.CreateAsync(postgres, broker, python, time); + var host = await env.StartHostAsync(s => s.Scheduler = true); + var latest = await CreateEchoScheduleAsync(host, "*/1 * * * *"); + var pinned = await host.Admin.CreateScheduleAsync("eco-agendado", EchoSchedule("*/1 * * * *") with { Name = "fixa", PinnedVersion = 1 }, + PsrEnvironment.Author, Ct); + await PsrEnvironment.PublishPipelineAsync(host, "eco-agendado", Scripts.SingleStep("eco", 1, [Scripts.Text("texto")])); + + time.Advance(TimeSpan.FromMinutes(1)); + await Eventually.WaitAsync(async () => await CountRunsAsync(env, latest) == 1 && await CountRunsAsync(env, pinned) == 1, TimeSpan.FromSeconds(20)); + + (await env.ScalarAsync($"SELECT pipeline_version FROM psr.runs WHERE schedule_id = '{latest.Id}'")).ShouldBe(2); + (await env.ScalarAsync($"SELECT pipeline_version FROM psr.runs WHERE schedule_id = '{pinned.Id}'")).ShouldBe(1); + } + + [Fact] + public async Task Spec006_AC13_NextRun_IsComputedInTheTimeZone() + { + var time = new FakeTimeProvider(new DateTimeOffset(2026, 10, 20, 12, 0, 0, TimeSpan.Zero)); + await using var env = await PsrEnvironment.CreateAsync(postgres, broker, python, time); + var app = await env.StartHostAsync(s => s.Worker = false); + await PublishEchoPipelineAsync(app); + + var schedule = await app.Admin.CreateScheduleAsync("eco-agendado", EchoSchedule("0 2 * * *"), PsrEnvironment.Author, Ct); + + schedule.Data.TimeZone.ShouldBe("America/Sao_Paulo"); + schedule.NextRunAt.ShouldBe(new DateTimeOffset(2026, 10, 21, 5, 0, 0, TimeSpan.Zero)); + } + + [Fact] + public async Task Spec006_AC14_PipelineWithFileInput_CannotBeScheduled() + { + await using var env = await PsrEnvironment.CreateAsync(postgres, broker, python); + var app = await env.StartHostAsync(s => s.Worker = false); + await PsrEnvironment.PublishStageAsync(app, "le-arquivo", "from psrunner import stage\n\n\n@stage\ndef run(inputs, ctx):\n return {}\n", + [Scripts.File("arquivo")], []); + await PsrEnvironment.PublishPipelineAsync(app, "com-arquivo", new PipelineDraftData + { + Inputs = [Scripts.File("arquivo")], + Steps = [new PipelineStepDefinition("le", "le-arquivo", 1)], + }); + + var error = await Should.ThrowAsync(() => + app.Admin.CreateScheduleAsync("com-arquivo", new ScheduleData { Name = "x", Cron = "0 * * * *" }, PsrEnvironment.Author, Ct)); + + error.Code.ShouldBe(PsrErrorCodes.ScheduleFileInput); + } + + [Fact] + public async Task ScheduleEditor_ValidatesAuditsAndRecomputesTheNextRun() + { + var time = new FakeTimeProvider(new DateTimeOffset(2026, 10, 20, 12, 0, 30, TimeSpan.Zero)); + await using var env = await PsrEnvironment.CreateAsync(postgres, broker, python, time); + var app = await env.StartHostAsync(s => s.Worker = false); + await PublishEchoPipelineAsync(app); + + (await Should.ThrowAsync(() => app.Admin.CreateScheduleAsync("eco-agendado", EchoSchedule("61 * * * *"), PsrEnvironment.Author, Ct))) + .Code.ShouldBe(PsrErrorCodes.ScheduleInvalid); + (await Should.ThrowAsync(() => + app.Admin.CreateScheduleAsync("eco-agendado", EchoSchedule("* * * * *") with { TimeZone = "Marte/Olimpo" }, PsrEnvironment.Author, Ct))) + .Code.ShouldBe(PsrErrorCodes.ScheduleInvalid); + await Should.ThrowAsync(() => + app.Admin.CreateScheduleAsync("eco-agendado", EchoSchedule("* * * * *") with { Inputs = new JsonObject { ["outro"] = 1 } }, PsrEnvironment.Author, Ct)); + await Should.ThrowAsync(() => app.Admin.CreateScheduleAsync("nao-existe", EchoSchedule("* * * * *"), PsrEnvironment.Author, Ct)); + + var created = await app.Admin.CreateScheduleAsync("eco-agendado", EchoSchedule("*/5 * * * *"), PsrEnvironment.Author, Ct); + created.NextRunAt.ShouldBe(new DateTimeOffset(2026, 10, 20, 12, 5, 0, TimeSpan.Zero)); + var updated = await app.Admin.UpdateScheduleAsync(created.Id, created.Data with { Cron = "0 * * * *" }, created.RowVersion, PsrEnvironment.Author, Ct); + updated.NextRunAt.ShouldBe(new DateTimeOffset(2026, 10, 20, 13, 0, 0, TimeSpan.Zero)); + (await Should.ThrowAsync(() => + app.Admin.UpdateScheduleAsync(created.Id, created.Data, created.RowVersion, PsrEnvironment.Author, Ct))).Current.ShouldNotBeNull(); + var disabled = await app.Admin.SetScheduleEnabledAsync(created.Id, false, updated.RowVersion, PsrEnvironment.Author, Ct); + disabled.NextRunAt.ShouldBeNull(); + var enabled = await app.Admin.SetScheduleEnabledAsync(created.Id, true, disabled.RowVersion, PsrEnvironment.Author, Ct); + enabled.NextRunAt.ShouldBe(new DateTimeOffset(2026, 10, 20, 13, 0, 0, TimeSpan.Zero)); + (await app.Admin.ListSchedulesAsync(new ScheduleQuery("eco-agendado"), Ct)).Items.Select(s => s.Id).ShouldBe([created.Id]); + await app.Admin.DeleteScheduleAsync(created.Id, PsrEnvironment.Author, Ct); + (await app.Admin.GetScheduleAsync(created.Id, Ct)).ShouldBeNull(); + + // The simulated clock does not move, so the events share one instant. + var actions = await env.ColumnAsync($"SELECT action FROM psr.audit_events WHERE entity_id = '{created.Id}'"); + actions.ShouldBe(["ScheduleCreated", "ScheduleUpdated", "ScheduleDisabled", "ScheduleEnabled", "ScheduleDeleted"], ignoreOrder: true); + (await env.ScalarAsync($"SELECT details->>'cron' FROM psr.audit_events WHERE entity_id = '{created.Id}' AND action = 'ScheduleUpdated'")) + .ShouldBe("""["*/5 * * * *", "0 * * * *"]"""); + } + + [Fact] + public async Task NewRequiredInput_WarnsOnPublicationAndRejectsTheFiringButKeepsTheSchedule() + { + var time = new FakeTimeProvider(HalfPastTheMinute()); + await using var env = await PsrEnvironment.CreateAsync(postgres, broker, python, time); + var host = await env.StartHostAsync(s => s.Scheduler = true); + var schedule = await CreateEchoScheduleAsync(host, "*/1 * * * *"); + await PsrEnvironment.PublishStageAsync(host, "eco2", Scripts.Echo, [Scripts.Text("texto"), Scripts.Text("obrigatoria")], [Scripts.Text("texto")]); + + var publication = await PsrEnvironment.PublishPipelineAsync(host, "eco-agendado", + Scripts.SingleStep("eco2", 1, [Scripts.Text("texto"), Scripts.Text("obrigatoria")])); + publication.Warnings.ShouldContain(w => w.Code == "SCHEDULE_MISSING_INPUT" && w.Message.Contains("obrigatoria", StringComparison.Ordinal)); + + time.Advance(TimeSpan.FromMinutes(1)); + var rejection = await host.Events.WaitForAsync(RunEventTypes.RunRejected, e => e.PipelineKey == "eco-agendado"); + + rejection.Error!.Reasons!.ShouldContain(r => r.Contains("obrigatoria", StringComparison.Ordinal)); + rejection.TriggerDetails!["scheduleId"]!.GetValue().ShouldBe(schedule.Id.ToString()); + var stored = (await host.Admin.GetScheduleAsync(schedule.Id, Ct))!; + stored.Data.IsEnabled.ShouldBeTrue(); + stored.LastError.ShouldNotBeNull(); + stored.NextRunAt.ShouldNotBeNull(); + (await CountRunsAsync(env, schedule)).ShouldBe(0); + } + + /// Now, moved to second 30 of its minute: each advance of one minute then crosses exactly one occurrence of "*/1". + private static DateTimeOffset HalfPastTheMinute() + { + var now = DateTimeOffset.UtcNow; + return now.AddTicks(-(now.Ticks % TimeSpan.TicksPerMinute)).AddSeconds(30); + } + + private static ScheduleData EchoSchedule(string cron) => new() { Name = "eco", Cron = cron, Inputs = new JsonObject { ["texto"] = "agendado" } }; + + private static async Task PublishEchoPipelineAsync(PsrHost host) + { + await PsrEnvironment.PublishStageAsync(host, "eco", Scripts.Echo, [Scripts.Text("texto")], [Scripts.Text("texto")]); + await PsrEnvironment.PublishPipelineAsync(host, "eco-agendado", Scripts.SingleStep("eco", 1, [Scripts.Text("texto")])); + } + + private static async Task CreateEchoScheduleAsync(PsrHost host, string cron) + { + await PublishEchoPipelineAsync(host); + return await host.Admin.CreateScheduleAsync("eco-agendado", EchoSchedule(cron), PsrEnvironment.Author, Ct); + } + + private static Task CountRunsAsync(PsrEnvironment env, ScheduleDetail schedule) => + env.ScalarAsync($"SELECT count(*) FROM psr.runs WHERE schedule_id = '{schedule.Id}'"); +} diff --git a/tests/PythonScriptRunner.Worker.Tests/StepExecutionTests.cs b/tests/PythonScriptRunner.Worker.Tests/StepExecutionTests.cs new file mode 100644 index 0000000..4b17932 --- /dev/null +++ b/tests/PythonScriptRunner.Worker.Tests/StepExecutionTests.cs @@ -0,0 +1,185 @@ +using System.Diagnostics.Metrics; +using System.Text.Json.Nodes; +using Microsoft.Extensions.Diagnostics.Metrics.Testing; +using Microsoft.Extensions.Time.Testing; + +namespace PythonScriptRunner.Worker.Tests; + +/// Spec 006: execution of queued steps (one message per step, retries, concurrency). +public sealed class StepExecutionTests(PostgresFixture postgres, BrokerFixture broker, PythonFixture python) +{ + private static CancellationToken Ct => TestContext.Current.CancellationToken; + + [Fact] + public async Task QueuedStage_RunsInTheWorker() + { + await using var env = await PsrEnvironment.CreateAsync(postgres, broker, python); + var host = await env.StartHostAsync(); + await PsrEnvironment.PublishStageAsync(host, "eco", Scripts.Echo, [Scripts.Text("texto")], [Scripts.Text("texto")]); + + var handle = await host.Queue.EnqueueStageAsync("eco", new { texto = "olá" }, cancellationToken: Ct); + + handle.Status.ShouldBe(RunStatus.Queued); + var run = await host.WaitForRunAsync(handle.RunId); + run.Status.ShouldBe(RunStatus.Succeeded); + run.Mode.ShouldBe(ExecutionMode.Async); + run.Outputs!["texto"]!.GetValue().ShouldBe("olá"); + run.Steps.Single().Attempts.Single().WorkerId.ShouldBe(host.InstanceId); + } + + [Fact] + public async Task Spec006_AC01_CalculoProgressao_RunsOneMessagePerStep() + { + await using var env = await PsrEnvironment.CreateAsync(postgres, broker, python); + var host = await env.StartHostAsync(); + await env.ImportFixtureAsync(host, "calculo-progressao"); + var inputs = new { a1 = 1, razao = 2, n = 4, tipo = "geometrica", coeficientes = new[] { 1, 0, 1 } }; + + var handle = await host.Queue.EnqueuePipelineAsync("calculo-progressao", inputs, cancellationToken: Ct); + var run = await host.WaitForRunAsync(handle.RunId, timeout: TimeSpan.FromMinutes(3)); + + run.Status.ShouldBe(RunStatus.Succeeded, run.Error?.Message); + await using var sync = await host.Runner.RunPipelineAsync("calculo-progressao", inputs, cancellationToken: Ct); + JsonNode.DeepEquals(run.Outputs, sync.Outputs).ShouldBeTrue($"{run.Outputs} != {sync.Outputs}"); + run.Outputs!["soma"]!.GetValue().ShouldBe(89); + (await env.ScalarAsync( + $"SELECT count(*) FROM psr.outbox_messages WHERE message_type = 'psr.execute-step.v1' AND payload->>'runId' = '{handle.RunId}'")) + .ShouldBe(4); + + await host.Events.WaitForAsync(RunEventTypes.RunSucceeded, e => e.RunId == handle.RunId); + // Events written in the same transaction share the instant; ties follow the logical order. + string[] logical = [RunEventTypes.RunQueued, RunEventTypes.RunStarted, RunEventTypes.StepStarted, RunEventTypes.StepSucceeded, RunEventTypes.RunSucceeded]; + var events = host.Events.For(handle.RunId).OrderBy(e => e.OccurredAt).ThenBy(e => Array.IndexOf(logical, e.Type)).ToList(); + events.Select(e => e.Type).ShouldBe( + [ + RunEventTypes.RunQueued, RunEventTypes.RunStarted, RunEventTypes.StepStarted, RunEventTypes.StepSucceeded, + RunEventTypes.StepStarted, RunEventTypes.StepSucceeded, RunEventTypes.StepStarted, RunEventTypes.StepSucceeded, + RunEventTypes.StepStarted, RunEventTypes.StepSucceeded, RunEventTypes.RunSucceeded, + ]); + events[^1].Outputs!["soma"]!.GetValue().ShouldBe(89); + } + + [Fact] + public async Task Spec006_AC03_TwoWorkers_ShareTheStepsAndRunEachOnce() + { + await using var env = await PsrEnvironment.CreateAsync(postgres, broker, python); + var first = await env.StartHostAsync(s => s.MaxConcurrency = 2); + var second = await env.StartHostAsync(s => s.MaxConcurrency = 2); + await PsrEnvironment.PublishStageAsync(first, "dorme", Scripts.Sleep, [Scripts.Number("segundos"), Scripts.Text("texto")], [Scripts.Text("texto")]); + + var handles = new List(); + for (var i = 0; i < 20; i++) + { + handles.Add(await first.Queue.EnqueueStageAsync("dorme", new { segundos = 0.3, texto = $"t{i}" }, cancellationToken: Ct)); + } + + var runs = new List(); + foreach (var handle in handles) + { + runs.Add(await first.WaitForRunAsync(handle.RunId, timeout: TimeSpan.FromMinutes(2))); + } + + runs.ShouldAllBe(r => r.Status == RunStatus.Succeeded); + runs.ShouldAllBe(r => r.Steps.Single().Attempts.Count == 1); + runs.Select(r => r.Steps.Single().Attempts.Single().WorkerId).Distinct().Order() + .ShouldBe(new[] { first.InstanceId, second.InstanceId }.Order()); + } + + [Fact] + public async Task Spec006_AC04_Retry_WaitsTheBackoffBetweenAttempts() + { + var time = new FakeTimeProvider(DateTimeOffset.UtcNow); + await using var env = await PsrEnvironment.CreateAsync(postgres, broker, python, time); + var host = await env.StartHostAsync(); + await PsrEnvironment.PublishStageAsync(host, "falha", Scripts.AlwaysFails, [], [], retry: new RetryPolicy(2, TimeSpan.FromSeconds(10), 2.0)); + + var handle = await host.Queue.EnqueueStageAsync("falha", null, cancellationToken: Ct); + await host.WaitForRunAsync(handle.RunId, r => Finished(r) == 1 && r.Steps[0].Status == StepStatus.Queued); + time.Advance(TimeSpan.FromSeconds(10)); + await host.WaitForRunAsync(handle.RunId, r => Finished(r) == 2 && r.Steps[0].Status == StepStatus.Queued); + time.Advance(TimeSpan.FromSeconds(20)); + var run = await host.WaitForRunAsync(handle.RunId); + + run.Status.ShouldBe(RunStatus.Failed); + var attempts = run.Steps.Single().Attempts; + attempts.Select(a => a.Cause).ShouldBe([AttemptCause.Initial, AttemptCause.AutoRetry, AttemptCause.AutoRetry]); + (attempts[1].StartedAt - attempts[0].FinishedAt!.Value).ShouldBe(TimeSpan.FromSeconds(10), TimeSpan.FromSeconds(1)); + (attempts[2].StartedAt - attempts[1].FinishedAt!.Value).ShouldBe(TimeSpan.FromSeconds(20), TimeSpan.FromSeconds(1)); + var retries = host.Events.For(handle.RunId).Where(e => e.Type == RunEventTypes.StepRetryScheduled).OrderBy(e => e.OccurredAt).ToList(); + retries.Select(e => e.RetryAt - e.OccurredAt).ShouldBe([TimeSpan.FromSeconds(10), TimeSpan.FromSeconds(20)]); + + static int Finished(RunSnapshot run) => run.Steps[0].Attempts.Count(a => a.Status == AttemptStatus.Failed); + } + + [Fact] + public async Task Spec006_AC07_PipelineLimit_KeepsTheOthersPendingInOrder() + { + await using var env = await PsrEnvironment.CreateAsync(postgres, broker, python); + var host = await env.StartHostAsync(); + await PsrEnvironment.PublishStageAsync(host, "dorme", Scripts.Sleep, [Scripts.Number("segundos"), Scripts.Text("texto")], [Scripts.Text("texto")]); + await PsrEnvironment.PublishPipelineAsync(host, "limitada", Scripts.SingleStep("dorme", 1, [Scripts.Number("segundos"), Scripts.Text("texto")])); + var pipeline = (await host.Admin.GetPipelineAsync("limitada", Ct))!; + await host.Admin.UpdatePipelineSettingsAsync("limitada", new PipelineSettings(1), pipeline.RowVersion, PsrEnvironment.Author, Ct); + + var handles = new List(); + for (var i = 0; i < 3; i++) + { + handles.Add(await host.Queue.EnqueuePipelineAsync("limitada", new { segundos = 1, texto = $"t{i}" }, cancellationToken: Ct)); + } + + handles.Select(h => h.Status).ShouldBe([RunStatus.Queued, RunStatus.Pending, RunStatus.Pending]); + var runs = new List(); + foreach (var handle in handles) + { + runs.Add(await host.WaitForRunAsync(handle.RunId, timeout: TimeSpan.FromMinutes(2))); + } + + runs.ShouldAllBe(r => r.Status == RunStatus.Succeeded); + runs[1].StartedAt!.Value.ShouldBeGreaterThanOrEqualTo(runs[0].FinishedAt!.Value); + runs[2].StartedAt!.Value.ShouldBeGreaterThanOrEqualTo(runs[1].FinishedAt!.Value); + } + + [Fact] + public async Task Spec006_AC08_MaxConcurrency_LimitsTheProcessesOfAWorker() + { + await using var env = await PsrEnvironment.CreateAsync(postgres, broker, python); + var host = await env.StartHostAsync(s => s.MaxConcurrency = 2); + await PsrEnvironment.PublishStageAsync(host, "dorme", Scripts.Sleep, [Scripts.Number("segundos"), Scripts.Text("texto")], [Scripts.Text("texto")]); + + var handles = new List(); + for (var i = 0; i < 5; i++) + { + handles.Add(await host.Queue.EnqueueStageAsync("dorme", new { segundos = 2, texto = $"t{i}" }, cancellationToken: Ct)); + } + + var attempts = new List(); + foreach (var handle in handles) + { + var run = await host.WaitForRunAsync(handle.RunId, timeout: TimeSpan.FromMinutes(2)); + run.Status.ShouldBe(RunStatus.Succeeded); + attempts.Add(run.Steps.Single().Attempts.Single()); + } + + var peak = attempts.Max(a => attempts.Count(other => other.StartedAt < a.FinishedAt && a.StartedAt < other.FinishedAt)); + peak.ShouldBe(2, string.Join(" | ", attempts.Select(a => $"{a.StartedAt:HH:mm:ss.ffffff}-{a.FinishedAt:HH:mm:ss.ffffff} {a.WorkerId}"))); + } + + [Fact] + public async Task Spec006_AC17_QueueWait_P95IsAtMostTwoSeconds() + { + await using var env = await PsrEnvironment.CreateAsync(postgres, broker, python); + var host = await env.StartHostAsync(s => s.MaxConcurrency = 8); + using var waits = new MetricCollector(host.Services.GetRequiredService(), "PythonScriptRunner", "psr.queue.wait"); + await PsrEnvironment.PublishStageAsync(host, "eco", Scripts.Echo, [Scripts.Text("texto")], [Scripts.Text("texto")]); + + for (var i = 0; i < 100; i++) + { + var handle = await host.Queue.EnqueueStageAsync("eco", new { texto = $"t{i}" }, cancellationToken: Ct); + await host.WaitForRunAsync(handle.RunId, r => r.Status != RunStatus.Queued); + } + + await Eventually.WaitAsync(() => Task.FromResult(waits.GetMeasurementSnapshot().Count >= 100), TimeSpan.FromSeconds(30)); + var sorted = waits.GetMeasurementSnapshot().Select(m => m.Value).Order().ToList(); + sorted[(int)Math.Ceiling(sorted.Count * 0.95) - 1].ShouldBeLessThanOrEqualTo(2000); + } +} diff --git a/tests/PythonScriptRunner.Worker.Tests/WorkerHostTests.cs b/tests/PythonScriptRunner.Worker.Tests/WorkerHostTests.cs new file mode 100644 index 0000000..7786f0c --- /dev/null +++ b/tests/PythonScriptRunner.Worker.Tests/WorkerHostTests.cs @@ -0,0 +1,49 @@ +using System.Net; +using Microsoft.AspNetCore.Mvc.Testing; + +namespace PythonScriptRunner.Worker.Tests; + +/// Spec 009: the standalone WorkerHost (D-107). +public sealed class WorkerHostTests(PostgresFixture postgres, BrokerFixture broker, PythonFixture python) +{ + private static CancellationToken Ct => TestContext.Current.CancellationToken; + + [Fact] + public async Task WorkerHost_MigratesAnswersHealthAndRunsQueuedSteps() + { + var database = await postgres.CreateDatabaseAsync(); + var amqp = await broker.CreateVirtualHostAsync(); + var workRoot = Path.Combine(Path.GetTempPath(), "psr-tests", "work", Guid.NewGuid().ToString("N")); + await using var factory = new WebApplicationFactory().WithWebHostBuilder(web => web + .UseSetting("ConnectionStrings:psr", database) + .UseSetting("ConnectionStrings:rabbitmq", amqp) + .UseSetting("PSR_AUTO_MIGRATE", "true") + .UseSetting("PythonScriptRunner:WorkRoot", workRoot) + .UseSetting("PythonScriptRunner:UvPath", python.UvPath) + .UseSetting("PythonScriptRunner:EnvironmentCacheRoot", python.EnvironmentCacheRoot) + .UseSetting("PythonScriptRunner:PsrunnerRequirement", python.WheelPath) + .UseSetting("PythonScriptRunner:Worker:MaxConcurrency", "2") + .UseSetting("Serilog:MinimumLevel:Default", "Warning")); + using var client = factory.CreateClient(); + + (await client.GetAsync("/health/live", Ct)).StatusCode.ShouldBe(HttpStatusCode.OK); + await Eventually.WaitAsync(async () => (await client.GetStringAsync("/health/ready", Ct)) == "Healthy", TimeSpan.FromSeconds(30)); + + var admin = factory.Services.GetRequiredService(); + var draft = await admin.CreateStageAsync(new NewCatalogItem("eco", "eco"), PsrEnvironment.Author, Ct); + var saved = await admin.SaveStageDraftAsync("eco", draft.Version, draft.Data with + { + Script = Scripts.Echo, + Inputs = [Scripts.Text("texto")], + Outputs = [Scripts.Text("texto")], + }, draft.RowVersion, PsrEnvironment.Author, Ct); + await admin.PublishStageAsync("eco", saved.Version, saved.RowVersion, PsrEnvironment.Author, Ct); + var handle = await factory.Services.GetRequiredService().EnqueueStageAsync("eco", new { texto = "host" }, cancellationToken: Ct); + + var runs = factory.Services.GetRequiredService(); + RunSnapshot? run = null; + await Eventually.WaitAsync(async () => (run = await runs.GetAsync(handle.RunId, Ct))?.Status == RunStatus.Succeeded, TimeSpan.FromSeconds(60), + () => $"run is {run?.Status}"); + run!.Steps.Single().Attempts.Single().WorkerId.ShouldNotBeNullOrEmpty(); + } +} diff --git a/tests/fixtures/html-para-email/pipelines/html-para-email.pipeline.json b/tests/fixtures/html-para-email/pipelines/html-para-email.pipeline.json new file mode 100644 index 0000000..8a73e68 --- /dev/null +++ b/tests/fixtures/html-para-email/pipelines/html-para-email.pipeline.json @@ -0,0 +1,75 @@ +{ + "$schema": "../../../../schemas/pipeline.v1.json", + "key": "html-para-email", + "version": 1, + "name": "HTML para PDF por e-mail", + "inputs": [ + { + "name": "html", + "type": "file", + "contentTypes": [ + "text/html" + ] + }, + { + "name": "destinatarios", + "type": "list", + "itemType": "string" + }, + { + "name": "assunto", + "type": "string" + }, + { + "name": "corpo", + "type": "string", + "required": false, + "default": "Segue o documento em anexo." + } + ], + "steps": [ + { + "key": "gerar-pdf", + "stage": "html-para-pdf@1", + "bindings": { + "html": "$pipeline.html", + "nome_arquivo": { + "const": "relatorio.pdf" + } + } + }, + { + "key": "enviar", + "stage": "enviar-email@1", + "bindings": { + "smtp_host": { + "const": "smtp.exemplo.com" + }, + "smtp_porta": { + "const": 465 + }, + "remetente": { + "const": "no-reply@exemplo.com" + }, + "destinatarios": "$pipeline.destinatarios", + "assunto": "$pipeline.assunto", + "corpo": "$pipeline.corpo", + "anexo": "$steps.gerar-pdf.pdf" + } + } + ], + "outputs": [ + { + "name": "pdf", + "source": "$steps.gerar-pdf.pdf" + }, + { + "name": "paginas", + "source": "$steps.gerar-pdf.paginas" + }, + { + "name": "message_id", + "source": "$steps.enviar.message_id" + } + ] +} diff --git a/tests/fixtures/html-para-email/stages/enviar-email/script.py b/tests/fixtures/html-para-email/stages/enviar-email/script.py new file mode 100644 index 0000000..9e67230 --- /dev/null +++ b/tests/fixtures/html-para-email/stages/enviar-email/script.py @@ -0,0 +1,39 @@ +import smtplib +import ssl +from datetime import datetime, timezone +from email.message import EmailMessage +from email.utils import make_msgid + +from psrunner import stage + + +@stage +def run(inputs, ctx): + remetente = inputs["remetente"] + mensagem = EmailMessage() + mensagem["From"] = remetente + mensagem["To"] = ", ".join(inputs["destinatarios"]) + mensagem["Subject"] = inputs["assunto"] + message_id = make_msgid(domain=remetente.rsplit("@", 1)[-1]) + mensagem["Message-ID"] = message_id + mensagem.set_content(inputs["corpo"]) + + anexo = inputs.get("anexo") + if anexo is not None: + tipo, _, subtipo = anexo.content_type.partition("/") + mensagem.add_attachment(anexo.read_bytes(), maintype=tipo, subtype=subtipo or "octet-stream", filename=anexo.name) + + host, porta = inputs["smtp_host"], inputs["smtp_porta"] + if porta == 465: + servidor = smtplib.SMTP_SSL(host, porta, context=ssl.create_default_context(), timeout=30) + else: + servidor = smtplib.SMTP(host, porta, timeout=30) + + with servidor: + if porta != 465 and servidor.has_extn("starttls"): + servidor.starttls(context=ssl.create_default_context()) + servidor.login(ctx.secrets["SMTP_USUARIO"], ctx.secrets["SMTP_SENHA"]) + servidor.send_message(mensagem) + + ctx.log.info("e-mail %s enviado para %d destinatário(s)", message_id, len(inputs["destinatarios"])) + return {"message_id": message_id, "enviado_em": datetime.now(timezone.utc)} diff --git a/tests/fixtures/html-para-email/stages/enviar-email/stage.json b/tests/fixtures/html-para-email/stages/enviar-email/stage.json new file mode 100644 index 0000000..9d1cc0b --- /dev/null +++ b/tests/fixtures/html-para-email/stages/enviar-email/stage.json @@ -0,0 +1,56 @@ +{ + "$schema": "../../../../../schemas/stage.v1.json", + "key": "enviar-email", + "version": 1, + "name": "Enviar e-mail", + "description": "Envia um e-mail por SMTP, com um anexo opcional.", + "idempotent": false, + "secrets": [ + "SMTP_USUARIO", + "SMTP_SENHA" + ], + "inputs": [ + { + "name": "smtp_host", + "type": "string" + }, + { + "name": "smtp_porta", + "type": "integer", + "required": false, + "default": 465 + }, + { + "name": "remetente", + "type": "string" + }, + { + "name": "destinatarios", + "type": "list", + "itemType": "string" + }, + { + "name": "assunto", + "type": "string" + }, + { + "name": "corpo", + "type": "string" + }, + { + "name": "anexo", + "type": "file", + "required": false + } + ], + "outputs": [ + { + "name": "message_id", + "type": "string" + }, + { + "name": "enviado_em", + "type": "datetime" + } + ] +} diff --git a/tests/fixtures/html-para-email/stages/html-para-pdf/requirements.txt b/tests/fixtures/html-para-email/stages/html-para-pdf/requirements.txt new file mode 100644 index 0000000..b40cf4d --- /dev/null +++ b/tests/fixtures/html-para-email/stages/html-para-pdf/requirements.txt @@ -0,0 +1 @@ +weasyprint diff --git a/tests/fixtures/html-para-email/stages/html-para-pdf/script.py b/tests/fixtures/html-para-email/stages/html-para-pdf/script.py new file mode 100644 index 0000000..32015e1 --- /dev/null +++ b/tests/fixtures/html-para-email/stages/html-para-pdf/script.py @@ -0,0 +1,10 @@ +from psrunner import stage +from weasyprint import HTML + + +@stage +def run(inputs, ctx): + pdf = ctx.new_file(inputs["nome_arquivo"], "application/pdf") + document = HTML(filename=str(inputs["html"].path)).render() + document.write_pdf(pdf.path) + return {"pdf": pdf, "paginas": len(document.pages)} diff --git a/tests/fixtures/html-para-email/stages/html-para-pdf/stage.json b/tests/fixtures/html-para-email/stages/html-para-pdf/stage.json new file mode 100644 index 0000000..3249960 --- /dev/null +++ b/tests/fixtures/html-para-email/stages/html-para-pdf/stage.json @@ -0,0 +1,37 @@ +{ + "$schema": "../../../../../schemas/stage.v1.json", + "key": "html-para-pdf", + "version": 1, + "name": "HTML para PDF", + "description": "Converte um HTML em PDF com o weasyprint.", + "requirements": "requirements.txt", + "idempotent": true, + "inputs": [ + { + "name": "html", + "type": "file", + "contentTypes": [ + "text/html" + ] + }, + { + "name": "nome_arquivo", + "type": "string", + "required": false, + "default": "documento.pdf" + } + ], + "outputs": [ + { + "name": "pdf", + "type": "file", + "contentTypes": [ + "application/pdf" + ] + }, + { + "name": "paginas", + "type": "integer" + } + ] +}