Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -41,9 +41,16 @@ jobs:
- name: Build
run: dotnet build --no-restore --configuration Release

# The persistence tests start PostgreSQL 16 with Testcontainers (Docker); GitHub's Windows runners
# cannot run Linux containers, so Windows runs the Core tests only.
- name: Test
if: runner.os == 'Linux'
run: dotnet test --no-build --configuration Release

- name: Test (Core)
if: runner.os == 'Windows'
run: dotnet test --no-build --configuration Release --project tests/PythonScriptRunner.Core.Tests

format:
name: .NET format
runs-on: ubuntu-latest
Expand Down
5 changes: 5 additions & 0 deletions Directory.Packages.props
Original file line number Diff line number Diff line change
Expand Up @@ -4,17 +4,22 @@
</PropertyGroup>
<ItemGroup>
<PackageVersion Include="JsonSchema.Net" Version="9.4.1" />
<PackageVersion Include="Microsoft.EntityFrameworkCore.Design" Version="10.0.13" />
<PackageVersion Include="Microsoft.EntityFrameworkCore.Relational" Version="10.0.13" />
<PackageVersion Include="Microsoft.Extensions.Configuration" Version="10.0.13" />
<PackageVersion Include="Microsoft.Extensions.Configuration.Abstractions" Version="10.0.13" />
<PackageVersion Include="Microsoft.Extensions.DependencyInjection" Version="10.0.13" />
<PackageVersion Include="Microsoft.Extensions.DependencyInjection.Abstractions" Version="10.0.13" />
<PackageVersion Include="Microsoft.Extensions.Diagnostics" Version="10.0.13" />
<PackageVersion Include="Microsoft.Extensions.Diagnostics.HealthChecks" Version="10.0.13" />
<PackageVersion Include="Microsoft.Extensions.Diagnostics.Testing" Version="10.11.0" />
<PackageVersion Include="Microsoft.Extensions.Hosting.Abstractions" Version="10.0.13" />
<PackageVersion Include="Microsoft.Extensions.Logging" Version="10.0.13" />
<PackageVersion Include="Microsoft.Extensions.Options" Version="10.0.13" />
<PackageVersion Include="Microsoft.Extensions.TimeProvider.Testing" Version="10.11.0" />
<PackageVersion Include="Npgsql.EntityFrameworkCore.PostgreSQL" Version="10.0.3" />
<PackageVersion Include="Shouldly" Version="4.3.0" />
<PackageVersion Include="Testcontainers.PostgreSql" Version="4.16.0" />
<PackageVersion Include="xunit.v3" Version="4.0.2" />
</ItemGroup>
</Project>
2 changes: 2 additions & 0 deletions PythonScriptRunner.slnx
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,10 @@
<Folder Name="/src/">
<Project Path="src/PythonScriptRunner.Abstractions/PythonScriptRunner.Abstractions.csproj" />
<Project Path="src/PythonScriptRunner.Core/PythonScriptRunner.Core.csproj" />
<Project Path="src/PythonScriptRunner.EntityFrameworkCore/PythonScriptRunner.EntityFrameworkCore.csproj" />
</Folder>
<Folder Name="/tests/">
<Project Path="tests/PythonScriptRunner.Core.Tests/PythonScriptRunner.Core.Tests.csproj" />
<Project Path="tests/PythonScriptRunner.EntityFrameworkCore.Tests/PythonScriptRunner.EntityFrameworkCore.Tests.csproj" />
</Folder>
</Solution>
36 changes: 35 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ com contrato de entrada/saída, trilhas (**Pipelines**), execução assíncrona
cron e painel Vue opcional.

> Projeto em desenvolvimento. Esta versão entrega o **Core Runner** (execução síncrona de Stages e de
> trilhas) e o SDK Python **`psrunner`**.
> trilhas), o SDK Python **`psrunner`** e a **persistência em PostgreSQL** (catálogo versionado e histórico).

## Como funciona

Expand Down Expand Up @@ -89,17 +89,50 @@ double[] termos = result.GetStepOutput<double[]>("termos", "termos"); // depur
A mesma trilha pode ser definida em C# com `AddPipeline("chave", p => p.Input(...).Step(...).Output(...))`.
Se um passo falhar, os seguintes ficam como `Skipped` e a execução termina `Failed`.

## Persistência (PostgreSQL)

Com o pacote `PythonScriptRunner.EntityFrameworkCore`, Stages e trilhas passam a viver no banco (schema `psr`) com
versões imutáveis (rascunho → publicado → arquivado), e **toda** execução fica registrada: passos, tentativas, cauda
de stdout/stderr e arquivos (em `bytea`, lidos e gravados em streaming).

```csharp
builder.Services
.AddPythonScriptRunner()
.AddEntityFrameworkCore(o =>
{
o.UseNpgsql(builder.Configuration.GetConnectionString("psr")!); // PostgreSQL 16+
o.Retention.Artifacts = TimeSpan.FromDays(7); // arquivos
o.Retention.Runs = TimeSpan.FromDays(90); // histórico
});
builder.Services.AddHealthChecks().AddPythonScriptRunnerDatabase();

var app = builder.Build();
await app.Services.MigratePythonScriptRunnerAsync(); // explícito; ou GeneratePythonScriptRunnerMigrationScriptAsync()
```

- **Catálogo:** `ICatalogAdmin` cria rascunhos (concorrência otimista), publica (verifica a sintaxe com o Python
declarado e trava as dependências com `uv pip compile --generate-hashes`), arquiva, exclui, testa rascunhos e
importa/exporta zip no formato de pasta.
- **Execuções:** `IRunManager` consulta (paginação keyset), cancela e reprocessa uma execução que falhou — os passos
que deram certo reaproveitam as saídas e o que falhou ganha uma nova tentativa. `RunOptions.IdempotencyKey` devolve
a mesma execução por 24 h.
- **Retenção:** o histórico é particionado (semana para execuções, dia para conteúdo de arquivos, mês para auditoria);
um job de manutenção cria partições à frente e remove as antigas. Uma trilha pode encurtar a retenção.
- **Auditoria:** ações humanas (`Actor.User(...)`) ficam em `psr.audit_events`; disparos automáticos ficam na própria execução.

## Requisitos

- .NET 10 SDK
- [uv](https://docs.astral.sh/uv/) no PATH (cria e cacheia um ambiente Python por conjunto de dependências)
- PostgreSQL 16+ apenas para o pacote de persistência (os testes usam Docker/Testcontainers)

## Estrutura

| Pasta | Conteúdo |
|-------|----------|
| `src/PythonScriptRunner.Abstractions` | contratos públicos (tipos de campo, `StageDefinition`, `RunResult`, `PsrFile`, interfaces) |
| `src/PythonScriptRunner.Core` | catálogos (código e pasta), validação, ambientes uv, executor de processos, runner |
| `src/PythonScriptRunner.EntityFrameworkCore` | PostgreSQL: catálogo versionado, histórico, arquivos, idempotência, outbox, auditoria, retenção |
| `python/psrunner` | SDK Python (`@stage`, `Context`, `File`, `StageError`, `psrunner.testing`, CLI) |
| `schemas` | JSON Schemas do `stage.json` e do `*.pipeline.json` |
| `tests` | testes .NET (unitários e de integração com Python real) e fixtures de trilha |
Expand All @@ -109,6 +142,7 @@ Se um passo falhar, os seguintes ficam como `Skipped` e a execução termina `Fa
```bash
dotnet build
dotnet test --project tests/PythonScriptRunner.Core.Tests
dotnet test --project tests/PythonScriptRunner.EntityFrameworkCore.Tests # precisa de Docker

cd python/psrunner
uv sync
Expand Down
55 changes: 55 additions & 0 deletions src/PythonScriptRunner.Abstractions/Exceptions.cs
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,61 @@ public VersionNotFoundException(string key, int version)
public int Version { get; }
}

/// <summary>The version exists but cannot run (it is a draft or was archived).</summary>
public sealed class VersionNotRunnableException : PsrException
{
/// <summary>Creates the exception.</summary>
/// <param name="key">Stage or pipeline key.</param>
/// <param name="version">Requested version.</param>
/// <param name="status">Current status of the version.</param>
public VersionNotRunnableException(string key, int version, VersionStatus status)
: base($"Version {version} of '{key}' is {status} and cannot run; only published versions run.")
{
Key = key;
Version = version;
Status = status;
}

/// <summary>Stage or pipeline key.</summary>
public string Key { get; }

/// <summary>Requested version.</summary>
public int Version { get; }

/// <summary>Current status of the version.</summary>
public VersionStatus Status { get; }
}

/// <summary>The run does not exist (or is beyond the retention).</summary>
public sealed class RunNotFoundException : PsrException
{
/// <summary>Creates the exception.</summary>
/// <param name="runId">Requested run.</param>
public RunNotFoundException(RunId runId)
: base($"Run {runId} was not found.")
{
RunId = runId;
}

/// <summary>Requested run.</summary>
public RunId RunId { get; }
}

/// <summary>The stored file does not exist (or is beyond the retention).</summary>
public sealed class ArtifactNotFoundException : PsrException
{
/// <summary>Creates the exception.</summary>
/// <param name="artifactId">Requested file.</param>
public ArtifactNotFoundException(ArtifactId artifactId)
: base($"Artifact {artifactId} was not found.")
{
ArtifactId = artifactId;
}

/// <summary>Requested file.</summary>
public ArtifactId ArtifactId { get; }
}

/// <summary>One invalid input.</summary>
/// <param name="Field">Field path (e.g. <c>coeficientes[2]</c>).</param>
/// <param name="Code">Stable code (e.g. <c>MISSING_REQUIRED_INPUT</c>).</param>
Expand Down
140 changes: 125 additions & 15 deletions src/PythonScriptRunner.Abstractions/Extensibility/Interfaces.cs
Original file line number Diff line number Diff line change
Expand Up @@ -2,24 +2,35 @@

namespace PythonScriptRunner;

/// <summary>Source of stage definitions (memory, folder or database).</summary>
/// <summary>Source of stage and pipeline definitions (memory, folder or database).</summary>
public interface ICatalog
{
/// <summary>Finds a stage version.</summary>
/// <param name="key">Stage key.</param>
/// <param name="version">Exact version, or <c>null</c> for the latest published version.</param>
/// <param name="cancellationToken">Cancellation token.</param>
/// <returns>The definition, or <c>null</c> when not found.</returns>
/// <exception cref="VersionNotRunnableException">The version exists but is a draft or archived.</exception>
ValueTask<StageDefinition?> FindStageAsync(string key, int? version, CancellationToken cancellationToken = default);

/// <summary>Finds a pipeline version.</summary>
/// <param name="key">Pipeline key.</param>
/// <param name="version">Exact version, or <c>null</c> for the latest published version.</param>
/// <param name="cancellationToken">Cancellation token.</param>
/// <returns>The definition, or <c>null</c> when not found.</returns>
/// <exception cref="VersionNotRunnableException">The version exists but is a draft or archived.</exception>
ValueTask<PipelineDefinition?> FindPipelineAsync(string key, int? version, CancellationToken cancellationToken = default);
}

/// <summary>Validates stage definitions (contract, limits and secret names).</summary>
public interface IStageValidator
{
/// <summary>Validates a stage definition.</summary>
/// <param name="stage">The definition.</param>
/// <returns>Errors (code <c>STAGE_DEFINITION</c>) and warnings.</returns>
ValidationReport Validate(StageDefinition stage);
}

/// <summary>Resolves the secrets declared by stages.</summary>
public interface ISecretProvider
{
Expand All @@ -29,46 +40,145 @@ public interface ISecretProvider
ValueTask<string?> GetSecretAsync(string name, CancellationToken cancellationToken = default);
}

/// <summary>Receives run transitions so they can be persisted. The default implementation does nothing.</summary>
/// <summary>
/// Receives every transition of a run so it can be persisted. The default implementation does nothing; the
/// EntityFrameworkCore package stores runs, steps, attempts, logs and files.
/// </summary>
public interface IRunRecorder
{
/// <summary>A run started.</summary>
/// <summary>A run is about to start (inputs already validated). Reserves the idempotency key.</summary>
/// <param name="run">Run data.</param>
/// <param name="cancellationToken">Cancellation token.</param>
/// <returns>The existing run when the idempotency key was already used (the new run must not start); otherwise <c>null</c>.</returns>
ValueTask<RunResult?> RunStartingAsync(RunStarting run, CancellationToken cancellationToken);

/// <summary>A failed run is being reprocessed: it is running again and skipped steps are pending.</summary>
/// <param name="run">Run data.</param>
/// <param name="cancellationToken">Cancellation token.</param>
ValueTask RunStartedAsync(RunStarted run, CancellationToken cancellationToken);
ValueTask RunResumingAsync(RunResuming run, CancellationToken cancellationToken);

/// <summary>A step started with its resolved inputs.</summary>
/// <param name="step">Step data.</param>
/// <param name="cancellationToken">Cancellation token.</param>
ValueTask StepStartedAsync(StepStarted step, CancellationToken cancellationToken);

/// <summary>An attempt finished.</summary>
/// <summary>An attempt (one Python process) started.</summary>
/// <param name="attempt">Attempt data.</param>
/// <param name="cancellationToken">Cancellation token.</param>
ValueTask AttemptStartedAsync(AttemptStarted attempt, CancellationToken cancellationToken);

/// <summary>An attempt finished (with its logs).</summary>
/// <param name="attempt">Attempt data.</param>
/// <param name="cancellationToken">Cancellation token.</param>
ValueTask AttemptFinishedAsync(AttemptFinished attempt, CancellationToken cancellationToken);

/// <summary>A step finished (succeeded with outputs and files, failed, cancelled or skipped).</summary>
/// <param name="step">Step data.</param>
/// <param name="cancellationToken">Cancellation token.</param>
ValueTask StepFinishedAsync(StepFinished step, CancellationToken cancellationToken);

/// <summary>A run finished.</summary>
/// <param name="run">Final data.</param>
/// <param name="cancellationToken">Cancellation token.</param>
ValueTask RunFinishedAsync(RunFinished run, CancellationToken cancellationToken);
}

/// <summary>A run started.</summary>
/// <summary>A run is about to start.</summary>
public sealed record RunStarting
{
/// <summary>Run identifier (its timestamp is the creation time).</summary>
public required RunId RunId { get; init; }

/// <summary>What is executed.</summary>
public required RunKind Kind { get; init; }

/// <summary>Stage or pipeline key.</summary>
public required string TargetKey { get; init; }

/// <summary>Stage or pipeline version.</summary>
public required int TargetVersion { get; init; }

/// <summary>How the run started.</summary>
public required RunTrigger Trigger { get; init; }

/// <summary>Who requested the run.</summary>
public required string TriggeredBy { get; init; }

/// <summary>Deduplication key given by the caller.</summary>
public string? IdempotencyKey { get; init; }

/// <summary>Validated inputs in wire format (files carry metadata only).</summary>
public required JsonObject Inputs { get; init; }

/// <summary>Input files by field name.</summary>
public IReadOnlyDictionary<string, IReadOnlyList<PsrFile>> InputFiles { get; init; } = new Dictionary<string, IReadOnlyList<PsrFile>>();

/// <summary>Steps that will run, in order.</summary>
public required IReadOnlyList<PlannedStep> Steps { get; init; }

/// <summary>Start time.</summary>
public required DateTimeOffset StartedAt { get; init; }
}

/// <summary>A step of a run that is about to start.</summary>
/// <param name="StepKey">Step key (the stage key for single-stage runs).</param>
/// <param name="Order">1-based position.</param>
/// <param name="StageKey">Stage key.</param>
/// <param name="StageVersion">Exact stage version.</param>
public sealed record PlannedStep(string StepKey, int Order, string StageKey, int StageVersion);

/// <summary>A failed run is being reprocessed.</summary>
/// <param name="RunId">Run identifier.</param>
/// <param name="TriggeredBy">Who reprocesses.</param>
/// <param name="ResumedAt">When.</param>
public sealed record RunResuming(RunId RunId, string TriggeredBy, DateTimeOffset ResumedAt);

/// <summary>A step started.</summary>
/// <param name="RunId">Run identifier.</param>
/// <param name="Kind">What is executed.</param>
/// <param name="TargetKey">Stage or pipeline key.</param>
/// <param name="TargetVersion">Stage or pipeline version.</param>
/// <param name="TriggeredBy">Who requested the run.</param>
/// <param name="StepKey">Step key.</param>
/// <param name="Order">1-based position.</param>
/// <param name="Inputs">Resolved inputs in wire format.</param>
/// <param name="InputFiles">Input files by field name.</param>
/// <param name="StartedAt">Start time.</param>
/// <param name="Inputs">Validated inputs in wire format.</param>
public sealed record RunStarted(
RunId RunId, RunKind Kind, string TargetKey, int TargetVersion, string TriggeredBy, DateTimeOffset StartedAt, JsonObject Inputs);
public sealed record StepStarted(
RunId RunId, string StepKey, int Order, JsonObject Inputs, IReadOnlyDictionary<string, IReadOnlyList<PsrFile>> InputFiles, DateTimeOffset StartedAt);

/// <summary>An attempt started.</summary>
/// <param name="RunId">Run identifier.</param>
/// <param name="StepKey">Step key.</param>
/// <param name="Order">1-based position of the step.</param>
/// <param name="AttemptId">Attempt identifier.</param>
/// <param name="Number">1-based attempt number.</param>
/// <param name="Cause">Why it started.</param>
/// <param name="TriggeredBy">Who started it.</param>
/// <param name="StartedAt">Start time.</param>
public sealed record AttemptStarted(
RunId RunId, string StepKey, int Order, AttemptId AttemptId, int Number, AttemptCause Cause, string TriggeredBy, DateTimeOffset StartedAt);

/// <summary>An attempt finished.</summary>
/// <param name="RunId">Run identifier.</param>
/// <param name="StepKey">Step key.</param>
/// <param name="Order">1-based position of the step.</param>
/// <param name="Attempt">Attempt outcome.</param>
public sealed record AttemptFinished(RunId RunId, string StepKey, AttemptResult Attempt);
public sealed record AttemptFinished(RunId RunId, string StepKey, int Order, AttemptResult Attempt);

/// <summary>A step finished.</summary>
/// <param name="RunId">Run identifier.</param>
/// <param name="Step">Step outcome (outputs and files when it succeeded).</param>
/// <param name="FinishedAt">End time.</param>
public sealed record StepFinished(RunId RunId, StepResult Step, DateTimeOffset FinishedAt);

/// <summary>A run finished.</summary>
/// <param name="RunId">Run identifier.</param>
/// <param name="Status">Final status.</param>
/// <param name="FinishedAt">End time.</param>
/// <param name="Outputs">Declared outputs in wire format.</param>
/// <param name="OutputFiles">Declared file outputs by name.</param>
/// <param name="Error">Failure, when the run did not succeed.</param>
public sealed record RunFinished(RunId RunId, RunStatus Status, DateTimeOffset FinishedAt, JsonObject? Outputs, RunError? Error);
public sealed record RunFinished(
RunId RunId,
RunStatus Status,
DateTimeOffset FinishedAt,
JsonObject? Outputs,
IReadOnlyDictionary<string, IReadOnlyList<PsrFile>>? OutputFiles,
RunError? Error);
Loading
Loading